ULTRA-HIGH PRECISION UAV-BASED LIDAR AND DENSE IMAGE MATCHING

This paper presents a study on the potential of ultra-high accurate UAV-based 3D data capture. It is motivated by a project aiming at the deformation monitoring of a ship lock and its surrounding. This study is part of a research and development project initiated by the German Federal Institute of Hydrology (BfG) in Koblenz in partnership with the Office of Development of Neckar River Heidelberg (ANH). For this first official presentation of the project, data from the first flight campaign will be analysed and presented. Despite the fact that monitoring aspects cannot be discussed before data from additional flight campaigns will be available later this year, our results from the first campaign highlight the potential of high-end UAV-based image and LiDAR sensors and their data fusion. So far, only techniques from engineering geodesy could fulfil the aspired accuracy demands in the range of millimetres. To the knowledge of the authors, this paper for the first time addresses such ultra-high accuracy applications by combing high precision UAVbased LiDAR and dense image matching. As the paper is written at an early stage of processing only preliminary results can be given here.


MOTIVATION
The paper describes the potential of ultra-high precision UAVbased LiDAR and dense image matching.The concrete application behind is the 3D data capture for deformation monitoring of a ship lock (including weir and hydropower plant building) and its surrounding area at the Neckar River in Hessigheim, north of Stuttgart, Germany (cf. Figure 1 and 2).Subsidence of about 1 mm/a up to 3 cm/a relative to the stable surroundings have been observed in the vicinity of the lock in the past few years.To monitor such movements, state-of-the-art engineer-geodetic monitoring typically applies point-wise measures on the respective structures by tachymeter, precise levelling, extensometer or alignment.In contrast, we aim at area wide measurement by UAV-based monitoring using image matching and laser scanning.Hence, we are able to detect changes in areas not monitored so far.In order to measure the subsidences and their spatial extent, both techniques have to be pushed to their limits.As we would like to compete with classical engineering-geodetic survey approaches, we strive for surface representations by dense 3D point clouds at highest possible quality.Thus, we aim on a ground sampling distance (GSD) of 3-5 mm and a point density up to 400 points/m².This requires data collection at low-altitude and slow speeds and thus opts UAV platforms for data collection.The combination of both active and passive remote sensing methods generates an area wide dense point cloud.In addition, selected points (for monitoring) are permanently signalized with color markings or can be precisely re-established through forced centering of checkerboard targets.As these points are also surveyed with engineering-geodetic methods they serve as benchmark for the photogrammetrically derived point coordinates.

TEST SET-UP
In the mid of March 2018 the first flight campaign was done.The test site includes the ship lock facilities, the Neckar River in the center, and the riparian area both sides of the river.Besides vegetated areas and farmland, a large market garden with greenhouses and residential areas of the Hessigheim village are part of the site (cf. Figure 2).The maximum site extension is around 570 m (east-west) x 780 m (north-south).Data with Riegl VUX-1LR LiDAR combined with two Sony Alpha 6000 oblique cameras were captured using the RiCopter multi-copter platform, provided by University of Innsbruck, Department of Geography.
Figure 1.The ship lock (monitoring object) and a photogrammetric control point marked with a forcedcentering checkerboard target on a pillar in the foreground.
PhaseOne iXU-RS 1000 nadir images have been captured using the CopterSystems CS-SQ8 copter that is optimized for PhaseOne camera payloads.18 consecutive flight missions were necessary to fully cover the area of interest.3775 images with 80/60 overlap are arranged in 146 mostly north-south oriented flight lines, as the Neckar River's cardinal orientation is almost north-south for this test area (cf. Figure 3).The maximum site extension is around 570 m (east west) x 780 m (north south).The green triangles in Figure 2 represent the photogrammetric control points (marked with forced centering checkerboard targets) whereas the yellow dots represent the LiDAR reference surfaces.As it is also visible in Figure 3, the river itself was not captured (except the ship lock and weir parts) as its width is around 100 m and thus images would only depict water.The mean flying height was around 40 m above ground level.This results in a nominal GSD of 3.7 mm for the nadir imagery.The effective resolution of the PhaseOne imagery was already estimated from independent evaluations based on Siemens star analysis.It is shown that the MTF10 value, which sometimes is seen as vanishing resolution, is equal to the nominal GSD in the center region of the images.The resolution slightly decreases when moving to the borders of the images, as this is typically the case for camera-lens systems.In order to guarantee most stable camera geometry, the 50 mm Rodenstock lens was fix focused to 40 m distance, which is equivalent to the mean aspired flying height above ground.Precise camera calibration is obtained from self-calibrating bundle adjustment.The LiDAR data acquisition was carried out in 4 flight sessions comprising 17 longitudinal (i.e.north-south) strips, 4 cross strips (east-west), 4 diagonal strips to cover the steep wooded slope in the south-eastern corner of the investigation area, and two diagonal extra flight lines for further block stabilization (cf. Figure 2).With a flying speed of 8 m/s, a nominal flying altitude of 50 m above ground level, a strip distance of 35 m, a pulse repetition rate of 820 kHz, a scan line rate of 133 Hz and a used scanner field-of-view of 70°, the resulting mean laser pulse density is 300-400 points/m² per strip and more than 800 points/m² for the entire flight block due to the nominal side overlap of 50%.These flight mission parameters guarantee a laser footprint diameter on the ground of less than 3 cm enabling a high planimetric resolution of 5 cm.The ranging accuracy, reported in the manufacturer's data sheet of the sensor, is 10 mm (Riegl 2018).The georeferencing of UAV image and LiDAR data is based on the available ground control points and directly measured GNSS/inertial exterior orientation elements.The central part of the area, i.e. the lock and weir itself, is circled by a network of 9 rigidly fixed pillars, normally serving as permanent points for the engineering-geodetic survey network.In order to cover the much larger area of interest, 10 control points are installed on tripods additionally.These points are not permanently marked; they are only valid for the correspondent flight campaign.Their precise position is derived from multiple GNSS static baseline surveys.As for the permanent pillar points, the tripods are occupied by forced centering checkerboards, which can be replaced by GNSS antennas or prisms for geodetic surveys.The size of the checkerboard targets (27 cm diameter) is chosen to be automatically measured (cf. Figure 1) in order to minimize errors in image point measurements.In addition, five additional reference surfaces, i.e. gable roof type shaped structures fixed on tripods with known position and orientation in space, were installed for the absolute orientation of the LiDAR data.Each roof surface of these reference surfaces is of size 40 cm x 80 cm (cf. Figure 4) and contains four retro-reflective targets.As the surfaces are located nearby photogrammetric control points, the retro-targets on the plane exposed to the tripod are measured using tacheometry.The geometry of each of these reference surfaces has been determined in lab survey beforehand.Thus, the final 3D orientation in space is derived from 7 parameter transformation.The maximum residuals after transformation are better than 1 mm.Both UAV sensor platforms also incorporate additional GNSS/inertial sensors for the direct determination of sensor's exterior orientation elements (APX-15 UAV combined with iXU-RS 1000 camera and Applanix APX-20 UAV in combination with VUX scanner).This direct trajectory determination is essential for the LiDAR scanning geometry but also helps for the nadir images.In particular, additional exterior orientations will help to stabilize the photogrammetric block where images mainly capture water only and, hence, tie point extraction may fail.

PROCESSING CONCEPT AND FIRST RESULTS
A LiDAR strip adjustment was done for the whole test area, whereas the photogrammetric bundle adjustment was conducted for the western part only, including the ship lock area and the adjacent village part at the western riverside.Due to the Neckar river, the photogrammetric block is naturally divided into a western and eastern part.The five cross lines flown over the Neckar weir do not allow for a rigid connection of the two parts to form one block.This is different for the LiDAR data, as the strips are wider and thus can bridge the Neckar river.The Neckar width changes from 90-120 m.The PhaseOne iXU RS 1000 with selected 50 mm Rodenstock lens provides 56 deg opening angle (long side) corresponding to a strip width of 43 m from 40 m flying height.Different to that, the laser scanner provides a 230° Field of View, enabling range measurements even above the horizon line up to a distant of at least 250 m: Although due to laser footprint size considerations the usable LiDAR strip width was reduced to 100 m, this is still enough to connect the strips from the eastern and western bankside.

Processing of PhaseOne nadir imagery
The PhaseOne nadir image data of the western block part consists of 10 individual flights, containing 1971 images altogether.The cardinal flight direction is north-south, except for one image block, which is flown in east-west covering the ship lock itself.
The georeferencing is based on 10 available ground control points.One additional point has been used as check point.As all of the points are signalized with checker boards of sufficient size, their image coordinates have been automatically measured.The mean reprojection error is around 0.06 pix.This is considerably better than the overall reprojection error which is mainly determined from tie point matching.Its value is 0.33 pix.The control point object coordinates are weighted individually according to their standard deviations from prior GNSS static network adjustment.The weights are between 2-5 mm (std.dev.).
In addition, weighted GNSS/inertial exterior orientation elements provided by the APX-15 UAV board are introduced.
The expected quality of directly measured GNSS/inertial position and orientation data after post-processing should be within 2-5 cm for position and 0.025-0.080deg for attitude, assuming sufficient GNSS constellation and short GNSS base lines (Applanix 2018).Unfortunately, the time synchronization between camera and APX-15 board was not done correctly when setting up the copter platform: The exposure times were registered with 0.1 sec accuracy only.Thus, the full potential of the APX-15 UAV board for direct georeferencing could not be exploited.Assuming that such synchronisation will reach up to 0.05 sec timing errors this will induce errors up to 20 cm into position, mainly in flight direction as the mean copter velocity during imaging was around 4 m/s.Attitude errors are expected for all three axes.
The position errors have been analysed by comparing the GNSS/inertial positions to the estimated perspective centre coordinates from control point based bundle adjustment.The mean offset per strip in north direction (corresponding with the main direction of flight lines) changes between -35 cm for northsouth and +33 cm for south-north flight lines.This is more than the expected 20 cm, which may indicate an additional systematic delay within timing.Within the flight line, the noise in north position differences is around 7 cm (std.dev.).The later limits the positioning accuracy.In principle this strip-dependent errors can be compensated by strip-wise offset (and drift) corrections as it is known from traditional GNSS-supported bundle adjustment.This strip-wise correction was not introduced so far.For the results presented, the positions have been considered with 25 cm std.dev.only.This neglects the strip-dependent effects and thus not fully exploits the potential of the directly measured exterior orientation elements.The directly measured attitudes are used with low weights only, with almost no influence in the final adjustment.Additionally, the camera geometry is obtained from physical parameter model for self-calibration as proposed by Brown (1971).Even though the image block is composed of 10 flight missions, flown on two consecutive days, the camera was modelled with one set of calibration parameters only.
The high image overlaps and low flying height in combination with the selected wide-angle lens allow for a theoretical 3D object point quality, which should be below 1 pix in horizontal and 1.5-2 pix in vertical direction.The mean residual from control points gives a first quality estimate.It is estimated from 9 control points.Values are 2.0 mm for east, 4.1 mm for north and 6.6 mm for vertical component.This is well within the expectations.As mentioned, only one of the available points has be used as check point.Its residuals are 5.5 mm, 0.9 mm and 5.6 mm for east, north and vertical.
Additional 33 benchmark points all with checker board signalization are available and already included as signalized tie points in the bundle adjustment but at the time of paper writing their reference coordinates were not available to the authors.These points have not been measured by static GNSS base lines observations but combined tacheometry and levelling.

LiDAR VUX1 processing
Parallel to the photogrammetric block adjustment, a LiDAR strip adjustment was carried out with time dependent trajectory correction (Glira et al. 2016, Mandlburger et al. 2017).Within a sophisticated calibration procedure, the parameters of the mounting calibration (lever arm and boresight alignment), a global datum shift, as well as trajectory corrections were estimated to minimize the discrepancies (point-to-plane distances of overlapping flight strips).For absolute orientation the reference surfaces are considered.Two different versions have been investigated within this LiDAR adjustment.The first version only considers a constant offset (Δx, Δy, Δz, Δroll, Δpitch, Δyaw) per strip, i.e. the trajectory information is not deformed, but only translated and slightly rotated as a whole.In the second version additional polynomial corrections are introduced, which adds flexibility to the a priori trajectory information from GNSS/inertial processing and further minimizes the differences between strips.The residual height error after strip adjustment amounted to 4 mm measured as the median of absolute strip height differences (  ) in smooth and open surface areas.
It should be mentioned, that LiDAR georeferencing primarily relies on the quality of direct georeferencing.For the APX-20 UAV GNSS/inertial board the post-processed accuracy is defined with 2-5 cm for positioning.The attitude is specified with 0.015-0.035deg, where the first refers to the roll and pitch, the later to the heading quality (Applanix 2018).Since the LiDAR was flown in 50 m above ground errors in roll / pitch will induce about 1.5 cm position shift on ground, which is little smaller than the errors in positioning but bigger than the targeted quality.This already shows that with the use of direct georeferencing only the expected accuracy could not be reached but additional reference data and strip adjustment is needed.
Table 1 shows some first accuracy analysis, derived from the marker points.All marker points have circular checker board signals with of 27 cm diameter.This defines a clear surface for the laser measurements.For a subset of these points GNSS reference measurements are available (GNSS check points, 8 points available).Coordinates of the remaining ones only have been determined from photogrammetric bundle adjustment of PhaseOne nadir imagery (marked tie points).Notice, that the later points are mainly distributed in the ship lock area, i.e. in a limited area of the test site.For the comparison, a sphere around the marker center (radius 10 cm) is searched for LiDAR points.From the six closest LiDAR points a plane is fitted.As the markers are horizontally aligned, this estimated plane also should be horizontal.The vertical distance (normal direction) between this plane and the marker point is calculated.The statistics are given in the table.The differences to the GNSS points can be seen as absolute check, whereas the differences to the photogrammetrically derived signalized marker points show the vertical agreement between well-defined photogrammetrically determined 3D points and LiDAR points.1. Vertical quality of LiDAR points from marker point differences.
Comparing the results from bias only and polynomial strip adjustment, there are only small differences from check point analysis.The mean difference from 8 points is within a few millimetres, which is in the range of the accuracy of GNSS points.Still, the variation from point to point differences is large, up to 3 cm, which significantly exceeds the required accuracy.
Comparing the LiDAR points to the heights of 33 photogrammetrically computed tie points (with limited distribution in the shop lock area only), the mean differences are even higher, even though the variation (std.dev.) is smaller compared to the check point differences.This indicates that there are still systematic deviations between the two data sets from photogrammetry and LiDAR.Investigation into first causes is work in progress.

Comparison of 3D point clouds from LiDAR and photogrammetry
In addition to the aspired deformation monitoring at the signalized targets, our projects aims at the area covering ultrahigh precision 3D data capture of the entire ship lock and its surrounding area.For this purpose 3D point clouds are made available both from dense image matching of the PhaseOne nadir imagery and the LiDAR VUX-1 data capture.Figure 5 allows a first visual interpretation of these point clouds for a small sample area in the built-up village area part of the test region.The RGBcolorized point cloud provided from multi-view dense image matching at the top, and the LiDAR point cloud for the same area colorized by the reflectance value at the bottom.Obviously, both approaches provide a dense and accurate representation of the respective surface geometry.However, as it is also visible in the profiles from the photogrammetric and LiDAR point cloud depicted in Figure 5, there are considerable differences in the results from both techniques.Dense multi-view-stereo-matching provides 3D information basically for each image pixel at considerable quality if sufficient image overlap and texture is available.In contrast, the polar measurement principle of LiDAR sensors is advantageous whenever the object appearance changes rapidly when seen from different positions.This for example holds true for semi-transparent objects like vegetation, which is especially important for our application aiming at the extraction of the bare earth for monitoring vegetated areas.The same effects can be observed in the profiles from the photogrammetric and LiDAR point cloud depicted in Figure 6.The blue points representing the LiDAR measurements provide a large amount of information for the tree, which is not available in the red points generated from multi-view-stereo matching.In contrast, these points from image matching feature a striking resolution for surfaces like the gable of the roof.This is visible in the close-up of the profile in Figure 6.Please mind, how good the roof ridge and the roof shingles are mapped, which can also be noticed from the next Figure 7.  was down-sampled to 5 cm (factor 2).This fits to the native sampling of the LiDAR points, when only one flight strip is considered.The GSD of nadir images is close to 4 mm as mentioned before.The DEM from dense image matching (Figure 7, top) is given as a 1 cm grid here, again down-sampled by a factor of two.Especially for the close-ups at the roof tiles, the higher resolution of the geometric reconstruction compared to the result from LiDAR is clearly visible.Again, notice the differences at trees.

CONCLUSION AND FURTHER WORK
This paper introduces an UAV-based LiDAR and image data project for the ultra-high precision mapping and monitoring of infrastructures or other objects.As the processing is still ongoing, this paper is not able to give a comprehensive quality estimation already.While the preliminary results are promising w.r.t.high spatial resolution and potential to fulfil the accuracy demand for aerial monitoring of very small subsidences, systematic effects are still visible in the LiDAR strip height differences and especially when comparing the LiDAR point cloud to the photogrammetric one.This is subject for the upcoming investigations, namely:  GNSS/inertial supported bundle adjustment of nadir image with strip-wise correction of strip dependent systematics, in order to better compensate for the noncorrect time synchronisation. Additional reference objects for LiDAR strip adjustment selected from photogrammetric point cloud after georeferencing. Combined LiDAR / photo adjustment.It is quite clear, that with a requested accuracy of better than 5 mm in object space, the limit of aerial mapping is close to reach.Still potential is left for refining the modelling of LiDAR and image data, in especially the combined adjustment of image and LiDAR seems to be straightforward.The same holds true for the integrated evaluation during point cloud generation from image matching and LiDAR measurement.While matured hardware and software tools are available for point cloud generation from image matching and LiDAR, up to now both approaches were considered as competing techniques with research efforts focussing on the individual improvement of sensors and algorithms.In our future work, we will also aim on the suitable combination of both data sources to further increase robustness, accuracy and reliability of 3D point clouds while aiming at ultrahigh precision applications from UAV-based data capture.

Figure 4 .
Figure 4. Photogrammetric control point (left) and LiDAR control surface (right) mounted on tripods.

Figure 6 .
Figure 6.Comparison of a profile through the photogrammetric (red) and LiDAR (blue) point cloud.

Figure 7 .
Figure 7. Shaded DEM from dense image matching (top, 1 cm grid width) and LiDAR DEM (bottom, 5 cm grid width).The marked region is zoomed-in.The Figure 7 depicts the shaded reliefs from LiDAR and photogrammetric point cloud for the same built-up area already shown in previous Figure 5.The corresponding ortho is shown in Figure 8.The GSD of LiDAR DSM depicted on top of Figure 7 was down-sampled to 5 cm (factor 2).This fits to the native