INTERACTION BETWEEN DIRECT GEOREFERENCING, CONTROL POINT CONFIGURATION AND CAMERA SELF-CALIBRATION FOR RTK-BASED UAV PHOTOGRAMMETRY

Unmanned Aerial Vehicles (UAV) are enjoying increasing popularity in the photogrammetric community. The Chinese supplier DJI is the market leader with about 70% of the global consumer UAV market. The Phantom model has been available for more than 10 years and its current version "RTK" is equipped with a 2-frequency GNSS receiver, as a basis for direct georeferencing of image flights, using RTK or PPK technologies. In the context of the paper, different case studies are investigated, which allow statements on the geometric accuracy of UAV image flights as well as on the self-calibration of the camera systems. In the first example, four DJI Phantom 4 RTK systems are examined, which were flown in a cross flight pattern configuration on the area of the UAV test field "Zeche Zollern" in Dortmund, Germany. The second example analyses the results of an open moorland area where the establishment of GCPs is extremely difficult and expensive, hence direct georeferencing offers a promising way to evaluate deformations, soil movements or mass calculations. In this example a DJI Matrice 210 v2 RTK drone has been used and the results of two different software packages (Agisoft Metashape and RealityCapture) are analysed. The third example presents a reference building that has been established by the Lower Saxony administration for geoinformation in order to evaluate UAV photogrammetry for cadastral purposes. Here again a DJI Phantom 4 RTK has been tested in a variety of flight configurations and a large number of high precision ground control and check points. The case studies show that the RTK option leads to sufficient results if at least 1 GCP is introduced. Flights without any GCPs lead to a significant height error in the order of up to 30 GSD.


INTRODUCTION
Unmanned Aerial Vehicles (UAV) are enjoying increasing popularity in the geodetic-photogrammetric community. The market -with complete systems offered by various manufacturers -is growing steadily. The Chinese supplier Da-Jiang Innovations Science and Technology Co, Ltd (DJI) is the market leader, with a current share of approx. 70% of the global consumer UAV market (Handelsblatt 2020). The Phantom 4 model (and its predecessor) has been available for more than 10 years and its current version "RTK" is equipped with a 2-frequency GNSS receiver (Figure 1a). With the DJI Matrice 210 v2 RTK another UAV is available, which is able to use 2 cameras simultaneously ( Figure 1b).
In connection with a reference station or alternatively through the connection via NTRIP (Network Transport of RTCM via Internet Protocol), via mobile radio or Wi-Fi hotspot, it is possible, for example, to use the SAPOS service (SAPOS 2020), and thus precise positioning in real time. The manufacturer's specifications (DJI 2020) for positioning accuracy are:  vertically: 1,5 cm + 1 ppm (RMS)  horizontally: 1 cm + 1 ppm (RMS) Since the system also provides raw data in RINEX format (Receiver Independent Exchange Format), it is also possible to determine positions using PPK (Post-Processed Kinematic). The GNSS raw data of a dual-frequency receiver (code and carrier phase observations as well as the ephemeris data) are the basis for subsequent evaluation, which usually leads to the determination of position solutions of higher accuracy. The user is thus offered extended possibilities for georeferencing UAV image flights. In addition to an (indirect) orientation using ground control points (GCP), both direct orientation on the measured image positions (external orientation -EO) and a combination of the two approaches (integrated orientation) is possible. In the context of the paper, different case studies are investigated, which allow statements on the geometric accuracy of UAV image flights as well as on the self-calibration of the camera systems. Table 1 gives an overview about flight parameters.

RTK image flight
Planning and execution of the image flight with a DJI RTK quadcopter system are carried out using a proprietary software package from the manufacturer (DJI GS PRO), which also contains the specifications for obtaining the GNSS correction data ( Figure 2). Figure 2. Definition of RTK settings in the GS PRO application (photo: DJI) During the image flight, for each image a frame number, a time stamp, the components of the lever arm between the antenna center and the image center of the CMOS sensor, the complete position data (in WGS84 or ETRS89 when using the SAPOS service HEPS), associated accuracy information and the RTK status are logged for each image (File Timestamp.MRK).
It should be noted here that the definition of the lever arm -as a vector between the antenna center and the projection centerdiffers from the usual definition in photogrammetry! Furthermore, the original satellite observation data as well as the ephemeris data are collected and stored in a PPKRAW.bin file in RTCM 3.2 format. Additionally, the system converts the satellite data on the fly into the RINEX format (Receiver Independent Exchange Format) and writes these data into a RINEX.obs file. This also provides all relevant information for a PPK evaluation. This can be carried out on demand, e.g. on the basis of the free RTKLIB software (Bäumker, 2014;Takasu, 2020;Przybilla, Bäumker, 2020).
For further processing, the available position data must usually be converted into a target coordinate system, in Germany often into the national coordinate system ETRS89. Since the height information is available as ellipsoid coordinates after the flight, a geoid undulation (currently in GCG2016 -German Combined Quasigeoid Model) must also be applied as a correction term. The results are heights in the German Main Altitude Network DHHN2016. "The horizontal variations" of the quasigeoid can take amounts up to 10 mm per km. Quasigeoid variations must therefore also be taken into account in local height determinations, e.g. with GNSS (BKG 2020a). In addition to a web application (BKG 2020b), BKG also sells a software as a desktop solution to solve this task.

Results of investigations
The following evaluations were carried out for four different Phantom 4 RTK systems, used on three different days. The evaluations were performed with the software Agisoft Metashape. All calculations are based on identical parameterization to ensure comparability of the results.

Quality of RTK measurements:
If the absolute orientation of a bundle block is to be carried out on the basis of measured external orientations, the question of the quality of the measured position data must first be clarified. The manufacturer's specifications listed in chapter 1 regarding the achievable accuracies are in a range that requires both an optimal satellite configuration and an undisturbed reception of the real-time correction data. It is not reasonable to assume that these basic conditions are valid at all times of a flight campaign. The manufacturer names four different quality levels for the RTK status, which is also displayed in the control app during the image flight (

Direct georeferencing with RTK:
The RTK measurements shown in Chapter 2.2.1 can be used as a basis for image orientation within the framework of bundle block adjustment (BBA), with the aim of reducing the number of necessary control points or even completely dispensing with them (Przybilla et al., 2015;Gerke, Przybilla, 2016).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition) The following investigations are based on typical datum configurations, which are composed of a combination of ground control points and measured external orientation (integrated orientation). In contrast to image sets of classical aerial photogrammetry, sufficiently accurate measurements of the orientation angles provided by a high-precision inertial measurement system are not available here. The configurations listed in Table 2 have been evaluated. Depending on the individual block, the maximum number of GCP varies between 45 and 50. For all configurations a uniform internal orientation (UNIFIED) was introduced for the two partial flights of the crossflight. A further calculation was performed with two separate interior orientations (SEPARATE) for a block orientation with observed EO and 4 GCP in the block corners. The aim of this is to detect a possible influence of changing camera focus. The RTK data collected during the flights with the DJI systems were introduced into the BBA as observations with their a priori accuracies. Additionally, the influence of different GCP configurations was evaluated. Fig. 4 shows the residuals of the observed external orientations after the bundle adjustment.
In Agisoft Metashape these values are denoted as "ERROR". The results shown in the figure are based on a datum definition consisting of the RTK measurements and additional 4 control points in the block corners. The residuals are in the order of magnitude of the observation accuracies (system A) for image flights with a high proportion of FIX solutions. The results for systems D indicate the (in parts) systematically bad RTK measurements by correspondingly high improvements.
The check of the block geometry, here in particular also of block deformations, is carried out via the residual deviations (RMSE) at the check points (CP, Figure 5). The figure very clearly shows the influences of the type of orientation on the respective block.
The following effects can be derived:

Direct orientation using measured EO
The deviations at the CP are, in relation to the ground coordinates, in the magnitude of the RTK accuracy (10-20 mm), but a significantly large deviation in height is shown. This is well over 100 mm for systems A and D and just under this value for system C. Only for system B the deviation is within the range of the observation accuracy. A reliable georeferencing for this variant (without GCP) is not recognizable.

Indirect orientation with maximum number of control points
As all GCP are used for referencing in this variant, it is not possible to control the system using independent CP. However, Figure 5 shows the high quality of the adjustment to GCP, which is approx. 0.5 -0.7 of the GSD. The variant considered here is associated with a very high terrestrial effort.

Indirect orientation with minimum number of control points
From the georeferencing using 4 GCP in the block corners it is clearly visible that no sufficient stability can be achieved in the image blocks. While the ground deviations of the CP are still within the range of the GSD, the height deviations exceed these by a factor of 15 -30. One reason for this result can be seen in the metric of the camera and the obviously insufficient possibilities for a simultaneous self-calibration (chapter 5).

Integrated orientation using measured EO and 4 GCP
The present results show the effectiveness of the integrated orientation based on the RTK measurements in conjunction with ground control points in the block corners. The deviations at the CP are within the range of the GSD, in some cases even below.
The results obtained are only slightly worse than the variant with a full control point referencing.
Integrated orientation using measured EO and 1 GCP While direct orientation by means of measured exterior orientation is characterized by significant height deviations in the available data sets, the positive effects of an additional ground control point (in the middle of the block) are clearly visible. The systematic height deviations at the CP are reduced to approx. 15 -30 mm and are thus on the same level of accuracy as the RTK measurements. The quality of the image orientation accuracy achieved here is sufficient for e. g. topographic applications.

Georeferencing of the image blocks using Post-Processed Kinematic (PPK):
DJI RTK systems offer, due to the availability of the original satellite observation as well as ephemeris data, the possibility of an improved position determination in post-processing (PPK). Figure 6 shows PPK evaluation for system D (RTK results see Figure 5). identical results. The presumed gain in accuracy through postprocessing is not detectable in the present data set. However, the PPK solution is of significant importance wherever a poor mobile radio infrastructure prevents the use of real-time correction services. Problems of limited satellite reception cannot be compensated by the PPK either. Extended information can be found in Bäumker, Przybilla (2020).

RTK image flight
A test area of moorland has been surveyed by DJI Matrice 210 RTK V2 in a strip-wise flight configuration using local ground reference station from DJI (Eilers, 2020). A GSD of 18mm has been achieved. For practical reasons only 8 signalised GCP could be provided (Figure 7). Special interest was drawn to the use of RTK data in order to avoid GCP or at least minimize the number of GCP due to high effort on site. The area is almost free of vegetation. Main application was to derive terrain deformations.
A systematic offset of about 40cm between the averaged position of the two antennas and the camera center has been observed which could be calibrated and applied for correcting the measured EO.

Investigations of GCP:
Using GCP only without RTK data results in systematic dome-shaped error distributions in object space as shown in Figure 8. When RTK data is introduced as weighted observations to the BBA, significant improvements of object coordinates can be achieved, even if only 1 GCP is used.
Using RTK observations only for EO leads to systematic positive or negative offsets in the resulting point clouds which may reach several meters in Z. Depending on the specific flight conditions and GCP configuration, different accuracies in X and Y can be observed at independent check points. Introducing RTK has resulted in Z deviations that are even smaller as for X and Y. As a summary, the accuracy on check points using RTK and GCPs lies in the order of 1.5-2 GSD for X,Y and 1-1.5 GSD for Z using the DJI Matrice.
It was further investigated how camera calibration and RTK/GCP variations interact. Using RTK without any GCP leads to a high correlation of principal distance c and Z position of the camera, thus to significant deviation and standard deviation of the principal distance with respect to optimal configurations. Again, introducing at least 1 GCP improves the precision of c from about 3.7 pixels down to 0.3 pixels. It has to be noted that a flat terrain as for moorlands usually leads to a weak calibration of the camera if no flight in different heights are conducted.

Investigations of pointclouds:
Assuming that a configuration with RTK and a maximum number (8) of GCP yields the best result in object space, the respective surface model can be regarded as a reference for relative comparison to surface models that are created from reduced GCP/RTK configurations. Figure 9 shows the results of cloud-to-mesh deviations for Agisoft Metashape calculations (white: GCP, red: CP). It is clearly visible that a reduced number of GCP, like 5 or 2, still results in acceptable results with medium deviations of about 1-2cm. Using only 1 GCP leads to a systematic tilt in deviations.
[m] Figure 9. Comparison of point clouds from Metashape; top left: 5 GCPs; top right: 2 GCPs; bottom left: 1 GCP 3.2.3 Comparison to RealityCapture: The SfM software RealityCapture is characterized by high-speed data processing (about 10 times faster than Metashape) and the optional integration of terrestrial laser point clouds. In our case the objective was to compare the results of bundle adjustment and self-calibration. Besides a number of operational differences and different parameter settings, both software packages generate comparable results for GCP count 8 and 5. However, for 2 or 1 GCPs the RealityCapture result yields 3D errors in the order of up to 1m together with a systematic, dome-shaped behaviour ( Figure 10, white: GCP, red: CP). Independently of GCP settings, the mean reprojection error lies in the order 0.5 px. A final investigation of these effects and for possible reasons will be carried out in the near future. [m]

CASE STUDY REFERENCE BUILDING
For the investigation and evaluation of UAVs in the cadastral administration, a comparison point field is currently established. This is located on a 6000 m² plot of land on which 33 control points of different heights and geometrical constellations are placed ( Figure 11). The precise survey is carried out using a geodetic network, separated in position and height. The adjusted coordinates were investigated by a test flight with a DJI Phantom 4 RTK (Kreyenkamp, 2020).   The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition)

CAMERA CALIBRATION
The cameras installed into the DJI drones cannot be regarded as metric, stable cameras. Hence, a simultaneous calibration is necessary to achieve reliable results in object space. Depending on the flight configuration and the 3D shape of the terrain, parameters of interior orientation may show high numerical correlations between each other, but also to the EO parameters.
In addition, the built-in rolling shutter (DJI Matrice with Zenmuse X5S) may cause additional systematic errors that need to be modelled within the bundle adjustment. As a summary, the case studies above indicate the following conclusions:  Cross-flights and flights of different heights improve camera calibration, especially of the principal distance, due to a better 3D configuration of the terrain.  Flights with oblique images also improve camera calibration due to higher depth in images.  A weak estimation of the principal point leads to possible scaling errors in Z.  The complete set of Brown parameters should be applied.  Modelling of rolling shutter shows improved results (only in Metashape)  Count and distribution of GCPs has no significant impact to camera calibration Within the use case "Reference Building" it was possible to investigate the calibration of the principal distance c (focal length) under different flying heights, gimbal inclinations, GCP and RTK conditions. Table 4 summarizes the variations of c for cross-shaped flights. It is visible that c can be estimated more or less consistently if RTK is used, regardless of flying height and gimbal inclination. For flights without RTK (numbers 5 and 6) c drops down significantly, hence a scaling problem in Z will arise as it has been discussed above ( Figure 13).
The DJI cameras show significant systematic residuals after selfcalibration. Figure 14 displays the result for the Phantom 4 camera which appears in similar figures for other projects as well.
The main reason is assumed to be the lens design which consists of aspherical lenses which could not be modelled by the standard set of Brown distortion parameters. Figure 15 shows the residuals for the Zenmuse camera where again systematic effects of larger than 1 pixel can be observed that may also be caused by the rolling shutter of this camera.  Figure 13. Variations of principal distance for different flights The DJI cameras show significant systematic residuals after selfcalibration. Figure 14 displays the result for the Phantom 4 camera which appears in similar figures for other projects as well.
The main reason is assumed to be the lens design which consists of aspherical lenses which could not be modelled by the standard set of Brown distortion parameters. Figure 15 shows the residuals for the Zenmuse X5S camera where again systematic effects of larger than 1 pixel can be observed. Note that the rolling shutter compensation in Metashape was activiated.

CONCLUSIONS
With the availability of RTK for UAV flights, an important step has been taken with regard to the economic use of UAVs in photogrammetry. The RTK drones under investigation provide sufficient possibilities for applications in the medium accuracy range (> 2 -4cm) to significantly reduce the terrestrial effort for georeferencing. However, datum definition without at least one GCP does not lead to acceptable results. This has been demonstrated by three field studies under different environmental and flight conditions.
The quality of the camera is worth mentioning. The concept of the system used in the Phantom 4 RTK is identical to that of the Zenmuse X4S. The camera consists of a global shutter and has a high level of stability, which is significantly higher than other systems from the manufacturer. From a photogrammetric point of view, despite these positive characteristics, it is not a metric camera, so in-situ calibration is urgently required.
Future work is addressed to a deeper analysis of systematic residuals in image space that, however, are not related to the RTK option.