KORTOWO TEST FIELD FOR TESTING PHOTOGRAMMETRIC PRODUCTS ACCURACY – DESIGN AND FIRST EVALUATION

The continuous development of sensors, methods and technologies in the modern digital photogrammetry requires testing the quality and accuracy of software, processing workflow and products. The paper presents a new test field for performance analysis of software processing and accuracy assessment of photogrammetric 2D and 3D data collection, mapping, 3D object reconstruction and modeling based on low-altitude imagery with particular regard to unmanned aerial vehicles imagery. The first experiment was carried out using images captured by Phase One iXU-RS 1000 medium format aerial digital camera and Light Detection and Ranging (LiDAR) point cloud acquired by RIEGL LMS-Q680i airborne laser scanner. The process of complex digital processing was performed in Agisoft Metashape packages. The subblock of 169 images and 16 signalized ground points measured by Global Navigation Satellite Systems in the WGS 84 coordinate system using the Real-Time Network method were adopted in the preliminary investigations. The root mean square error RMSEXYZ on check points in the bundle block adjustment was equal to 0.032 m. Vertical deviations between digital elevation model and LiDAR point clouds belong to the range from -0.020 m to 0.020 m which is related to RIEGL LMS-Q680i accuracy and precision. Georeferenced orthomosaic was generated with ground sampling distance (GSD) equal to 0.020 m, which was the same as the GSD of input images. The high accuracy of obtained processing results is related to accuracy of initial data, and it proves the usefulness of Kortowo test field.


INTRODUCTION
In the last few years, there has been a rapid development of sensors (various spectral range) and innovative image processing techniques in digital photogrammetry and computer vision. Besides, the novel technology of unmanned aerial vehicles (UAVs) has made huge progress and have opened perspectives for a wide range of photogrammetric application in new fields.
Commercially off-the-shelf (COTS) software packages focusing on UAV image processing work in the composed and often in an autonomous processing chain (Jaud et al., 2016;Hung et al., 2019;Remondino et al., 2017). They are applied for the automatic estimation of exterior and interior image parameters using the bundle block adjustment (BBA) method with selfcalibration (Oniga et al., 2018), for computation of the dense point clouds (Remondino et al., 2017), for surface reconstruction in form of the digital surface model (DSM) (Jaud et al., 2016;Uysal et al., 2015) and finally for generation of the orthomosaic (Mesas-Carrascosa et al., 2014). These software systems automatically generate large numbers of tie points, based on different image matching approach and implementation of SfM (Structure from Motion) method (Westoby et al., 2012).
Nowadays, the scope of problems and research tasks in the UAVs photogrammetry technology is very wide (Crommelinck et al., 2016;D'Oleire-Oltmanns et al. 2012;Gevaert et al., 2017;Pajares, 2015;Remondino et al., 2014;Yao et al., 2019;Zongjian, 2008). Traditionally, photogrammetry has always focused on the evaluation of the accuracy and precision of the mapping. An excellent confirmation of this trend was the test field created by the Institute for Photogrammetry of the University of Stuttgart (Cramer, Haala, 2010) under the auspices of German Society of Photogrammetry, Remote Sensing and Geoinformation (DGPF). The objective of this test was to evaluate the sensor technical attributes and their relevance to the specific applications and to investigate the software processing chain in the preparing process of photogrammetric products. Similarly, an important task is also testing the accuracy potential (Colomina, Molina, 2014;Haala et al. 2011;Harwin, Lucieer, 2012;Remondino et al., 2012;Zhou et al., 2018) of new measurement technologies and final products based on UAVs imagery and consequently searching for new areas of its practical applications (Gindraux et al., 2017;Mesas-Carrascosa et al. 2016).
Depending on the type and purpose of the UAV images target processing, different types and configurations of test fields have been used to study. A thorough calibration of the camera system mounted onboard UAV platforms is very important and is performed most often in the laboratory using a small volumetric test field and planar pattern (Hastedt, Luhmann, 2015) or the 3D test field (Oniga et al., 2018), which consists spatially distributed, coded and non-coded targets (Cramer et al., 2017). In the data collection or mapping case, the test fields are mainly characterized by a small area and homogeneous surface (Eling et al., 2015) or also topographical differentiated terrain (Cramer et al., 2017;Wierzbicki et al., 2015), with evenly located ground control points (GCPs) and check points (ChPs). Due to the limited scope of field surveys, the used approaches usually had a small number of GCPs and ChPs (Barry, Coakley, 2013;Saponaro et al., 2019;Wierzbicki et al., 2015). The test fields with an area of several square kilometers and varying terrain elevation and topography, as well as dense GCPs network, requiring a large number of surveys, are in practice very rarely realized. A phenomenon is the test area presented in the work of Haala et al., 2011. A fundamental dataset for research in the field of the present UAV photogrammetry is the "ISPRS benchmark for multiplatform photogrammetry" (Nex et al., 2015) created under the auspices of the ISPRS and EuroSDR, which concerns two test areas, each less than 0.1 km 2 .
Current researches are concerning the accuracy evaluation of UAV-based photogrammetric 2D and 3D data collection and large-scale mapping. They were realized in incomparable technical conditions of projects and using various statistical parameters for results assessment (Barry, Coakley, 2013;James et al., 2017b) relating to the particular stages of the processing (James et al., 2017a;Mesas-Carrascosa et al., 2016;Murtiyoso et al., 2018). The root mean square errors (RMSEXYZ) on ground control and check points was the most commonly used metrical information of accuracy. Besides, the quality and accuracy of DSMs and orthomosaics were analyzed using RMSZ and RMSXY deviations, respectively.
Based on our previous experience (Gabara, Sawicki, 2018), we decided to design and create a multi-purpose test field. The main aims of the new test field are photogrammetric software performance benchmark and accuracy evaluation of photogrammetric products based on low-altitude imagery with particular regard to UAV imagery.
The paper focuses on presenting the test field and the first results of data collection based on images captured through medium format aerial digital camera and point cloud acquired using an airborne laser scanner.

KORTOWO TEST FIELD
In order to the accuracy assessment on UAV-based photogrammetric data collection, with particular emphasis on the evaluation of respective stages of digital images processing and computation, the new very high accuracy test field (B = 53.75500, L = 20.45785) was designed and constructed in the Kortowo campus of the University of Warmia and Mazury in Olsztyn (Poland).
The complete test area covers about 2 km 2 (2.150 × 0.950 km) with 68 evenly distributed GCPs ( Figure 1). The test site is characterized by diversified terrain elevation and different topographic objects, i.e. buildings, roads, sports fields, lakes, forest, parks, green areas.
In the beginning, the different shapes and colors of targets were tested. For this purpose, the simulation of the nadir and oblique photos in different lighting conditions were carried out using the DSLR Nikon D5100 camera. The assumption of the test was to achieve real condition of aerial capturing. Figure 2 shows the experiment.
The GCPs centers were stabilized on solid ground by geodetic nails. The artificial GCPs were marked by a white and black color paint using a precise metal template. The circle shape of the target has an internal and external diameter, respectively to 0.18 m and 0.45 m (Figure 3). The GCPs size allows the acquisition and processing of digital images with a small ground sampling distance (GSD) to 1 ÷ 2 cm level. The shape and color of the GCPs have been adapted for automated measurement of image pixel coordinates using the center-weighted method (centroid operator). Definitive, the coordinates of GCPs were measured with high accuracy using two methods: static GNSS with the estimated accuracy to mXY = 0.005 m and precise leveling with received mean accuracy to mZ = 0.0006 m as an adjustment results in GEONET software (http://www.geonet.net.pl).
The designed Kortowo test field allows testing the quality and accuracy of all photogrammetric products generated from lowaltitude images acquired mainly from UAVs, i.e. the bundle block adjustment with camera self-calibration, systems calibration consisting of various sensors, dense point clouds, DSMs, orthomosaics, true orthophoto maps, 3D object reconstruction, 3D city models with different typologies of buildings for level of detail, mainly in the LOD3 and LOD4, 2D and 3D data collection in order to large-scale basic and thematic mapping. Besides, it is possible to test the accuracy of derived products obtained from dense point clouds from airborne laser scanning (ALS).

TEST DATA ACQUISITION
The reference data for further accuracy assessments were obtained by the Phase One iXU-RS 1000 camera and RIEGL LMS-Q680i laser scanner in the first flight mission.

Phase One iXU-RS 1000
In the first experiment Phase One iXU-RS 1000 medium format digital camera for aerial photography of Phase One Industrial company was used to obtain the block of photos. Table 1 presents the basic technical parameters of Phase One iXU-RS 1000.

RIEGL LMS-Q680i
For the LiDAR (Light Detection and Ranging) data acquisition, RIEGL LMS-Q680i long-range airborne laser scanner was used.

Flight mission and data acquisition
The first flight mission on the test field was performed in cooperation with Polish company Visimind Ltd Sp. z o.o. in Olsztyn, using Hughes MD 500 helicopter ( Figure 4). The sun was about 56⁰ high above the horizon.
Initial camera preferences were defined manually. The lens focus was set to infinity, and the aperture was set to 5.0. The light source and white balance were set to Auto. Due to poor lighting conditions, ISO sensitivity and shutter speed were changed by the operator in manual mode in the range of respectively to 100 ÷ 500 and 1/1000 ÷ 1/2500 sec.
During the flight 583 nadir photos in the 12 strips were captured. The planned flying altitude was ca. 220 m above ground level (AGL), with ground pixel size to GSD = 0.02 m. The 7 photo strips were carried out approximately in the SN and NS directions. In addition, 5 cross photo strips oriented in EW and WE directions were done: 2 at the beginning, 2 at the end, and 1 in the middle of the photo block. The exposure interval was 1 second. Projected forward and side overlap was respectively to p% = 80% and q% = 60%, and the base-toheight ratio was to ν = 0.16. The ground frame size of single image was to 235 × 175 m.
RIEGL LMS-Q680i was set to max. laser pulse repetition rate of 400 kHz. The acquired LiDAR point cloud of the test area had the density to 25 points/square meter (pts/m 2 ).

TEST AND ANALYSES
For the preliminary investigations, the subblock of 169 images (6 strips NS and SN and 2 strips EW and WE) and 16 signalized ground points measured by GNSS RTN were used. The localization of image projection centers and image orientations are presented in Figure 5. The points pixel coordinates were measured automatically using a centroid operator. For the test purposes, the bundle adjustment with selfcalibration, dense point cloud, digital elevation model (DEM) and the orthomosaic were computed.
The data processing was carried out on workstation with the processor Intel® Core™ i9-7940X, 128 GB RAM DDR4-3200 MHz memory, MSI 1080 Ti graphic card and Samsung 960 Pro SSD hard drive.

Bundle Block Adjustment
The bundle block adjustment (BBA) was carried out in two variants. In variant 1, all ground points were used as a GCP. The variant 2 included 7 GCPs and 9 ChPs. The parameters of laboratory camera calibration were used in the aerotriangulation solution as approximated. The computation factor of 1000 tie points and 2000 key points per 1 Mpix was used for matching purposes.
Furthermore, the 'marker accuracy', which is the measurement precision of the 3D GCP coordinates was set to 0.05 m. The 'projection accuracy' and 'tie point accuracy', which are the precision in image space that point observations are made to, and for computations, they were set to 0.2 pix and 1 pix, respectively.  Table 3. BBA results

Digital Elevation Model and orthomosaic
The depth maps for images and the dense point cloud were calculated using ultra high quality and aggressive filtering options. The generated DEM ( Figure 6) for processed subblock of images is characterized by the following parameters: The CloudCompare (Girardeau-Montaut, 2019) application was used to analyze the vertical deviations (VD) between generated DEM and the adjusted LiDAR point cloud. The last squares planar fitting of 3D points (James et al., 2017b) was applied to achieve a better approximation of true distances. The results are presented in Figure 7.
Ca. 1.8 out of 2 billion of VD belong to the range ± 0.020 m, which is related to RIEGL LMS-Q680i accuracy and precision. The outliers are localized mainly on trees.
For the orthorectification in Metashape software, the same GSD size of the orthomosaic and acquired images (GSDortho = GSD = 2.0 cm) was assumed.
To generate an orthomosaic, the DEM was used as a surface, and the mosaic option was set to blending mode. The obtained result is presented in Figure 8.

CONCLUSIONS
Testing the accuracy potential of new measurement technologies and final products based on UAVs imagery is a very important task. It is not usually possible to carry out such tests under real production conditions. For this reason, the Kortowo test field was designed and created to evaluate the photogrammetric software and the accuracy of photogrammetric products based on low-altitude imagery with particular regard to UAV imagery.
High resolution (100 megapixels) images (GSD = 2 cm) captured by Phase One iXU-RS 1000 medium format aerial digital camera and LiDAR point cloud (25 pts/m 2 ) acquired using RIEGL LMS-Q680i airborne scanner were used in the first experiment.
The processing results obtained in the Agisoft Metashape suite proved the high accuracy of photogrammetric data collection. The RMSEXYZ on check points in the bundle block adjustment was equal to 0.032 m. Vertical deviations between DEM and LiDAR point clouds belong to the range |VD| < 0.020 m which is related to RIEGL LMS-Q680i accuracy and precision. The high accuracy of obtained processing results is related to accuracy of initial data, and it proves the usefulness of the Kortowo test field.
For further study, new artificially signalized ground points on the roofs of the buildings as well as natural, well-defined ground points (water gate valves, manholes, curbstones) will be measured using the static GNSS or GNSS RTN method. In addition, a new photogrammetric reference dataset (high-resolution imagery and LiDAR) is going to be obtained using a manned aircraft. Furthermore, the different types of UAVs and sensors will be used for tests. The obtained results allow for comprehensive performance analysis of software processing and accuracy assessment of photogrammetric products based on UAVs imagery.