QUALITY EVALUATION OF 3D BUILDING MODELS BASED ON LOW-ALTITUDE IMAGERY AND AIRBORNE LASER SCANNING POINT CLOUDS

The term “3D building models” is used in relation to the CityGML models and building information modelling. Reconstruction and modelling of 3D building objects in urban areas becomes a common trend and finds a wide spectrum of utilitarian applications. The paper presents the quality assessment of two multifaceted 3D building models, which were obtained from two open-access databases: Polish national Geoportal (accuracy in LOD 2 standard) and Trimble SketchUp Warehouse (accuracy in LOD 2 standard with information about architectural details of façades). The Geoportal 3D models were primary created based on the airborne laser scanning data (density 12 pts/sq. m, elevation accuracy to 0.10 m) collected during Informatic System for Country Protection against extraordinary hazards project. The testing was performed using different validation low-altitude photogrammetric datasets: RIEGL LMS-Q680i airborne laser scanning point cloud (min. density 25 pts/sq. m and height accuracy 0.03 m), and image-based Phase One iXU-RS 1000 point cloud (average accuracy in the horizontal and in the vertical plane is respectively to 0.015 m and 0.030 m). The visual comparison, heat maps with the function of the signed distance, and histograms in predefined ranges were used to evaluate the quality and accuracy of 3D building models. The aspect of error sources that occurred during the modelling process was also discussed.


INTRODUCTION
The photogrammetric 3D modelling and reconstruction of scenes and objects in a varied spectrum of applications are currently performed based on laser scanning (Oude Elberink and Vosselman, 2011;Rutzinger et al., 2009;Pilarska et al., 2017;Li et al., 2019) or image data (Remondino et al., 2011;Remondino et al., 2014;Dahlke et al., 2015;Gabara and Sawicki, 2019a). Image-based dense point clouds are generated using dense image matching. Some of the most commonly implemented techniques are the commercial SfM (Structure from Motion), the scalable MVS (Multi-View Stereo) approach, and SGM (Semi-Global Matching) method (Westoby et al., 2012;Pepe et al., 2019;Remondino et al., 2011). Photogrammetric approaches enable a highly redundant bundle block adjustment (BBA), simultaneous digital camera selfcalibration and automatic 3D scene reconstruction. The dense point clouds and advanced data processing are allowing to recognise and completely reconstruct 3D objects and then to measure and extract geometric and semantic information.
Currently, 3D building reconstruction becomes a common trend. They are feeding national and private databases and geoportals. Every day, they are used for property visualization, urban analysis (Batty et al., 2000), urban design (Zhang and Zhu, 2004), real-time disaster management (Rezaeian and Gruen, 2011), solar potential estimation (Zhang et al., 2015) or are being a part of building information modelling (BIM) and augmented reality (AR) (Wang et al., 2013, Biljecki et al., 2015. For this reason, they have been a focus of attention for national mapping agencies and city authorities, which are usually involved in projects and campaigns for possessing 3D building databases and map out genuine presentations of these data. The most popular and utilized standard of 3D building presentation is CityGML 2.0 (City Geography Markup Language) defined by Open Geospatial Consortium (2012). It assumes that the level of detail (LOD) (Clark, 1976;Goetz, 2013;Biljecki et al., 2014) concept is used to differentiate the geometric and semantic representations of 3D building models (Biljecki et al. 2016). While LOD could be used to evaluate the visual presentation and complexity of the models, another aspect that is taken into considerations before practical using this kind of data -is their accuracy. Therefore, important research tasks concern the evaluation of the accuracy and quality of image-based dense point clouds generation and processing aiming at the 3D object reconstruction (Suveg and Vosselman, 2004;Oude Elberink and Vosselman, 2011;Pilarska et al., 2017;Gabara and Sawicki, 2019a, b).
In the literature, two methods of 3D building models accuracy evaluation are described. The first method is realized through the comparison of the prepared and reference models with the assumption that both models are available in vector and raster format, e.g., pixel-based and object-based evaluations (Rutzinger et al., 2009). This kind of approach was realized in the ISPRS benchmark on urban object detection and 3D building reconstruction (Rottensteiner et al., 2014).
In the second method authors dealing with the point clouds acquired through an image-based approach or by airborne laser scanning (ALS). Dorninger and Pfeifer (2008) in the evaluation of 3D buildings, proposed using distances between the 3D models and the LiDAR point cloud, rather than using points acquired by tacheometer, which was described as a partial method in terms of quality assessment. A more complex evaluation was presented by Oude Elberink and Vosselman (2011). Their approach was related also to airborne laser scanner data of about 20 pts/m 2 , but in addition to the method mentioned before, they presented two new features. First was the distance between each corner and nearest point (assigned to roof plane), to check how the assumptions and constraints on the edge locations according to the data. The second condition was to check if laser segments fit a certain target graph or not.
Other authors (e.g., Ostrowski et al., 2018) have used a similar methodology of accuracy assessment, but in the special case study, they have extended it by project accuracy conditions. Nowadays, most researchers have focused on testing many simple 3D models of buildings using the same LiDAR point cloud which was used for 3D reconstruction and modelling while this study aims to evaluate the accuracy of multifaceted 3D models of buildings using different validation datasets and to focus on the error sources that occurred during the modelling process.

DATA AND METHODOLOGY
Test objects are localized (53°45'01.5"N, 20°27'26.0"E) in Kortowo campus of the University of Warmia and Mazury in Olsztyn (Poland). In order to the accuracy evaluation of 3D models, two large-size utility buildings representative of contemporary architecture were selected. Both buildings are characterised by a diversified shape structure of the façades and roofs (different geometrical primitives, flat and cylindrical surfaces) and various texture (amorphous, porous and transparent). The object visualizations at different stages of testing are presented in Figures 1-5. In the experiment, it was assumed that for the evaluation the multiple datasets possessed from different sensors and databases will be used.

3D Building Models:
The accuracy assessment is provided using existing 3D models of buildings, which were gathered from two sources: 1. Polish national Geoportal (geoportal.gov.pl) 2. Trimble SketchUp Warehouse Models in the first database were created as a part of the Public Administration Centre for Spatial Analysis project CAPAP (Ostrowski et al., 2018). The project started in November 2018 and assumed that after the acquisition of LiDAR point cloud for the entire country it had been divided into 5 parts for which contractors prepared 3D models of buildings (in total 15 million) in LOD 2 according to CityGML. For this purpose, national mapping services provided: 1. Polish Database of Topographic Objects BDOT10K, which includes building footprints 2. Digital elevation model (DEM) generated on the basis of ALS point clouds, which allows to locating 3D building models footprint on the proper elevation 3. Roofs ALS point clouds for preparing models with the correct height The part of CAPAP, which contains the preparation of 3D models of buildings in LOD 1 standard ended in July 2019, while buildings in LOD 2 covers about 50% of the country. Figure 1 presents 3D models of test objects. The main accuracy requirements of 3D models inspection (Ostrowski et al., 2018) are described in Table 1.  The second database is related to the Trimble SketchUp application. It was chosen because of the good visual quality of models which are provided in Google Earth for Olsztyn city. In the SketchUp application, 3D models could be created by anyone. Models used in this case study were created in march 2014 after Trimble released SketchUp 2014 version which was integrated with BIM. Figure 2 presents models of two test objects from Trimble SketchUp Warehouse.

Figure 2.
Sketchup 3D models of test objects.

CAPAP ALS Point Cloud:
The 3D models of buildings were created based on the ALS data possessed during ISOK (Informatic System for Country Protection against extraordinary hazards) project. The point cloud density was realized in two standards 4 pts/m 2 and 12 pts/m 2 . Both standards assumed elevation accuracy on the level of 0.10 m. The second standard was realized in city areas. The priority is to show a comprehensive evaluation of the models, so the source data needs to be included in the tests. The complex description of the ISOK requirements was prepared by Kurczyński and Bakuła (2013). Figure 3 presents the ISOK point clouds.

RIEGL LMS-Q680i ALS Point Cloud:
Since researchers (Oude Elberink and Vosselman, 2011) have pointed out that computing distance between the point cloud and the plane which is fitted to these points could be misleading, because of small residual, in the tests additional ALS point cloud was used. In flight mission, the RIEGL LMS-Q680i longrange airborne laser scanner was used. The point cloud was acquired in July 2019 and evaluated by Gabara and Sawicki (2019b). The point cloud is characterized by a minimal density of 25 pts/m 2 and an accuracy of about 0.03 m.

Image-Based Phase One iXU-RS 1000 Point Cloud:
To take a full picture of the building 3D models, the point cloud generated based on images acquired was engaged in the evaluation. The time of acquisition is the same as in the case of the additional RIEGL LMS-Q680i ALS point cloud. Images were acquired using Phase One iXU-RS 1000 medium format digital camera. Because test objects form a small section (coverage area about 0.55 km 2 ) of the test field (Gabara and Sawicki, 2019b), so the images and ground control points (GCPs) were selected to this region (121 photos) and processed using Agisoft Metashape v. 1.7.1 application. The root mean square errors (RMSE) and reprojection error obtained in the BBA are presented in Table 2. Figure 4 presents the computed dense point cloud of test objects and Figure 5 shows the point cloud in a wider context. The accuracy assessment of the whole block of images and comparison with LiDAR point cloud is described in (Gabara and Sawicki, 2019b

Methodology
The proposed workflow in the case study of measurement and processing for 3D building models accuracy evaluation is presented in Figure 6. Due to CAPAP project requirements assume that planes were fitted to 12 pts/m 2 ALS point cloud, some of the roof structures could be omitted. To check this case, two multifaceted objects which have complicated roof planes with parts of glass were selected. Furthermore, a lot of big technical facilities are located on the roofs of tested objects, which may cause additional errors during automatic plane fitting to ALS point cloud. Because of that, in this case study point clouds were used for analyses and validation of the results of automatic algorithms.  first step, we have to read in AutoCAD Civil 3D the CityGML file with 3D building models from Polish Geoportal (Figure 7). The second step was building solid models using imported wireframes. The assessment of 3D buildings was divided into two steps: primary visual comparison which shows the main differences between objects, and main comparison, which uses photogrammetric measurement data for geometric accuracy evaluation. For further steps, all point clouds were cut to the region of interest area. Then the CloudCompare (Girardeau-Montaut, 2020) application was used to determine the differences between models and point clouds. As a result, the signed distances from the compared point clouds to the 3D models were determined. After that, the distributions of signed distances in predefined ranges were computed and analysed. The ranges are related to direct geodetic measurement accuracy (0.05 m, 0.10 m), project accuracy (1.00 m) and distribution of differences.

RESULTS AND DISCUSSION
The data processing in all of the used software was carried out on a workstation with the processor Intel® Core™ i9-7940X, 128 GB RAM DDR4 memory, MSI 1080 Ti graphic card and SSD NVME hard drive.
The first part of the analysis contains the visual comparison of both 3D building models from Polish Geoportal and Sketchup. Figure 8 shows the superimposition of 3D models. The plan view is presented in the first column and the front view in the second one. The top row shows object 1 and the bottom row presents object 2. The Geoportal model is indicated as a solid block (blue, red and white lines and parts), while the grayscale object is the SketchUp model. It should be noted that during the visual inspection multiple outliers appears. In the case of test object 1, the corner parts of the SketchUp model are higher than the Geoportal model, while the centres lower parts of the building are correlated. In some parts, the SketchUp model looks longer and wider. Probably it is related to the level of detail because the model prepared for Google Earth service contains glass parts on its façades. The same things could be observed on the main part of the roof, which is composed of steel and glass is shifted in the horizontal plane. However, it should be emphasised that differences between models exclude 1D, 2D and 3D shifts. It can be stated that the passageway between the oval part and the main part of the building is wrongly modelled in SketchUp.
On the comparison of 3D models of object 2, the main differences occur on the glass roof elements and because of additional ventilation infrastructure was not modelled in SketchUp. But it should be stated that inconsistency is observed in the Geoportal model. Only some parts of the ventilation system have been modelled. Probably it is related to the CAPAP project ratings, which quantify the max. value of outliers. Another thing is the lack of the 90-degree angle on the left side of the SketchUp model (white model, row 2, column 2). When looking at both images of test object 2 it can be speculated that the SketchUp model is simply rotated, but walls on both sides are shifted in regard to the Geoportal model. were compared with both models (Geoportal 3D model, SketchUp 3D model). Figure 9 presents heat maps of 12 computation parts in the signed distance function.
As we can expect ISOK LiDAR point cloud fits in both Geoportal models with high accuracy (below 0.10 m and in corners below 0.20 m). The outstanding elements are located mainly on walls that have his decoration elements above the roof plane. Some changes could be observed when the 3D model was compared to the additional RIEGL LMS-Q680i ALS point cloud. In both test objects, the distance between them rises to range 0.20 m÷1.00 m.
When the SketchUp model was compared to test point clouds, the best results have been achieved with the image-based Phase One iXU-RS 1000 point cloud. Probably it is caused by a higher level of detail. While worst results have been achieved with less dense LiDAR point clouds, especially on the edges of the 3D model (parapet wall). In the best scenario for the SketchUp model (image-based point cloud) the most of the distances do not exceed 1.00 m for test object 2 and for test object 1 they exceed 1.00 m. Places marked as 1.00÷2.50 m are related to the passageway and building oval part (modelling error) for object 1, and the ventilation system for object 2. In scenarios with ALS point clouds, test object 1 consists of the same modelling error that was mentioned before and test object 2 looks similar to the relation of the Geoportal 3D model and additional RIEGL LMS-Q680i ALS point cloud.
For further analysis, the heat maps were created. Then the histograms of the computed heat maps in predefined ranges with the inscribed ISOK accuracy requirements ( Figure 10) were prepared. During the analysis of the histograms, the primary statements about the same level of detail of the model and the validation test set were confirmed. In the first case (Geoportal models) of object 1 and 2, the higher level of detail of Phase One iXU-RS 1000 point cloud than the 3D model caused the appearance of higher differences, especially on parapet walls and technical facilities. The façade of object 2 where is the entrance to the building particularly shows the results of the generalization. While in the evaluation of the same objects using SketchUp models the entrance to the building shows differences in the range of 1 m.  Michelin et al. (2013) have mentioned three categories of 3D building models errors. The first group is footprint error and it is related to mistakes in input data (incorrect building outline, mission inner court, inaccurate footprint, non-existent building). The second group is reconstruction errors and it is strictly connected with reconstruction algorithms (under and oversegmentation, geometrical errors of a roof, wrong Z coordinate translation). Vegetation errors are the third group and it described situations when tree canopy occlude building and it is not possible to reliably compute the 3D model. In the context of this case study, a new group of errors, related to 3D objects have occurred. This group contains issues related to object complexity, e.g., parapet walls, multipart roofs with large height differences, roof surface type, e.g., glass, and big technical facility, which occlude big parts of the roof. Furthermore, the footprint errors group should be named object error and exceed by data acquisition technique errors (various errors for different methods, e.g., ALS point cloud noise on the contours of roofs) and LOD that can be achieved. (3) -image-based Phase One iXU-RS 1000 point cloud.

CONCLUSIONS
In this study, the quality assessment of two multifaceted 3D building models was performed. The test objects were obtained from two open-access databases: Polish national Geoportal (accuracy in LOD 2) and Trimble SketchUp Warehouse (accuracy in LOD 2 with information about architectural details of façades). The Geoportal 3D models were primary created based on the airborne laser scanning data (density 12 pts/sq. m, elevation accuracy to 0.10 m) collected during Informatic System for Country Protection against extraordinary hazards project. The testing was performed applying different validation low-altitude photogrammetric datasets: RIEGL LMS-Q680i airborne laser scanning point cloud (min. density 25 pts/sq. m and height accuracy 0.03 m), which was adjusted using Polish Active Geodetic Network ASG EUPOS points, and the point cloud based on Phase One iXU-RS 1000 imagery (average accuracy in the horizontal and in the vertical plane are respectively to 0.015 m and 0.030 m) and artificially signalized ground control points network. Each additional measurement is encumbered with its own errors and is not correlated with measurements on the basis of which the 3D models were generated, which eliminates the rational criticism related to using the same LiDAR point cloud for creating the 3D model and its validation.
The comprehensive 3D building models analysis was done using the visual comparison, heat maps with the function of the signed distance, and histograms in predefined ranges. The 3D building models from Polish Geoportal meet the accuracy requirements of the CAPAP (Ostrowski et. al., 2018). In the case of models downloaded from Trimble SketchUp Warehouse, the accuracy requirements were exceeded. It was related to some modelling errors as with the lack of 90-degree between object walls. In projects like CAPAP, only a small part of the entire 3D building models dataset is used for testing accuracy, our suggestion is to for future verification of 3D building models, the manual inspection using independent measurements for a small part of objects should be provided. It could be realized using low-altitude unmanned aircraft systems surveys, which allow fast 3D inspections even in the LOD 3 standard.
Furthermore, the study shows that groups of errors sources mentioned by Michelin et al. (2013) should be exceeded by a new group of errors, related to 3D objects, which contains issues related to object complexity, e.g., parapet walls, multipart roofs with large height differences, roof surface type, e.g., glass, and big technical facility, which occlude big parts of the roof. Besides, the footprint errors group should be named object error and exceed by data acquisition technique errors (various errors for different methods, e.g., ALS point cloud noise on the contours of roofs) and LOD that can be achieved.