EXPLORING THE POTENTIALS OF UAV PHOTOGRAMMETRIC POINT CLOUDS IN FAÇADE DETECTION AND 3D RECONSTRUCTION OF BUILDINGS

The use of Airborne Laser Scanner (ALS) point clouds has dominated 3D buildings reconstruction research, thus giving photogrammetric point clouds less attention. Point cloud density, occlusion and vegetation cover are some of the concerns that promote the necessity to understand and question the completeness and correctness of UAV photogrammetric point clouds for 3D buildings reconstruction. This research explores the potentials of modelling 3D buildings from nadir and oblique UAV image data vis a vis airborne laser data. Optimal parameter settings for dense matching and reconstruction are analysed for both UAV imagebased and lidar point clouds. This research employs an automatic data driven model approach to 3D building reconstruction. A proper segmentation into planar roof faces is crucial, followed by façade detection to capture the real extent of the buildings’ roof overhang. An analysis of the quality of point density and point noise, in relation to setting parameter indicates that with a minimum of 50 points/m, most of the planar surfaces are reconstructed comfortably. But for smaller features than dormers on the roof, a denser point cloud than 80points/m is needed. 3D buildings from UAVs point cloud can be improved by enhancing roof boundary by use of edge information from images. It can also be improved by merging the imagery building outlines, point clouds roof boundary and the walls outline to extract the real extent of the building.


INTRODUCTION
Buildings are vital structuring elements for urban planning, emergency response and disaster management applications; they are an important feature in a spatial decision support system (Xiao et al., 2012). In the third dimension (3D), buildings can be used as urban parameters to monitor and evaluate city planning indicators (Rebelo et al., 2015). 3D buildings show geometry and appearance of reality; they enable contextual spatiotemporal understanding of built-up scenes.
Traditional photogrammetry has demonstrated the ability to reconstruct 3D scenes. Nevertheless, manual stereo pair feature extraction has proven to be tedious and time consuming for expansive built-up areas. Airborne Laser Scanning (ALS), has come a long way to leverage traditional image-based 3D reconstruction. ALS point clouds are dense, accurate, penetrate vegetation and give ready 3D data. However, reconstruction of 3D models representative of the real ground scene remains to be a challenge (Malihi et al., 2016). High cost of scanner acquisition, and limitation to capture the roof and parts of a building only visible from an aerial perspective plague the use of ALS to accurately model 3D buildings. ALS cannot record data on slate roofs, roof covered with water and glass materials; the ALS beam can also be diverted by solar panels. Moreover, ALS lacks accuracy at the edge of the building due to laser sampling. Maltezos & Ioannidis (2015) argue that, ALS point clouds could give false 3D models since buildings are easily confused with smooth canopy.
On the other hand, contemporary digital photogrammetry has revived image-based data collection and is a suitable alternative for 3D reconstruction of built-up scenes. The use of Unmanned Aerial Vehicles (UAV) for image data acquisition is widely spread due to their low-cost, flexibility and high-efficiency. They are viewed as a midway option between higher resolution ground-based images and the lower resolution data acquired from satellites and other airborne platforms.
Irrespective of the pros of UAV application, the cons are quite an array and can be separated into three broad categories: operational restrictions such as violent weather conditions, high, rugged and rapidly undulating terrain, limited spatial coverage, radio connection stability, landing services; political readiness such as public approval and safety measures; and regulation restrictions such as privacy, region coverage and flying height.
Both UAV and ALS point clouds suffer similar shortcomings; occlusion and shadows (Li et al., 2013). According to Haala & Kada (2010), the development of a fully automated algorithm for 3D modelling of buildings remains a challenge. To add to this discourse, Xiong, (2014) summarizes the complexities of 3D building modelling as: • Complex scenes -The environment to which the buildings are found is a mixer of many objects thus hard to distinguish; • Complex building shapes -Some buildings have complicated shapes and a lot of fixtures on the roof; • Complex boundaries -Some incomplete shapes missing due to missing 3D points; and • Lack of data -These are caused by occlusion, slate roofs, water on roofs, and shadows. In Figure 1(a), the arrows show scene features void of points due to occlusion and shadow; comparably, in 1(b), the same features are either fully captured or void of points like the roof.
The developments in computer vision and photogrammetry allows for the extraction of geometrically accurate point clouds from overlapping imagery and automatic scene interpretation. The motivation is triggered by increased quality of digital cameras, flight planning flexibility as well as amplified innovation in image matching algorithms. According to Fabio , accurate image matching is one of the keys to proper 3D modelling.
This research aims at 3D modelling of buildings from UAV dense point clouds. We explore the use of oblique images to enhance roof boundaries, facades and wall outline to reveal the true extents of buildings.

RELATED WORKS
For two decades now, lidar and aerial image point clouds have been used for automatic 3D building reconstruction with different level of details (LOD) and approaches, model-driven and data-driven. Haala & Kada, (2010) give an update of the current state-of-the-art to 3D building reconstruction from laser, aerial images and a fusion of both. Later, Xiao et al., (2012) attempt automatic detection of 3D buildings from aerial images. Oblique airborne images were used; facades positioning with same view direction were used to recognize buildings. One key assumption was that facades are a composition of vertical planes. Onyango et al., (2017) estimate orientation parameters of oblique UAV imagery using keypoint intensity features on building facades. Resolution and scale differences are observed not to be an image matching deterrent.
Tutzauer & Haala, (2015) used a combination method of dense point clouds from mobile and aerial images to reconstruct and enrich the building facades; a grammar-based approach was used for the building reconstruction in parts which were not covered by images. Another approach was applied by Verdie et al. (2015); multiple classification of building categories like ground, roofs, or façades was done. Zebedin et al., (2008

METHODOLOGY
An integrated approach that takes point clouds and existing 2D map boundary as inputs to reconstruct the 3D building is proposed in this study. Xiong, Oude Elberink and Vosselman, (2016) demonstrate an automatic 3D building reconstruction approach and correction of 3D building models using the planar faces segmentation algorithm and roof topology graphs. This approach is also explored in this research using oblique images to extract accurate building extents from the facades. Figure 5 shows the workflow overview.

Photogrammetric Workflow
The initial processing mainly entails keypoint detection and matching, and estimation of camera interior and the exterior orientation parameters. Bundle block adjustment was done to refine 3D positioning and accurate image scaling. Automatic tie points are used as an input to the next processing step of point cloud densification. The point density can be processed in three different scales: original image scale, a point for every pixel; half image scale, a point for every four pixels -this is the recommended image scale in Pix4D; and quarter image scale, a point for every 16 pixels.  argue that on aerial acquisition, laser gives 1-25 points/m 2 dense point clouds, while UAV imagery with a GSD of 10cm can produce a dense point cloud of 100 points/m 2 . For accurate comparison, all point clouds were clipped from the same area with the ALS point clouds. The Nunspeet UAV point clouds processed in different scales were considered as separate input datasets. A statistical analysis of point spacing for each input dataset was done. Point spacing refers to the linear distance between individual points whereas point density is defined as the number of points per m 2 or m 3 . A statistical analysis on maximum and minimum Z of the UAV point cloud was also done.

UAV Point Cloud Accuracy Assessment
To evaluate the accuracy and completeness of 3D building models from UAV point clouds, accuracy assessment was done from various points of view.

Structure from Motion (SfM):
Pix4D gives a quality report on the accuracy of the point clouds generated after the processing stage. In the initial processing stage, Pix4D uses the images and GCPs to identify specific feature in the images as key points. The GCPs are assessed for Root Mean Square error.

Positional Accuracy:
To compare the UAV and ALS positional accuracy, the two datasets are displayed in Cloud Compare (CC) for visual interpretation of the two-point clouds.
The UAV point clouds are displayed in RGB and the ALS point clouds are displayed in a height colour code. The positional accuracy is assessed visually by looking at the position of the building in UAV and the position of the ALS buildings as well.

Internal Accuracy Assessment by Fitting a Plane:
This was done by comparing the distance from the fitted plane and the UAV point clouds, same for the ALS point clouds. To give the noise to the fitted plane in Cloud Compare, the algorithm does a least square best fit of a set of 3D points by applying principal component analysis (PCA) (Pearson, 1901). Noise residuals were also compared in Python and R. The samples were taken from building roofs of different colours.

External Accuracy Assessment:
In this research, ALS point cloud is used as the reference, and is thus used to assess the accuracy of the UAV point clouds. It was not possible to do ground truthing, which is the best external accuracy assessment for survey data. Thus, for external accuracy assessment, the two datasets were loaded and compared for cloud-to-cloud distances in Cloud Compare software. A threshold of 20cm was set to be the maximum distance between the two-point clouds. The distance of each point in the photogrammetric point cloud is compared to the nearest point in the reference ALS point cloud.

Accuracy Assessment by Running
Profile: a profile is run through same area of the two Nunspeet datasets. Surface undulation is extracted, and finally horizontal and vertical discrepancies between the UAV and ALS points is assessed.

2D Building Edge Extraction
The proposed method uses 3D point clouds and a 2D cadastral map as inputs. Digitization to extract 2D building outlines from the orthomosaic was done. About 35 main houses in the common area of the two datasets were extracted. The 2D outlines act as a constraint to the extent of the 3D buildings and as the base for projecting walls of the buildings.

Point Cloud Filtering
3D building reconstruction is affected by missing point cloud information, noise, shadows and trees among others. UAV images with trees cannot generate point clouds under those areas, only trees points will be captured. This hinders 3D reconstruction since planar segmentation will mistake the trees as building roof segments, thus giving spikes as a result in the final 3D buildings.

Classification:
The unsupervised machine learning algorithm in Pix4D uses colour information and geometry; a trained model is then applied which predicts the label of each point then assigns it to one of the five classes (High vegetation, buildings, human made objects, roads and ground). For this data, trees were removed from the classes and the data exported. Tall trees which cover the roof are problematic to 3D modelling. Filtering of the ground and non-ground in the ALS cloud points was done using Lidar360 software.

Normalized DSM:
The normalized digital surface model (nDSM) is the difference of digital surface model (DSM) and the generated digital terrain model (DTM). Normalizing the UAV and the ALS point clouds makes it possible to get the local height of the buildings which can be determined from 0level (bottom) to the rooftop. This is the height which the building should be at and not the included DTM height.

Noise Filtering:
Noise is any unwanted detail that makes part of the building reconstruction points; it can be grass on the roof or other litters which if not removed, may give a wrong geometry due to errors in surface reconstruction. In CC, the noise filter algorithm is used to remove unwanted outliers. The algorithm considers the underlying plane and not the distance to the neighbouring points; it uses least square distance for best plane fit.

Defining the Real Buildings Extent
The nadir aerial view is used to outline building boundaries. In most cases, the roof edge does not reflect the real building extent especially where there is a roof hang or gutters. The real extent of buildings is defined by facades of the building. Oblique images are needed to capture facades. The L'Aquila images captured the buildings from all the five views (nadir, oblique North, oblique East, oblique South, and oblique west), thus being perfect to realize this objective.

Planar Segmentation
Segmentation is very important in the detection of roof outlines and for the reconstruction of buildings. Segmentation is meant to cluster point clouds with similar characteristics into homogenous regions. Due to large number of points on the roof, planar roof faces can be detected automatically. In this research we use the segmentation algorithm by Oude Elberink & Vosselman, (2009). To detect these planar points, the Hough transform was extended.
According to Elberink & Vosselman, (2009), lines connecting two roof faces and height jump are part of a topological relationship between two neighbouring segments and if there are segmentation errors, roof faces and height jumps cannot be detected. If within a segment an intersection line or a height jump is detected, the segment is split into two parts.

Automatic Facades Detection from UAV Point Clouds
To automatically detect the facades from the UAV point clouds, an assumption is made that the walls are 90 degrees vertical to the ground. In setting of segmentation parameters, this assumption is factored in. Walls, just like the roof faces are searched by segmentation and detected automatically. If the keep-roof parameter filters the walls, then a keep-walls parameter can filter the roof.

3D Building Evaluation
Dorninger & Pfeifer, (2008) present an orthogonal vertical difference between the 3D model and the reference point clouds. Elberink & Vosselman, (2011) however, argue that the perpendicular distance from the point clouds to the 3D models might be misleading since most points are close to the models. In this research, two quality checks were done, a visual interpretation and an overlay of the segmentation contours since an optimal segmentation defines the final structure.

UAV Dense Point Clouds 5.1.1 Nunspeet Dataset:
It is clear that processing at varying scales produce different point densities; which is determined by the number of matched pixels. It is also observed that the point clouds from the aerial view are uniformly distributed, and no clusters or holes are in between the point clouds.

L'Aquila and City Hall Dortmund Dataset:
A mismatch of the walls and the roofs is evident. This can be due to imperfect geolocation of the images and lack of GCPs on the ground. Figure 7 illustrates the mis-match by showing 3 roof layers and two walls. In between tall buildings there are some missing points which were not generated for the walls. This can be attributed to occlusion and image alignment failures.
To align the mis-aligned walls and roofs of all the projects, each project was brought into Cloud Compare and each was aligned to the other by picking 4 common points. This is a point cloud registration by applying a transformation matrix. The merged project had over 99 million points and almost all the walls were captured except the areas in between the close buildings and ones next to tree canopies and other types of occlusions. The walls were more than 1m thick (a challenge on where is the exact location of the wall plane). It can be noted that the manual tie points did not work for a perfect alignment. A lot of noise persists even after alignment in the L'Aquila dataset; the walls are very thick, doors and windows are not clearly defined, a lot of filtering is necessary. Also, distribution of points is not uniform, a lot of holes in the data is clear, and some areas have clustered distribution of points. Conversely, the City Hall Dortmund dataset is well aligned; the windows and doors are perfectly aligned. All facades were captured smoothly. The City Hall building yielded over 7million points for 312 images, and the 110 images yielded less noisy point clouds. Reducing the number of image overlaps still gives acceptable results with reduced noise. The extreme right snippet in Figure  8 shows the cross-section of the wall point clouds. The challenge of many images is the noise, as it can be seen. The cross-sectional wall point clouds thickness is 82cm which is a challenge to plane fitting.

UAV versus ALS Point Cloud Density
The point clouds were compared with respect to point spacing, point density, number of points in the clipped area and positional precision as well as minimum and maximum Z. The photogrammetric point cloud is observed to be denser than the ALS. The statistics in Table 1 show that the point spacing of the photogrammetric points is less than that of ALS. The Z values shows how well the two-point clouds align vertically. The difference of the minimum Z values is 0.453m. This can be attributed to the UAV and ALS having a different Z reference datum or GCP Z value refinement. It can also simply be a function of noise in the point cloud. The maximum Z value the difference is 1.61m. In addition to the reasons above, this can be accounted for by difference in time of data capture.

UAV Triangulation RMS Errors
The photogrammetric point cloud has a low triangulation RMS error for GCPS. According to ASPRS accuracy standards for digital geospatial data ASPRS, (2014), the accuracy standards for aerial triangulation errors can be 3 times the GSD of the images. The 10 GCPs show quite good residuals in XYZ, the highest being 1.24cm in X. Thus, the RMS errors are good compared to the GSD of 1.65cm with a mean error of 1cm. This can be attributed to good contrast and high resolution of the images, and accurate GCPs.

UAV and ALS Point Cloud Positional Accuracy
An overlay of the two point clouds with the image building information outlines shows that the two overlay perfectly horizontally (See Figure 9). This makes it possible to check whether the Lidar point cloud and the photogrammetric point clouds cover the same space especially around the buildings. The RGB coloured buildings are of the UAV point clouds, the blue points are the ground points for the ALS, the dark green points are the roof points of the lower buildings and green points are the higher roofs points. Figure 9. UAV and ALS point clouds overlay

Accuracy Assessment by Fitting Plane
The aim of fitting a plane in the UAV and ALS point clouds is to assess the amount of noise. Four random samples are assessed (See Figure 10). The results are so smooth in CC that further comparison is done in Python and R programming platforms. The difference in the roof's standard deviations for the UAV point clouds is due to the fact that the images of the UAV data are not uniform throughout the area of coverage.

Distance Comparison Accuracy Assessment
The results show that values higher than 20cm maximum threshold distance were excluded, the mean distance was 0.162m and the standard deviation was 0.046m. Normally, the deviations are an indication of the low quality of UAV point clouds in certain areas or less dense point clouds of the lidar especially on the roof, ground and on the trees. ALS points were sparse; thus, chimneys and dormers were not captured. As the differences are mainly concentrated on the trees and on limited portions of the scene, these results confirm the suitability of the UAV for surface reconstruction.

Accuracy Assessment by Running Profile
The positional accuracy of the UAV data seems same with the ALS when looking at the length of the data across the two buildings as shown in Figure 12. The height accuracy is in the same range as looked through the height of the roof and the tallest chimneys. The UAV points give a smooth chimney canopy than the ALS points. The point clouds of both data give more less the same profile and a model reconstruction will give the same.

Classification:
The ALS point clouds were filtered to ground and non-ground points. The UAV point clouds were filtered through trees up to the buildings only; closer inspection reveals misclassification. Some building patches were classified as high vegetation or road surface. This can be attributed to roof colour or tree canopy occluding the building or casting shadows. To avoid missing information from UAV point cloud, it is not advisable to over filter.

Normalized DSM:
Results show that the approximate height of the consecutive three buildings is 7.6m. This is evident by looking at the profile of the two datasets as shown in Figure 11. The profile confirms the same building height for the UAV and ALS, this can be another accuracy assessment for the volume in the final 3D model.

Noise Filtering:
Noise filtering is done by least squares for the best plane fit. Some chimneys were longer than 1m; a radius of 2m was considered and a relative error of 2m. As shown in Figure 12, it reduces points density and completely trims all the chimneys for both UAV and ALS point cloud. Figure 12. Upper image (before filtering); middle image (after filtering); and the lower image showing removed chimney clouds

Defining Building Extents
The role of facades in defining the real extents of a 3D building is addressed in this section. After clipping the buildings to expose the walls, and superimposing it on the orthomosaic, it is evident that the two don't have the same extent (See Figure 13). An outline of both facade edges displayed with the digitized roof outline, show totally different extents. The real extent is defined by the facades/walls of the building. Figure 13. Real extent of a building as shown by its facades Results show that the real extent of a 3D building can be achieved by UAV multi-oblique images capturing the facades. It is observed that by clipping the building to a certain height, can exclude the roof and then, manually or by edge filters the façades boundary is extracted. By displaying the facades edge and building image information, the real extent is achieved. In cases where there is no roof hand, the image building information and the facades edge will have the same extent.

Planar Segmentation
Roof planes are correctly detected. During segmentation, small roof segments not meeting a minimum number of points are removed. By comparing the roof segments and the orthomosaic, the accuracy is expressed by the completeness and correctness of the segmentation contours.
In this research, the 0.5 image scale was chosen to work with for the 3D buildings reconstruction. The half image scale point cloud is the middle ground of quality and computational cost. Also, after filtering of noise it offers the perfect cloud to capture target roof fixtures. The original image scale generates most points thus creating more surfaces during segmentation process while the quarter image scale, after filtering the noise, removes all the roof fixtures including the wanted ones like the dormers. 5.6.1 UAV Segmentation Parameter Setting: Different parameter settings yield different results per scene. Varying the seed radius, grow radius, maximum distance to grow, and minimum segmentation size results into over-segmentation and under-segmentation of roof planes. Figure 14 below shows the orthomosaic snippet used for segmentation quality check. In Figure 15 (a), chimneys and dormers are visible, oversegmentation is circled green while under-segmentation purple. In (b), more surfaces are seen due to many points within the increased seed radius. Finally, in (c), more surfaces are over-segmented. Table 3 shows the varied parameters during segmentation (meters).  there's over segmentation. By reducing the minimum segmentation size to 10 and maintaining the maximum distance to grow at 0.3, under segmentation is observed in some roofs. A seed radius of 1.0, grow radius of 1, maximum distance grows of 0.2, minimum segment size of 10, and flatness 0.75 were observed to give optimal segmentation results. See Figure 16. Further comparison of the optimal segmentation contours after noise filtering shows that over and under-segmentation errors for this section alone gives 98% accurate planar segments for the ALS and 94% for the UAV.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France The parameter settings depend on point density for optimal segmentation. In this research, the less dense ALS point clouds, the minimum segment is less (10 points) than the minimum segment in UAV point clouds which is denser (30 points).

Façade Detection
The planar faces segmentation algorithm implemented in this research filters non-planar points from the point cloud. It successfully detects building facades for simple buildings. The main challenge of automatic façade detection for this algorithm is the point density. For big buildings like the City Hall Dortmund with more than 7 million points, planar faces are not detected due to noise and computing limitation. Only roof contours are automatically detected.

Figure 17. City Hall segmentation and façade detection
Automatic façade detection is possible for buildings with less than 7 million points. See Figure 18. Figure 18. Detected facades for a 'simple' building.

3D Building Reconstruction
3D modelling was carried out for two datasets by using the same planar faces segmentation algorithm with different parameter settings. The study examined the modelling correctness of the buildings with reference to the orthomosaic by visual interpretation and inspection. For the modelling accuracy, buildings models were one by one chosen from each side (ALS and UAV). It is important to note that over and under segmentation errors are propagated to this modelling phase. Figure 19 shows the 3D building reconstruction results of UAV and ALS using similar parameter settings. Figure 19. Snippet of final UAV (left) and ALS (Right) 3D buildings UAV image buildings were observed to have more roof features like dormers and chimneys. Due to UAV dense point clouds, some buildings depict more partitioning than in reality as a result of more surfaces recovery from the data during segmentation. Results of the ALS 3D buildings reconstruction was as a result of one parameter setting giving only one bad reconstructed model. For UAV point clouds, two different parameter settings had to be done in different areas of the same dataset to get correct buildings, and a third parameter setting done on the bad reconstructed buildings.

DISCUSSION
The UAVs systems can provide very high-resolution data for it can be manipulated to fly lower with high overlaps, the higher the flying height, the bigger the GSD and the poor the resolution of the images acquired. The GNSS/INS on-board provide automated navigation and photogrammetric images orientation, GPS and IMU are used for direct-georeferencing but ground control points (GCPs) are needed to refine the accuracy of the models. The main challenges for ALS are wall occlusions and point density but this can be improved by taking nadir and oblique multi-view UAVs images in areas that cannot be accessed by ALS.
To capture the facades and take care of occlusions, oblique images with bigger overlaps are needed. Dataset 2 has demonstrated the use of many overlapping nadir and oblique images. However, big overlaps should be executed with the knowledge that it comes with extra cost. Bigger overlaps translate to many flight lines which mean many images will be captured. A trade-of between data accuracy and completeness with acquisition and computational cost should be made.
Automatic classification addressed the challenge posed by tree canopy. Other classes are exported without tall trees. The planar faces segmentation algorithm applies the keep-roof parameter to keep the roof at 0-75 degrees slope and filter others by connected component at 90 degrees slope.
3D buildings from UAVs can be improved by enhancing roof boundary by use of edge information from images. It can also be improved by merging the imagery building outlines, point clouds roof boundary and the walls outline to extract the real extent of the building. This way, the planar segmentation approach used in this research somewhat accounts for missing point cloud data. Figure 20 shows and example of a scene missing data and how 3D building reconstruction filled the missing roof segment. This clearly demonstrates that the missing data can be reconstructed without omission of the segment or distortions of the roof outline. However, it depends on the amount of the missing segment, if the whole segment has no data, the roof will take a flat shape at the wall's height. Figure 20. Showing missing data, UAV and ALS 3D models respectively 7. CONCLUSION Although photogrammetric point cloud is considered less accurate, the advances in technology has made it possible to create highly accurate maps from drones for a wide range of applications. The findings in this research show that, UAVs can be the suitable platform for improvement of 3D buildings from a combination of multi-view and oblique UAV images.
It can be concluded that there can be more than one parameter setting within one dataset for both UAV and ALS. It is not easy to get the geometry of all the buildings correctly within the entire area with only one parameter setting. For the wrongly reconstructed buildings one can get another parameter setting.
There is no need for oblique multiple overlaps of above 80% for 3D building reconstruction, the nadir images with traditional overlaps of between 60% to 75% is sufficient for the roof points. Also, the 0.5 image scale is good enough for the roof features and optimal segmentation without over/ under segmentations depending on the parameter settings. From the results, the deduction is that with a minimum of 50 points/m 2 , most of the planar surfaces are reconstructed comfortably. But for smaller features than dormers on the roof, a denser point cloud than 80points/m 2 is needed.
Further research into automatic façade detection is needed; if the proposed planar segmentation approach can automatically detect the roof planar contours and snap them to the footprint maps perpendicularly, then the facades contours can be snapped to the 2D planes parallel to the walls. An alternative on the walls may also be to reduce the image scale during point cloud generation.