QUALITY ANALYSIS OF 3 D SURFACE RECONSTRUCTION USING MULTI-PLATFORM PHOTOGRAMMETRIC SYSTEMS

In recent years, the necessity of accurate 3D surface reconstruction has been more pronounced for a wide range of mapping, modelling, and monitoring applications. The 3D data for satisfying the needs of these applications can be collected using different digital imaging systems. Among them, photogrammetric systems have recently received considerable attention due to significant improvements in digital imaging sensors, emergence of new mapping platforms, and development of innovative data processing techniques. To date, a variety of techniques haven been proposed for 3D surface reconstruction using imagery collected by multiplatform photogrammetric systems. However, these approaches suffer from the lack of a well-established quality control procedure which evaluates the quality of reconstructed 3D surfaces independent of the utilized reconstruction technique. Hence, this paper aims to introduce a new quality assessment platform for the evaluation of the 3D surface reconstruction using photogrammetric data. This quality control procedure is performed while considering the quality of input data, processing procedures, and photo-realistic 3D surface modelling. The feasibility of the proposed quality control procedure is finally verified by quality assessment of the 3D surface reconstruction using images from different photogrammetric systems.


INTRODUCTION
Over the past few decades, accurate 3D surface reconstruction has been established as a necessity for a variety of mapping, modelling, and monitoring applications such as urban planning, environmental monitoring, infrastructure monitoring, cultural heritage documentation, indoor localization, and disaster management (Campos et al., 2015;Poullis and You, 2011;Kwak et al., 2013;Kemec et al., 2009;Zou et al., 2012).The required data for the reconstruction of 3D surfaces satisfying the needs of the aforementioned applications are generally collected using different passive and active digital imaging systems (e.g., laser scanning systems, high-end photogrammetric systems, and range cameras) mounted on static or mobile airborne and terrestrial platforms.Among these systems, laser scanners have received more attention due to their fast and accurate data acquisition capability, flexibility, and applicability in different atmospheric conditions (Arefi, 2009).In spite of their proven feasibility for 3D data acquisition, these systems cannot be efficiently employed for semantic large-scale modelling applications due to their high initialization costs, the need for users with high-level of technical expertise, and the lack of descriptive information along the scanned surfaces (Pu and Vosselman, 2009).To overcome these limitations, significant research attempts have recently been made to propose innovative image processing techniques (e.g.multi-view dense matching procedure) and revive photogrammetric systems as efficient and feasible tools for the collection of required data for 3D surface reconstruction (Haala, 2011;Mathias Rothermel et al., 2012;Wiechert et al., 2012).The significant advancements in image processing techniques, together with progressive hardware developments (e.g., low-cost high-resolution digital cameras and multi-camera systems), and emergence of new cost-effective mapping platforms (terrestrial platforms, Unmanned Aerial Vehicles -UAVs, and mobile robots) have strived to shift the focus of attention from laser scanning systems to modern photogrammetric systems for 3D modelling activities.Although these systems have been noticed tremendously due to their cost saving and ability to provide more detailed information from the scanned scenes, they still are not widely used in different applications due to concerns about quality of delivered mapping products and 3D models (Lari and El-Sheimy, 2015).This concern originates from the use of lower-grade sensors onboard these platforms which might pose several processing challenges and deteriorate the quality of final products (Nex and Remondino, 2014).In order to verify the feasibility of newly-developed photogrammetric data processing techniques in dealing with these challenges, well-established quality control procedures are needed to evaluate the quality of 3D reconstructed surfaces using the images collected onboard emerging photogrammetric systems (i.e., UAV-borne and terrestrial photogrammetric systems).Hence, this paper introduces a quality analysis procedure for the evaluation of image-based 3D surface reconstruction.This quality analysis procedure is implemented by qualitative and quantitative evaluation of the reconstructed 3D surfaces in three successive steps: In the first step, the quality of input data for 3D surface reconstruction is investigated while considering the inaccuracies in raw system measurements (georeferencing and imaging sensors' measurements), the performance of initial processing steps (i.e., pose estimation and dense image matching) and data gaps caused by nearby objects during data acquisition.In the second step, the quality of extracted 3D surfaces is investigated using a novel quality control procedure for the assessment of point cloud processing procedure (i.e., segmentation).Furthermore, geometric quality of the extracted 3D surfaces from imagery-based 3D point clouds is investigated using a novel quality control technique based on the estimated roughness factor of those surfaces.A surface matching between the reconstructed surfaces and control surfaces extracted from other data sources is also performed to verify the quality of the reconstructed surfaces using images.In the third step, the quality of the reconstructed 3D surfaces is inspected from visualization point of view.The reconstructed 3D surfaces can be textured photo-realistically or represented by their boundaries as wireframe models.These 3D models will be also investigated for possible occlusions or problems wireframe modelling procedures.Figure 1 shows the outline of the proposed method for quality analysis of 3D surface reconstruction using multi-platform photogrammetric systems.Figure 1.The outline of the proposed procedure for quality analysis of 3D reconstruction using photogrammetric data This paper starts by the introduction of the proposed approach for the quality analysis of 3D surface reconstruction using photogrammetric data.Afterwards, the feasibility of the proposed quality control procedure is verified through quality assessment of the 3D surface reconstruction using images from UAV-borne and airborne photogrammetric systems.A comparative analysis of the reconstructed surfaces from imagery collected onboard the emerging photogrammetric systems and overlapping surfaces extracted from laser scanning data is also performed to confirm the feasibility of these systems for the acquisition of data required for accurate 3D surface reconstruction.The implemented experimental results prove the competency of modern photogrammetric systems for the collection of geospatial data utilized for 3D modelling and monitoring applications.Finally, concluding remarks and recommendations for future research work are summarized.

QUALITY CONTROL OF INPUT DATA
The first step in quality assessment of 3D surface reconstruction using photogrammetric data (collected on different platforms) is to investigate the quality of acquired data.The collected data using these systems usually include multi-spectral images and positioning and orientation (POS) information provided by georeferencing sensors onboard these photogrammetric platforms.The quality assessment of input data for 3D surface starts by quality control of the collected data by onboard sensors.Subsequently, initial processing procedures (i.e., accurate pose estimation and multi-view dense matching procedures) are implemented to generate 3D point clouds from multi-view stereo images.Therefore, quality analysis procedures need to be developed for the assessment of generated 3D information using the collected images and POS information.Finally, the quality of the generated point clouds should be investigated from completeness point of view.Hence, point density quantification procedures are implemented to evaluate data completeness for 3D surface reconstruction and identify areas with lower point densities.In the following subsections, the utilized procedures for the quality evaluation of input data for photogrammetric 3D surface reconstruction will be discussed, in detail.

Quality control of raw system observations
Since the quality of 3D surface reconstruction from photogrammetric data is highly dependent on the quality of the input imagery and provided POS information, an accuracy analysis procedure is implemented in the first step to ensure the quality of the utilized data before further processing.This procedure is implemented to identify low-quality images (i.e., the images that suffer from problems such as noise, distortions, low radiometric quality, shadows, etc) and images which exhibit texture-less surfaces or surfaces with repeated textures (Remondino et al., 2013) (Figure 2).The existence of such problems in the collected images will lead to the generation of noisy point clouds or failure in feature extraction and image dense matching procedures.In order to solve first group of problems, image manipulation, geometric and radiometric calibration techniques can be performed.However, for the second set of problems no effective solutions have yet been found.The quality of the collected POS information will be assessed using a well-distributed network of ground control points.The coordinates of these control points are re-estimated using image coordinate measurements and compared to their reference coordinates.The calculated differences in object space represents the overall accuracy of the photogrammetric system's direct georeferencing components (Cramer et al., 2000).

Quality analysis of initial processing steps
The quality control of input data for 3D surface reconstruction is continued by quality control of initial processing steps for the generation of 3D point clouds from overlapping imagery.These procedures mainly comprise pose estimation and multi-view dense matching.The pose estimation procedure aims at accurately estimating the position and orientation of the utilized camera/cameras at exposure moments using known image features.The quality of pose estimation procedure is highly dependent on the accuracy of camera calibration, quality of feature extraction procedure from imagery, and the initial POS information approximations (Yoon et al., 2006).
On the other hand, the objective of dense matching procedure is to generate 3D information from the scanned surfaces based on the matched pixels in overlapping imagery.The accuracy of this procedure and the quality of the delivered 3D point clouds depends on the quality of the captured imagery and quality of camera calibration, pose estimation, and matching steps (Remondino et al., 2013).Figure 3 shows the impact of a poor camera calibration procedure on the quality of the generated 3D point cloud.
(a) (b) Figure 3.The impact of poor system calibration on the quality of the generated 3D point clouds using: (a) inaccurate and (b) accurate camera calibration parameters

Data gaps
Although the introduced pre-processing procedures strive to provide complete accurate 3D point clouds over the scanned scenes, there are still several issues that cause gaps within the provided data.These gaps may originate from occlusions within the images, mismatching problems due to repeated or low quality image textures, or problems in accurate POS information estimation.In order to investigate these data gaps and areas with low point density in the generated point clouds, new point density quantification techniques will be applied (Lari and Habib, 2013).These techniques estimate the local point density indices while considering the 3D relationships among the neighbouring points and the nature of their enclosing surfaces.In other words, they reflect point density variations along the scanned surfaces.The areas with lower point density indices will be considered as data gaps and will be investigated for the possibility of visibility in the other captured images.Figure 4 represents a point density map derived for a generated point cloud according to these techniques.

QUALITY CONTROL OF DATA PROCESSING PROCEDURES
The generated 3D point clouds does not provide semantic information about the type and characteristics of the scanned features.Therefore, these point clouds should undergo processing techniques to deliver the required products for different mapping, modelling, and monitoring applications.In order to verify the validity of these products, quality control techniques need to be developed for the assessment of these processing procedures.Among these processing procedures, segmentation is considered to be the most important one.The objective of this procedure is to identify features (i.e., surfaces) of interest and reduce scene complexity within the 3D generated point clouds.The far majority of existing point cloud segmentation techniques do not provide built-in procedures for the evaluation of the processing outcome.Therefore, quality control technique introduced in Lari et al. ( 2014) is used to identify possible problems affecting point cloud segmentation without user intervention and suggest alternatives for the resolution of these problems.
In addition, the quality of extracted surfaces can be investigated with respect to control surfaces.This quality control procedure is performed based on surface-by-surface matching and surface roughness comparison between the reconstructed surfaces and control surfaces.In the following subsections, the introduced procedures for the quality control of point cloud processing will be discussed, in detail.

Quality control of point cloud processing
As mentioned before, different approaches have been proposed for the extraction of primitive surfaces from 3D point clouds.
Since none of these approaches are exempted from possible anomalies, a quality control procedure is required to evaluate the validity of the segmentation outcome.In this section, a recently developed technique for quality control of point cloud processing (i.e., segmentation) is described.One should note that this technique is implemented without the need for reference data and can be applied regardless of the utilized processing procedure.As a pre-requisite tool for the quality control of the extracted surface, roughness measure/factor is used (Wöhler, 2012).The roughness factor for a given surface is defined as the deviation of the segmented points within that surface from the best fitted plane/line to those points.This deviation is estimated as in Equation (1).
Where ndi is the normal distance between the i th point within the j th extracted surface and the best-fitted plane/line/cylinder through the constituent points and n is the number of clustered points in the extracted surface.The possible issues in the processing outcome are then detected based on the estimated surface roughness factors as follows: (1) Non-Segmented Points The unincorporated points are defined as those belonging to the extracted surfaces, while they could not be clustered during the processing procedure.The instances of this problem can be included in their neighbouring extracted surfaces if their normal distance to the best-fitted surface through the points of that cluster is not more than a pre-specified threshold which is determined based on the roughness factor of the neighbouring extracted surfaces.
(2) Over-Segmentation The over-segmentation problem occurs when an individual surface is segmented into several surfaces.This problem might happen due to using strict processing thresholds or problems in surface parameterization procedure.To identify instances of this problem, the similarity between surface parameters (normals to neighbouring planar surfaces and directional parameters of neighbouring linear/cylindrical surfaces) are firstly investigated.If the parameters of the investigated surfaces are deemed to be similar, those surfaces are merged together as a single surface.(3) Under-Segmentation The under-segmentation problem occurs when multiple surfaces with different parameters segmented a single surface.This problem might happen due to using relaxed processing thresholds and improper problems in surface parameterization procedure.In order to detect instances of the undersegmentation problem, the average surface roughness factors (ASRF) for all extracted surfaces are firstly estimated.For each of the extracted surfaces, if its surface roughness factor is more than a pre-defined threshold (e.g., 2-3 times of the average surface roughness factor of all derived surfaces in the same category), that surface is suspected to be undersegmented.For the suspected under-segmented surfaces, the segmentation process is repeated while changing the processing thresholds to check if they can be segmented into their constituent surfaces.
(4) Invading/Invaded surfaces This problem occurs when an extracted surface is invading/being invaded by another neighbouring surface/surfaces.This issue might happen due to falsely evaluated surface parameters or poorly-chosen processing thresholds.To identify instances of this problem, the points which belong to the boundary of each extracted surface will be examined to determine if they can be incorporated into their neighbouring surfaces.If the normal distance between the boundary point in question and the best-fitted plane/line/cylinder through the constituents of a neighboring surface is less than the normal distance between that point and best-fitted plane/line/cylinder through the points of its original surface, that boundary point will be incorporated in its neighbouring surface.This procedure proceeds until no more points can be transferred from invading to invaded surfaces.Figure 5 shows how the introduced quality control procedure improves the outcome of point cloud processing.

t control surfaces
The quality control of the extracted surfaces from the generated point clouds can be also performed with respect to control surfaces extracted from other data sources (e.g, laser scanning data).In order to perform such an evaluation, a few corresponding surfaces between two point clouds are identified.Once the correspondence between two point clouds established, optimal rotation and translation parameters between the two point clouds will be estimated.The estimated transformation parameters assist for the identification of other corresponding surfaces between the two point clouds.The corresponding surfaces are then compared according to their estimated surface parameters or roughness factors.The deviation of the extracted surface parameters is utilized as a measure for the assessment of the quality of the extracted surfaces.Figure 6 shows how the quality control of an extracted planar surface with respect to a control planar surface is performed.

Control Surface
Surface Normal Surface Normal 6. Quality control an extracted planar surface with respect to a control planar surface is performed

QUALITY CONTROL OF 3D SURFACE RECONSTRUCTION
The extracted surfaces from the generated point clouds can be represented as wire-frame model using their boundaries or textured using the captured images to provide a realistic 3D view of the scanned scene.The quality control of the reconstructed 3D models can be carried out in two steps: qualitative evaluation of the reconstructed 3D model and identification of occluded surfaces within captured images.The qualitative evaluation of the 3D model is performed through visual inspection of the reconstructed 3D surfaces to find possible defects and imperfections.In addition, an occlusion detection procedure can be implemented during the texturing of the extracted surfaces to detect the parts of the surfaces that cannot be textured using the captured images.In the following subsection, the utilized approach for the identification of possible occlusions within the reconstructed 3D model will be described in detail.

Occlusion detection
In this section, an occlusion detection procedure introduced in Lari and Habib ( 2015) is described to check the visibility of the extracted surfaces within the captured imagery.This procedure also aims to investigate if the extracted surfaces are occluding or being occluded by other surfaces within the established 3D model.This occlusion detection procedure is established through the following steps: In the first step of this procedure, the suitability of available images for the texturing of an extracted surface is analysed while considering the angle between the surface normal and the optical axes of available images (α).For an extracted surface, the image in question is considered appropriate for the texturing procedure if the surface normal makes an acute angle with the optical axis of the image (Figure 7).Once this suitability analysis is carried out for all surfaces, a list of the appropriate images that could be used for the texturing of a given region is established.
Figure 9. Identification of the occluded part of the region of interest

EXPERIMENTAL RESULTS
In this section, experiments using real multi-platform photogrammetric data are conducted to verify the feasibility of the introduced approach for quality analysis of 3D surface reconstruction.The first experiment is implemented to investigate the feasibility of the proposed quality control procedure for the evaluation of extracted surfaces from a point cloud generated from images captured onboard a UAV-borne photogrammetric system.The UAV-borne images (167 images) utilized in this experiment have been collected using a Sony Nex 7 camera in Dortmund, Germany.In this experiment, the generated point cloud is initially processed for the extraction of individual planar surfaces (Figure 10).The introduced quality control procedure is then utilized to evaluate the quality of the extracted surfaces through point cloud processing and resolve problems affecting the quality of point cloud processing.Table 1 presents the frequency of possible issues occurring in processing outcome.
Figure 10.The extracted surfaces through point cloud processing procedure

Unincorporated points 38%
Over-segmentation 19% Under-segmentation 2% Invading/invaded surfaces 1.5% The second experiment is carried out to evaluate the quality of 3D surface reconstruction using the images collected onboard a UAV with respect to control surfaces derived from an airborne LiDAR data processing.The utilized imagery data in this experiment have been collected using a GoPro Hero3 camera onboard a UAV system and control LiDAR data has been collected using an Optech ALTM 3100 airborne LiDAR system.The quantitative evaluation of the extracted surfaces in two point clouds is performed by comparison of the estimated surface roughness factors for all of the extracted surfaces in both point clouds.The third experiment provides photo-realistic 3D models generated using images collected onboard airborne and UAVborne photogrammetric systems.The utilized airborne imagery in this experiment has been collected over University of Calgary campus, Canada using a RC30 camera and the utilized UAVborne imagery have been captured by a GoPro Hero3 Camera.Figure 14 and 15 show different views of photo-textured realistic 3D models generated using the provided multi-platform photogrammetric data.

CONCLUSIONS AND RECOMMENDATION FOR FUTURE RESEARCH WORK
In this paper, we have described a new methodology for quality analysis of 3D surface reconstruction using images collected onboard multi-platform photogrammetric systems.These reconstructed 3D models have been acquired through automated point cloud generation from the collected imagery, 3D point clouds processing and surface extraction, and texturing of the extracted surfaces.The quality control of 3D models is then implemented from three different aspects: quality of input data, quality of data processing, and quality of 3D surface reconstruction and texturing.The quality control of input data is performed by the evaluation of the captured imagery and collected POS information onboard photogrammetric systems.The quality of initial processing steps for pose estimation and 3D point cloud generation is also assessed in this step.In the next step, the quality control of further processing steps for surface extraction from 3D point clouds is investigated.The quality of the extracted surfaces is also verified by comparing the surface parameters of those surfaces with surface parameters of the matched control surfaces extracted other data sources.In the third step, the quality of the reconstructed 3D model is evaluated from visualization point of view.The possibility of surface occlusions by other surface for image-based texturing of the reconstructed 3D models is also investigated in this step.
In conclusion, this paper provides a comprehensive platform for quality assessment of the reconstructed 3D models using data provided by photogrammetric systems.One should note that the quality analysis procedure presented in this paper is not limited to 3D surfaces reconstructed using photogrammetric data and can be applied for 3D models extracted from other data sources (e.g., LiDAR and range systems).Future research work will be concentrated on the investigation of other issues that might affect the quality of 3D surface reconstruction to provide a more comprehensive quality analysis framework.

Figure 2 .
Figure 2. Examples of low quality images: (a) shadowed image, (b) image with high radial distortion, (c) image with low radiometric quality, and (d) image with repeated texturesThe quality of the collected POS information will be assessed using a well-distributed network of ground control points.The coordinates of these control points are re-estimated using image coordinate measurements and compared to their reference coordinates.The calculated differences in object space represents the overall accuracy of the photogrammetric system's direct georeferencing components(Cramer et al., 2000).

Figure 4 .
Figure 4.The point density map of a generated 3D point cloud for the identification of data gaps The segmented point (a) before quality control procedure and (b) after quality control procedure 3.2 Quality control of the extracted surface w.r.
Figure 11 shows the extracted surfaces from airborne LiDAR point cloud and imagery-based point cloud and Figure 12 represents the accumulation of the extracted surfaces with respect to their estimated surface roughness factors.The comparison of the estimated values in both point clouds shows that higher quality surfaces have been extracted from LiDAR point cloud.The extracted surfaces from (a) LiDAR point cloud and (b) imagery-based point cloud (a) (b) Figure 12.Comparative analysis of the quality of the extracted surfaces from: (a) LiDAR-based point cloud and (b) imagerybased point cloud The quality assessment of the extracted surfaces from imagerybased cloud is then implemented based on the control surfaces derived from overlapping LiDAR point cloud.Figure 13 represents the deviations of the extracted surfaces from imagery-based point-cloud with respect to their matched control surfaces from LiDAR point cloud.

Figure 13 .
Figure 13.The distribution of the extracted surfaces from imagery-based point cloud based on their deviations from matched LiDAR-derived control surfaces Figure 15.Two different views of a photo-realistic 3D model generated from UAV-borne photogrammetric data Qualitative evaluation of the reconstructed 3D models through visual inspection of Figures 14 and 15 verify the quality photorealistic 3D surface reconstruction using overlapping images while considering the possible occlusions by different surfaces within the images.One should note that in these textured 3D models, the occluded areas are visualized in black color.

Table 1 .
Derived quality control measures for point cloud processing outcome