INTEGRATION OF IMAGE DATA FOR REFINING BUILDING BOUNDARIES DERIVED FROM POINT CLOUDS

Geometrically and topologically correct 3D building models are required to satisfy with new demands such as 3D cadastre, map updating, and decision making. More attention on building reconstruction has been paid using Airborne Laser Scanning (ALS) point cloud data. The planimetric accuracy of roof outlines, including step-edges is questionable in building models derived from only point clouds. This paper presents a new approach for the detection of accurate building boundaries by merging point clouds acquired by ALS and aerial photographs. It comprises two major parts: reconstruction of initial roof models from point clouds only, and refinement of their boundaries. A shortest closed circle (graph) analysis method is employed to generate building models in the first step. Having the advantages of high reliability, this method provides reconstruction without prior knowledge of primitive building types even when complex height jumps and various types of building roof are available. The accurate position of boundaries of the initial models is determined by the integration of the edges extracted from aerial photographs. In this process, scene constraints defined based on the initial roof models are introduced as the initial roof models are representing explicit unambiguous geometries about the scene. Experiments were conducted using the ISPRS benchmark test data. Based on test results, we show that the proposed approach can reconstruct 3D building models with higher geometrical (planimetry and vertical) and topological accuracy. * Corresponding author


INTRODUCTION
With the development of sensor technology, Airborne Laser Scanning (ALS) demonstrates the high potential for the acquisition of accurate three dimensional dense point clouds in a rapid manner.Consequently, point clouds are often employed for the reconstruction of complex 3D building models (e.g.Maas and Vosselman, 1999;Brenner et al., 2001;Suveg and Vosselman, 2004;Kim et al., 2006;Sampath and Shan, 2010;Lafarge et al., 2010;Haala and Kada, 2010;Huang et al, 2013;Lin et al., 2013).Although, ALS point clouds enable for highly automated processing workflow, the major drawback of point clouds is its' sparsely distributed point nature, which still reduces the geometric accuracy, more specifically the planimetric accuracy, of model boundaries.In order to enhance the accuracy of building models, refinement processes have been applied in some reconstruction schemes.Due to the difficulty of enhancing accuracy of models by point clouds itself, most practices which deal with refinement have used external data sources.In this regard, image data is shown as the most prominent source because it has complementary properties than ALS point clouds.Integration of these two data sources enhances the quality of building models in many ways, for instance LoD, segmentation, planimetric and as well as topological accuracy.Once data is properly integrated, image information can be used to increase the accuracy of roof point segmentation.Khoshelham (2005) use a single image for refining segmentation of point clouds.Rottensteiner (2010) illustrate that the segmentation can further be enhanced by integrating multiple images and it solves contrast issues of individual images.
The goal of this paper is to present a new method for enhancing, mainly, the planimetric accuracy of roof models reconstructed from point clouds.The method proposed by Perera and Maas (2014), i.e., closed circle analysis, is used for the reconstruction of initial roof models.Straight lines are detected using the Burn's line extraction algorithm (Burns et al., 1986) from multiple images.The already derived initial roof outlines are primarily used as guides for deriving more precise 3D line segments from image data.For this, model edges are projected to the image space to match with the extracted image line segments.For the construction of boundary line segments in object space from the stereo (and multiple) images, this study follows the geometric relationships of line features at projective geometry as proposed by (Heuel, 2004;Ok et al., 2011).The matching process is further enhanced by defining some constraints.
The performance of the process is evaluated with the ISPRS benchmark test data Results show that a significant improvement on planimetric accuracy can be achieved with the proposed method.

RELATED WORK
In the approach proposed by Ameri and Fritsch (2000), image information is used to refine the building edges detected from the DSM that has been derived by point clouds.Similarly, in Mcintosh and Krupnik (2002), edges extracted from aerial images are matched and are used to refine DSM derived from point clouds aiming to accomplish sharp discontinuation of urban buildings.In contrast, Ma (2004) proposes a postrefinement approach for the initial building models, which is derived by clustering method from ALS point clouds.Stereo imagery is mainly used in this post-refinement process.Chen et al. (2005) integrate building boundaries extracted from aerial images and roof planes derived from point clouds for reconstructing enhanced building edges with the help of split and merge strategy.Lee et al. (2008) extract initial building regions from the point clouds and merge them with the regions segmented from images to reach coarse roof edges.Precise roof boundaries are then extracted based on the coarse result through line segment matching and perceptual grouping.Cheng et al. (2011) derive the principal building orientation and building roof edges are thus extracted robustly from aerial images.To determine 3D roof boundary segments, they integrate point clouds and 2D boundary segments extracted from multiple images.A strategy to recover boundaries which are lost is also included in their approach.Habib et al. (2010) propose a semiautomated way for enhancing the accuracy of roof patches generated from point clouds by integrating point clouds and stereo imagery.
In ALS and image data integration, co-registration of one data set to another is first needed to achieve.Currently, many data registering schemes have been developed and their applicability associates mainly with the processing workflow.Habib et al. (2001) perform single photo resection to estimate orientation parameters using a modified iterative Hough transformation.Schenk and Csatho (2002) illustrate that the images can be integrated with point clouds through the break lines and surface patches extracted from the point clouds.To this end, linear features need to be first extracted from the images as building roofs can be described by linear features.In linear feature extraction, burns line extractor (Burns et al., 1986) and line growing -Förstner operator (Förstner, 1999) are widely being used.From the extracted linear features, corresponding features can then be searched and matched.As the features extracted from image space are in 2D, matching of corresponding linear feature in stereo or multi-view images would lead to transform them in to 3D object space.
Current practices of line matching procedures can be divided mainly into two groups: (i) matching of individual line segment and (ii) matching of group (mostly pairs) of line segments (Schmid and Zisserman, 1997).In either way, during the matching process, the search space has to be restricted to reduce the complexity.In order to filter out irrelevant candidates, algorithms in the first group mainly apply basic geometric parameters of line segments such as orientation, length, and mid-point (e.g.Collins et. al., 1998;Noronha and Nevatia, 2001;Suveg and Vosselman, 2004;Kim and Nevatia, 2004;Jaw and Perny, 2008).Group of line matching is also a quite popular approach (e.g. Park et. al., 2000;Kim and Nevatia, 2004;Ok at el., 2010a).Using relaxation labelling, Zhang and Baltsavias (2000) adapt pair wise searching strategy while Kim and Nevatia (2004) illustrate how to use pair-wise approach for multi-view images based on the epipolar geometry.

Reconstruction of initial roof models
As proposed in Perera and Maas (2014), the reconstruction of roof models from point clouds is done as a modeling of roof corner vertices.Here, first inner roof corners and associated in-ner skeleton is fixed which is followed the modeling of outer roof corners or roof outer-boundaries.
Describing the process in detail, as can be seen from Figure 1, the convergence of ridge and step-edge lines produce inner roof corners and is represented by the shortest closed cycle (SCC) in a roof topology graph.
To fix such corners, corresponding SCCs should be first extracted.This is done by decomposing the RTG into possible SCCs using Dijkstra's algorithm as shown in Figure 1.Graph edges in the SCC relate which feature line to be intersected and which not to be intersected.Based on the attributes of SCCs, relevant features lines are efficiently accessed from databases.The SCCs with edges labeled as ridge-line are chosen for fixing convergences solely by ridge-lines (Figure 2).Since the convergence produces a single intersecting point in 3D, the fixing can be considered a weighted least square process.The weight is assigned as a sinus angle between two roof planes.
In the RTG, the remaining SCCs (length > 3) refer to the corners associated with mixed-feature lines (Figure 3).These corners generally lie at different height levels because of the available step-edges.Since the convergence of each corner arises at different height level, the SCCs are split into disjointed paths and converted to directed path graphs.The feature lines relevant to each path are fixed separately (Figure 3b).Sample results in Figure 3(d, e) demonstrate that the procedure is more robust and leading to topologically and geometrically valid roof top vertices.

Fixing of outer-boundaries via outermost cycle
As shown in figure 4, some of the outer-boundaries meet outer ends of the inner roof skeleton.If one can recover the exact locations, where the inner roof skeleton meets the outerboundaries, then the outer roof corners can be accomplished.However, location of gutters and eave lines should first be corrected in order to establish geometrically correct outer corners at their correct locations.Constraints on gutter symmetries, aligning of outer-boundaries by sweeping and so on can be applied to achieve this.In the process of maintaining the roof geometries, the top hierarchy can be given to preserving the gutter symmetries because of its global look.As such, individual boundary alignments can be performed based on the preserved gutters.
Figure 4: Convergence of outer edges with the inner roof skeleton (red inner skeleton, blackouter edges that intersect inner skeleton).
Figure 5 illustrates that the method explained in above sections is able to reconstruct different complex roof models.

Model refinement
To automatically extract straight line segments from aerial photographs, the Burns line extractor (Burns et al., 1986) is used in this research.Although the line extraction is done at sub-pixel accuracy, the Burns line extractor -as in other line extraction methods -gave many irrelevant line segments related to unwanted objects such as vegetation patches, road features and border of shadow regions (Figure 7).In order to increase the efficiency of the refinement process, first unnecessary extracted lines are dropped out.This is done by projecting the initial models to the image space.Then, based on a buffer range, line segments which come into the buffer area is chosen assuming the ideal boundary line is within the defined buffer.For the easiness, the projected model lines and selected extracted lines (lines that come into buffer range) indicate as reference and candidates.In order to construct an object space line segment (3D) for a given reference, it is first necessary to find corresponding candidates from the stereo pair images.In the case of multiple images, one way to perform this is to match candidates geometrically, relevant to two successive images at a time.Based on the projective geometry, how the plane intersection can be achieved in a homogeneous vector space is illustrated by (Heuel and Forstner, 2001) and (Heuel, 2004;Ok el al., 2010b).However, in this research, Euclidian coordinate space and the intersection of two or many planes are done to reconstruct object space line segments.Each step of the object space line reconstruction process is described in the following sub sections.

Constructing object space line segments
The matching algorithm intersects pairs of viewing planes from a pair of images at a time, and compares the constructed 3D line with the given constraints (i.e.defined threshold for distance to plane).From all the possible candidate pairs, the best matched candidate pair or the best 3D line representing the known constraints are selected for a given image pair.Similarly, all possible image pairs are also compared one after the other through their list of candidates to find other corresponding pairs of candidates or 3D lines.In case of two images or one image with a roof plane, the constructed 3D line is taken straightaway as the goal is to construct the 3D line segment.But for multiple images, unique candidates are first chosen by removing redundancies in the selected pairs and multiple plane intersection is finally performed based on the least square theories.

Constraints
Sometimes, from the closely located adjacent parallel candidates appearing in a pair of images (Figure 9), the chosen candidate pair could be leading to a faulty result.Besides, the outcome could represent a building ground edge or another edge somewhere in a side wall, or even an arbitrary line in space, instead of the respective roof edge (Figure 9c).The perpendicular distance to the roof plane from the midpoint of the constructed 3D line segment is used for accepting the outcome (Figure 9d).Although, we obtained 3D line segments from the above method, some of the derived line segments may still contain some defects.One example is directional deviation of line segments due to the shadow effect (Figure 10).Missing line segments can also be seen relevant to some model edges defined by images.
For a successful refinement of a building model, these issues should be overcome.With the aim of minimizing such defects and filling missing boundary edges, an additional step next.

Rectification of image line segments
In order to eliminate effect of missing edges, first it is identified whether it is relevant to a gutter or eave lines.If it is a gutter line, then if that boundary edge in the initial model follows rough gutter symmetry, then based on the line segment relevant to opposite gutter and one which is derived from images, the missing edge is re-established.Similarly if a defective edge is found as shown in (Figure 10), then, by analyzing the adjacent eave derived from images the exact eave for the defective eave edge is rectified.

RESULTS & DISCUSSION
The performance of the method is analysed in the context of the ISPRS WG III/4 "Test Project on Urban Classification and 3D Building Reconstruction", which allows us to use external reference data.For the analysis, we use only the test site "area3" which contains residential area having small detached houses.
Figure 11 is the average overall evaluation results in terms of roof planes, and it illustrates that the correctness, completeness and quality of refined roof planes are increased slightly by the refinement process.Since the refinement process considers only the available buildings in the initial modelling results, i.e. excluding the missed buildings (FN) caused in the feature extraction, it is impossible to expect significant improvements on the completeness and quality.Table 1 Illustrates that the planimetric accuracy of roof models is increased, though it is not a substantial number.Figure 12 demonstrate the per-pixel evaluation results before and after refining the roof outlines.It is recognisable that false positive (FP) and negative (FN) errors along the roof outlines are now reduced.However, large blue regions are still remained as it is.Actually it is a limitation of the guided line searching process.Label images of the difference of DSMs (see Figure 13) elaborate that the heights errors close to the step-edge locations are restricted to a very narrow area.This is because the planimetric uncertainty of step-edges has decreased.Figure 14(a) illustrates an effect due to the erroneous recognition of attached vegetation as the roof plane, leading to a FP error.This was successfully avoided based on the assumption "polygon which doesn't deliver any line segments from image data may represent an erroneously reconstructed roof".Figure 14(b) illustrates erroneous overhangs caused by under-segmentation.In reality, these were effects due to the merging of close vegetation with the roof planes.All these effects were rectified during the refinement process.
Figure 14: Per-pixel evalaution of some selected buildings (yellow -true positive, red and bluefalse positive and negative).

CONCLUSION & OUTLOOK
The results show that the refinement process further improves the indices completeness and quality.The correctness has reached to its top having100% value.This tells that now no more false positive roof planes.Since the approach is unable to find boundary lines for the building that are missed in initial roof models, those blue regions can be left out from the analysis.Then, one can say the results are fully improved.Of course, there are still very narrow false positive and negative errors which caused many reasons.Effect due to poor contrast difference extraction of the ideal boundary lines would be problematic.Shadow effect further reduces the positional accuracy of lines.At the time of constructing object space line segments, some errors could occur and sometime, when rectifying the missing and defective boundary edges, prediction could relies on erroneous line segments.
Planimetric accuracy is reached to 0.5 m range and which could hard to accomplish with the point clouds only as this range is roughly equal to the point space of laser scanner data which we used.The uncertainty around step-edge locations are not substantially decreased as their planimetric locations are now acceptable.
However, results can be further improved and further experiments are needed concerning the available false positive and negative narrow regions.

Figure 1 :
Figure 1: Subdivision of shortest closed cycles: (a) internal skeleton of the roof model; (b) relevant RTG; and (c) decomposed SCC and their associated inner roof corners (dark thick line -roof outline, red lineridge lines, solid linesgraph edges).

Figure 3 :
Figure 3: Convergence given by mixed feature lines: (a) inner roof corners associated with step-edges; (b) dis-joint paths at each height level; (c) having many intersection positions; (d)

Figure 6
Figure6is an analyzed sample result of reconstructed building models based on the method explained in above sections, which are well fitted with benchmark reference data provided by ISPRS.The green color is because the majority of model planes correspond to low height residuals.This indicates that the vertical accuracy of the resulting roof models is adequate even for very high accuracy engineering applications.In contrast to the high vertical accuracy, a comparatively low planimetric accuracy can be seen from the resulting models.As shown in Figure6(b), both false positive and negative errors may arise in roof outlines which can hardly be corrected by exploiting only the

Figure 7 :
Figure 7: Extracted line segments using the Burns line extractor

Figure 8 :
Figure 8: Viewing rays coming from end-points of a line segment (same object) in the image space (a, b and a , b represent the same object and in object space it is AB).

Figure 9 :
Figure 9: Perpendicular distance to model plane: (a) and (b) subset of two neighboring images and their extracted line segments; (c) and (d) constructed object space line segment without and with distance to plane constraints (white boxes indicate desired area of interest, yellow linesextracted edges, red and green lineswithout and with the distance to plane constraint).

Figure 11 :
Figure 11: Completeness, correctness and quality of model reconstruction before and after refinementsper-roof plane.

Figure 13 :
Figure 13: Difference of DSM as label image: before and after refinement is shown is left and right side respectively.