IDENTIFICATION OF ERRORS IN 3 D BUILDING MODELS BY A ROBUST CAMERA POSE ESTIMATION

This paper presents a method for identification of errors in 3D building models which are results of inaccurate creation process. Error detection is carried out within the camera pose estimation. As observations, parameters of the building corners and of the line segments detected in the image are used and conditions for the coplanarity of corresponding edges are defined. For the estimation, the uncertainty of the 3D building models and image features are taken into account.


INTRODUCTION
A great demand for 3D building models contributed to development of various automatic and semi-automatic methods, as well as commercial software for 3D building model creation.However, an error free creation is often not possible.Particularly, while creating the model from aerial images, the roof overhang is not modeled.
Such errors can cause problems in many of applications of the 3D building models.In this paper, we concentrate on the texturing application.Texturing is commonly used in urban scenes to increase the visual attractiveness of the 3D building models (Weinhaus and Devarajan, 1997, Groneman, 2004, Hegarty and Carswell, 2009).Fac ¸ade and roof textures not only support the visual interpretation of the mapped scenes, but also can be source of information about the building envelope, for example about the position of windows.
In the last years also not only the appearance of the building envelope was of interest, but also it's image in thermal infrared (TIR) domain (Hoegner et al., 2007, Chandler, 2011, Borrmann et al., 2012).Exploring the TIR images of the building envelope can contribute to thermal inspections of buildings and to assessment of the energy efficiency of the buildings.Challenging are large area inspection where a manual geo-location and analysis may be very time consuming and expensive.As a solution an automatic alignment of the TIR images ia texture mapping using terrestrial image sequences (Hoegner et al., 2007) and in combination with airborne oblique imagery (Iwaszczuk et al., 2011) was proposed.
Using airborne oblique imagery, all building faces -roofs and fac ¸ades can be captured.Knowing the exterior orientation of the images, textures can be extracted by the projecting the model into the image.This allows to select image regions representing each face.The exterior orientation can be easily determined from the navigation device mounted on the flying platform together with the camera.However, often the navigation data and the 3D building model are not accurate enough for a precise texture mapping.Such errors in the exterior orientation and 3D building models lead to extraction of an incorrect image section as the texture.This can cause displacements in the localization of objects detected in further processing (e.g.heat leakages).The inaccuracy of the exterior orientation parameters can be reduced by the a model-to-image matching (Iwaszczuk et al., 2012).
The errors in the model more difficult to deal with and do not allow for a precise coregistration and correct texture extraction, particularly for the fac ¸ades in case of unmodeled roof overhang.Therefore, the errors in the 3D building model have to be identified and corrected to ensure the best possible geometry and the best possible thermal textures.
Several methods for verification of 3D building models can be found in the literature.Approaches for the verification using point clouds (Hebel and Stilla, 2011) or aerial images (Huertas andNevatla, 1998, Nyaruhuma et al., 2012) were proposed.
In (Huertas and Nevatla, 1998) a multiple step method is used to register the model with the new image, validate the model, find structural changes and update the model accordingly.For the verification only one image is used.In an another work (Nyaruhuma et al., 2012) mutual information between the model edges and the image is used to identify changes of the investigated scene.First the mutual information is calculated for the whole image and then the amount of the information is veryfied for each edge using statistical tests.

PAPER OVERVIEW
Presented work is a continuation of our previous work presented in (Iwaszczuk et al., 2013).In contrast to this previous contribution, we take the uncertainty of the 3D building model and of the extracted image features into account in this paper.Moreover, we implement a robust estimation and use it to detect errors in the 3D building model.
In contrast to other works we don't concentrate on the change detection between two different points in time, but aim to find outliers in the building geometry.What is more, we don't treat every pixel separately, but see an model edge as an object defined by a set of parameters verified in the estimation process.
In Section 3. we explain the methodology used for the pose estimation of the thermal camera based on the corresponding model edges and line segments detected in the image.In Section 4. we describe the detection of errors in the 3D building model by implementing an outlier detector within the estimation.In Section 6. we presents achieved results and discuss their quality, as well as the limitations of our approach.

LINE-BASED CAMERA POSE ESTIMATION
In the presented approach, line segments are selected to be used for matching.The corresponding image line segments are assigned to the model edges using an accumulator approach.The model edges are projected into the image using the initial exterior orientation taken from the navigation device and moved and rotated in the image creating a 3D accumulator space.For each position of the projected model in the image, the number of fitting lines segments in the image is counted.Correspondences which voted for the accumulator cell with most line correspondences are used for the optimal pose estimation.
The mathematical model is formulated using homogeneous representation of the image and model features.The condition for coplanarity of building corners defining an edge and a line segment in the image is utilized for this purpose (Fig. 1).In addition to the functional model, the stochastic model is also defined in order to take the uncertainty of the 3D model and image features into account.The covariance matrices of the homogeneous entities are often singular.To be able to cope with such covariance matrices, additional constrains for the observations are needed.Optimal exterior orientation parameters are estimated using Gauss-Helmert model.
Figure 1: Principle of assignment of image line segments to 3D model edges

Finding correspondences
In the presented approach, line segments are selected to be used for matching.The corresponding image line segments are assigned to the model edges using an accumulator approach.The model edges are projected into the image using the initial exterior orientation taken from the navigation device and moved and rotated in the image creating a 3D accumulator space.For each position of the projected model in the image, the number of fitting lines segments in the image is counted.Correspondences which voted for the accumulator cell with most line correspondences are used for the optimal pose estimation.

Optimal camera pose estimation
Estimation of the exterior orientation parameters in the projective space is formulated using the complanarity of lj, X1i and X2i, where X1i and X2i are the endpoints of a building edge corresponding to line segment lj detected in the image.Coplanarity of lj, X1i and X2i is expressed as incidence of the projected points x 1i and x 2i with the line lj.The projected points x 1i : x 1i = PX1i and x 2i : x 2i = PX2i, where P is the projection matrix.Then, the incidence conditions l T j x 1i = 0 and l T j x 2i = 0 write l T j PX1i = 0, (1) (2) These two equations are directly adapted in the Gauss-Helmert model as conditions for the observations and parameters.
Also in the projective space the uncertainty of the image features and 3D building model can be introduced.The covariance matrix for a 3D point X represented in homogeneous coordinates X can be directly derived from the cavariance matrix Σ X X for the Euclidean representation X of this point as However, due to redundancy in the homogeneous representation, the cavarinace matrix ΣXX is singular (Förstner, 2004) which leads to restrictions in the optimization.To solve this problem, all entities has to be spherically normalized (Kanatani, 1996), so that l s j = Ns(lj)1 , X s 1i = Ns(X1i) and Y s 1i = Ns(Y1i).(In the rest of this Section, the index s is omitted assuming the homogeneous coordinates to be spherically normalized.)This normalization has to hold also during the estimation.Therefore, also the constrains for the observations are needed.
To find optimal solution for the unknown parameters β = [ X0, Y0, Z0, ω, φ, κ], the optimization method for homogeneous entities presented in (Meidow et al., 2009a) and (Meidow et al., 2009b) is adapted for this functional model.For this purpose, the Lagrange function is minimized, where λ and ν are the Lagrangian vectors.In contrast to (Meidow et al., 2009a) and (Meidow et al., 2009b), here the restriction for the estimated parameters h1( β) = 0 is not needed, because the estimated parameters are defined directly as exterior orientation parameters X0, Y0, Z0, ω, φ, κ.The observation vector for each pair of corresponding lines writes y ij = [lj, X1i, X2i] T , where l = [a, b, c] T is the homogeneous representation for the image line segment and X1i, X2i is the homogeneous representation of the corners of the corresponding 3D building edge.The covariance matrix Σll is assumed to be known as result of the line fitting or as result of error propagation knowing the covariance matrices of the end points of the detected line segment.This done using where S is the skew-symmetric matrix The solution for the unknown parameters fulfilling eq. 9 is calculated in a iterative manner, similar as described in (Meidow et al., 2009b).For all conditions the Jacobians are calculated at the approximate values of β as and used for normal equation where (16) The index τ denotes the current iteration.Then the residuals are computed as

Conditioning the coordinates
Switching from the Euclidean to the homogeneous representation for point x in 2D or X in 3D is usually effected by adding 1 as an additional coordinate (homogeneous part).Hence, for a 2D point in Euclidean space x = [u, v] T , the equivalent homogeneous representation is x = [u, v, 1] T , and for a 3D point in Euclidean space X = [U , V , W ] T , the equivalent homogeneous representation is X = [U , V , W , 1] T .In many photogrammetric applications, particularly in aerial photogrammetry, the points are given in geodetic coordinate systems (e.g., Gauss-Krüger, UTM), where the values for U and V is in order 10 6 .Computations with such inconsistent number can lead to numerical instability of the computations.To solve this problem, the homogeneous entities should be conditioned.Similar to the conditioning proposed by (Heuel, 2002), also here, the entities are conditioned before optimizing, by checking the condition where x h i is the homogeneous and xO i the Euclidean part of a homogeneous entity xi.If max hO < fmin the conditioning factor is calculated as In case of very large Euclidean part xO compared to the homogeneous part x h , f calculated as shown in eq.20 can be smaller than the machine accuracy h .Hence, if f < h then f should be calculated as ( (Heuel, 2002)).Next, each entity is conditioned using matrices for the 2D points, for the 2D lines and so that the conditioned coordinates x c , l c and X c are calculated as and where fim is the conditioning factor for the 2D image entities and f mod is the conditioning factor for the 3D entities.Conditioning entities changes also the transformation matrix.Here the transformation matrix is the projection matrix P which can be reconditioned using

IDENTIFICATION OF ERRORS IN THE MODEL
The inaccuracies in the model should result in large residuals in the adjustment.This can be used to identify these inaccuracies.For this purpose, an outlier detector using statistical test on corrections to the observations is implemented within the adjustment.This is done by using weight matrix for computation of the weight coefficient matrix for the corrections to observations Then the vector of standardized corrections for the observations is calculated.
Statistical tests are used to identify the outliers using the critical value which allows rejecting a hypothesis with the significance level α.If V i > h then this observation is considered as an outleir.
Observations identified as outliers are excluded from the estima- The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-3, 2014ISPRS Technical Commission III Symposium, 5 -7 September 2014, Zurich, Switzerland tion and the link to the corresponding model edges is stored.Then the procedure is repeated for every frame of the image sequence.Model edges which are frequently identified as outlieres are labeled as possible errors in the 3D model.

RESULTS
The results are presented on an oblique thermal infrared (TIR) image sequence taken from helicopter flying over a densely built up urban area.The frame rate was 25 frames/sec.which gives a high overlap between the frames.100 following frames (with frame IDs 13141-13240) and three buildings were selected for the first tests.The 3D building model was created manually in a commercial software by fitting predefined building primitives.Hence, no roof overhang is modeled.The significance level was set to α = 0.1 for the experiments.
Each observation is represented by 11 parameters, 3 related to the image line, and 8 related to model corners.In the presented method, each parameter is treated separately.It means that for each observation, one or more parameters can be identified as an outlier.It can be also distinguish between outliers in the image feature and outliers in the 3D building model.However, the usage of this fact is not sufficient for outlier detection in the 3D model, because the inaccuracies in the line detection can also affect the errors in the parameters of the model points.Tests showed that both errors coexist.Outliers in the line parameters were detected in less cases and in most cases they were coexisting with a errors in the parameters of model points.The outliers in image line parameters were detected for 151 edges in 100 frames but only in 21 cases they did not coexist with outliers in the parameters of model points.

CONCLUSION
In this paper we presented a method for model-to-image matching with optimal pose estimation and outlier detection withing the estimation.The novelty of the presented methodology consist of the usage of the linear features and their uncertainty in the adjustment process which allows for more reliable outlier detection.Using the implemented method, buildings or building parts being potentially badly modeled can be identified.However, this identification is useful only for further manual post-processing.
In the future, more work should be done on more automatic confirmation of an error in the building and automatic improvement of the geometry.

Figure 2 :
Figure 2: Results on outlier detection in the frame no.13209: a) outliers detected based on analysis of parameters of model points; b) outliers detected based on analysis of parameters of image lines; c) outliers detected based on analysis of all parameters.Color coding: dark cyan -detected image line segments without correspondences, bright cyan -projected model edges without correspondences, dark green -detected image line segments with correspondences, bright green -projected model edges with correspondences, dark orange -detected image line segments with correspondences detected as outliers, bright green -projected model edges with correspondences detected as outliers

Figure 3 :
Figure 3: Projected 3D building model in frame 13209 before (red) and after (green) adjustment