Stochastic Reasoning for Uav Supported Reconstruction of 3d Building Models

The acquisition of detailed information for buildings and their components becomes more and more important. However, an automatic reconstruction needs high-resolution measurements. Such features can be derived from images or 3D laserscans that are e.g. taken by unmanned aerial vehicles (UAV). Since this data is not always available or not measurable at the first for example due to occlusions we developed a reasoning approach that is based on sparse observations. It benefits from an extensive prior knowledge of probability density distributions and functional dependencies and allows for the incorporation of further structural characteristics such as symmetries. Bayesian networks are used to determine posterior beliefs. Stochastic reasoning is complex since the problem is characterized by a mixture of discrete and continuous parameters that are in turn correlated by nonlinear constraints. To cope with this kind of complexity, the implemented reasoner combines statistical methods with constraint propagation. It generates a limited number of hypotheses in a model-based top-down approach. It predicts substructures in building facades-such as windows-that can be used for specific UAV navigations for further measurements.


INTRODUCTION
The detailed reconstruction of buildings becomes more and more important.In many critical situations building models that are only enhanced by textures are not sufficient.Semantics such as the attribution of a window or door are important for many scenarios, such as rescue operations or the calculation of energy balances.The standardized data model CityGML (Gröger et al., 2008) allows for representing building components (such as windows or doors) in different level of details (LOD) that however up to now have mainly been modeled manually.
With the ability to collect data by unmanned aerial vehicles (UAVs) data sets of measurements become easily available.Most approaches for an automatic reconstruction of buildings rely on high-resolution measurements -such as 3D point clouds from laserscans, or features extracted from images.However, on the one hand, observations of all building facades are not alwayas available at first.Occlusions or missing data avoid to build a comprehensive model.On the other hand, it is desirable that the interpretation of data acquisited by an UAV is performed in realtime in order to control further recordings.With huge amounts of data that have to be analyzed this requirement is difficult to satisfy with common approaches of building reconstruction.
To this end, we developed a reasoning method that predicts unknown substructures in buildings based on sparse observations.Hereby, we follow a model-based top-down approach and exploit the fact that it is easier to verify or falsify hypotheses by additional UAV recordings than to reconstruct models from observations in a bottom-up way.The implemented reasoner benefits from the incorporation of extensive prior knowledge and accelerates the acquisition and interpretation of data.The presented work is part of the project "Mapping on demand" that is supported by the German Research Foundation (DFG) and whose aim is to build 3D building models in real time during acquisition of data by an UAV.With the provided predictions the UAV is able to navigate to targets where new measurements restrict the set of hypotheses.This paper illustrates the approach with the prediction of a row of windows in a building facade.Figure 1 depicts the prediction for a facade of the Poppelsdorf Castle in Bonn, Germany.Prior knowledge is incorporated as probability density functions and constraints on the model parameters.The facade width f is, for example, correlated with the number n of windows, the width w of the windows, the distance d between these windows and (1) Table 1: Constraints that characterize a row of windows the distances l and r between the facade margin and the leftmost and rightmost window, respectively (see equation (1) in table 1).Parameters such as width of windows or height of floors follow certain architectural restrictions and design decisions.
Observations such as a footprint, i.e. the widths of the facades, and possibly measurements of a single embrasure are the input of the reasoner.Based on these sparse observations the reasoner calculates the best hypotheses consistent with the given observations that can be verified or falsified by further observations in a topdown manner.As can be seen in figure 1d) the observed width of the given facade allows for three or four windows.The occluded window on the right can be predicted.The measurements of the embrasure belong obviously to the first window but may be part of the left or right embrasure.An additional observation would restrict the space of hypotheses.
The building model is characterized by discrete as well as continuous parameters.These are correlated by non-linear equations and follow certain regularities so that their domains can be further restricted by probabilitiy density functions.Bayesian networks are used in order to reason within the uncertain domain of buildings.However, the non-linearity of the problem and the hybrid character leads to a complex task that has to be tackled adequately in order to met the requirements of online UAV interpretations.Therefore, we present a new approach that combines statistical methods with constraint propagation.While the constraint propagation component exploits its strength in solving combinatorical problems with non-linear constraints, Bayesian networks are used to reason with uncertain data.While the constraint program instantiates the discrete parameters and thus linearizes the problem, the Bayesian network calculates the posterior belief given the observations and determines the most probable explanations for the given facade.
The remainder of this paper is structured as follows: The related work is introduced in section 2. Section 3 discusses the hybrid building model that is the basis of the reasoning process which is presented in section 4.

RELATED WORK
The importance of detailed building reconstruction was emphasized by (Haala and Kada, 2010) who give an overview of current approaches in the context of automatic city modeling.
A model-based top-down process is one way to reconstruct buildings.It is used in various approaches but requires in contrast to this work sufficient data.(Schmittwilken and Plümer, 2010) used prior knowledge for the reconstruction and classification of facade parts in 3d point clouds.For the estimation of boundaries the approach is based on an informed sampling.(Becker, 2009) introduces a grammar based approach for the reconstruction of buildings and integrates a bottom-up and top-down modelling of 3d buildings.(Pu and Vosselman, 2009) identify building parts such as walls, roofs or windows from high density point clouds and are able to predict occluded facade parts by incorporating background knowledge.
Bayesian networks as used in the presented work are well established in various fields such as financial risk management or discrete number of windows (n), correspondence from embrasure measurement to a window (c), side of embrasure measurement (left/right) (s), ratios between model parameters (r1, r2, r3), indices of mixture components continuous facade width (f ), left facade margin (l), right facade margin (r), window width(w), distance between windows (d), embrasure measurement (e) Table 2: parameters modelling a row of windows medical decision support but are as well applied for computer vision and 3D modelling.In contrast to most applications we have to cope with a hybrid model, i.e. a model with discrete as well as continuous parameters that in general results in a computational complex task.(Lauritzen and Jensen, 2001) developed an efficient algorithm which is able to tackle the problem of hard inference in restricted hybrid networks.However, with non-linear constraints -as in the presented building model -the algorithm is not appropriate for our applicaton.
Constraint programs are popular tools to solve combinatorial problems.However, several approaches were developed that extend the framework by a stochastic component.(Flerova and Dechter, 2010) adapt combination and marginalization operators to find the m best solutions for optimization tasks in graphical models.Intervals with cumulative distribution functions are used by (Saad et al., 2010) in order to model a degree of knowledge for uncertain data.In order to address uncertainty, our approach combines the classical constraint propagation with Bayesian networks and thus benefits from the strength of both paradigms.
Various approaches extract architectural features such as symmetry from man-made objects but do not take semantical information into account.(Pauly et al., 2008) present a framework for detecting regular or repeated structures in point-or mesh-based models.The algorithm is based on an analysis of pairwise similarity transformations.Neither prior knowledge is incorporated nor are results enhanced by semantical information.(Mitra and Pauly, 2008) describe algorithms that discover symmetry or repeated structures in 3D models and discuss their use for shape exploration and manipulation in the context of architectural design.They reinforce that regular structures are often the result of economical, manufacturing, functional, or aesthetic considerations.(Thrun and Wegbreit, 2005) present an algorithm that detects symmetry in partial 3D views in order to reconstruct occluded surfaces of the analyzed objects.

HYBRID BUILDING MODEL
Man-made objects such as buildings are often characterized by regularities.Geometric constraints, e.g.orthogonality or parallelity, are present in most buildings (Loch- Dehbi and Plümer, 2011).Building parameters such as the width or height of windows are restricted by architectural properties and can be described by distributions and functional dependencies.
The presented approach benefits from such a prior knowledge about typical structures and distributions of building facades that is based on an extensive data analysis.The building model is characterized by discrete as well as continuous parameters (cf.where the ith Gaussian distribution N (µi, σ 2 i ) is weighted by wi.The use of mixture models enables us to use well-studied reasoning techniques such as Bayesian networks where the model is defined in a directed graph with conditional probability distributions and functional dependencies (Koller and Friedman, 2009).Herewith, the reasoner is able to generates a few good hypotheses for given observations that support the UAV in the acquisition and interpretation of new data.
Bayesian networks are powerful tools to reason with uncertain data.While there exist efficient inference techniques for discrete problems we have to tackle the problem that the domain of buildings is characterized by discrete as well as continuous parameters.This is especially a hard task since building models are characterized by nonlinear equations and discrete variables have apriori unknown states.However, the constraints have the special form i dici, as a product of discrete parameters di and continuous parameters ci.The presented approach is therefore divided into solving the discrete problem first and determining the continuous model parameters afterwards.The following section delineates the corresponding reasoning process.

REASONING
The implemented reasoner expects a few observations as input and outputs the best hypotheses, i.e. instantiations for discrete and continuous model parameters.To this end, we incorporate prior knowledge and combine Bayesian networks with constraint logic programming that results in the following three reasoning steps: 1. incorporate prior knowlegde 2. propagate constraints 3. apply Kalman Filter Figure 2 illustrates the consequence of the reasoning steps in the presented method.The reasoner begins with an infinite number of hypotheses and narrows the space of hypotheses in each of the three reasoning steps to a small number of hypotheses.

Incorporation of prior knowledge
As described in section 3 prior knowledge includes functional as well as statistical constraints.It further allows for the integration of architectural prior knowledge that can be extracted from footprints or data bases such as the type of buildings or symmetry properties.Domains of parameters are thus be restricted beforehand.
Basic observations about buildings are available in a database of about 9 million buildings extant in North-Rhine-Westfalia, Germany, including their footprints and height.However, footprints could just as well be extracted from open source projects such as Open Street Map or provided by interpretation of data previously selected by the UAV.
Footprints allow for further analyses in order to extend the prior knowledge and enhance the reasoning process.For instance, a Figure 2: The presented reasoning process restricts the space of hypotheses in three reasoning steps: incorporation of prior knowledge, constraint (logic) programming and Kalman filter method is applied that enables to identify and model hierarchical structures in building footprints.Hereby axis as well as translation symmetries within footprints are identified and modeled using formal grammars (Dehbi and Plümer, 2011).The latter enable with a set of rules the representation of symmetry hierarchies with respect to various underlying global and local symmetry axes.Figure 3 illustrates the axis symmetries identified from the footprint of the Poppelsdorf Castle in Bonn.The symmetric substructures are highlighted with the same color as the underlying symmetry axis.In order to identify symmetries in footprints the concept of symmetric polylines is learned in a supervised way.Consequently the reasoner makes use of these information as background knowledge in order to restrict the search space of hypotheses.
In case symmetry properties can be incorporated into the reasoning process, less observations are needed for hypotheses of high quality.The fact that symmetric building parts are often similar constructed helps to restrict the space of hypotheses.
As an example for statistical prior knowledge figure 1b shows the distribution for the distance between the left margin of a facade and its first window.The prior knowledge is characterized by a Gaussian mixture with four components and extremely small variances.In a first step, component distributions are used to derive thresholds, i.e. they are represented by intervals of the form [µi − λσi, µi + λσi] with means µi, standard derivations σi and appropriate λ (3 or 4).In the same way, it can be observed that continuous model parameters follow certain architectural constraints and can be further restricted by ratios rj with a finite number of states (cf.table 1).Thus, the reasoner has not to deal with the apriori infinite solution space of variables.

Constraint logic programming
Before applying inference techniques with Bayesian networks we are interested in possible instantiations of the discrete parameters, e.g. the number of windows or the component of the Gaussian mixture.The problem is described by a set of constraints C = C1, ..., Cm on variables X = x1, ..., xn with associated domains D = D1, ..., Dn.It can thus be defined as a constraint satisfaction problem (CSP) whose solution is an instan- The constraints that define the building model (cf.table 1) restrict the domains of unknown parameters so that the final solution leads to a small number of qualified hypotheses.The output of the CLP component is used afterwards as evidence for statistical reasoning.The building model is now restricted to single Gaussian distributions instead of mixtures.Above, instantiations of discrete parameters transform the non-linear constraints to linear ones.For further reading on constraint processing the reader is referred to (Dechter, 2003) and (Marriott and Stuckey, 1998).

Kalman filter
As a result of constraint propagation, the stochastic component has to reason within a specially structured Bayesian network: a state-observation model with a n-dimensional state vector x ∈ R n representing the model parameters and a m-dimensional observation vector o ∈ R m that can be described by the mapping o = M x with a measurement matrix M ∈ R nxm .For such state estimations the Kalman filter is an efficient algorithm for calculating the posterior (Thrun et al., 2005).It assumes that state transition and measurement can be described linearly and initial beliefs are represented by multivariate Gaussian distributions: Gaussian distributions represented by µ and Σ are carried over from the constraint solver of the reasoner and the posterior is computed in a correction step implemented by a Kalman filter: where Q ∈ R mxm is the Gaussian noise of the observations and Id is the identy matrix.Figure 4 shows exemplarily the distributions for the left margin in different reasoning steps.The Gaussian mixture (red) for this continuous model parameter serves as prior knowledge and has four components.After instantiating the discrete parameters such as the component of the Gaussian mixture the constraints are linearized and the pdf reduces to a single Gaussian mixture (blue).The value of the continuous model parameter is finally predicted by using the correction step of the Kalman filter (green) that yields means and variances for the building model according to the given evidence.The output of the algorithm is a small set of ranked hypotheses with instantiated discrete and continuous parameters.20 datasets of facades with different observations such as facade width or embrasures were evaluated.The expected hypothesis -compared to the corresponding ground thruth data -was among the top six generated hypotheses.Results got more precise by adding a single embrasure observation whereby the expected hypothesis was on average the best hypothesis.The UAV is able to incorporate these predictions in order to determine new flight paths, distinguishing between the predictions and enhancing the 3D building model.For example, as can be seen in figure 1d, the information whether the measured embrasure is a right or a left one or the extraction of an additional embrasure would help to decide between different hypotheses.
In order to cope with a wide range of building facades the reasoner is implemented in an object oriented and modular manner.
For example each single facade is represented by an internal domain model and can be connected to other facades in a complex facade.For extensibility and flexibility, domain parameters and their characterizing constraints and distributions are further defined in a backend module so that the reasoner works independent from its input parameters.Thus, new problems can be addressed easily.

CONCLUSION
To conclude, the reasoner supports the acquisition and interpretation of data by an unmanned aerial vehicle.Based on LOD 1 models it restricts the space of hypotheses by predicting substructures in facades and exploits the fact that it is easier to verify or falsify hypotheses by the UAV engine than to reconstruct models from observations in a bottom-up way.On the one hand, the acquisition of high-density measurements is no longer required.On the other hand, occluded building parts can be predicted.The considered building model is characterized by continuous as well as discrete parameters that in turn are correlated by non-linear constraints.To tackle the resulting problem of hard inference, the presented approach combines constraint logic programming with Bayesian networks that in turn benefit from an extensive prior knowledge such as Gaussian mixture models, functional dependencies or derived information such as symmetries.Thus, the implemented algorithm enables the generation of first hypotheses based on sparse observations that can be refined subsequently by the UAV.

Figure 1 :
Figure 1: predicting substructures in building facades: a) facade image of Poppelsdorf Castle, b) distribution (Gaussian mixture) as prior knowledge, c) footprint and measurements of embrasures as observations, d) resulting hypotheses for a row of windows

Figure 3 :
Figure 3: Derived prior knowledge: Global and local axis symmetries of the Poppelsdorf Castle, Bonn, identified from the footprint.The symmetric substructures are highlighted with the same color as the underlying symmetry axis (vertical dotted lines) tiation of the variables, i.e. an assignment of values for each variable (x1, a1), ..., (xn, an) with (a1, ..., an) ∈ D1 × ... × Dn so that all constraints are satisfied.Constraints (logic) programming (CLP) is well suited to solve those combinatorial problems with nonlinear equations and are used during the reasoning process to instantiate the discrete parameters and linearize the problem.

Figure 4 :
Figure 4: distributions for left margin: gaussian mixture as prior (top) and selected component for determining the posterior (bottom)