TOWARDS AUTOMATIC RECONSTRUCTION OF INDOOR SCENES FROM INCOMPLETE POINT CLOUDS : DOOR AND WINDOW DETECTION AND REGULARIZATION

In the last years, point clouds have become the main source of information for building modelling. Although a considerable amount of methodologies addressing the automated generation of 3D models from point clouds have been developed, indoor modelling is still a challenging task due to complex building layouts and the high presence of severe clutters and occlusions. Most of methodologies are highly dependent on data quality, often producing irregular and non-consistent models. Although manmade environments generally exhibit some regularities, they are not commonly considered. This paper presents an optimization-based approach for detecting regularities (i.e., same shape, same alignment and same spacing) in building indoor features. The methodology starts from the detection of openings based on a voxel-based visibility analysis to distinguish ‘occluded’ from ‘empty’ regions in wall surfaces. The extraction of regular patterns in windows is addressed from studying the point cloud from an outdoor perspective. The layout is regularized by minimizing deformations while respecting the detected constraints. The methodology applies for elements placed in the same plane.


INTRODUCTION
Automated generation of 3D indoor models from point clouds continues to be an active research topic.Updated and detailed spatial models of indoor environments are being increasingly demanded for a wide range of applications, including gaming, tourism, construction control, navigation, emergency response, and location-based services.Among the main challenges connected with automated indoor model generation there are the possible complexity of building layouts and the abundant presence of objects such as pieces of furniture and wallhangings resulting in severe clutters and occlusions.
In the last years, point clouds have gained a lot of popularity and have become the main data source of information for building modelling.Indeed, point clouds are a very dense and accurate set of three-dimensional points representing reality.Methodologies automating the reconstruction from point clouds can be classified in two major categories: data-driven (bottomup) approaches and model-driven (top-down) approaches.While model-driven approaches use previous knowledge for searching the most appropriate model among a specific library and fitting it to the data, data-driven approaches are based on directly extracting features such as points or edges from data.Consequently, the performance of data-driven approaches is highly dependent on data quality and therefore, the existence of noise, clutters and occlusions, may ultimately condition the completeness and correctness of the final indoor model up to a large extent.Most of data-driven methodologies extract the elements (walls, floors, ceilings, window, door, etc.) of each building independently and therefore, the representation only relies on those points corresponding to the individual element.However, manmade environments generally exhibit some regularities (e.g., all windows have the same shape, they are aligned and uniformly distributed along specific directions, etc.) that are not taken into consideration if the building's elements are separately extracted.This context-based information is crucial for obtaining a consistent model less dependent on data quality.
This paper presents a data-driven method for indoor reconstruction starting from point clouds collected by Indoor Mobile Mapping Systems.The main assumption of this method is the Manhattan-World geometry.According to this assumption, elements are characterized by a prevalence of planar primitives and straight lines with orthogonal intersections.The large majority of existing residential buildings in modern cities are in line with this assumption.In particular, the presented data-driven procedure tries to cope with the presence of significant amount of clutters and occlusions, that may be widely observed in indoor environments and may cause significant lack of data.However, some inaccuracies still persist, which can be solved by considering context-based information.In particular, this work, presents a procedure to identify similar building elements (specifically doors and windows) and restore regularities.In particular, the presented methodology can be subdivided into two main steps: first, detected objects are grouped according to three regularity constraints (i.e., same shape, same alignment, same spacing), then the regularized layout is obtained by using the constraints inferred in the previous step and enforcing positional and shape changes to be minimal.
The paper is structured as follows.Subsection 1.1 reviews related work on data driven methodologies for indoor reconstruction with a specific focus on methodologies using regularization for building modelling.Section 2 briefly explains the method implemented for recognizing windows and doors in indoor scenes.A detailed description of the regularization approach is provided in Section 3. The results of the application of this methodology are presented and discussed in Section 4. Finally, Section 5 is devoted to draw some conclusions and to address future developments.

Related-work
Using the classification presented in Zlatanova et al. (2013) methods dealing with indoor modelling can be grouped, according to the primitives they are reconstructing: linearprimitive, planar-primitive and volumetric-primitive based.Linear-primitive methods are based on the preliminary extraction of each floorplan, followed by extrusion assuming walls as planar and vertical surfaces.These methods are generally applied to single floors independently and without clutter.Indeed, clutters may result in problems with the recognition and reconstruction of doors and windows as (Oesau et al., 2018;Ochmann et al., 2016).Planar-primitive approaches consist of submitting point clouds to a planar primitive detection followed by classification (Sanchez et al., 2012;Díaz-Vilariño et al., 2017).However, even if these algorithms work well for extracting planar patches from laser data, they do not consider the complete occlusion problem.Volumetric-primitive detection generally imposes a stronger regularity.Those methods classified in this group can be applied either to 2D (Furukawa et al., 2009) and 3D (Bassier et al., 2017) data sets.Also grammar-based approaches were used to reconstruct MW indoor spaces based on some defined rules for splitting and merging volumetric primitives (Khoshelham and Díaz-Vilariño, 2014); Becker et al., 2015).Detection of regular patterns was mainly focusing on building façades.In particular, those techniques were designed for discovering in single and multi-view images (Muller et al., 2007;Musialski et al., 2010;Nan et al., 2012;Al Halawani et al., 2013;Huang et al., 2014, Xu et al., 2015).Those techniques generally rely on specific characteristics of facades structures such as regularity and orthogonality, to infer regular patterns and to complete missing parts.In many cases (Muller et al., 2007;Musialski et al., 2010, Al Halawani et al., 2013) authors assumed that the façade is governed by a hidden global rectilinear grid.Even if such assumption may hold for façade reconstruction it may be too strong in the case of detection of indoor regularities such as repetition of doors.Indeed, in those cases repeated patterns are weaker and other regularities may appear, such as all doors have the same shape and are aligned with respect the to the ground.Fewer works are focusing on identifying regular patterns in a 3D space.Mitra et al. (2006) and Shen et al. (2011) present a pair-matching based approach to detect symmetries in 3D shapes and point clouds.In Pauly et al. (2008) a general regularity detection method for 3D models is presented.This approach can be used for extracting a single façade pattern.However, in this case similarities in the model are detected by considering a local similarity measure of the point cloud curvature which is more prone to output outliers.In Triebel et al. (2006) a Markov Network approach that requires training is used to label points as windows.In Friedman and Stamos (2011) and Mesolongitis and Stamos (2012) methods for detecting regularities in building façades are presented.However, they are mainly addressed to detect only one feature type (e.g., windows) and require scan points being organized in 2D vertical scan-lines which may be a limiting condition if multiple scans are fused together.Böhm (2008) published a method for completion of point clouds from terrestrial laser scanning, which is done by iteratively utilizing the repetitive information typically present in urban buildings.Another approach aiming at a similar goal was introduced by Zheng et al. (2010), who proposed an interactive method for completing holes in scans of building façades.This method exploits repetitions to consolidate the imperfect data, based on denoising and completing pending parts.Another interactive tool for assembling architectural models was introduced by Nan et al. (2010).In this system, the user defines simple building blocks (Smart-Boxes), which snap to common architectural structures like windows and balconies.They are assembled through a discrete optimization process that balances between fitting the point-cloud data and their mutual similarity.In combination with user interaction, the system can reconstruct complex buildings and façades.In Previtali et al. (2013) regularities are imposed on building facades assuming that elements are distributed in an irregular lattice schema.Repeated candidates are identified according to a voting scheme and the final regular lattice is estimated minimizing a score function.Fewer works are focusing on enforcing regularities in the case of indoor environments.

OPENING DETECTION
This paper presents an automatic method for modelling Manhattan-World (MW) indoor scenes on the basis of Indoor Mobile Mapping Systems (IMMS) data incorporating some contextual-based information (namely, detection of regular patterns) into a data-driven approach.In particular, the methodology presented here focuses on the identification of regularities among building elements (windows and doors) and on the definition of the final pattern respecting the previously identified constraints.However, for the sake of clarity an overview of the entire methodology is presented first in Subsection 2.1.In Subsection 2.2 a specific focus is given to the opening detection as starting point of the regularization algorithm.

Indoor reconstruction overview
The core of the indoor reconstruction methodology is the labelling problem of structural cells in the 2D floorplan.In the case of highly cluttered scenes, by assuming the prevalence of orthogonal intersections between walls typical of MW scenes, the indoor completion is formulated as an energy minimization problem using graph cuts.The workflow of the developed methodology is presented in Figure 1.
The first step consists in the detection and estimation of planar surfaces representing walls, ceilings and floors.In particular, the extraction of the potential indoor primitives is carried out by using a modified RANSAC implementation for extraction of planar surfaces (Previtali et al., 2014).The extracted primitives are then refined to reduce under-and over-segmentation issues.However, due to occlusions and clutters, some walls may be missing in the data set.For this reason, an automated procedure is implemented to look for pending elements in a plausible way.To achieve this, the developed algorithm incorporates some architectural priors on indoor scenes, notably the prevalence of orthogonal intersection between walls as in MW building assumption.For this reason, some 'pending walls' are added to prevent complete missing parts.In particular, according to the MW assumption those 'pending walls' are guessed to orthogonally intersect with 'detected walls.' Starting from 'detected' and 'pending' walls the indoor building model is derived by applying a decomposition and labelling procedure based on cell-complex labelling.To this end, the floorplan is decomposed into adjacent cells creating a 2D arrangement which are labelled as 'indoor' and 'outdoor'.At this stage the indoor reconstruction can be formulated as the optimal labelling of the cell complex and this way can be efficiently handles as an energy minimization problem using graph cuts to finalize the creation of the floorplan.
Figure 1.Workflow of the indoor reconstruction procedure.

Opening detection from ray-tracing
The methodology described in this paper starts from those openings detected by implementing a ray-tracing algorithm (Alsadik et al, 2014).The detection of openings in indoor scenes is a relatively complex task due to the existence of other objects such as pieces of furniture causing occlusions that can be detected as false positives.To detect real openings from occlusions, an occupancy map is generated based on a visibility analysis from a ray-tracing algorithm (Adan et al, 2011).In particular, once all the indoor structure is determined with wall elements, openings may be looked for.Detecting the boundaries of openings, such as windows or doors in a wall is also a complex task.While in the reconstruction of building façades windows are generally detected as holes in the point cloud representing the façade itself, this does not generally hold for the indoor environment.Indeed, also clutters and occlusions may produce significant holes in the point cloud which have to be distinguished from real openings.To this end, a ray-tracing labelling is applied to generate an occupancy map.
The methodology is implemented for surfaces conforming one plane.Once the plane is identified, the point cloud is discretized into regular cells and an occupancy map is generated based on whether inlier points are detected at each voxel location ('occupied' voxel) or not ('empty' voxel).Afterwards, a raytracing algorithm is implemented to distinguish from 'empty' and 'occluded' voxels.If the ray traced from the observer position to the 'empty' voxel under evaluation is intersecting another 'occupied' voxel, the 'empty' voxel will be labelled as 'occluded' (Fig. 2).On the contrary, if the ray is not intersecting any occupied voxel, it will be labelled as real 'empty'.For the discretization of the voxel space a cylindrical buffer of 3 voxels is considered around the identified ray.The main drawback of this approach is given by the fact that each element is separately detected.However, windows and doors generally present some regularities.For example, windows have generally the same shape, they are aligned and uniformly distributed along specific directions.Similarly, doors are aligned with the ground and have similar sizes.By processing each room independently, those regularities are not taken into consideration.To identify these regularities, the first step is the identification of each element into a specific class (i.e., either 'window' or 'door').The workflow of the developed methodology is presented in Figure 3. Once the building candidate features are detected they are subdivided according to their class into windows and doors.
First of all, windows are detected using a methodology conceived for indoor point clouds only.In spite of that, windows are elements that can be observed from an outdoor perspective, and this assumption is further exploited to extract regularities between elements.Indeed, an outdoor perspective may allow an easier identification of regular patterns similarly to the analysis of façades (Previtali et al., 2013).Therefore, in the case of windows, the observer point is placed outdoors.
Once the observer point is determined, windows are clustered according to the façade they belong to.Indeed, it is supposed that windows belonging to the same façade may present some similarities.Once the façade is determined the point cloud is analysed from this point of observation and gaps are identified as previously discussed (Figure 2).To filter out void areas determined by partition walls, the aspect ratio of extracted features is evaluated.Indeed, partitions' walls determine from an outdoor point of view some very slander gaps.By evaluating the aspect ratio, and defining a proper threshold those gaps can be eliminated and windows are kept.
The same procedure is applied to doors.In this case, doors are modelled from the corridor to enable regularity extraction.Doors are studied from an indoor point of view placed on the centre of the corridor.In particular, it is worth to mention that by using the presented approach only doors opened or having a sidewall can be detected.Doors that belong to the same plane of the corridor wall are not recognized.

REGULARIZATION OF DETECTED OBJECTS
Once a set of building elements is detected (i.e., either windows or doors) O={o 1 ,…, o N }, the main aim of the presented methodology is to infer a set of y starting from them R={r 1 ,…, r M }.In particular each object o i can be described as a rectangle where the position of the bottom left corner is (x i , y i ), its height is h i and its width is w i , so: o i ={x i , y i , h i , w i }.In this paper we are looking for three specific sets of constraints:  'same shape;'  'same alignment;' and  'same spacing.'  In particular, if two objects o i and o j are characterized by the 'same shape' constraints, this means they share the same width and height.So this constraint can be formulated as follows: (1) Similarly, if two objects o i and o j have the 'same vertical alignment' or the 'same horizontal alignment,' this constraint can be formulated as: Finally, the 'same spacing' constraints are defined on two element pairs.These can be either in the horizontal or in the vertical direction.In particular, assuming that o i and o j have the same spacing of o n and o m in the horizontal direction and objects are organized in an ascending order with respect to the x coordinate (x i <x j <x n <x m ), his constraint can be formulated as:

=>
(3) A similar definition can be given also for the 'same spacing' constraints in vertical direction: As previously anticipated the first step of the proposed regularization method is the detection of possible regularities between detected objects.Given a set of objects O the main goal is to identify a set of candidates sharing a specific type of similarity.The three types of similarity are separately evaluated.First, the 'same shape' constraint is evaluated.In particular, the whole group of detected objects is clustered, taking into consideration h i , w i , by using the mean-shift clustering algorithm (Comaniciu and Meer, 2002).This exploits the object shape using as bandwidth the user-defined tolerance α.Objects with different shapes will be recognized as different clusters.
The main advantage of mean shift algorithm is that it is a nonparametric clustering technique which does not require prior knowledge of the number of clusters, and does not constrain the shape of the clusters.For all objects belonging to the same clusters, pairwise constraints defined in Eq. ( 1) are set up.
Secondly, the 'same alignment' constraint is investigated.Similar to the previous constraint, two different kinds of clustering are applied: the first one by considering the 'horizontal alignment' (x) between objects and the second one the vertical alignment (y).Indeed, objects aligned with respect to one of the main axis have the same x coordinate (horizontal alignment) and/or the same y coordinate (vertical alignment).Also in this case the clustering is performed by using mean shift algorithm.For all objects belonging to the same clusters, pairwise constraints defined in Eq. ( 2a) and/or Eq. ( 2b) are set up.
Finally, the 'same spacing' constraints are evaluated: in this case, first of all the spacings between consecutive objects are calculated (i.e., objects are organized into an ascending order along one direction) and the spacings are clustered.Also in such a case, for all objects belonging to the same clusters, pairwise constraints defined in Eq. ( 3) and/or Eq. ( 4) are set up.
Since in many cases the spacing between objects is obtained as a multiplier of a basic dimension ('unit') the cluster with smaller spacing is used as reference and others are checked it they are the integer multiplier of the 'unit' (to do this test, the mean values of the different clusters are used).A user-defined bandwidth tolerance α is used.In the case the following constraint are added: where k is an integer value.
Once the constraints between objects are identified, the second step of the proposed approach is aimed at regularizing the layout under the previously identified constraints.The regularization is performed by using an approach similar to the one presented in Dang et al. (2014).In particular, the regularization problem is addressed by transforming the initial layout, L, into a regularized one, L*, by minimizing the following score function: where the first term represents the element's position while the second one represents the element's shape.The parameter ω balances the influence between the position and the shape constraints.
In addition to the previously identified constraints some additional lower and upped bound constraints are added to limit possible changes of the element's position.In particular, the maximum allowed width change inside a cluster is limited to ( ̅ ̅ ) where ̅ is the average height of the cluster and  is the RMSE (Root Mean Square Error) of the cluster.Similar constraints are added also for object height.By solving the quadratic programming problem of Eq. ( 6) the regularized layout is obtained.Similarly to Jiang et al. (2016), constraints are added sequentially to avoid potential conflicts.If any conflict is detected the constraint is discarded.

EXPERIMENTAL RESULTS
The methodology described in the previous sections has been tested in a real case study: a university building indoor surveyed with the IMMS Viametris IMS3D (www.viametris.com/).This data set is provided by the ISPRS Scientific Initiative Benchmark on Indoor Modelling (Khoshelham et al., 2017).Figure 4 shows the point cloud and BIM model of the case study.As it can be observed in the figure, windows are just placed on one side of the building.
In particular, the procedure for regularizing the detected windows/doors in the corridor has been tested.The different layouts have been independently processed for windows and doors.Only the setup of a few parameters has been the bandwidth α for the mean-shift clustering and the weight ω to balance the influence of the position/shape constraints in the sore function.The selection of the bandwidth α is influenced by several factors: 1. the required level regularization: a higher bandwidth value will determine higher regularization; 2. the noise level in the input data: higher level of noise in the input data can be compensated with a higher regularization; and 3. the accuracy of the methodology used for identification of building features.
On the other hand, a higher value of ω will give preference to the position changes with respect to the shape changes.For the case study afforded here, the parameters have been set to α=0.03 cm and ω=0.8.The detected regularizing constraints are shown in Figure 5 and Figure 6 for building windows and corridor doors respectively.In particular, it is possible to observe that the three detected feature classes share the alignment constraint (features inside each class are aligned) and also the number of 'same shape' features is high (two 'same shape' classes for windows in a total range of five features and five 'same shape' classes for doors in a total range of sixteen).Conversely the number of 'same spacing' regularities is quite low especially for doors (two regularities detected in a total range of 14).This observation confirms that in the case of indoor regularities the assumption that regularities follow a lattice structure is generally too strict and more flexible approaches, like the one presented in this paper, are needed.Once the regularities are detected the final regularized pattern is computed (see Figure 7).

Windows
Corridor doors 1 Corridor doors 2

CONCLUSIONS
This paper presented an optimization-based approach for detecting regularities (i.e., same shape, same alignment; and same spacing) in building indoor features.In particular, the developed methodology takes as input a set of building features (e.g., windows, doors) and detects regularity constraints based on some a priori knowledge about the building's geometry.The layout is then regularized by minimizing the deformations of the initial layout while respecting the detected constraints.
Future work will try to extend the presented 2D regularizing methodology to wider 3D patterns.In addition, level of detection of regularities can be increased.Indeed, in this paper we focused on the level of building components (doors and windows).At a higher level it would be also possible to detect and recognize similarities between different rooms and in the organization of the floorplan.Detection of such similarities will be addressed in the future developments.

Figure 3 .
Figure 3. Workflow of ray-tracing algorithm adopted for the detection of openings.

Figure 4 :Figure 5 :Figure 6 .
Figure 4: Data set adopted for testing the procedure described in this paper: (a) Point cloud with the IMMS sensor trajectory (red line); (b) BIM model.

Figure 7 .
Figure 7. Regularization results: initial (black) and regularized (red) position of for the building features.

Figure 8
Figure8shows the final reconstruction for the data sets used.It is worth to notice that all rooms have been successfully reconstructed and all doors and windows detected.

Figure 8 .
Figure 8. Results of the reconstruction of the indoor environment adopted as case study: (a) the reconstructed model without ceiling; and (b) a wireframe view of the same model.