Detecting Openings for Indoor/Outdoor Registration

: Indoor/Outdoor modeling of buildings is an important issue in the ﬁeld of building life cycle management. It is seen as a joint process where the two aspects collaborate to take advantage of their semantic and geometric complementary. This global approach will allow a more complete, correct, precise and coherent reconstruction of the buildings . The ﬁrst issue of such modeling is thus to precisely register this data. The lack of overlap between indoor and outdoor data is the most encountered obstacle, more so when both data sets are acquired separately and using different types of sensors. As an opening in the fac¸ade is the unique common entity that can be seen from inside and outside, it can help the registration of indoor and outdoor point clouds. So it must be automatically, accurately and efﬁciently extracted. In this paper,we start by proposing a very efﬁcient algorithm to detect openings with great precision in both indoor and outdoor scans. Afterwards, we integrate them in a registration framework. As an opening is deﬁned by a rectangular shape composed of four segments, two horizontal and two vertical, we can write our registration problem as a minimization of a global robust distance between two segment sets and propose a robust approach to minimize this distance using the RANSAC paradigm.


Context
The topic of building reconstruction draws increasing interest and has various applications such as urban planning, virtual tourism. BIM (Building Information Model) is also becoming a de facto standard to handle the whole life cycle of buildings and 3D city models are central to manage smart cities and in particular to realize environmental simulations. Combining indoor and outdoor modeling of buildings remains a challenging task. The first issue of such modeling is to precisely register the indoor and outdoor data sources. Openings are the most obvious common entity to link the inside and outside data. As such, it can help the registration of indoor and outdoor point clouds. Therefore, in this paper, we propose to detect rectangular openings with high precision and integrate them in a registration framework.
1.2 State of the art 1.2.1 Openings detection Many works have dealt with the windows detection problem. A hole based windows extraction method was proposed in Pu and Vosselman (2007), where a segmentation process was performed in order to group laser points in planar segments. A TIN (the Triangular Irregular Network) of the wall segments is generated after recognizing them. the windows points, extrusion points and door points are identified by classify the hole points. An automatic approach to facade and window detection from mobile LiDAR Data by combining two strategies was proposed in Wang et al. (2011) . The first strategy used Principle Component Analysis (PCA) to cluster point clouds into potential facade region. while the second extract the facade by applying RANSAC (Random * Corresponding author Sample Consensus) plane fitting to potential facade region. Finally, the windows detection is achieved in two main steps: potential window point detection and window localization. The authors of Tuttas and Stilla (2011) have proposed a new method for detecting windows in airborne multi-aspect laser scanning data which represent a sparse point clouds. They started by detecting facade planes using point normals and region growing algorithm. Then, the windows are detected using the indoor points (points lying behind the detected facade planes) which have a special interest due to their regularities in appearance. Another method for window and door detection has been proposed in Nguatem et al. (2014), which apply a model selection based on Bayes factor and use a Monte Carlo simulation to generate and score competing models. This method has proven its ability to distinguish between various window and door shapes. A robust algorithm to locate opening corners for terrestrial LiDAR points has been proposed in Li et al. (2018). This algorithm searches and analyzes the facade with a globally wide sliding window that reduce the influence of noise in horizontal and vertical directions. The changes in facade elements was inspected by computing gradients in each sliding window, applying statistical analysis and deriving the symmetry information. The authors of Recky and Leberl (2010) have proposed a modified gradient projection method designed for automatic processing of complex building facades. This algorithm is able to detect many different windows types without a learning step. An automatic method to detect windows and estimate their parameters in 3D LiDAR point clouds has been proposed in Aijazi et al. (2014). This method fuses information from both symmetrical and temporal correspondences to estimate window features. A new algorithm for openings detection in indoor point clouds has been proposed in Assi et al. (2019a). Holes corresponding to openings were detected after identifying walls in 2D binary image. Each opening was submitted to an energy function with two terms: data and coherence. Those functions help the authors to determine if an opening is due to a window/door or an object obstructing the acquisition.

LiDAR scans registration
Integrating multi-platform, multi-angle, and multi-temporal data is a core step in several fields of research such as computer vision, remote sensing and 3D modeling. To achieve this integration, a registration step is necessary. For Terrestrial Laser Scanning (TLS), there are two main families of registration methods depending on the number N of input point clouds: pairwise (N = 2) and multi-view (N > 2) registration. In this work we are interested in the pairwise category. The most used approach to solve this problem is feature-based registration which can be decomposed in three main steps: extracting geometric features (points, curves, planes, and/or surfaces), selecting the correspondences, and using selected correspondences to estimate the optimal transformation Dong et al. (2018). The ICP algorithm (Iterative Closest Points) Besl and McKay (1992) and its proposed improvements Low (2004) Grant et al. (2012) are considered as a reference for the iterative approximation methods. The transformation parameters can be iteratively estimated by generating pairs of correspondences and minimizing the error metric (point to point distance, point to plane distance, . . . ). A major disadvantage of ICP is its sensitivity to initialization: if the initial data are not close enough, ICP converges towards a local minimum corresponding to a wrong registration. In order to minimise the search space for correspondence between two point clouds and to increase the accuracy of the registration, the authors of Rabbani et al. (2007) have used standard shapes, especially planar and cylindrical surfaces to introduce two solutions suitable in industrial environment: an indirect solution which separates object fitting and registration steps, a direct solution which simultaneously determines the shape and pose parameters of the objects as well as the registration parameters. An NDT (Normal Distribution Transform) algorithm and its variants are proposed to deal with the problem of 3D points cloud registration. A certain cost is obtained that evaluates the alignment quality between the two point clouds, according to the desired transformation between the two point clouds and using a set of Gaussian distributions with different probability density functions (PDFs) to represent the referenced point clouds Dong et al. (2020) Magnusson et al. (2007). A registration workflow simulating the well-known Random Sample Consensus method (RANSAC) was proposed in Al- Durgham et al. (2013) to register overlapping terrestrial laser scans by using 3D linear features to establish an automatic matching strategy.

Indoor/Outdoor registration
The registration of indoor and outdoor scans is a challenging problem for Indoor/Outdoor building modeling. Two registration and georeferencing methods for exterior and interior point clouds, in the case of heritage building recording, are tested in Murtiyoso and Grussenmeyer (2018). The first corresponds to the independent georeferencing method, in which the indoor and outdoor scans were georeferenced separately. The second is a keypoint based registration method. Those tie points concern mainly openings such as windows, which are evenly distributed throughout the building. Windows are manually segmented in the indoor and outdoor point clouds. Then keypoints are detected for both sides of the windows, resulting in two sets of keypoints. The work presented in Assi et al. (2019b) has proven that the use of key points rather than the entirety of a point cloud for an ICP algorithm can improve its performance to register indoor and outdoor scans. An approach for automatically aligning the nonoverlapping indoor and outdoor parts of a 3D building model computed from image based 3D reconstructions has been pro-posed in Koch et al. (2016). This alignment has been performed by identifying corresponding 3D structures that are part of the interior and exterior model (e.g. openings like windows).
The authors of Djahel et al. (2021) have proposed a new planar polygons based registration algorithm to efficiently register indoor and outdoor scans. The proposed algorithm starts by clustering polygons according to their normal direction, then by their offset in the normal direction. The polygon correspondences (hypotheses) are found using the obtained clusters. An optimal transformation is estimated for each hypothesis and the best one is selected according to a quality criteria. However, this method cannot recover translations along the façade plane if only a single wall is seen from outside through the windows (which is a rather common case).

Data
This study was carried out using two terrestrial LiDAR data acquisitions on the Zoological Museum of Strasbourg: • Mobile outdoor acquisition by the Stéréopolis MMS (Mobile Mapping System) which gives access to the sensor topology (adjacency between successive pulses in the same line and between lines) 2.
• Static indoor acquisition by a static Laser scanner with no access to sensor topology 1.

OPENINGS DETECTION
According to Pu and Vosselman (2007), we can define the following characteristics for an opening: • Most openings have a rectangular vertical shape of limited extent (a few meters) positioned within a vertical plane (the wall) The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France • Openings which are not covered with curtains cause holes on the wall segments where the Laser beams traverse the wall plane.
• Openings which are covered with curtains reflect the Laser beam which produces Lidar points which are not on the wall plane.
According to these properties we propose a robust method to detect openings in LiDAR data compatible with any acquisition mode (mobile outdoor or static indoor) which is summarized in figure 3.

LiDAR data segmentation and facade selection
RANSAC is the most used method for LiDAR data segmentation. But, it has some limitations such as under-segmentation (gropuing points from nearly co-planar but disjoint surfaces in a single planar primitive). In order to overcome the limitation of RANSAC, we chose to use the two solutions proposed in Djahel et al. (2021) After the planar polygons extraction, we have selected the facades as large vertical polygons.

Evidence of openings detection
As the LiDAR beams usually cross the facade through openings, we propose to start by detecting the evidences of openings as the intersection points of these beams and the detected facades using Ray Tracing: 1. For each point Pi, we trace a ray Ri from the LiDAR optical center O to Pi.
2. we find the intersection points P j i of Ri with the supporting planes Pj of each façade polygon Fj.
3. If one P j i lies inside the polygon and the distance between Pi and Pj > dmin, we add P j i to the list Ej of evidences of openings on wall j. In Section 2.2, we use dmin = 0.1m in order to exclude noisy points that may still represent points of the facade.

Outline openings extraction
For each wall plane Pj the evidences of openings Ej are grouped in vertical rectangles: • Extract the connected components of Ej with a distance threshold.
• Estimate the minimum bounding rectangle of each component.
• For each rectangle, transform each 2D corner to a 3D point and create the four 3D line segments corresponding to its edges to get a 3D representation of our shape as shown in Figures (5, 4).

OPENINGS BASED REGISTRATION ALGORITHM
Registration is usually performed by matching features. In the present work, the features are the two sets of vertical openings rectangles detected from indoor and outdoor scans. Selection of correspondences is a crucial step for a successful registration because a bad choice can lead to a bad estimate of the optimal transformation. In our case, the openings are not characteristic enough to match them robustly independently. As an opening is defined by a rectangular shape composed of four segments, two horizontal and two vertical, we can write our registration problem as a minimization of a global robust distance between two segment sets. To achieve this minimization with high accuracy and robustness, we must select the right matches and choose the good optimization algorithm.

Distance between segment sets
We propose to define a distance between two sets of 3D segments in a way that minimizing this distance will favor important overlaps between segments and small distances over these overlaps while being robust to outliers, which is to be expected as it is possible that openings extracted form one data set will have no counterpart in the other. This will implicitly also minimize angles between the segments as an important angle with The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France an important overlap implies a large distance. Given two sets of 3D segments we start by defining a relative overlap between two segments over the projection p on the bisector of the two segments: We define this bisector as the line passing through the closest point Pi,j to (A 1 i B 1 i ) and (A 2 j B 2 j ) and oriented by the bisector vi,j of d 1 i and d 2 j . Calling: (3) we have: The projection on the bisector is: and the projected points have a linear coordinate: The length of the projected segment is then defined as and the overlap length of the projected segments as: We then define a distance between a segment L1 ∈ L1 and a segment set L2: where d thr is a distance threshold above which a line is just considered an outlier and dist is a metric distance between segments, for instance: For each line pair, this distance is 0 if L1 is completely overlapped by segments of L2 with 0 distance over these overlaps. Finally we can write our final symmetrized robust distance between 3D segments sets as: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France

RANSAC optimization
RANSAC has proven its robustness and efficiency as an optimization algorithm in several applications. In this part, we describe a new RANSAC based on two matching strategies. The first consists in matching the vertical planes representing the walls of the two scans whereas the second consists in matching the selected segments in the matched walls. The advantage of the integration of these two matching strategies is to simplify the selection of the two pairs of segments and optimize the calculation time of the algorithm by reducing the number of iterations. At each RANSAC iteration: • We randomly select a wall plane from each scan.
• We randomly select one vertical segment and one horizontal segment in each selected wall plane.
• We associate the vertical segments and the horizontal segments between the two selected sets.
• We compute the transform (rotation and translation) that best aligns the matched 3D segments using the method of Section 3.3.
• For this transform, we estimate the global robust distance between all segments of the two sets using equation (14).
The final registration is given by the transformation that minimizes the global robust distance.

Transform estimation
Once two pairs of segments {vi, vj} (vertical) and {hi, hj} (horizontal) are associated, we estimate the rotation that best aligns the corresponding two 3D lines. Let us call the unit director vectors of vi, vj, hi and hj. We start by creating orthonormal bases O i = (x i , y i , z i ), where: We then compute the rotation R that aligns the associated clusters as the base change matrix between O i and O j : The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France To estimate the translation, we start by defining the point to line distance: assuming that d is normalized, and again calling [d]× the matrix of the cross product with d. We look for the translation t that minimizes: The minimum is reached when the gradient is null: Calling:

Openings detection
Our proposed method to detect openings can be applied for both indoor and outdoor scans regardless of the density of the 3D point cloud. The first evaluation was carried out on more than 50 outdoor scans, the obtained results have shown the robustness and accuracy of our framework as more than 98 % of the openings of each scan are detected as shown in Figures(4,5. Afterwards, we tested our approach on indoor scans where we have detected around 80 % of openings as shown in Figure 6. The performance of our algorithm is lower on interior scans due to the presence of curtains for most of the windows that prevent rays from traversing the wall plane from indoor to outdoor while it traverses the façade plane from outdoor to indoor. Compared to hole based methods, which cannot precisely extract window outlines in most cases, and binary image based methods which require a lot of parameters like pixel size, our algorithm is very efficient to detect openings, whatever the acquisition platform (indoor static mode and outdoor mobile) and the point clouds density.

Indoor/Outdoor registration
The second part of our contribution consists in proposing an openings based registration algorithm to address the challenging problem of indoor/outdoor registration with a unified formalism. Our proposed algorithm works without any constraints on the initial position of the two scans, unlike iterative methods which require a good approximation of the initial transformation to be able to converge towards the correct solution. The key step of our algorithm is the detection of openings which was carried out using an automatic method applicable in indoor and outdoor data. The obtained results have proven the performance of our algorithm to register indoor/outdoor data whatever the initial position and orientation as shown in figures 7. The particularity of our algorithm lies in the effect that it matches the openings as a set of 3D segments. we defined a global robust distance between two segments sets and we proposed a robust approach to minimize this distance using RANSAC pardigm. The proposed method in Djahel et al. (2021) has some limitations which can be improved using our algorithm. If we have three pairs of polygons (with independent normals), we can recover a 6D (rotation+translation) transform between two scans. In some configurations, we are able to find only two degrees of freedom for the translation (this situation can happen if we match an indoor polygon which represents a small part of the ceiling with two outdoor polygons). Using our proposed algorithm, we can find the missing degree of freedom.
Implementation: Our algorithm was implemented in C++, using several software libraries: • The CGAL library https://www.cgal.org/) to check if the intersection point of the LiDAR ray and the supporting planes of the facade inside the facade polygon.

CONCLUSION AND FUTURE WORK
In the present work, we have proposed a method based on openings detection and matching to address the challenging problem of indoor/outdoor registration. In the first part we proposed an automatic method to detect the openings in the LiDAR data whatever the acquisition mode and the density. In the second part we have proposed an efficient algorithm to register the indoor and outdoor scans using the detected openings as features. We register the openings detected from both inside and outside by minimizing a global robust distance between the corresponding segment sets and we proposed a robust approach to minimize this distance using the RANSAC paradigm. Our preliminary tests have exhibited the potential of the proposed methods (openings detection + openings based registration). We can propose some improvements in the future works: • Allowing more flexible openings shape, for instance by extracting corner points for each connected component and connecting them using parametric B-spline curves.
• Leveraging a semantic segmentation of the input point clouds to improve the accuracy and robustness of openings detection.

ACKNOWLEDGMENT
This work has been supported by the Building Indoor/Outdoor Modelling ANR-17-CE23-0003 BIOM project. The authors would like to thank Mrs. Tania LANDES who provided them with the indoor data.