DETECTION OF INDOOR ATTACHED EQUIPMENT FROM TLS POINT CLOUDS USING PLANAR REGION BOUNDARY

: Laser scanning technology is useful to create accurate three-dimensional models of indoor environments for applications such as maintenance, inspection, renovation, and simulations. In this paper, a detection method of indoor attached equipment such as windows, lightings, and fire alarms, from TLS point clouds, is proposed. In order to make the method robust against to the lack of points of equipment surface, a footprint of the equipment is used for detection, because the entire or a part of the footprint boundary shapes explicitly appear as the boundary of base surfaces, i.e. walls for windows, and ceilings for lightings and fire alarms. In the method, first, base surface regions are extracted from given TLS point clouds of indoor environments. Then, footprint boundary points are detected from the region boundary points. Finally, target equipment is detected by fitting or voting using given target footprint shapes. The features of our method are footprint boundary point extraction considering occlusions, shape fitting with adaptive parameters based on point intervals, and robust shape detection by voting from multiple footprint boundary candidates. The effectiveness of the proposed method is evaluated using TLS point clouds.


INTRODUCTION
3D indoor models generated from laser-scanned point clouds are used in applications such as maintenance, inspection, renovation, and simulations. Many automatic 3D indoor modeling methods have been proposed, and the main target of the modeling is basic structures of the indoor scene, such as ceiling, floor, and walls, in many articles (Díaz-Vilariño et al., 2015, Hong et al., 2015, Monszpart et al, 2015, Macher et al., 2017, Previtali et al., 2018, Shi et al., 2019, Takahashi et al., 2019. Fitting planes to 3D points and lines to projected points are often used to find the basic structures. The Manhattan worlds assumption in which the objects follow orthogonal or parallel relationships under a coordinate frame are also used to create useful models and to realize efficient and stable modeling (Hong et al., 2015, Monszpart et al., 2015, and additional regularities, such as regular arrangements which often appear in man-made environments, are also used (Previtali et al., 2018, Takahashi et al., 2019. Using these methods, useful 3D models of the basic structures of indoor scenes can be automatically obtained from laser scanned point clouds. On the other hand, equipment attached to the walls and ceilings, such as windows, lightings, sprinklers and fire alarms are also important for maintenance and inspection of the indoor environment. Drawings of the ceiling including the equipment of several facilities are created from measurement of real environments for inspection. 3D scanning technology is sometimes used to create the drawing. However, recognition of the equipment is manually done, and the work is quite time consuming for large facilities. Unfortunately, automatic recognition of the attached equipment, the methods which can be used in practical work, have not been developed. Therefore, in this paper, a method for detecting indoor equipment attached to the walls and ceilings such as windows, lightings, and fire alarms from point clouds acquired by a terrestrial laser scanner (TLS).
For window detection, holes or space occupancy information in wall planes are often used (Previtali et al., 2018，Shi et al., 2019. However, non-negligible points in the glass area of windows often exist caused by the dirt or screens. In addition, large parts of windows may be occluded by other objects such as columns and curtains (e.g. see Fig. 7). To detect objects in the point clouds, recently, machine learning and classification can be used for several outdoor and indoor environments. After the supervised learning, the classifiers recognize scanned points of not only basic structures but also desks, chairs, and tables (Babacan et al., 2017, Qi et al., 2017a, Qi et al., 2017b, Su et al., 2018. Furthermore, model-based object recognition can be used for detecting specific shape objects from point clouds (Johnson, Hebert, 1999, Drost et al., 2010, Date et al., 2012, Salti et al., 2014. However, in the point clouds acquired by TLS, the points of the attached equipment are often lacking due to their surface reflectance properties of transparency and specularity. Therefore, it is difficult to apply the 3D modelbased recognition method and machine learning to attached equipment detection. In this paper, we propose a method for detecting equipment attached to walls and ceilings (base surfaces) from point clouds using the footprint which is shape of the equipment on the base surfaces. The entire or part of the footprint boundary always appears explicitly as the boundary points of ceilings and walls, and their shapes are simple. Therefore, they are useful to detect the equipment. The contributions of the paper are to propose a simple algorithm for detecting attached equipment based on planar region boundary points in TLS point clouds, to provide some techniques of point cloud processing for detecting footprints stably considering occlusions, point intervals, and noises, and to show effectiveness of the boundary point-based detection method through experiments.

Overview
As described in Section 1, we focus on the footprint of the equipment to be detected, because the points on the equipment surfaces are not often obtained due to the surface reflectance properties of them. The footprint is shape of the equipment on the base surfaces (walls and ceilings). They can be easily found as a part of boundary of base surfaces, and their shapes are usually quite simple. Therefore, more stable detection of the attached equipment is achieved by using footprint compared with 3D points of the equipment.
The major equipment to be detected and type of footprints are summarized in Table. 1. The footprint boundary shapes of almost attached equipment are rectangle and single or multiple circles. Therefore, the method is designed to detect equipment with circular or rectangle footprint boundaries under the assumption that the base surfaces are planes. In this paper, methods for detecting circular footprint boundary for equipment on the ceiling, and rectangular footprint boundary for windows on the wall are described. To detect points of the equipment's footprint boundary (footprint boundary points) correctly, occlusions in the scanning are considered. Primitive fitting considering point intervals is developed to detect equipment on the ceilings, and voting is used to find window footprints which consist of boundary points of multiple planar regions.
The flow of the method proposed in this paper is shown in Fig.1. In the method, TLS point clouds and shape type and dimension of footprint boundary of the equipment to be detected are given by the user as input. First, boundary points of planar regions are extracted from point clouds. Then, the footprint boundary points are detected considering occlusion. Finally, positions of the equipment are detected using footprint boundary shape fitting considering point interval for equipment on single planar region (equipment on the ceiling) and voting for the ones on multiple planar regions (windows). Most of the process is applied to point clouds acquired in one scan to accelerate the point cloud processing using 2D structured points. Finally, the detection results of each scan are merged. Detection methods for equipment attached to the ceiling and windows on the wall are described in Sections 2.2 and 2.3, respectively.

Planar region and boundary point extraction
The ceiling points are extracted using region growing. First, 2D structured point clouds (Masuda, Tanaka, 2010) are generated to realize efficient neighbor search in region growing, and the normal vector of each point is estimated using outer product of difference vectors to neighbors considering Euclidean distance (Che, Olsen, 2017). Then, the region growing is applied to the structured point clouds. In the region growing, thresholding for differences of the estimated point normal and vertical direction, and difference of z-coordinate of points are used as the growing conditions. The regions with enough number of points above the scanner are detected as ceilings. Finally, sets of the connected boundary points of each ceiling region are extracted by traversing neighboring boundary points on the structured point clouds. We denote a set of connected region boundary points by Bi.  The basic idea is shown in Fig. 2. Lower convex equipment yields occlusion boundary points as shown in Fig. 3. Such points should not be used in the detection because they have no useful information for the footprint of equipment. Occlusion boundary points can be detected by distance comparison of neighboring points in structured points as shown in Fig. 3. Let point i be the boundary point of a region R, and di be the distance between the point i and the scanner. If a neighbor point j of the point i is not included in the R and dj is smaller than di, the point i is recognized as occlusion boundary point. The occlusion boundary points can be defined by Eq. (1).
where N(i) = neighbor points of i

Detection of equipment
The equipment is detected by fitting the given footprint boundary shape to each connected footprint boundary point. In this paper, the method for circular footprint detection is described. The fitting problem differs from general line/surface fitting to point clouds because all points exist outside the target footprint shape (Fig. 2) and gaps between points change according to distance from the scanner (Fig. 4). In our method, RANSAC (Fischler, Bolles, 1981) with adaptive parameters is used to detect the target footprint shape. RANSAC consists of model definition from samples (hypothesis) and consensus calculation under a given tolerance (verification). In our method, the diameter of fit circle and the tolerance are adaptively determined by the point interval estimated from the height of the ceiling and distance from the scanner. The point interval near the point i is estimated by Eq. (2). As shown in Fig. 4, the interval increases while the distance from the scanners increases. In the RANSAC, pi is added to the radius of the circle model in hypothesis, and pi is used as the tolerance in the verification. In our implementation, for too small pi near the scanner, we use constant value e instead of pi. If inliers are more than outliers, the boundary points are recognized as the footprint of the target equipment. Finally, the inscribed circle is fitted to determine the position of the equipment. The above process is applied to each point cloud from different scans, and the results are merged. If detected equipment overlaps with the other, the closest one to the scanners is adopted as the output.

Detection of Windows
Region and footprint boundary candidate point extraction First, using region growing on the structured point cloud, vertical planar regions in the point cloud are extracted. Then, the largest vertical planar region is extracted as a wall region, and the related sub-regions which are parallel and close to the  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B2-2020, 2020XXIV ISPRS Congress (2020 wall region are also extracted. This process is repeated until all regions which are considered as walls are extracted. Here, we denote a wall region k and its related sub-regions by Wk. From planar regions in Wk, boundary points Bk are extracted, and occlusion boundary points obtained by using same method describe in Section 2.2 are removed from Bk. Finally, remaining neighboring boundary points in Bk are connected as the footprint boundary candidate points. As a result, a set of the connected footprint boundary candidate points Ck is obtained for each wall k. The following detection process is applied to each Ck.

Detecting window rectangle
In the point clouds, the boundary of the window often consists of small planar regions of window frames. In addition, Ck includes several region boundary points of not only windows but also subdivided window frames, window screens, pillars and part of walls. To detect rectangle of the target window from boundary points of multiple regions, voting for the center position of the rectangle is used. First, all points in Ck are projected onto the wall plane and polygonised using Douglass-Peucker algorithm (Douglass, Peucker, 1973). Then, vertical and horizontal straight-line segments of the polygon with enough length (more than few centimeters) are extracted as candidates of window boundary. The sets of vertical and horizontal segments are denoted by Lv and Lh, respectively. Next, as shown in Figure 5, the voting for the center position of the window candidate is done. In this process, first, a regular grid for storing votes is defined on the wall plane. Then, the voting for cells of the grid is done by possible line segment pairs (lm, ln) which consist of a vertical line segment lm in Lv and horizontal ln in Lh. Here, the overlap length of the line segments of the line segment pair and the target rectangle t estimated by the pair is voted (added) to a grid cell corresponding to the center of t as shown in Fig. 5. Finally, from the cell with the maximum vote which is larger than a given threshold, rectangles are sequentially detected considering overlaps of extracted rectangles. were connected with the outside boundary of the ceiling region due to lack of points at the distant region from the scanner. Additional laser scanning will reduce the lack of detection. In the figure, filled red circles are false detection of the lightings. Seven false detections were observed from other attached equipment with similar footprints (monitoring camera) and the corner of square boundaries from rails of partitions. Figure 7 shows the results of the window detection. The point clouds (11 M points) from one scanner position was acquired using FARO Focus 3D S120, and the point interval at 10 m from the scanner is about 12 mm. The scanner is set 2.6 m away from the wall, and the width of the wall is approximately 8.7 m. Figs. 7 (b) and (c) show the extracted planar regions and line segments of candidates of window boundary. The window frame consists of several small regions caused by the lack of points due to reflectance property of the material. Figure 7 (d) shows detected windows by the proposed method. In this experiment, all windows are correctly detected robustly against to the points on the glass area, large occlusions, and small steps of the window frames. However, positional errors of a few centimeters in some windows were observed. Accuracy improvement is included in future works.

CONCLUSIONS
In this paper, a method for detecting equipment attached to walls and ceilings from TLS point clouds was proposed. In order to realize robust detection from the lack of scan points of the equipment, the method finds points of footprint boundary of the equipment in the point clouds, and shape fitting and voting for the footprint boundary points using given footprint boundary shape of target equipment were used to detect target equipment. For extracting appropriate footprint boundary points from the region boundary of base surfaces (ceilings and windows), a method to recognize points caused by the occlusions was developed. The footprint shape fitting by RANSAC with adaptive parameters was proposed to find footprints robustly from boundary points with different point intervals. Moreover, the voting from multiple footprint boundary candidates was also proposed to realize robust shape detection from multiple candidates of footprint boundary points. In the experiments, a The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) detection rate of over 90% for lightings and fire alarms on the ceiling was achieved; all windows of a wall were also detected from point clouds including points in the glass areas and large occlusions. The future work includes developing a method for detecting several types of equipment, accuracy improvement of window detection, and automatic detection without information of target footprint shape.