AUTOMATION OF WINDOWS DETECTION FROM GEOMETRIC AND RADIOMETRIC INFORMATION OF POINT CLOUDS IN A SCAN-TO-BIM PROCESS

The detection of openings based on point clouds is generally based on geometric features namely the X, Y and Z coordinates of points. Existing methods exploit geometric features but ignore most of the time other attributes as colour and intensity. Such information may be however particularly interesting to recognise and segment openings in facades. In this paper, the use of radiometric information, namely colour and intensity, is investigated for the segmentation of windows from façade point clouds. The assumption is made that windows are significantly different in terms of materials and colours to be distinguished from facade walls. The exploitation of colour and intensity are considered separately. A histogram analysis method is proposed for the exploitation of intensity information whereas a region growing method based on colorimetric distances is considered for the exploitation of colours. In order to refine the results, geometric information, and more precisely depth of points with regards to façade walls, are combined to radiometric information. Several combinations of attributes are considered and provide promising results for windows segmentation based on point clouds.


Openings detection in point clouds
The scan-to-BIM process consists in capturing the as-is conditions of buildings in form of point clouds and in creating asbuilt BIM from these data. This process remains largely manual due to, on the one hand, a very large amount of data, and on the other hand, to the challenge that constitutes the reconstruction of occluded parts of the scene. Several studies deal with the automation of as-built BIM creation. A semi-automatic approach was already developed for the segmentation and reconstruction of walls, ceilings and slabs of existing buildings from point clouds (Macher et al., 2017). Our interest will now focus on windows. In this paper, a semi-automatic approach is proposed for the detection of windows from point clouds acquired on façades.
The detection of openings based on point clouds is generally based on geometric features namely the X, Y and Z coordinates of points. The assumption is often made that windows correspond to empty spaces in the point clouds of façades. After extracting planes composing façade point clouds, Boulaassal et al. (2009) perform, after a planar segments detection, a Delaunay triangulation so that points located on opening's boundaries can be extracted.
As highlighted by Jarząbek-Rychard et al. (2020), existing methods exploit geometric features but ignore most of the time other attributes like colour or intensity. These information might be particularly interesting for the identification of openings, namely windows and doors. Windows of a façade can have a very different colour compared to that of a façade wall as well as they are composed of a different material (wood or aluminium) leading to intensity variations. The use of such attributes could contribute to detect openings which are difficult to detect based only on the geometry. Indeed, the detection based only on * Corresponding author geometric feature can be difficult if the depth of windows is not very high and if windows are occluded, for example by curtains.
This paper studies the promising use of radiometric information for the purpose of detecting windows. The intensity value of the laser beam provided by the laser scanner as well as a colour information (R, G, B) issued from the photographs taken by the laser scanner are considered. Some scanners are not equipped with a digital camera and thus don't provide colour. And even if the scanner is coupled with a digital camera, many users frequently don't activate the option of scanning with RGB information in order to save time during acquisition. Therefore, the exploitation of colour and intensity were considered separately.
The aim of our study is to segment as automatically as possible windows and doors from a façade point cloud thanks to colour and intensity information and then to refine results with geometry and more particularly the depth of points related to façade plane. The results could then be used to reconstruct windows in a BIM model. One should note that the exploitation of thermal features which can be acquired in parallel with a thermal camera was already considered in Macher et al. (2020).

Dataset used for the study
The point cloud presented Figure 1b  The laser scanner used is equipped with a digital camera which allows to take photographs and colorize point clouds in a postprocessing step. One should note that this laser scanner doesn't provide the HDR (High Dynamic Range) function for the photographs.

RELATED WORKS
Some methods consider the exploitation of images for radiometric information exploitation. These images can result on the one hand from the combination of photogrammetry and laserscanning techniques as illustrated by Del Pozo et al. (2015). On the other hand, orthoimages can be created by orthoprojecting point clouds on a plane (Quintana et al., 2018). Based on these images and orthoimages, well known image processing methods can be applied.
Regarding the exploitation of point clouds and their associated radiometric information, several methods can be found in the literature. Zhan et al. (2009) propose a segmentation algorithm based on colorimetrical similarity and spatial proximity. A region growing based on the Euclidean distance in the colorimetric space is first performed to define coarse regions. Then, region merging and refinement are performed. Aitelkadi et al. (2014) exploit at the same time geometry, colour and intensity for the extraction of facade details. Intensity presents the benefit not to be sensitive to lightning conditions. As remarked by Zhan et al. (2009) and Aitelkadi et al. (2014), shadows and reflective surfaces strongly disturb algorithms based on colour information.
An unsupervised windows extraction approach from photogrammetric point clouds with thermal attributes is proposed by Lin et al. (2019). The segmentation of point clouds is conducted by the generation of multiscale supervoxels, a region growing and an energy optimization. Jarząbek-Rychard et al.
(2020) propose an approach for a supervised extraction of façade openings (windows and doors) from photogrammetric 3D point clouds attributed to RGB (red, green, blue) and thermal infrared (TIR) information. Lichti (2005) uses near infrared in combination to colour and emphasizes that there is a parallel between aerial acquisitions (satellite imagery and LiDAR) and terrestrial laser scanning. The R, G, B and near infrared information can be seen as four channels which provide a spectral information in addition to spatial information. Lichti (2005) chooses to apply a supervised classification through the maximum likelihood classifier which is frequently considered in remote sensing.

DISCUSSION ABOUT INTENSITY AND COLOUR
The intensity constitutes the first component of the radiometric information. The intensity returned by objects and measured at scanner level represents the amount of light received after reflexion of an incident laser beam on a surface. This information is usually coded between 0 and 255 or between 0 and 1. When looking at a point cloud colorized by intensity, structures and objects can be generally identified. The hypothesis is made that each type of materials has its own intensity signature, and that this intensity has only few variations in all the points of a same object. Voegtle et al. (2008) investigated the effects of object materials and colours -typically encountered on building facades -on laser scan measurements. This study highlights that the intensities values increase from black to white and a same behaviour is observed for colour sheets. Moreover, intensity values are also strongly linked to object materials. The intensity is also sensitive to the scanner position and more particularly to the incidence angle formed by the laser beam with the surface and the distance between the scanner and the acquired object. Lichti (2005) and Soudarissanane et al. (2009) established that beyond 65 degrees, the influence of incidence angle can't be neglected. Moreover, an incidence angle close to the normal combined with a relatively short scanner-object distance can lead to zones of high intensity. Figure 2 illustrates this phenomenon. A high intensity was reached in the red frame because the distance scanner-object was small (4 meters approximately), and the incidence angle was close to the normal of the surface. This shows that the relation between intensity and range is not calibrated for the laser scanner used for the acquisitions.
The R, G and B values associated at each point are for their part extracted from digital images acquired on the scanned object with a digital camera which is often integrated in the laser scanner. This information is recovered by linking each laser point (X, Y, Z) with the corresponding pixel in the image (x', y'). The colorization of points is performed for each scan independently. Thus, regarding overlapping zones, colorimetric variations can be encountered due to, on the one hand lighting changes during acquisitions, and on the other hand automatic lighting equalizations. These variations constitute a brake for colour exploitation.
Whether for images or point clouds segmentation, RGB channels are often converted to HSV (Hue, Saturation, Value) channels (Sural et al., 2002;Pierdicca et al., 2020). The HSV representation is known for more closely aligning with the human perception of colours. The HSV colour space separates out the luminance from the colour information. The colours are represented as three independent variables which allows to take into account variations (shadows and different light conditions).

INTENSITY EXPLOITATION
After the presentation of the data, the method developed to exploit intensity information for extracting windows is described.

Data
As highlighted in the previous section, intensity is sensitive to the scanner position and more particularly to the incidence angle formed by the laser beam with the surface and the distance between the scanner and the acquired object. For this reason, a filter has been applied so that points for which incidence angle can't be neglected are removed (beyond 65 degrees). This filter is applied to each scan based on the scanner positions. Regarding the distance between scanner and object, the three scans under study are placed approximately at the same distance from the façade and far enough so that phenomenon describing Figure 2 is not encountered. The segmentation process is obviously applied to each scan individually because intensity values differ from one scan position to another. Figure 3a presents a point cloud of a part of the façade acquired from one scan position and colorized with intensity values.

Developed methodology
The developed methodology considers the analysis of the intensity histogram, namely the identification of peaks describing different elements. A similar methodology is used in Macher et al. (2020) for the exploitation of thermal attributes of point clouds. It involves three main steps: histogram analysis and generation of the curve which approximates the histogram, extraction of points corresponding to each peak by an iterative process, identification of points belonging to façade walls and openings.
Since the histogram is by construction non-continuous, a kernel density estimation, also termed the Parzen-Rosenblatt method, is used to approximate the histogram by a curve. This method consists in the calculation of the probability density function describing the sample. The extraction of peaks in the histogram is performed based on local maxima and minima of the curve. The highest peak of the histogram is extracted the first by considering the highest maxima and its closest minimums. Peaks are extracted in an iterative process.
The assumption is made that the first extracted peak corresponds to the façade wall. Indeed, façade walls is composed of the main part of the points. The other peaks correspond to points for which the intensity values vary significantly from those of the façade walls and, among them, points belonging to openings.

Application
The methodology is applied to the point cloud presented Figure  3a. In Figure 3b, the grey points correspond to points belonging to the first extracted peak whereas red points correspond to points belonging to other peaks.
As expected, most of the points of the façade wall belong to the first extracted peak. Some points still remain in the façade notably at the bottom of the façade where another material and texture (not smooth as for the main part of the façade wall) are used, and at the borders of the façade.

COLOUR EXPLOITATION
After the presentation of the data, the method developed to exploit colour information for extracting windows is described.

Data
Point clouds of scanner stations are considered separately. Indeed, regarding overlapping zones, colorimetric variations can be encountered. Figure 5a presents the point cloud, already presented in Figure 3a, colorized with RGB information provided by the photographs taken simultaneously to the point cloud by the laser scanner. HSV representation is often used for segmentation purposes, that is why RGB channels are converted to HSV channels to perform the segmentation process. HSV channels are in the range 0 to 1.

Developed methodology
The developed methodology consists, in the HSV space, in separating into regions the elements of the point cloud according to the colorimetric distance. The two steps of the methodology are illustrated in Figure 4.
Similarly to a nearest neighbour classification in a spectral space, a region growing algorithm based on the Euclidean distance in the colorimetric space is first performed to define coarse regions. Colorimetric distances between the first point and the other points in the point cloud are calculated. Points for which the colorimetric distance is lower than a threshold are grouped to form a region (Figure 4a). The algorithm repeats the operation until the point cloud is empty.
The merging of regions is then considered. It consists in calculating the average colorimetric coordinates of each region. If the distance between the average values of two regions is lower than a threshold the regions are merged (Figure 4b).
(a) Region growing (b) Region merging The assumption is made that the façade wall has a homogeneous colour and that the main region corresponds to the façade wall.

Application
The methodology is applied to the point cloud presented Figure  5a. A threshold of 0.3 was chosen heuristically for both steps of the methodology. A total of 4 regions are detected. The region containing most of the points correspond to façade wall. Figure 5b presents the three other regions segmented (one colour per region). The main region is not represented for visual purpose and because it is not the region of interest. The orange points are related to white cardboards fixed on some windows. The blue and green points are related to windows frames but also some parts of the façade. The points in the façade are due to shadows on the façade (red frames in Figure 5a). Moreover, the colour used for the bottom part of the façade as well as some details in the façade is close to the colour of the window frames.  Figure 3 and Figure 5 show interesting results for the use of radiometric information. Nevertheless, the results still need some refinement because of some variations in radiometric information due among other reasons to lighting conditions. Moreover, it must be defined which detected peaks or region belong to the class describing openings.

COMBINATION OF RADIOMETRY AND GEOMETRY
The objects of interest are windows which are part of the façade wall. They are often behind the façade plane, sometimes at the same level but rarely in front of the façade. Thus, it seems suitable to use geometric information and more particularly to use the depth of points, i.e. their depth in relation to the façade plane.
For each point cloud, a Principal Component Analysis (PCA) is performed to rotate the facade so that the Y coordinates correspond to the depth of point as illustrated in Figure 6. A plane is also fitted to the point cloud to determine the average depth of the façade.

Intensity and geometry
The refinement of the result of the segmentation based on intensity is performed by considering the average depth of the façade. The developed methodology allows identifying elements which are different from the façade in terms of intensity, but they are not necessary openings. A filter is applied to points extracted so that only points behind the wall are kept. Figure 7 presents the result of the segmentation based only on intensity. One should note that for the left part of the façade, many points of the façade were extracted. This may be due to the fact that the scanner was closer to the façade wall than the other scan stations. However, by taking into account the geometry, those points are removed as shown in Figure 7b.

Colour and geometry
In order to exploit colour and geometry, the developed methodology for colour exploitation was adapted to integrate also depth. Instead of calculating colorimetric distances in a 3D space composed of the HSV channels, distances are calculated in a 4D space where depth is the fourth dimension. The depth is normalized to be combined to HSV channels. After the segmentation, the detected regions are classified thanks to the depth of the façade plane.
The aim of this combination is to eliminate the shadow effect appearing in Figure 5. Figure 7c and 7d present the results obtained based respectively on colour information and on the combination of colour and geometry.

CONCLUSION AND FUTURE WORKS
In this paper, the use of radiometric information was studied for the segmentation of windows from facade point clouds.
Promising results are obtained since windows, which have different intensity and colour values compared to façade walls, can be segmented semi-automatically based on radiometric features combined with geometric features.
The developed methodology for intensity exploitation is based on the analysis of the histogram of the repartition of intensity values. The method works provided that the laser scanner is not too close to the façade wall and that a filter considering the incidence angle is applied. For the segmentation based on colour information, a region growing method based on colorimetric distances is considered. As shown in the paper, shadows can disturb the results when considering only colour information for the segmentation. Point clouds acquired with other scanners will be considered in future works. Indeed, results might change when using another scanner (other range, other calibration of camera, other intensity correction related to the range). The laser scanner used for the study doesn't provide an HDR function which may avoid shadows effects on photographs. The developed methods will be also applied to this kind of data and on façade point clouds produced by photogrammetry.
In future works, facades with different colours and materials will be also considered so that general recommendations about the use of radiometric information for windows segmentation can be made. The developed algorithmic method will be compared to machine learning and deep learning techniques. The segmentation will be considered not only for the exterior of buildings but also for indoor of buildings. The interaction between indoor/outdoor seems interesting when dealing with windows.
Finally, in order to reconstruct and use windows in BIM models, the segmented windows will be processed to identify the types of windows and to extract insertion points in façade walls. This information will allow to insert automatically windows in BIM models.