AUTOMATION OF THERMAL POINT CLOUDS ANALYSIS FOR THE EXTRACTION OF WINDOWS AND THERMAL BRIDGES OF BUILDING FACADES

In the context of building renovation, energy audits of buildings are performed in order to reduce energy consumption and greenhouse gas emissions. Infrared cameras allow analyzing precisely the energetic performances and the thermal comfort of a building and the analysis of thermal images of buildings represents a key step on the way to the reduction of energy consumption. In the last few years, the combination of thermal and metric information has been increasingly studied. The geometry provides the exact location of defects and enables accurate energy analysis. This paper focuses on the automatic analysis of facades point clouds with thermal attributes. The aim of our work is to extract windows and thermal bridges by considering both geometric and thermal features. The extraction of objects or zones of interest in thermal data consists in the detection of significant temperature variations. The developed approach, mainly based on the automatic analysis of histograms, is applied to a point cloud acquired by a Mobile Mapping Unit. * Corresponding author


INTRODUCTION
In the building sector, the BIM process responds to both an economic and an environmental challenge. The objective of its use is to reach better performing, better quality housing, notably environmental, and at lower cost. In highly urbanized cities, the rehabilitation of the existing buildings is an essential measure to reduce energy consumption and greenhouse gas emissions. In France, buildings participate in 45% of the energy consumed and are responsible for 20% of greenhouse gas emissions. Because of its ability to coordinate all the information needed to diagnose the building and plan rehabilitation work, the digital model of a building is an optimal solution.
The creation of a digital model of an existing building requires the acquisition of the current state of the building. Laser scanners are widely used for this purpose. Moreover, in the context of the renovation of a building, it is necessary to perform an energy audit of the building. Infrared cameras allow analyzing precisely the energetic performances and the thermal comfort of a building. As highlighted by Scaioni et al. (2017) the thermal analysis of buildings represents a key step on the way to the reduction of energy consumption.
In the last few years, the combination of thermal and metric information has been increasingly studied. The geometry provides the exact location of defects and enables accurate energy analysis. Since the geometric resolutions of infrared cameras and the number of strong features in thermal images are low, generating an accurate 3D model exclusively from thermal images is quite challenging (Hoegner et al., 2018). That's why the information provided by a thermal camera is usually combined with data from laser scanning or photogrammetry. Data fusion can be considered either by creating multi-sensors systems or by combining data from different sensors in a later on. In our study, the combination of geometric and thermal features will be studied for the automatic extraction of windows and thermal bridges in building facades seen from outside. Our paper will be composed of four main parts. A related works section will present some existing methods for the extraction of objects or zones based on either thermal images or thermal point clouds. The dataset used for our study will be then presented namely thermal point clouds of different building facades located in Strasbourg (France). The acquisition system will be described, and the quality of the data will be discussed, not only regarding the radiometric (thermal) data, but also the geometric data provided by the laser scanner. Next, the developed approach for the extraction of windows and thermal bridges from geometric and thermal features will be presented. Finally, the results will be analysed and future works will be proposed.

RELATED WORKS
The inspection of a building requires commonly the manual interpretation of thermal images to detect heat loss (Gade and Moeslund, 2014). This involves subjectivity and relies on the expertise of the operator. In the last few years, several works aim at minimizing this subjectivity by automating the process.
In most of the works thermal images are processed directly. Garrido et al. (2018) propose a procedure for the automation of thermographic building inspections mainly focused on thermal bridges. This procedure uses both geometric characteristics and temperature differences with the surroundings. Besides thermal bridges, methodologies are also developed to detect and characterize pathologies in structures (Garrido et al., 2019).
With the rise of sensor combination and data fusion, especially the combination of laser scanning and thermal imaging, new techniques of inspection are being developed based on point clouds with thermal attributes. Lin et al. (2019) developed an unsupervised windows extraction approach based on point clouds with thermal attributes. The segmentation of point clouds is conducted by the generation of multiscale supervoxels, a region growing and an energy optimization. Jarząbek-Rychard et al. (2020) highlight that existing methods exploit geometric features but ignore most of the time other attributes. They study the combination of thermal information with other available characteristics of data for a classification performed directly in 3D space.

DATASET
In this paper, we will use thermal point clouds of different building facades located in Strasbourg (France) acquired by a mobile system installed on a vehicle. The acquisitions were carried out by the University of Vigo (Spain) in July 2015 from the 14 th to the 16 th at different times of the day.

Acquisition system and generation of thermographic 3D point cloud
A full description of the acquisition system and the procedure for the generation of thermographic point clouds is available in Lagüela López et al. (2014). A Thermographic Mobile Mapping Unit was used for the joint acquisition of geometric 3D data and thermographic data of big-scale areas. This Mobile Mapping Unit consists on a scanning system from Optech (Lynx Mobile Mapper SG1) and a thermographic camera from Xenics (Gobi 384 Scientific, resolution of 384x288 pixels, field of view of 50° x 40°). The system is installed on a vehicle and is coupled with a GNSS (Global Navigation Satellite System) sensor for the positioning of the vehicle as well as an IMU (Inertial Measurement Unit) for defining its attitude.
The production of thermographic 3D point clouds involves the generation of 3D point clouds acquired by the LiDAR system and the projection of thermographic images on this point cloud thanks to the synchronization of GNSS, IMU and thermographic camera systems. The process of those data will of course rely on the quality of the data, not only regarding the radiometric (thermal) data, but also the geometric data provided by the laser scanner. To ensure the quality of the data, on the one hand the radiometric information and on the other hand the geometric information have been assesses.
Thermal data have been compared to measurements taken with a handheld thermal camera. And the geometry provided by point clouds acquired with the mobile mapping system were compared to point clouds registered through a static terrestrial laser scanning system. This comparison shows a relative precision of 5 cm in average for different datasets.

Dataset under study
For our study, the sample of a thermal point cloud presented Figure 1 is used. This thermal point cloud was acquired in the Goethe Street in July 2015 at 6 AM. In such an hour in the morning, there was no direct solar radiation on building facades and the temperature of the wall was quite homogeneous. In Figure 1, the colors correspond to different temperatures and the colormap is going from 10°C (blue) to 25.5°C (red). The density of the point cloud of this sample is about one point each 5 cm.
Several comments can be done about this figure. Firstly, one can notice that the windows have a color which is different from their neighboring objects since they emerge from the thermal point cloud. Secondly, points with a high temperature appear along the tiles between floors and form linear zones of the building envelop which present high thermal variation. This coincides with the definition of thermal bridges. On the basis of these observations we decided to study how to automate the extraction of information from thermal point clouds like windows and thermal bridges. The lower part of the facades as well as building roofs could not be scanned, due to the limited incidence angle of the mobile mapping system and its short distance to the façade. The approximate size of the sample is 10 m height (two and a half storeys) and 85 m long.

DEVELOPED APPROACH
The developed approach is mainly based on the automatic analysis of histograms namely on the extraction of peaks from histograms. Two main steps are considered to extract windows and thermal bridges: a global analysis and a local analysis. The analyses of thermal and geometric attributes have both their advantages and are combined to optimize the result. The point cloud used to develop the approach is presented Figure 1.

Pre-processing
Two pre-processing steps are carried out. First, in order to remove objects in front of the buildings, namely trees in our case, as well as roofs, a plane fitting algorithm is used. The points which are far from the determined plane corresponding to the facade are removed. Secondly, a Principal Component Analysis (PCA) is performed to the remaining point cloud to rotate the facade so that the Y coordinates correspond to the distance from the plane. This depth information will help us for the extraction of windows based on geometric attributes.

Histogram analysis
The developed approach considers the analysis of the histogram of the thermal attributes of points but also the analysis of the histogram of the depth of points. The aim is to identify peaks describing different elements. The histogram analysis involves the creation of the histogram and the associated curve and the extraction of points corresponding to each peak by an iterative process. The two steps are described by taking a histogram of thermal information as example.

Creation of the histogram and the associated curve:
A histogram is created based on the considered point cloud. It is by construction non-continuous. Thus, a kernel density estimation, also termed the Parzen-Rosenblatt method, is used to approximate the histogram by a curve as illustrated Figure 2. This method consists in the calculation of the probability density function of the sample.

Extraction of peaks:
In order to extract peaks, local minima and maxima of the curve are used. An iterative process is considered: first the highest maxima is selected. The two neighbouring minima are then searched. The extraction of the peak is carried out according to a search radius SR defined as follow: where T max is the temperature of the maximum and T mink the temperature of the minimum on the right (T minright ) or on the left (T minleft ), as shown in Figure 2a. This radius allows avoiding, during the extraction of one peak, the disappearance of close neighbouring peaks.
Once a peak and the corresponding radius are determined, the points whose temperature values are included in the search radius are extracted. A histogram is then created with the remaining peaks. Figure 2b shows the histogram Figure 2a from which the first peak has been extracted. All peaks are extracted in the same way.

Global analysis
A histogram analysis was performed for thermal and geometric attributes by considering all the points of the facade.

Analysis based on thermal attributes:
The histogram analysis was carried out for the temperature values of points.
The extracted peaks are classified as follows: the first peak, extracted corresponds to the facade wall since this is the major part of the façade i.e. the biggest population of the sample. Thus, one can store an interval of temperatures for the wall. Next, the peaks with an interval of temperatures higher than the temperature of the wall are classified as thermal bridges. Finally, in order to identify the peak or the peaks corresponding to windows, all peaks for which the mean position of points along Y coordinates is behind the wall are selected. Figure 3a shows the points of windows obtained with a global analysis of thermal attributes.

Analysis based on geometric attributes:
In order to extract the points belonging to windows, a global analysis is also performing based on the depth of points with regards to the facade plane. Generally, the main part of the windows is located behind the facade plane, thus all peaks for which the mean Y coordinates is behind the wall are considered as windows. Figure 3b presents the result of this analysis.  Figure 3b, some windows cannot be detected because there are no points behind the wall (windows hidden by curtains or windows in the same plane as the wall). The benefit of thermal attributes is that windows can be detected in such cases.
The result obtained with thermal attributes is good for two façades. For the others, the windows are only partially detected. Moreover, one can see the impact of the trees in front of the facades. Some parts are colder because of the trees making the extraction of windows more difficult (see red frames in Figure  3a). The thermal attributes are quite sensitive that's why a local analysis is also performed. The global analysis allows having approximations of the intervals of temperatures of the different elements.

Local analysis
Regarding the size of the facades, a locally use of the methodology seems to be more appropriate for analysing repartition of points against temperatures. Indeed, façade materials and types of windows are not the same for the entire facade. Thus, the point cloud is cut into sub point clouds. 30 sub point clouds where created of around 6 by 6 meters. Figure  4 presents the result of histogram analyses for two sub point clouds. One colour corresponds to one extracted peak. The same colour between the two examples doesn't mean that the points belong to the same kind of element. For each extracted peak the minimum, the maximum and the mean temperatures as well as the mean depth of the points are stored to identify the elements of interest in the further step.

Combination of thermal and depth information
As highlighted, the analyses of thermal and geometric attributes have both their advantages. Moreover, a local analysis of thermal attributes provides more reliable results than a global analysis. Thus, the last step of our approach consists in combining the different results obtained in order to take benefit from both sets of attributes.

Extraction of windows:
Each extracted peak of each sub point cloud is examined. Considering a sub point cloud, the point clouds of the extracted peaks are first compared to the result of the global analysis thanks to the depth of points. If there are points in common, the point clouds are merged. If not, the peaks corresponding to windows are determined by looking to the min, mean and max values of temperatures. The global analysis allowed having an idea about temperature intervals of elements.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)

RESULTS
The developed approach was applied to the point cloud presented Figure 1. The final result is presented in Figure 5 and in Figure 6. The extracted windows are represented in blue whereas the thermal bridges are represented in red. The result is promising since the main part of the elements of interest was extracted automatically. Some windows at the bottom of the building were difficult to extract or were not extracted at all. This is probably due to the fact that, because of the limited field of view of the camera, only a half of the windows is visible.
Moreover, there is a lack of point because of masks in front of the building. Regarding thermal bridges, hotter part of the facades was detected notably between storeys and below balconies.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)

CONCLUSION
In this paper, an approach was proposed to extract automatically windows and thermal bridges from point clouds with thermal attributes. This approach is mainly based on the analysis of histograms and the extraction of peaks of histograms. The analyses of thermal and geometric attributes have both their advantages and the combined use of both sets of attributes showed promising results.
In the future works, the developed approach will be applied to other datasets and assessed based on manually segmented samples. Moreover, other information will be also taken into account for the extraction of windows and doors, namely the laser intensity and the colour (RGB values) of the points.