COMBINING TIR IMAGES AND POINT CLOUDS FOR URBAN SCENES MODELLING

The simulation of the energy balance at the scale of a street become a huge issue. The accurate geometry of the area, detailed atmospheric measurements, and eco-physiological information are necessary to perform faithful simulations. In addition, thermal images showing surface temperatures are also of highly interest to analyse the energy balance between façades and trees. In this context, the aim of our project is to contribute to the development of a methodology for merging 3D geometry of an urban scene with surface temperatures. In this paper, a methodology is proposed for the texturing of a point cloud from a static laser scanner with IR images from a thermal camera. The main idea of the developed methodology is to take part of the ability of the thermal camera to acquire simultaneously RGB and IR images. After determining the relative position and orientation of IR and RGB sensors, the orientations of IR images are deduced from the orientations of RGB images. Then, the point cloud and RGB images are registered by selecting homologous points manually. Finally, the point cloud is textured with the IR images.


INTRODUCTION
It is well known that urban vegetation has beneficial effects on urban comfort and the health of inhabitants. It acts as a microclimate regulator by providing shade for pedestrians and facades, thereby reducing the surface temperature of buildings and the ground. The intensity of this phenomenon depends on the characteristics of the tree (species, density of leaves, size, transpiration, geometry of the crown, proximity to buildings, alignment trees or park trees). Street trees can have an effect of 2°C to 3°C on air cooling and more than 10°C on surface cooling (Gillner et al., 2015).
Even if the processes are known, simulating the energy balance at the scale of a street remains a major challenge, because it supposes to faithfully reproduce the geometry and the physical characteristics of the surface, as well as all the processes of energy exchange that take place between these elements.
Consequently, it is necessary to acquire accurate geometry of the area, detailed atmospheric measurements, and eco-physiological information on the behaviour of different trees. In addition, thermal images showing surface temperatures are also of highly interest to analyse the energy balance between façades and trees. In this context, the aim of our project is to contribute to the development of a methodology for merging 3D geometry of an urban scene with surface temperatures. Moreover, these 3D models must be adapted regarding the level of detail to the expectations of urban microclimatic models (Bournez et al., 2019).
In this paper, a methodology is proposed for the texturing of point cloud from a static laser scanner with IR images from a thermal camera. After a related work section, the area of interest and the thermal camera under study are presented. The developed * Corresponding author methodology is then explained, and first results are shown. Those results allow us to propose several future works for our project.

RELATED WORK
In the field of thermal infrared (TIR) for urban trees, the number of papers published in the last 2 decades increased slightly and focus mainly on remote sensing data for assessing urban heat islands (Del Pozo et al., 2020). But when adding the keyword "laserscanning", the number of papers doubled from 8380 in 2001 to 17700 in 2021. Indeed, at city scale, Mobile Mapping Systems (MMS) equipped with laser scanners are increasingly used to provide, in a very fast way, accurate and dense 3D data of a street and its elements, in particular building facades and individual trees. When equipped with thermal cameras, they are used for energy studies in urban environments (Lagüela et al., 2014).
Generally, at street scale, TIR data are acquired with handheld cameras, either from radiometers or from cameras. Compared to RGB cameras, the geometric resolution of infrared cameras is quite low as well as the number of identifiable features in thermal images. Therefore, creating an accurate 3D model exclusively from thermal images is challenging (Hoegner et al., 2018). That's why the information provided by a thermal camera is usually combined with data from other sensors, like laserscanning data.
TIR data and 3D geometry are acquired most of the time independently and are combined in a later on, through mesh or point cloud texturing. Only rarely, multi-sensors systems provide both data registered instantly. The main drawback of these systems is that the monitoring of thermal behaviour of an object over time requires to capture the same geometric information every time.
When acquired separately, as in our case, the drawback remains in the registration of both datasets. It may require the measurement of ground control points. Some authors propose to create a point cloud with thermal infrared data associated to each point. For instance, Zhu et al. (2021) performed the alignment using keypoint detectors in both datasets.
Another way to proceed is the mapping of TIR images on 3D building models (Previtali et al., 2013), however it becomes challenging in the case of complex geometries. Other studies take benefit from the orientation of RGB images to register TIR images on the point cloud (Macher et al., 2019).

Area of interest
The area of interest covers three streets lined with alignment trees of three different species (Figure 1). The streets are located in Strasbourg (France) in a typical urban environment. The main street is about 130 meters long. The trees of interest are equipped with several sensors to determine over time eco-physiological information related to them. Atmospheric measurements such as temperature and humidity are also performed.

3D acquisitions with laserscanning systems
In order to simulate the energy balance at the scale of a street, besides meteorological measurements and eco-physiological information, it is also necessary to perform an accurate acquisition of the 3D geometry of the streets, especially of trees and facades. Since a tree has a variable geometry, acquisition campaigns will be carried out and repeated for specific phenological stages during our project. Typically, an acquisition must be carried out before the leave budding and after the leave growth.
Several terrestrial laser scanner systems have been used to acquire the 3D geometry of the scene under study: a mobile laser system (GeoSLAM ZEB-REVO RT) and three static systems (FARO Focus 3D X330; Trimble X7; Z+F IMAGER 5016). The last measurement campaign was performed in March 2022 with the laser scanner Z+F IMAGER 5016. This scanner is able to perform registration in real time. However, for guarantying reliable registration between several stations, this scanner needs to rely on large overlapping areas. That is why, 15 scan stations were required to scan the area of interest. For the trees of interest, additional stations have been placed around the tree to be able to model as accurately as possible those trees. Figure 2 presents the positions of the laser scanner along the streets as well as a planar view of a scan position. One should note that for static laser scanning, whatever the laser scanner used, a similar point spacing is chosen that is one point each 6 mm at 10 m.
(a) (b) Figure 2. Acquisition of the streets with static laser scanner: (a) laser scanner positions; (b) planar view of a scan position.

Camera specifications
For the acquisition of TIR images, the thermal camera used is the FLIR T560 with sensor resolution of 640x480 pixels and a field of view of 42x32°. With the lens of 42°, the focal length indicated by the manufacturer is 10 mm and the detector pitch is 12 µm. The resolution of the IR images is quite low but pretty good considering an infrared sensor.
Beside IR sensor, the camera is also composed of an RGB camera and can capture both thermal and RGB images at the same time. The RGB images are used for a visual purpose only by enhancing the clarity of thermal images with the addition of visual details in real time. Thus, a very few information is given by the manufacturer about the RGB sensor (5MP, FOV 53°x41°).
The quality of RGB images is not sufficient to generate an accurate point cloud through multi-view processing. However, in our study, the RGB images will be used to deduce the thermal images' relative orientation.

Geometric and thermal calibrations
Several calibrations must be performed before taking benefit of the images for 3D measurements: the geometric calibration of the two sensors (IR and RGB sensors) and the thermal calibration of the IR sensor.

Geometric calibration of IR and RGB sensors:
For the geometric calibration of RGB cameras, usually a typical checkerboard with white and black squares is used. In order to perform the geometric calibration of the IR sensor, a checkerboard was created specifically (Figure 3a). The checkerboard is composed of squares of two materials with significantly different emissivity (aluminium and black stickers). After experimenting with several materials, aluminium proved its efficiency due to its low emissivity compared to the behaviour of black matt stickers. Therefore, a good contrast is observed on the checkerboard in TIR images (Figure 3b). The geometric calibration of both sensors consists in taking a set of photographs from several viewpoints around the dedicated checkerboard. The multiple images of the checkerboard allow determining the camera parameters.

Radiometric calibration of IR sensor:
For the radiometric calibration, two black bodies were used. First, the black body Mikron M345 was used to assess the homogeneity of the temperature measurement on the sensor. Secondly, the black body Landcal P80P was used to assess the temperature measurement namely the difference between the black body temperature and the temperature measured by the thermal camera. Temperatures of the black body between -2°C and +40°C were used to carried out this assessment. The black body can set a temperature with a precision of 0.1°C. Table 1  The temperature differences are pretty low. According to the manufacturer, the T560 thermal camera provides an accuracy of ±2°C and offers the temperature sensitivity needed to detect differences smaller than 0.03°C. One should note that for our project, the temperature variation is more important than the absolute temperature value because the aim is to analyse temperature variations over time, whether on façades or trees.

Relative position and orientation of IR and RGB sensors
In our study, we take benefit of the RGB sensor to orient images of the IR sensor. Indeed, the relative orientation of IR images alone is difficult due to the low resolution of images. However, if the relative position and orientation of IR and RGB sensors is known, the orientations of IR images can be deduced from the orientations of RGB images.
The Stereo Camera Calibrator tool provided by MATLAB (MathWorks) was used to determine the relative position and orientation of IR camera related to RGB camera (Figure 4). Regarding the results, the rotation values are low (between -0.0034 and +0.0024 radians) meaning that the two sensors have nearly the same orientation. There is nearly no translation in X and the translation in Z of 6 mm is probably due to the difference between the focal lengths of the sensors. Finally, a translation of about 2.3 cm is observed in Y. This is consistent with the distance that one can be measured directly by hand on the camera.

DEVELOPED METHODOLOGY
Several investigations were performed to merge 3D geometry namely point clouds with thermal images for facades and trees. For these first investigations, only a small part of the area of study was considered, i.e., a building façade of about 60 meters long and the trees in front of this façade.
The main idea of the developed methodology is to use the RGB images taken simultaneously with IR images: on the one hand to orient IR images, and on the other hand to merge the oriented IR images with a point cloud acquired by laserscanning.

Acquisition of IR and RGB images
The camera can capture both thermal and RGB images at the same time. It was configured to save the thermal image and the visual image as separate JPEG files.
For the acquisition of images, two configurations were tested. First, a traditional photogrammetric approach was considered. To acquire the façade and the trees in front of it, several photograph strips were carried out. Figure 5a shows the positions of the acquisition stations as well as the photographs taken, as blue frames.
Secondly, the camera was put on a pan head to performed panoramas (Figure 5b). In this configuration, fixed positions of the camera are considered. An IR and RGB image are taken each 10° in each position. The idea was to adopt the same behaviour as a static laser scanner to facilitate data merging. The camera was placed approximately at the same position as the laser scanner (Figure 5c).

Preprocessing of IR images
The colour mapping of IR images must be fixed. The thermal camera maps temperatures to colours, according to a scale determined automatically based on the visible temperature range. Thus, the colours of a same surface may differ between two IR images, which will cause discontinuity for texturing.
Moreover, to access to temperature measured by the thermal camera, the relation between temperatures and colour used to represent the temperatures must be known. Unfortunately, FLIR doesn't provide such information. That's why, a csv file with a temperature matrix is exported for each IR image thanks to the software Research Studio (FLIR). This allows us to generate IR images with our own transfer curve and to manage properly the temperature information.

Relative orientation of IR images
Considering the good resolution of IR images considering an infrared sensor, the orientation of IR images with photogrammetric techniques was first investigated. Despite the pretty good resolution, the orientation of IR images failed most of the time especially when there are not a lot of temperature variations. On this basis, the RGB images were used as a support for the orientation of IR images as explained in our previous work (Macher et al., 2019).
For the purpose of image orientation, the commercial software Metashape (Agisoft) was used. This software proposes a multicamera system approach. This approach works if the sensors are synchronized, and their relative exterior orientation doesn't change during the capturing process. This is the case with the thermal camera since IR and RGB images are taken simultaneously, and the two sensors are fixed.
The RGB images are defined as "master" in the software, that is, the relative orientation is performed on those images. The relative position and orientation of IR sensor related to RGB images determined previously has of course to be informed to deduce the relative orientation of IR images.
Besides the relative position and orientation of the cameras, the size of image pixel for both sensors can be indicated to help the orientation. This value is known for the IR sensor. However, the image pixel size is more difficult to access for the RGB sensor since the manufacturer give only a few information about this sensor. Further tests will be carried out to determine this value accurately.

Texturing of laserscanning point clouds
After image orientation, a point cloud from a static scanner of the considered area is imported in Metashape. For texturing properly the point cloud, the block of IR images has to be positioned regarding the point cloud. To do so, homologous points were selected manually in the point cloud and RGB images. Point selection is for now manual but this step is intended to be automated in future works. Once again, the RGB images can help in the process for keypoint selection.
Some tests were carried out for the texturing of mobile point clouds. These tests are not satisfying that's why static laser scanner data were considered instead. Indeed, in mobile point cloud it is difficult to select characteristic points because of the low density of mobile point clouds. The registration of IR images and the mobile point cloud is thus not very accurate causing texturing problems.
Once the IR images are registered with the point cloud, the 3D point cloud can be textured with the IR images. Since the relation between colour and temperature is perfectly known, a conversion tool was developed in MATLAB to convert the RGB information attributed to each point into a scalar field with the temperature. Thus, a "thermal point cloud" is obtained, namely a point cloud with a temperature information attributed to each point. The results are presented in the next section.

Image relative orientation and registration with the point cloud
When using the traditional approach, the relative orientation of images works fine. Two blocks were defined and registered to the point cloud with well distributed points.
Regarding the use of panoramas, i.e., when the camera has fixed positions, the orientation of the images for one panorama works well. In Metashape, the user can indicate to the software that the camera is fixed. In this specific case, the IR images can even be oriented without RGB images (Figure 6a). Figure 6b presents a panorama of IR images taken in the study area.
(a) (b) Figure 6. Orientation of IR images for a panorama: (a) orientation of images in a 3D view; (b) Resulting panorama.
However, the orientation of panoramas with each other is a more difficult task. Additional points were added manually by the user for the orientation of panoramas together.
The registration of panoramas with the point cloud is also not ideal. The panoramas were made close to a wall hiding the façade (following the laser scanner positions) and higher point of views are missing so that a well distribution of homologous points is not guarantee. Consequently, it leads to problems in the texturing step as shown in Figure 7. The texturing problem can be seen when looking to the windows of the façade. One should not that, in future works, the panoramas are intended to be used individually and linked directly to laser scanner positions. This solution will avoid orientation and registration problems. Figure 8 presents the result of the texturing of the static point cloud with IR images when using the traditional approach for the acquisition. A temperature is associated to each point in a scalar field. The result is promising, the textures seem well applied regarding the façades and more particularly the windows where temperature variations are observed. As illustrated in Figure 9, one can observed some texturing problems on the perpendicular façade. This is related to the texturing of the point cloud. Indeed, in this case, points belonging to two façades are in the field of view of the IR camera. Normally, the perpendicular façade can't be textured since there is no IR information about it. This may be solved by considering a mesh or by texturing each scan independently with a panorama taken in the same position. Figure 9. Texturing problem with a point cloud.

CONCLUSION AND FUTURE WORKS
In this paper, a methodology was proposed for the texturing of point cloud from a static laser scanner with IR images from a thermal camera. This methodology takes part of the ability of the thermal camera to acquire at the same time RGB and IR images. After determining the relative position and orientation of the IR sensor related to the RGB sensor, the relative orientation of IR images can be deduced from this of the RGB images. Then, the point cloud is registered with the images so that the point cloud can be textured with IR information. The result of the developed methodology is promising but can be improved.
In future works, the automation of the registration between the point cloud and the images will be studied. As static laser scanner is used to acquire the point cloud, images acquired by the laser scanner will be exploited to register IR images. The idea is to carry out a panorama with the thermal camera approximately at the same position as the scan stations. The registration with the 3D geometry may be performed with the IR panorama. But if it doesn't work, the RGB panorama based on images from the thermal camera can be also used as a support, as it is the case in this paper. In this context, an image matching process aiming the detection of keypoints in the images will be considered to automate the registration. The registration of RGB images in laser scanner panoramas is proposed in the literature (Bruno et al., 2022) but is not considered yet for IR images.
For the texturing step, the use of meshes will be also investigated. Indeed, one must integrate thermal information to climatologic models, and textured meshes may be more suitable than thermal point clouds. However, some questions arise on how to determine the temperature to assign to a triangle of a mesh.
Finally, the modelling and texturing of trees will be studied more specifically. Finding a way to merge 3D models of trees with TIR data is challenging, because of the complex and variable geometry of a tree, and the temporal variability of its temperature.