Fusion of thermal and visible point clouds : Application to the Vaches Noires landslide, Normandy, France

In this paper, we present a methodology to fusion 3D visible and thermal infrared (TIR) information on a coastal landslide area located in Normandy, France. A reﬂex and TIR camera on-board an Unmanned Aerial Vehicle are utilized to generate a 3D visible and a thermal model using Photogrammetry. A Python-written algorithm is then used to associate the thermal scalar on the TIR model to the closest point on the visible point cloud, before applying α -blending to ease the visualization of both data sets. This methodology leads to the generation of an integrated 3D thermo-visible model, allowing the direct analysis of the surface temperatures, visible data and geometric conﬁguration of the landslide.


INTRODUCTION
The monitoring of coastal areas is a major preoccupation for scientists and stakeholders in the context of climate change (Zhang et al., 2004). The morphological modification of the coastline involves various geological processes (rockfalls, debris flows, landslides, mudslides,...), depending on the nature of the lithology (rocky, clayey,...), the geometrical configuration of the coast, as well as marine and continental forcings (Cambers, 1976;Costa et al., 2019).
High performance and low-cost methods have been developed in a tremendous way since 10 years for coastal research, with the development and miniaturization of sensors implemented on-board Unmanned Aerial Vehicles (UAVs). Indeed, these aerial systems are particularly suited for mass movements studies over large areas (typically, several km 2 ), thanks to their flexibility, operability and complementarity with aircrafts and satellites (especially in cloudy/rainy weathers). In this context, the main scientific objectives are to perform punctual or repeated observations of the coastline for geological hazards mapping, geomorphological investigations or slope deformations quantification (Scaioni et al., 2014). Most of these studies are based on Digital Elevation Models (DEM), Digital Surface Models (DSM), 3D point clouds or orthophotos analysis, obtained using visible digital cameras or LiDAR (Torrero et al., 2015;Eker et al., 2018;Lindner et al., 2016). Eventually, near infrared or hyperspectral sensors may be implemented to determine vegetation indices (Berni et al., 2009). * Corresponding author Thermal Infrared imagery (TIR) allows to observe the surface temperatures of a scene. It was successfully used to characterize surface properties (Watson, 1975;Brenning et al., 2012), fractured areas (Frodella et al., 2017) or hydrological processes (Antoine et al., 2020), but its use remains very limited in coastal studies. This consideration arises from the facts that 1) thermal images may be often analyzed in conjunction with visible images acquired on the same area, 2) thermal processes are transient, depending on the diurnal or seasonal cycles and 3) the procedure to locate a thermal image on given area (geo-referencing, co-registration, ...) may be difficult to implement, due to the presence of complex topography and to the lowest resolution and dynamic range of TIR images (256 colours), compared to visible systems.
Many factors can affect the surface temperatures of the top centimeters of the soil, including albedo or emissivity, but thermal inertia is the key parameter controlling the diurnal variations of the thermal field (Mellon et al., 2000). It represents the capacity of the material to conduct and store solar radiations away from the surface during the day and emits this energy during the night. Thermal inertia depends on the nature of the soil, particle size and consolidation degree (Mellon et al., 2000). When complex topography is present (for instance, with crest and valleys), the surface temperatures may also be influenced by the geometry of the surfaces, due to their orientation regarding the incident solar radiation (Whiteman et al., 1989;Eppelbaum et al., 2014;Tadé, 2004). As a consequence, advanced analysis of TIR imagery on coastal areas relies on the joint interpretation of thermal, visible and geometric data. Very recently, Ma et al. (2020) integrated TIR imagery with Terrestrial Laser Scanning(TLS) and Particle Tracking Velocimetry (PTV) for characterization of a laboratory landslide model. In this paper, we propose a methodology to fusion 3D thermal and visible point clouds obtained using a TIR and a visible camera embedded onboard an UAV. The acquisition was realized on a coastal area called the Vaches Noires cliffs (Normandy, France), which is known for its permanent mass movements.
The objective of this work is to present an integrated 3D model of the Vaches Noires cliffs composed of visible data and enriched with a thermal scalar field, in order to ease the interpretation of the observed surface temperatures. After presenting the acquisition and processing procedure to obtain the point clouds, the methodology to fusion both models and to realize α-Blending is described. Finally, the potential use of such 3D model is discussed in the last section.

STUDY SITE
The Vaches Noires cliffs are located on the Normandy coast, close to the city of Villers-sur-Mer. It has a typical badland morphology with the alternance of valleys and crests. The Vaches Noires cliffs are composed of a multilayered geological formation of limestone, clay and marl. Two water tables are identified as factors of erosion in the upper part of the landslide. The interaction between water and geological layers leads to i) to intense fracturation at the upper part of the cliffs, ii) the occurrence of mudslides on the valleys and iii) to the accumulation of debris deposits at the base of the structure. Finally, oceanic tides and winter storms are responsible for the continuous erosion of the beach and of the disaggregation of the sediments present at the basal part of the structure.

ACQUISITION AND PROCESSING
The acquisition was performed in April 2016 around 1 p.m.. The weather was clear and no wind was observed. Visible and thermal data were obtained using a Nikon D7000 Reflex camera (focal : 28 mm, sensor size : 35.9 x 24 mm, 4.8 µm pixel size, Angle of View : 53 • ) and a FLIR A320 thermal camera (focal : 9.6 mm, sensor size : 8 x 6 mm, 25 µm pixel size, Field of view : 25 • x 18.8 • ), acquiring data in the range [7.5 µm -14 µm]. The UAV used for the survey is a Copter 1B helicopter ( c SURVEY Copter, figure 2).   The survey was completed by the positioning of 18 white squareshaped plastic targets, both noticeable on the photos and the thermal images and distributed over the landslide (figure 5). These targets were geo-referenced using a differential GPS with a precision of 1.5 cm in the X and Y direction and 2 cm in the Z direction.  The processing of each visible and thermal dataset has been achieved using the commercial c Agisoft Metashape photogrammetric software. It consists of the following classical workflow : • Manual geo-referencing of the targets on each visible and thermal images; • Retrieval of tie-points between each pair of image; • Calculation of the relative position of the cameras and of the sparse point cloud; • Densification of the sparse point cloud.
During the processing, for each model, a particular attention has been paid to the evolution of the precision in the X, Y and Z directions, which is associated 1) to the manual geo-referencing of each target, 2) to the location and the number of used targets (the precision was checked for example by suppressing or adding some targets), 3) to the optimization of the camera optical parameters and 4) to the calculation of the point clouds (reprojection error). Finally, this processing leads to the generation of a visible model containing 20 millions points (figure 6) and a so-called "thermogrammetric" dense point cloud with 300000 points, i.e. a number of points 60 times lower than the visible model ( figure 6). The total error calculated by Metashape is 1.6 cm, 1.6 cm and 3 cm in the X, Y and Z direction, respectively for the visible model. For the thermal model, the obtained values are 5.8 cm, 4.6 cm, and 12.9 cm in the X, Y and Z directions, respectively.  2019)). It shows that both models are relatively close to each other in distance (peak of the distribution centered on -0.06 m, mean for the Gaussian fit : 0.15 m, standard deviation : 1 m) and consequently that thermal images seem to well-fitted the complex topography of the Vaches Noires cliffs. However, some discrepancies appear on the lowest parts of the cliff (blue colour) and on the right side of the figure (red colour). This result may be explained by the presence of dense vegetation, lower dynamic range of the thermal images and lower spatial resolution compared to classic visible images. Moreover, vegetation contributes to roughness which is one of the least well-constrained parameter in point cloud building (Lague et al., 2013). Such effect could have been removed with help of classification tools (Brodu, Lague).

FUSION OF POINT CLOUDS
The visible model is taken as a reference because of its density. A scalar temperature is assigned to each point of this reference model as following.
3D models are previously structured into a kd-tree form, which allows for a fast browse within the densified point cloud (20 million of points). Once the data is structured, for each point of the visible cloud, a search based on an euclidean distance criteria of the nearest neighbour in the thermal cloud is performed. This search is greatly facilitated by the kd-tree structure, so this operation only takes minutes for the entire reference point cloud. Since the density ratio of point clouds is 1 out of 60, the scalar field associated with the visible cloud has a histogram profile, i.e. on average about sixty neighbour points have the same thermal value. Figure 8 is obtained using the point clouds displayed on figure 6. The algorithm implemented for the merging of the .ply formatted data is Python-written and open-source. A simple user interface working on Windows has been developed for a more friendly and distributable use of the code. This interface is freely available for testing on other data sets on request.

α-BLENDING
The geo-localized visible cloud is enriched by a 3D temperature information. We present here a first exploitation of this cloud based on the building of a visualization under c CloudCompare in order to highlight phenomena that are not observable only in the visible spectrum.
In computer graphics, α-blending is a classical method on pictures, that consists in merging a image with a background by using a convex combination ruled by a parameter α. The result is called a composite and the two images can be seen as the combination of two layers.
Here, we thus perform an α-blending of the clouds, so the same method apply in 3D. The parameters α of the combination allows us to set thresholds for the user in order to weight the color scale of the thermal scalar field and the visible cloud in order to highlight phenomena, like for example the hydraulic behaviour of the massif. Figure 9 shows an example of α-blending of a visible and a thermal clouds. Figure 9. α-blending with infrared and visible clouds.

POTENTIAL USES OF AN INTEGRATED 3D THERMO-VISIBLE MODEL
The methodology presented in the previous sections are based on classical data (visible, TIR imagery) and processing techniques (photogrammetry, α-blending). Nevertheless, the combined utilization of these tools strengthens our interpretations of surface temperatures, in particular in areas exhibiting complex topographies (coastal areas, volcanoes, geothermal areas, desert dunes, glaciers but also civil engineering structures). Most of the TIR studies in Earth Sciences are based on single images analyses (or simple mosaicking), making difficult the use of high resolution TIR methods over large areas. In fact, the difficulties associated 1) to the difference of perception between thermal and visible images and 2) to the difference of resolution between both sensors represent important constraints for advanced processing and interpretation of the thermal signal. Recently, some studies geo-referenced and orthorectified thermal images on a photogrammetric model made with photos or on a LiDAR model to obtain 3D temperature maps (Lewis et al., 2015;Harvey et al., 2016). Besides, although these works are primordial to understand the link between thermal fields and topography, the orthorectification of separated thermal images on 3D models represents an important processing effort. Finally, the cost of a LiDAR may also be prohibitive and do not easily permit the comparison with visible data. This study proposes a workflow which allows : • to enhance the resolution of the thermal model, by addition of a high resolution visible model; • to ease the production of temperature maps over large areas and their comparison with visible data, taking into account topography; • to facilitate the direct quantitative analysis of surface temperatures and the location of potential thermal anomalies on an integrated 3D model; • to highlight the influence of the surface geology (nature of rocks, fracturation degree, hydrogeology, ...) and topography on the thermal field. Figure 10 displays the interpretation of the α-blended model presented in section 6. Surface temperatures can directly be analyzed in conjunction with the visible data and the geometric configuration of the Vaches Noires cliffs. It definitely permits the characterization of the landslide in terms of thermal inertia (and thus lithological nature : induration degree, particle size, ...), topography, vegetation and hydrogeology. Such information may be useful to complete in situ geological studies in the inaccessible parts of the landslide. First, the influence of the slope orientations is clear on the 3D model, with the observation of heated and shadowed faces on the crests. Second, the essential part of the vegetation appears cooler than the soil, due to the evapo-transpiration effect (Whitehead et al., 1986;Sabins, 2007). Third, the base of the landslide is composed of debris deposits coming from the erosion of the upper parts of the cliffs. At mid-day, these unconsolidated deposits are revealed by relatively high temperatures, due to their low thermal inertia (Watson, 1975;Mellon et al., 2000). Conversely, compacted sedimentary layers appear cooler than debris deposits due to higher thermal inertia. Finally, water resurgences are also observable, being warmer than the sand, at the lowest parts of the landslide and confirmed by in situ observations. Although preliminary, this analysis shows that the knowledge of the landslide may be enriched by the 3D thermo-visible model, in terms of location of unconsolidated areas and hydrogeological systems. Finally, once well-calibrated on the Vaches Noires cliffs, such methodology may be extended to other parts of the Normandy coast, in order to extrapolate the results obtained on the landslide.
Several limitations can be pointed out concerning our methodology. In this study, the thermal camera was used with factorycalibrated temperatures. Thus, the absence of field calibration prevents us from obtaining realistic absolute field temperatures (although thermal contrasts can still be analyzed). A solution may be to calibrate the thermal camera just before the survey using a material with a known emissity. Despite topographic discrepancies between thermal and visible data, the joint model is clearly interpretable from a thermal point of view. However, these differences may be reduced using a thermal camera with a bigger sensor and with a better dynamic range. Such joint 3D model implies to have both visible and thermal cameras on-board a UAV. This configuration may be rapidly limiting in term of payload and thus flight times, especially over large areas. Thermal surveys can be disturbed by bad atmospheric conditions, for instance strong wind causing uncontrolled UAV movements or high humidity content blurring thermal images. Consequently, clear-sky observations in calm atmospheric conditions must be considered for better acquisitions. Eventually, the flight can be realized just before sunrise to take advantage of the lowest solar radiation residuals. Finally, the processing and analysis of large 3D multi-spectral models remain a very challenging task. Future works will have to consider this fact, in particular by optimizing the resolution of the acquired images.

CONCLUSION
Photogrammetry was used in the visible and infrared spectrum to produce a joint 3D model of the Vaches Noires cliffs. As this 3D model is the result of the merging of two types of acquisition, a common representation of the data was produced with the objective to have all the information on a single 3D model. The proposed visualization makes it possible to weight the information jointly in order to highlight the physical phenomena at work on the studied site. The different tools used in this paper remain classical. Besides, the methodology consisting of the use of thermogrammetric data and of the combination of thermal/visible data is not conventional in Earth Sciences. In this context, the use of a UAV allows to go towards 1) more flexible acquisitions, adapted to the different sensors constraints and 2) advanced interpretations by assessing the correlations between topology, thermal behaviour and surface properties of the environments. Finally, other datasets such as field and UAV-based albedo measurements may be very useful to complete thermo-visible data and enrich our understanding of thermal processes within soils.