THERMAL AND OPTICAL DATA FUSION SUPPORTING BUILT HERITAGE ANALYSES

The recent developments of passive sensors techniques, that have been able to take advantage of the technological innovations related to sensors technical features, sensor calibration, the use of UAV systems (Unmanned Aerial Vehicle), the integration of image matching techniques and SfM (Structure from Motion) algorithms, enable to exploit both thermal and optical data in multidisciplinary projects. This synergy boost the application of Infrared Thermography (IRT) to new application domains, since the capability to provide thematic information of the analysed objects benefits from the typical advantages of data georeferencing and metric accuracy, being able to compare results investigating different phenomena. This paper presents a research activity in terrestrial and aerial (UAV) applications, aimed at generating photogrammetric products with certified and controlled geometric and thematic accuracy even when the acquisitions of thermal data were not initially designed for the photogrammetric process. The basic principle investigated and pursued is the processing of a photogrammetric block of images, including thermal IR and optical imagery, using the same reference system, which allows the use of co-registration algorithms. Such approach enabled the generation of radiance maps, orthoimagery and 3D models embedding the thermal information of the investigated surfaces, also known as texture mapping; these geospatial dataset are particularly useful in the context of the built Heritage documentation, characterised by complex analyses challenges that a perfect fit for investigations based on interdisciplinary approaches. * Corresponding author


INTRODUCTION
Thermal remote sensing, i.e. the acquisition, processing and interpretation of the data acquired primarily in the thermal infrared region of the electromagnetic spectrum, is being exploited since decades in several application domains at different levels of detail. A thermographic image correspond to a raster map that represents the variations in radiation intensity; a deep connection between infrared radiation and temperature exists according to the rule of black body radiation, which state that all objects emit infrared radiation based on their temperatures (Kuenzer & Dech 2013). As far as close-range thermal remote sensing is concerned, the typical application is the non-destructive diagnostics of materials and surfaces at very large map scales, focusing mainly on the thematic information rather than on the geometric accuracy, but improving so much the capabilities to detect alterations or damages on historical buildings and Cultural Heritage (CH) assets (Grinzato et al. 2002). In fact, as will be reported in the next paragraphs, the use of IRT has recently supported the evolution of close range survey techniques together with digital photogrammetry and terrestrial laser scanner (TLS), combining range and image based data as well that thermal for the complete documentation of the CH, including archaeological sites (Cabrelles et al. 2009). At smaller map scale, satellite thermal sensors are generally exploited in land cover mapping (Sun & Schulz 2015), agriculture analyses, water quality studies, urban thermal field (Small 2006). Since a few years, UAV platforms have been equipped also with thermal sensors, closing the gap between satellite and closerange sensors in terms of level of detail of the thermal analysis from aerial platforms. The paper is focused on the integration of optical and thermal data (by means of a rigorous photogrammetric approach), with the goal to enrich typical mapping products (e.g. orthoimagery and 3D models) with thermal information, allowing the extraction of added-value information supporting built heritage analyses. Two specific case studies will be presented and critically discussed, investigating both terrestrial and UAVbased solutions. Specifically, through the integration of the information obtained using a multi-sensor solution, a methodological approach has been tested combining geometric and radiometric information in a single workflow, aimed at investigating the potential of thermography analysis both as a decision tool and as a synergistic element for the analysis and documentation of the territory and the built environment.

Photogrammetric applications using thermal data
As reported in the previous section, IRT is successfully applied in several fields and professional sectors, and in recent years many researchers have been investigating the possibility to enrich 3D spatial data with thermal information, exploiting innovations in the processing algorithms, sensors enriched 2D/3D products and metric information generation. Although the thematic interpretation of the thermal data can be achieved through single images, attempts have been made for a long time to obtain mosaics of images in the form of orthoimages, to analyse extended portions of the objects or surfaces. Contemporarily, the obvious purpose was also to extract geometric measurements from orthoimages, or perform texture mapping, i.e. reprojecting oriented thermal data on 3D models derived from photogrammetric techniques or laser scanning. In case the surface is sufficiently flat, and a 3D solution is not needed, the homographic transformation can be applied. The valid alternative investigated even before the automation of the image matching and bundle block orientation process was to exploit the collinearity equations. Such approach was based on image orientation using tie points manually identified and improving the solution of the orientation of the block of thermograms by integrating visible images in the block adjustment . Nowadays, the potentialities connected to the combined use of visible and thermal data in the framework of photogrammetric processing using semi-automatic techniques for image orientation, are demonstrated by the fact that many SfM photogrammetric software developers (both commercial and open-source) have implemented new algorithms and specific templates in order to allow managing and processing thermal images. (Javadnejad et al 2019;Gonzalez-Aguilera et al. 2018) As reported in the previous section, a significant boost to the development of new strategies connected to the use of thermograms for photogrammetric purposes is connected to the exponential growth of new low-cost commercial solutions in the UAV market, as well as new performing thermal cameras which can be installed on aerial platforms. Thanks to the constant evolution of UAV systems and thermal sensors, in the last few years a relevant growth has been observed in research experiences involving the photogrammetric use of thermal images in many application domains. Among others it is worth mentioning archaeological remote sensing (Hill et al. 2020), building inspection (Dlesk and Vach 2019), volcano monitoring (Wakeford et al. 2019) and, of course, Cultural Heritage documentation (Güleç Korumaz 2019).

OPEN ISSUES AND METHOD
Despite the promising results, there are still several issues regarding the possibility to process thermal images using a SfM approach. One of the main experienced issue concerning the use of these kinds of data for photogrammetric purposes is connected to the difficulty to unambiguously identify traditional photogrammetric markers (which are often made of plastic or paper) in thermal images, due to their high emissivity (paper emissivity ≈ 0.85-0.99; plastic emissivity ≈ 0.95-0.99) that negatively impact on their detectability given the similar thermal behaviour to other common materials. GCPs (Ground Control Points) measured with traditional topographic techniques (GNNS methods or total stations) are essential in order to ensure a proper georeferencing and allow accuracy check on final models to be performed. When the surveyed object is rich of detectable features, this necessity is generally solved by measuring natural points clearly detectable in thermal images, contrarily employing markers made of materials characterised by a low emissivity which makes them appear colder compared to the surrounding ground (for example markers made of aluminium) (Hill et al. 2020). Another strategy is the use of geo-tag information acquired by UAV embedded GNSS receivers during the acquisition phase (Wakeford et al. 2019). However, the latter method is generally less accurate and, when possible, it is preferable to provide measured GCPs (unless RTK-enabled UAV solutions are adopted). In the case studies described in this paper, involving historical built heritage, the problem is lowered and the use of natural GCP has been successful. Another observed issue is connected to the relatively low spatial resolution of thermal sensors, which often corresponds to a lower level of detail of the acquired imagery. In addition, TIR (Thermal Infrared) images can be blurred and smoothed out (Javadnejad et al. 2019) and in some cases processing them using a SfM approach may be challenging. Due to the aforementioned issues, the number of extracted tie points can be relatively low (Jarząbek-Rychard et al. 2019), causing topological errors and the unproper reconstruction of the 3D model. This is certainly the most relevant problem, which led the research towards a synergistic workflow, by solving the photogrammetric orientation through bundle block adjustment using the datasets of the visible and thermal data separately, and subsequently use the thermal images oriented to perform the texture mapping on the geometric model derived from visible images. this strategy is made possible by using the same 3D reference system, and therefore ensuring a suitable coregistration of the two photogrammetric projects. Another well-known issue concern the format of thermal images: the thermal images analysis is often based on processed images (rather than raw dataset) in which the recorded temperatures are visualized as range colour. To preserve the original emissivity values, and since most IR sensor are sensitive to large ranges of temperatures, a thermal image can be expressed by a raster with rational values for each pixel, i.e. encoded in a 16-bit or 32-bit data format. ).
In the proposed case studies, the conversion to temperature values has been carried out manually for the terrestrial dataset ( Figure 4), while it was performed automatically by the photogrammetric Pix4Dmapper software for the UAV dataset

Case studies
The first case study concerns a coffered wooden ceiling in a XV century church (Figure 1), where an interdisciplinary research team of the Politecnico di Torino performed a multi-sensor 3D survey and diagnostic investigations for the study of the seismic vulnerability of the building in the framework of analyses carried out after the 2016-17 earthquake in central Italy (Grazzini et al. 2019). The IRT investigation performed by the Diagnostic Lab of the Architecture and Design dept. highlighted the traces of the recent intervention of reinforcement of the attic through the insertion of a slab with beams and brick planks, introduced after the 1997 earthquake. The aim of this case study is to clarify if it is possible to derive an orthophoto using thermal images not acquired with a photogrammetric strategy, thus without a proper overlapping. The second case study is related to an accurate photogrammetric UAV survey with a multi-rotor UAV platform also equipped with a thermal camera. The study area is an abandoned village in an alpine valley (Bout du Col, Prali -TO, altitude 1700 m a.m.s.l.)), the subject of a now traditional didactic experience in the teaching area of Architecture at Politecnico d Torino which studies the possibilities of regeneration of small alpine settlements (Cuneo et al. 2019). Since the beginning, the experience was aimed at testing the fusion possibilities of thermal and visible data in 2D and 3D photogrammetric products. Figure 2 shows village. A subsequent study as well as the analyses based on georeferenced thermal data presented in this paper confirm a the presence of areas with a denser building density (Cortese 2019).

Data acquisition and processing
A close-range photogrammetric survey has been carried out in order to collect digital images of the coffered wooden ceiling and perform a 3D reconstruction. The DSLR (Digital Single-Lens Reflex) camera used during the survey is a Canon EOS 5DSR equipped with a Zeiss 25 mm lens. The main specifications of the employed camera are shown in Table 1. In parallel, several thermal images have been acquired for nondestructive diagnostics, though not for photogrammetric purposes, using a FLIR SC660 thermal camera (main specifications in Table 2).

Model
Canon As far as the true colour dataset is concerned, the geometry of the acquisition has been planned in order to pay particular attention on the overlapping between images, following the well-known photogrammetric acquisition principles. Three photogrammetric stripes have been performed with nadiral camera configuration in order to cover the entire surface of the wooden ceiling of the church. Since the indoor environment was poorly lighted, long exposure times were required (30 seconds) in order to achieve adequate results in terms of image quality. For this reason, the use of a photographic tripod would have been mandatory, in order to minimize vibrations and, consequently, avoid as much as possible noise in the collected images; nevertheless, to increase the acquisition distance and to increase the coverage of each image, the camera has been placed directly on the floor. The thermal images of the coffered ceiling have been acquired for diagnostic purposes without following any photogrammetric criteria, confirming that photogrammetric principles are not yet widespread in other communities (Waldhäusl & Ogleby 1994) So, for this reason, the overlapping between most of the images is low and not adequate to perform a 3D reconstruction of the surveyed surface. Nevertheless, the overlapping of three of the collected thermal images was acceptable for image relative orientation and sparse cloud matching. In Figure 3 it is possible to observe the estimated overlap scheme of both data acquisitions. Both true colour images and thermal images have been processed using the photogrammetric SfM-based software Agisoft Metashape. Before proceeding with thermal images processing, it was necessary to export the acquired thermograms from R-JPEG (Radiometric-JPEG) format to TIFF (32-bit floating point) format using the software FLIR ResearchIR. Therefore, it was possible to process the thermal images in Metashape software considering the information about temperature data embedded in each pixel of the converted image ( Figure 4). Figure 4. Example of exported thermal image with its histogram displayed in QGIS platform (temperature range from 19 °C to 24 °C).
As previously reported, due to bad lighting conditions inside the church, it was impossible to acquire direct measures of natural points on the surface of the coffered wooden ceiling using a total station. To provide a reference data for georeferencing and accuracy assessment of the models, a set of points to be used as GCPs and CPs (Control Points) has been extracted from a georeferenced LiDAR point cloud, previously acquired with a laser scanner (Faro Focus 3D X 330, accuracy ±2 mm).
Concerning the true colour dataset, it was relatively easy to identify 10 natural points that could be used for accuracy check purposes (6 as GCPs and 4 as CPs). Instead, for the termal dataset, the detection process of an adequate set of points has been more challenging due to the already mentioned difficulties to univocally identify features in the thermal images by visual interpretation. Nevertheless, it was possible to identify four points situated at the corners of the wooden fillets of the coffered ceiling on both LiDAR point cloud and thermograms. Therefore, these points have been used as GCPs during data processing. As previously reported, the thermograms covered a relatively limited portion of the ceiling (since they were not acquired for photogrammetric purposes) and considered the difficulty to identify unambiguous points in that area, the four extracted points have been only used as GCPs. RMSE after the bundle block adjustment is reported in the Due to the low number of thermograms used during the photogrammetric process and to their low resolution (640 x 480 pixels), the tie points extracted from three images were only 132. Therefore, the dense point cloud generation has been performed on both datasets (Table 4). The dense cloud from optical images is constituted by ca 10 millions of points with an estimated GSD (Ground Sample Distance) of 1 mm/pixel; the point cloud obtained from thermograms is composed by ca 125,000 points with an estimated GSD of 7.4 mm/pixel.  Table 4. Main details of the photogrammetric process.

Results
True colour dataset allowed to generate a much denser point cloud and to achieve high detailed 3D mesh of the reconstructed surface, as it is possible to observe in Figure 5a. Instead, the point cloud generated from thermal images is characterised by a very low density and high noise (Figure 5b). Furthermore, the recess resulting by the presence of a chandelier generated an occlusion and consequently a hole on the point cloud. The mesh generated from the thermal point cloud is extremely noisy and characterised by several topologic errors. Due to its low geometric resolution, the 3D model is not particularly suitable for the proper generation of a thermal orthomosaic. However, a high-resolution mesh (ca 2,000,000 of faces) (Figure 6a) has been triangulated from the optical dataset. Thanks to the integration between the thermal data and the true colour 3D model (which have been processed in the same reference system and in the same photogrammetric project), it was possible to process both optical and thermal orthoimagery, following a data merging strategy. In the second case the thermal images, oriented in the same coordinate system of the 3D model, have been projected on the high detailed 3D mesh obtained from the true colour dataset. In this way it was possible to integrate the thermal information with higher spatial resolution data derived from optical images (Fig. 7).

Acquisition and processing
A UAV mapping campaign with multispectral sensors has been carried out in a high mountain context, with the aim of documenting the historical-architectural heritage of the Alpine village of Bout du Col, Prali (TO), which is currently in a state of neglect. A 3D metric survey has been performed using the UAV platform DJI M210 V2 quadcopter equipped with a thermal imaging camera (DJI Zenmuse XT2, Figure 8a). This sensor is based on an uncooled FLIR Tau 2 microbolometer thermal sensor (the main specifications of the camera are reported in Table 5). The camera is stabilized by a three-axis gimbal joint that allows it to rotate horizontally 320° and tilt from +30° to -90°. An optical camera (Zenmuse X4S, Figure  8b) was also used to acquire true colour data (main specifications in Table 6).  Before the UAV acquisitions, a topographical network has been measured using integrated GNSS and total station techniques. A set of points has been measured in order to provide a topographic reference during the photogrammetric process. The UAV flights have been planned on the basis of satellite images and DTM (Digital Terrain Model) covering the area of interest. The height difference between the lowest and the highest areas of the hamlet and the presence of high vegetation required a minimum flight altitude of 60 m. The UAV surveys have been carried out in two separate times: the thermal acquisition with the Zenmuse XT2 camera has been carried out at twilight in order to acquire a thermal response of the ground as truthful as possible, avoiding the reflections by direct solar radiation and the discontinuity caused by shadows. (Kuenzer & Dech 2013). Six photogrammetric strips have been performed with nadiral camera configuration acquiring an estimated area of 0.012 km2. During this flight, 96 thermograms have been collected (Figure 9) and measured GCPs (blue crosses) -Zenmuse XT2 thermal acquisition.
The following day a second acquisition has been carried out using the Zenmuse X4T optical camera. In this case the survey has been performed in the late morning in order to take advantage of the best lighting conditions which allowed to use faster shutter speeds and lower ISO avoiding as much as possible noise in the images. Two flights have been carried out ( Figure 10) planning a large overlapping between acquired images: a single grid flight with nadiral camera configuration and a double grid flight with oblique (45°) camera configuration in order to cover both the upper areas of the hamlet (the rooftops) and the facades of the buildings. During these two flights, an estimated area of 0.171 km 2 has been acquired and 235 frames have been collected. (Figure 10) Figure 10. Flight plan (green lines), camera positions (red dots) and measured GCPs (blue crosses) -Zenmuse X4S acquisition.
The acquired data have been processed using the Pix4Dmapper software, since it implements some specific templates for thermal UAV photogrammetry and specifically supports the thermal sensor used during the survey (FLIR Tau 2). The parameters of the thermograms acquired by the drone in R-JPG format at the maximum available resolution (640 x 512 pixels) have been modified using the Flir Tools + software. A distance parameter of 60 meters has been set and a generic emissivity value of 0.92 has been estimated, considering that the buildings under investigation are mainly composed of different types of stone materials. The thermograms have been imported in the Pix4Dmapper platform and processed using the "Camera Thermo Map" template. In a first step this template allows a preliminary rough orientation of the images using geo-tag information embedded in Exif data. During the second step 10 GCPs have been identified on the thermograms and used in order to perform a second orientation of the images, refining the accuracy of the tie point extraction and the bundle block adjustment. This procedure created a "warning" message in the accuracy report, regarding the high values of RMSE between the first and the second alignment (with differences ranging to a few tens of centimetres from a few meters). That underlines the poor accuracy of the GPS module installed on board of the quadcopter, especially on the Zcoordinate axis, and the necessity of using GCP for accurate georeferencing as expected.
According to the operational workflow, 96 thermograms have been correctly oriented with overlapping higher than 5 thermograms for each point with an estimated GSD of 5.9 cm/pixel (Table 8) and an estimated RMSE on the 10 GCPs of 0.031 m (Table 10). 165,327 tie points have been extracted allowing to generate a dense point cloud constituted by ca 1 million of points (Table  8). Also, a 3D mesh has been generated, even if characterised by a low resolution and a low level of detail which caused some topological errors in 3D reconstruction ( Figure 11).

N. Images
Tie points Dense cloud Estimated GSD [cm/pixel] 96 165,327 1,015,029 5.9 Table 8. Main details of the photogrammetric process (thermal dataset). The same photogrammetric workflow has been carried out also for the optical dataset acquired by the X4S camera: also in this case a first image alignment based on images geo-tag has been performed, followed by a second one (more accurate) involving the use of 10 GCPs. In the accuracy report are observable RMS positioning errors comparable to those obtained in the thermal dataset analysis. Texturized mesh (thermal data, IRON palette).
As reported in Table 9, 235 images have been processed and 2,419,592 tie points have been extracted at the end of bundle block adjustment with an RMSE on the 10 GCPs of 0.028 m. A dense point cloud constituted by more than 28 millions of points has been generated with an estimated GSD of 2.18 cm/pixel (Table 10). From the point cloud a 3D mesh has been triangulated and, as expected, in comparison to the thermal model it is characterised by higher resolution and geometric definition. (Figure 12

Results
In addition to the lower resolution of the mesh triangulated from the thermal dataset, it is necessary to take into consideration the topological errors affecting the reconstructed 3D surface. Such errors could cause reprojection miscalculations during the generation of the thermal orthomosaic or the 3D model texturization. For this reason, using an approach similar to the one observed for the terrestrial case study, a data fusion strategy between thermal data and optical data has been proposed combining the data acquired by different sensors. As previously observed, this procedure uses the geometric data of the mesh obtained from the visible dataset as reference geometric surface to be used for the reprojection of the radiometric data acquired by the thermal camera. This approach enables the generation of a topologically correct thermal orthoimagery as well as the texturization of the 3D model displaying temperature values. From the comparison between the two 3D models (respectively derived by only thermal and both visible-thermal datasets) shown in Figure 13 it is possible to underline that the higher resolution of the second model contributes to successfully solve projection miscalculations caused by topological errors of the thermal-only mesh.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 13. (a) Texturized thermal mesh (the thermal dataset has been used for both mesh generation and texturization). (b) Texturized thermal mesh resulting from data fusion between optical dataset and thermal dataset.
Starting from the historical cadastral map by Antonio Rabbini (Rabbini, 1092) and a morphological analysis of the area, on the basis of the shape of the hamlet and the remaining buildings, it has been suggested that Bout du Col was characterised by a greater presence of structures in the past, that nowadays have been lost. Therefore, the added value products (thermal orthoimage and 3D model) have been deeply investigated in order to find any evidence of anthropized traces and remainings of buildings. From the study of the processed thermal data, seven areas with possible building evidences have been finally identified. ( Figure  14). Figure 14. Seven areas (red dashed polygons) were deemed worthy of more detailed investigations according to their thermal discontinuity as well as in relation to their shape and orientation, fitting an anthropized pattern.
In order to investigate more deeply the aspects that have emerged during the previous analyses, the thermal orthomosaic has been compared to a georeferenced excerpt from the Rabbini Cadastre map, the historical cadastre commissioned by Camillo Benso Count of Cavour and realised by Antonio Rabbini (at the time Cadastre director of Ministry of Finance) between 1855 and 1870. Comparing the historical map and the thermal orthoimagery ( Figure 15), some consistencies between the detected areas and the historical buildings documented by Rabbini Cadastre can be observed. Certainly, further investigations would be necessary, but these preliminary results are extremely interesting and underline how these non-invasive technologies could help historians and researchers operating in Heritage fieldwork to investigate our past.

CONCLUSIONS AND FUTURE PERSPECTIVES
Starting from the information obtained using multi-sensor surveys, a methodological approach based on a photogrammetric workflow has been tested and validated with the goal to merge the 3D geometric information derived from visible imagery with temperature information from IR images. The approach allows added-value information supporting built heritage analyses to be extracted. This has been possible thanks to the possibilities offered by new tools to manage in a combined way the co-registration of data from different sources, based on images that bring with them different types of information, including 3D data and temperatures of the inspected materials and surfaces. There are many potentialities connected to these co-registration processes and, in addition, an aspect that should be underlined is related to the use of detectable natural points as GCPs. In the framework of the tested data fusion strategy, as previously mentioned, georeferencing is fundamental in order to perform a proper co-registration between the different source data. Since several researchers operating in Geomatics fieldwork are studying new methods and developing new rapid mapping strategies (e.g. related to UAV and Mobile Mapping Systems), the use of natural points instead of markers can be seen, in a time saving perspective, as an opportunity, allowing to save the time required from the preparation of specific low emissivity makers and from their positioning on the surfaces of the surveyed object. In addition to the rapidity of this method, the advantage of using non-invasive diagnostic technologies must be considered, especially during operations involving assets intrinsically vulnerable as the ones belonging to our built heritage. As confirmed by the research experience presented in this paper, the possibility to study fragile objects or areas with limited accessibility without the need of a direct contact is a very valuable advantage. The proposed data fusion strategy enable the generation of both orthoimagery and 3D textured models, that may represent a very effective tool to carry out noninvasive investigations on intrinsically vulnerable assets belonging to CH. In particular, as demonstrated by the presented case studies, these innovative techniques successfully support synergistic efforts to carry out interdisciplinary and multidisciplinary studies on the built heritage.