3D AND HYPERSPECTRAL DATA INTEGRATION FOR ASSESSING MATERIAL DEGRADATION IN MEDIEVAL MASONRY HERITAGE BUILDINGS

Cultural Heritage (CH) is a domain which has been greatly affected by climate change in the past decades. At the same time Information and Communication Technologies (ICT) have been greatly exploited to contribute to the holistic documentation, to support conservation and preservation actions. In order to move further on from the interdisciplinary approach to the holistic approach on Cultural Heritage the fusion of data from various sensors is the next goal. This paper focuses on the exploitation and integration of close-range 3D and Hyperspectral data from four Cultural Heritage buildings of Rhodes in order to assess material degradation. The methodology and data processing for this integration are presented as well as the useful and promising results of this approach which lead to further analysis and future work. The research is conducted within the framework of an EU funded project. * Corresponding author


INTRODUCTION
Recent studies highlight the impact of climate change and geohazards on historic areas, Cultural Heritage sites and monuments. Towards this direction, European Union has decided to fund a relevant 4-year research project (contr. No 821054) called HYPERION: Development of a Decision Support System for Improved Resilience & Sustainable Reconstruction of historic areas to cope with Climate Change & Extreme Events based on Novel Sensors and Modelling Tools. This project is developing a holistic system for monitoring the impact of climate change on endangered monuments, by deploying a network of suitable sensors, which will report by sending relevant data to a central system. For that purpose, 3D models of the selected monuments in Rhodes, Greece, Venice, Italy, Granada, Spain and Tønsberg, Norway were needed. This paper focuses on the exploitation and integration of close-range 3D Optical and Hyperspectral data acquired in June 2020 from the four cultural heritage buildings on Rhodes (Figure 1). A series of methods using data from multiple sources is proposed and implemented in order to extract, identify and assess material features and degradation on these cultural heritage buildings. The data were integrated to highlight the variability of material degradation on the surface of the buildings at macro and micro scale, while the results and findings are discussed. Hyperspectral images have been widely used in remote sensing and surveillance, while lately many efforts have been reported to combine and integrate multi-sensor data to provide substantial information about both geometric features and material properties in various scientific fields (Brusco et al. 2006, Erenoglu et al. 2017. Compared to RGB images, hyperspectral data have also shown great potential in 3D computer vision, particularly for the applications that require fine analysis of the spectral responses of objects (Zia et al. 2015).

Study areas
The island of Rhodes in Greece is situated at the South Eastern part of the Aegean Sea and the city of Rhodes, where the four pilot areas are located, has a history of 2400 years. The first pilot area is the Saint Nikolaos lighthouse and fort ( Figure 1A) at the entrance of the Mantraki port, which is a part of the medieval fortifications. It includes consecutive reinforcements and expansions after the destructions it suffered from sieges. It was built in 1464-1467 while the walls around the fort were built in 1480. The fort was transformed into a lighthouse in 1863. The second pilot area, the Naillac Pier at Saint Paul's rampart ( Figure 1B) was also built around the same period in 1400 on the Hellenistic pier and it was also part of the fortification of the "Great Port", the Commercial Harbour of Rhodes. The third pilot area is the Roman Bridge ( Figure 1C) built across the stream of Rhodini. It is one of the few ancient bridges surviving in Greece dated in the Roman period, but with later repairs at the piers of the arches. The bridge is still in use today, as a two-way motor carriage is running over it, leading to the deterioration of its static competence. Although consolidation works have been carried out on the banks of the stream, the flow of the water is blocked impeded by the dense vegetation and the elevation of the initial riverbed by the modern one, made of reinforced concrete. The fourth pilot area are the Grave enclosures in Rhodini ( Figure 1D) dated in the Hellenistic period, in the 3 rd -2 nd century BC. The environmental conditions and the microclimate of the area, in which critical role plays the stream of Rhodini, in combination probably with extreme natural disastrous events, such as earthquakes, have caused severe damages to the tombs ranging from the erosion of their stucco-coated facades to fissuring, detachment and collapse of the monuments.

Photogrammetric and laser scanning data acquisition
For the initial 3D modelling and representation of the Cultural Heritage buildings the combination of geodetic, photogrammetric and laser scanning data acquisition and processing methods have been applied. Digital images have been taken in different ways according to the size, complexity and level of detail of each monument using high resolution fullframe cameras and two UAS combined with two low resolution multispectral cameras. Furthermore, terrestrial laser scanning has been conducted to accurately determine the surface of the Cultural Heritage buildings used as reference for the routine and post-disaster monitoring, as foreseen by the research project. In particular, for the data acquisition process the following equipment were used: Two Integrated Total Stations (Pentax R-323NX & Leica TCR 405), two time-of-flight pulse-based 3D Laser Scanners (Leica BLK 360 & Faro HDR), two full-frame DSLR cameras (Canon EOS 6D & Sony A7RIII) with various lenses (24mm, 135mm, 28-75mm) and two Unmanned Aerial Systems (DJI Phantom 4 Pro & Mavic 2 Pro). The first step of the documentation of the Cultural Heritage buildings in Rhodes was the determination of the reference coordinate systems that were used to conduct the necessary measurements. The use of arbitrary local coordinate reference systems in each Cultural Heritage site georeferenced with the minimum constraints ensured the avoidance of deformations of the shape or size of each monument. A trigonometric network in form of a traverse was established at each pilot area to measure the ground control points (GCPs), the laser scanner targets and any additional supportive measurements. In order to properly and fully document the Cultural Heritage buildings a large number of digital images were taken using both the highresolution cameras mostly handheld but also with a special telescopic tripod and the UAS by planning and executing manual and pre-programmed flights. The imaging parameters like the image overlap, camera angle, GSD and flight altitude were properly adjusted each time according to the need and complexity of each pilot area. The data acquisition process using the UAS can be challenging or even impossible to achieve due to several restrictions. The weather conditions may not make it possible to plan and execute a flight, the vegetation and terrain may also be a restriction since the aircraft is not able to fly near any obstacles and high trees may cover the Cultural Heritage buildings leading to lack of information as well as the flight time limitation should always be taken into account. These restrictions made it essential to acquire images with the handheld cameras to overcome the eventual lack of data. Moreover, multiple scans were performed for each Cultural Heritage building in order to acquire the necessary data for the production of the accurate 3D surface along with the SfM-MVS procedure. All the acquired data for the 3D documentation of the Cultural Heritage buildings are presented at

Hyperspectral data acquisition and pre-processing
The hyperspectral data acquisition has been carried out using the HyperView multi sensor hyperspectral sensing platform by 3D-one. The HyperView system ( Figure 2) is a dual head system combining one Visual (VIS) snap-shot camera with 16 bands within the spectral range 470-620 nm, and one Near Infrared (NIR) camera with 25 bands within the spectral range 600-975 nm, which are connected on a EP12 board.

Figure 2. The HyperView multi sensor hyperspectral system
The system is designed to be lightweight and can be installed on a UAV after some extra modifications and necessary adjustments. For the time being no appropriate UAV is available and therefore the hyperspectral image acquisitions took place in situ placing the system on a tripod at various Cultural Heritage sites, depicting different materials and corrosion states. These particular cameras are equipped with a mosaic CCD and if someone looks at the raw image of each camera, one would see an image like the one presented in Figure 3.

Figure 3. A raw HyperView image
The trick that these cameras do is that instead of acquiring all spectral bands for every pixel they acquire only one band per The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) pixel, while they acquire all the bands in small windows, 4x4 for the VIS head and 5x5 for the NIR head. This is done by appropriate filters that have been added over every pixel of each camera CCD. Thus, the image pixels do not have a uniform spectral response. Rather, they have a complementary spectral response, which if transformed in an appropriate way could produce an image with all image pixels containing a spectral signature from all the camera bands. Special ansi C programs, which perform the following tasks, have been created for this scope. These programs: (1) decode the raw images, (2) produce band sequential lower resolution hyperspectral images with every pixel containing all image bands, (3) perform appropriate intra band shifts in order to correct the different viewing angles within the 4x4 and 5x5 acquisition windows, (4) correct a systematic NIR sensor pattern, (5) optionally rearrange the image bands in order to be in spectral order, (6) create a full resolution panchromatic image by applying an appropriate Gaussian filter on the initial raw image, (7) perform pansharpening of the lower resolution HS images using the Local Mean Matching pansharpening method (De Bethune et al. 1998), and (8) convert radiance images to reflectance images based on the pixels of a reference surface, if one was used during acquisition.
During the Rhodes measurements a spectralon reference surface was not available and therefore a bright white paper coupled with a photographic gray card were used as reference. Laboratory measurements following the campaign compared their spectral response to that of a spectralon surface in order to correct the spectral response of the acquired images and provide correct reflectance images. The laboratory measurements were carried out using the HyperView dual-head system and a PSR3500+ spectroradiometer in order to study any differences encountered. According to these spectral measurements the white paper, which has been used, presented some variations in the blue and red part of the spectrum, while the photographic grey card had a much better response while reflecting 60% of the incoming radiation. Thus, the grey card has been used for the conversion of all HyperView images from radiance to reflectance taking into consideration that the reference surface radiance had to be multiplied by 10/6. For the Rhodes pilot area, 1540 hyperspectral images have been acquired in total, occupying around 6.5GB of storage space. During acquisition, several aperture settings have been selected to ensure correct exposure. Moreover, three to five frames have been acquired for every single acquisition. The preceding two reasons lead to this big number of acquired images from which only the best acquisitions had to be chosen for processing. A small linux bash script has been used to quickly calculate statistical values and previews for all the images, using GDAL (gdal.org) and the linux convert utility. By both statistical and optical evaluation of the images only 53 images have been kept for processing. As already mentioned, the HyperView system was not mounted on a drone. Instead, the image acquisitions have been carried out with the HyperView cameras mounted on a tripod. Therefore, the images do not cover all the Cultural Heritage buildings but just characteristic parts of them. Nevertheless, facade acquisitions provide more details about the materials as well as their corrosion state than the vertical acquisitions that could have been carried out using a drone. The resolution for most of the acquired images range from less than 1x1 mm to 10x10 mm, depending on a) the distance of the cameras to the captured objects, which could not be fixed in most cases and b) the technical specifications of the two cameras. The final selected hyperspectral images have been preprocessed (i.e., decoded, corrected for internal sensor shifts, pansharpened), transformed to reflectance, and co-registered, as the VIS and NIR sensors provided images from a slightly different viewing angle.

Photogrammetric processing
The data processing procedure focuses on one hand on the exploitation of the photogrammetric data in order to produce the 3D polygon meshes and on the other hand on the integration of the hyperspectral data in order to produce 3D Textured Models.

3D models
The data processing procedure was divided in two parts. First the digital images were processed using the Image Based Modelling (IBM) software Agisoft Metashape v.1.6.5 following the common workflow as in every documentation process. The captured images for each pilot area were loaded in chunks, separating the aerial and the ground images. The images were examined, evaluated and some of them were excluded. The next step was the automatic detection of the pre-marked targets and the manual detection of the laser scanner targets to facilitate the merging of the chunks, the alignment, scaling and georeferencing of each 3D point cloud. After the successful alignment of the images, the dense point cloud was generated and subsequently extracted to be further processed with the Geomagic Wrap 2017 software. Then the data processing continued with the registration of the scanned point clouds using the Faro Scene and Cyclone Register 360 (BLK Edition) software applying the Cloud-to-Cloud method since the overlap between the scans was enough, while the aligned scans were then georeferenced by identifying the targets and setting the coordinates. The point cloud of each Cultural Heritage building was further processed using the Geomagic Wrap 2017 in order to reduce inevitable scanner errors (noise) and reduce the number of points leading to a smoother and accurate 3D Model. Finally, the dense point cloud from the IBM software was used to fill the eventual gaps in the scans and generate the final point cloud which was then converted to a 3D mesh.

RGB and HS 3D textured models
Since the hyperspectral data were acquired for specific parts of each Cultural Heritage building, it was decided to trim the final 3D surface for each pilot area and isolate the meshes that were covering just these smaller areas. Then these 3D meshes were imported once again into the IBM software to produce the 3D Textured model using only the RGB high resolution digital images (Figure 4) as well as an orthomosaic for each part of the Cultural Heritage building. In addition, the DEM was also produced for each one of these trimmed 3D meshes. The exploitation of the hyperspectral images required first their transformation and translation in QGIS software. Then the images were aligned with the RGB digital images using the premarked targets as common points, if any of them were visible, or characteristic points at the structure that were easily identified in both the RGB and HS images. More than 6 common points in each HS image were used to properly and accurately align them with the RGB images. Since the HS images were successfully aligned, the 3D Textured models and the orthomosaics were produced enabling the use of the HS images ( Figures 5 & 6) along with the 3D data.

Hyperspectral data processing
The preprocessed hyperspectral images have been used for the extraction of the spectral signature of the Cultural Heritage building materials and biodeterioration appearances.

Analysis of spectral signatures
Spectral signatures of various samples from the hyperspectral images have been studied. Signatures have been collected for (1) building material types, (2) bio-degradation types or other corrosion factors according to the ICOMOS-ISCS illustrated glossary on stone deterioration patterns (ICOMOS-ISCS, 2011), and (3) material loss condition. Small polygons have been selected for the calculation of the spectral signatures for pure appearances. Later on, these pure spectra were used as endmembers during spectral unmixing. The stone materials were classified according to their colour, based on the work of Papagianni et al, 2004. For the signatures of these materials differences were encountered mainly in the visual spectrum and more particular in the green-red region. The spectral signatures of the various corrosion factors presented more variations. Eflorescence, daub and salt crystallization presented a signature similar to this of the reference surface. Vegetation appearing on the walls presented a strong increase in the Near Infrared (NIR) spectrum with a slight drop around 880 nm. Moss, which was the most common bio-deterioration factor in the specific monuments, also presents increase in the NIR part of the spectrum but less than the leaves of the bigger plants. Dry moss presents only a slight increase in the NIR region. As far as material loss is concerned, it was found that the spectral signatures present a general decrease in reflectance when the material loss is more intense (Figure 7). This reflectance decrease is caused by the rougher texture and the occurring shadows and not changes within the stone material. Thus, spectral signatures for material loss cannot be selected and will not be discussed further.  Roman bridge, some parts of the moss were wet, presenting very low reflectance especially in the NIR region. Signatures of the same appearances present similar shapes but different reflectance magnitudes due to irradiance variations, which are still apparent despite the radiance to reflectance conversion. Continuum removed spectra has also been studied in order to detect useful absorption features, but it did not prove to be useful in this case study.

Spectral unmixing
The signal detected by a sensor into a single pixel is frequently a combination of numerous disparate signals that jointly occupy the pixel by forming a macroscopic mixture. These disparate signals correspond to distinct substances that are called endmembers or pure pixels and the fractions in which they appear in a mixed pixel are called abundances. A very common procedure when working with hyperspectral imagery is the unmixing process, which aims at detecting the endmembers in a given scene as well as their abundances. The accuracy of spectral unmixing is highly dependent on the selection of appropriate, representative endmembers (Tompkins 1997). In many cases, selection of appropriate endmembers is carried out manually. Then supervised unmixing process is performed. The hyperspecral imagery collected in Rhodes has been processed with various unmixing methods (linear spectral unmixing, matched filtering, mixture tuned matched filtering, constrained energy minimization etc.) in a supervised way, providing the collected spectral signatures as endmembers. The unmixing method that provided the best results was the mixture tuned matched filtering (MTMF). MTMF is an advanced spectral unmixing algorithm in which it is unnecessary for all materials within a scene to be known or to have identified endmembers (Boardman et al. 1995), and it combines the best aspects of the linear spectral mixing model and the statistical matched filter model. MTMF performs partial unmixing only by finding the abundance of a single, user-defined endmember, by maximizing the response of the endmember of interest, and by minimizing the response of the composite unknown background, thus 'matching' the known signature (Williams and Hunt 2002). The MTMF method includes three main steps (Mehra et al, 2013): (1) an MNF transformation of apparent reflection data (Green et al.1988), (2) matched filtering for abundance estimation, and (3) mixture tuning to identify infeasible or falsepositive pixels (Boardman1998). The output of MTMF is a set of rule images given as MF and infeasibility scores for each pixel related to each endmember. In order to be sure that the abundance images contain only correct values, the pixels with high MF scores and low infeasibility values have to be selected.
In our experiments, pixels with MF score above 0.5 and infeasibility below 20 have been selected as correct for each material type (Figure 10).

Integration of 3D and Hyperspectral data
The integration of the 3D and the HS data was achieved by coupling the 3D and the hyperspectral processing results in order to build a basic reasoning on which are the most vulnerable parts of the CH buildings. In our case study we have compared the 3D estimated material loss results with the HS unmixing results for the building materials as well as the biodeterioration factors which are apparent on the CH buildings. This work presents limited and quite preliminary integration results from the St. Nikolas Fort and the Roman Bridge. Consequently, initial conclusions on this subject are produced.
The coupling of the 3D and the HS has been carried out in QGIS and the comparison of the results was performed using cross classification matrices between the 3D resulting classes and the HS resulting classes (after converting the unmixing abundances to material or biodeterioration classes).

Evaluation of 3D geometric documentation
One of the main objectives of this paper is the estimation and assessment of the material degradation by utilizing the geometric information and the 3D point clouds and meshes of each pilot area at indicative parts, where HS data were acquired as well. Two approaches have been used and evaluated for this scope.

Planar surface approach
A simple approach was followed in order to assess the material degradation is by trying to adjust a planar surface at the part of the monument that presented no degradation. After defining that plane, the distances between the plane and the 3D Point Cloud or Mesh would indicate the depths of the degradation and the material loss for each stone at the Cultural Heritage buildings. Practically, for each case both the 3D point cloud and the 3D Textured Model were imported in the CloudCompare Software (CloudCompare 2.12 alpha, 2021). Then, the qMPlane Plugin (http://www.cloudcompare.org/doc/wiki/index.php?title=MPlan e_(plugin)) was used to define an appropriate plane by picking numerous points at the 3D point cloud where no degradation was obvious according to the 3D Textured model (Figure 11). Figure 11. qMplane Plugin: Defining the best fit plane by picking points on the 3D point cloud.
The software automatically creates and applies a scalar field for the visualization of the normal distance between the point cloud and the plane leading to a first estimation of the highest and lowest degradation value. Further normal distance measurements against the plane can be performed to acquire accurate and specific results about the depth of the degradation so as to estimate and evaluate the material loss and the level of degradation for each part of the monument in particular ( Figure  12). These measurements can easily be exported in a .csv file and the comparison between the visual degradation and the measured distances is quite easy when observing the 3D Textured model and the scalar field ( Figure 13).

Integration results
The integration of the 3D and HS results was achieved by comparing the maps of Figures 16 and 17  Then, it is easy to interpret that in the total appearance of each material on the specific wall, the marbles and the grey stones present low material loss values, the reddish-grey stones present higher risk of material loss and the red stones suffer the most severe material loss. In this way it is possible to evaluate which parts of a CH building is more vulnerable depending on the dominant materials in them.

Vulnerability based on degradation type
Similarly, Table 4 shows the correspondence of the biodeterioration type to each material loss class. When normalizing the values of Table 4 according to the share of each biodeterioration type on the examined wall surface it is possible to draw useful conclusions about the effect that each biodeterioration type has on the CH wall. The values in Table 5 show that the less material loss is observed within the salt crystallization and the dry moss categories, while slightly higher vulnerability is observed within the dense and wet moss areas. It should be pointed out though, that the most severe material loss in the Roman Bridge area is observed within the areas that are not covered neither by moss nor by salts and this is obvious in Figure 16. Thus, despite their vulnerability differences moss and salts seem to act as a protective shield against material loss.

Discussion
The spectral information and signatures are linked to the detailed geometry of the Cultural Heritage buildings leading to the analysis, mapping and assessment of materials and degradation. The HS data analysis revealed that, with the selection of appropriate endmembers, it is possible to map the different building materials, as well as biodeterioration and other corrosion factors. Spectral unmixing and appropriate transformations produced material degradation maps. The integration of the 3D surface texture with these maps significantly improved our knowledge about building material type, degradation type, degradation level and their relations to material loss. The results presented in this work are preliminary and cannot be considered as objective by any means. More test sites have to be analysed and the results should be compared and integrated.
There are several technical and methodological improvements or optimizations that could help not only derive safer results, but also build a robust decision making system that could help identify the most vulnerable parts of a CH building. As far as it concerns the technical improvements, the main improvement would be to use a hyperspectral sensor with wider spectral range, as the Middle Infrared region of the spectrum is considered important for many biodeterioration types. Thermal images could also add very useful information especially in the case of excessive moisture conditions. On the methodological part, machine learning and/or object based image analysis could help in achieving better integration and automation of the CH vulnerability assessment procedures and our team is currently investigating the utilization of both machine learning and OBIA.

CONCLUSIONS
A holistic interdisciplinary approach has been proposed, developed, and implemented for the documentation and surface degradation assessment of Cultural Heritage buildings, involving Non-destructive Techniques (NDT). Detailed 3D Textured models in combination of hyperspectral imagery, have enabled the production of material type and degradation maps. The results are very promising and further analysis will be carried out. It has been shown that HS unmixing results can indicate where detailed 3D textured models should be produced to efficiently document material degradation and loss.
It is planned to involve machine learning algorithms for identifying the areas with material degradation, and thus automate the whole procedure.