A proposed multimodal convolutional neural networks (CNNs) soluble salts mapping system of the subsurface: The case of the wall painting at the royal box at Herodium, Israel

In 2008, excavations at Herodium revealed magnificent secco wall paintings and stucco decorations adorning the central chamber at the top of the royal theatre. The wall paintings, dated to the first century B.C.E., have been preserved up to a height of 6 meters. However, shortly after the discovery, salts weathering and structural faults caused severe damages to the decorations. The conservation process to restore the wall paintings lasted almost a decade. These efforts helped stabilize the state of wall painting, but in a very fragile manner, while the deterioration factors are still present, any slight change in the condition of the enclosure, could damage the paintings. This study is aimed at assisting the conservators in developing a tool that will offer a glance to the hidden threats at the subsurface, and by that help protect historic monuments from salt weathering. This paper will describe an innovative methodology with particular emphasis on novel multimodal convolutional neural networks (CNNs) technologies to process data of non-destructive testing (NDT) for detection and mapping soluble salts at the subsurface of ancient wall paintings. Prior to preforming the system protocol in situ, a laboratory simulation was carried out to study thermochemical behaviour of soluble salts, chlorides and sulphates, within different subsets. The preliminary results of the simulation will be presented in this paper.


INTRODUCTION
Salt weathering is one of the most detrimental agents in the deterioration of historic wall paintings. Presence of water in any amount transfers the salt solution within the structure to various locations. When the water evaporates, the salts recrystallize in the porous material and form efflorescence followed by volumetric expansion, which results in deterioration of the surface material. Moreover, the thin layer of the painting is characterized with high open porosity that allows high accessibility of salt in different forms (Doehne, 2002;Granneman et al., 2019;Martínez-Martínez et al., 2020;Matteini, 1991;Sfarra et al., 2016). Unfortunately, it is mostly noticeable once it reaches the surface and has already caused damage (Arnold & Zehnder, 1991). The case study of this research, the wall paintings at Herodium, is an example of the serious destruction caused by salt weathering. After discovered in 2008, the paintings have suffered severe damages caused by salt crystallization. To address the problem, expert conservators from the Israel Antiquities Authority were called to restore the paintings and stabilize the environmental conditions of the enclosure. These efforts lasted almost a decade. The conservation process was based on condition assessment that included analysis of the decoration surface, as well as inspection of moisture and water penetration into the structure (Neguer & Davidoff, 2015). Though, with the salts still present in the walls, any slight change leading to wet and dry cycles, could cause outbursts of efflorescence that will damage the paintings. Nevertheless, a recent water penetration report showed that the conditions in the enclosure are still unstable (Cahana, 2020). With the salts still present in the walls, any slight change leading to wet and dry cycles, could cause outbursts of efflorescence that will damage the paintings (Arnold & Zehnder, 1991;Charola & Bläuer, 2015).
Therefore, prior knowledge of the subsurface conditions and monitoring salt solution movement patterns, could help to identify developing problems in early stages and prevent unnecessary harm (Arnold & Zehnder, 1991;Di Tullio et al., 2010). This, however, is a challenge when keeping the integrity of the monument is a priority, and destructive methods are often excluded (Johnston et al., 2019). Today, soluble salts and moisture distribution analysis and diagnosis studies using non-destructively methods mostly investigate the surface and its superficial layers (Di Tullio et al., 2010;Hoła & Sadowski, 2019;Sfarra et al., 2016). Thus, full mapping of salt and moisture content beyond the superficial layers has yet to be conducted, and consequently destructive soluble salts at the subsurface are hidden from the eyes of the expert.
Considering those problems, this proposed protocol, will aim at investigating the effectiveness of multimodal convolutional neural networks (CNNs) system in detecting, mapping and evaluating subsurface soluble salts. At this aim, the study will be performed in situ on the once severely damaged and now restored wall paintings of Herodium, Israel. In the framework on the proposed study data will be collected with two fully NDT methods -Infrared Thermography (IRT) and Evanescent Field Dielectrometry (EFD).
The NDT methods were chosen for their capabilities of achieving successful results in the diagnostic of the deterioration agents that trigger salt weathering. The first is widely known in the field of art conservation Infrared Thermography (IRT). It was chosen for its advantages -non-destructive nature, portable use, qualitative investigation of the surface and the subsurface features and quantitative analysis of the thermal properties (Mercuri et al., 2015;Moropoulou et al., 2013). In particular, mapping of moisture distribution deep in the subsurface has been shown in recent study at a ranging depths of dozens of centimetres (Chaban et al., 2020). The second is Evanescent Field Dielectrometry (EFD), a diagnostic method based on dielectric spectroscopy at microwave frequencies (Di Tullio et al., 2010). With the ability to measure the dielectric contrast in materials insitu, it allows a non invasive detection of dishomogeneity and material discontinuities in the subsurface. For this reason, EFD is a powerful tool for analysing the sub superficial moisture content and mapping soluble salts presence within the wall. It has succefully shown that EFD has the abilty to produce significant information on moisture and salts contents. Notably, with the data collected it was possible to explicate the levels of salinity in the moisture with evaluation of the Salinity Index (Marie-Victoire et al., 2018), though, The depth can range up to a couple of centimetres (Di Tullio et al., 2010;Marie-Victoire et al., 2015). While each method shows remarkable capabilities, when mapping and investigating soluble salts beyond the superficial layers of the surface it seems that each method is lacking some ability the other obtains. In recent years, Methods integration have become more and more common in the field of art conservation (Delegou et al., 2019;Kilic, 2015;Raimondi et al., 2015;Sfarra et al., 2016). In particulare, this approach is being taken further these days with deep learning systems (Hoła & Sadowski, 2019). Due to the rapid development and the efficient performance of deep learning, variety of convolutional neural networks (CNNs) have emerged. Consequently, the ability of high accuracy recognition and classification has grown. Its applications have been widely applied in many fields and in recent years it entered the field of art conservation (Wang et al., 2019). More importantly, art conservation studies have been useing deep learning for weathering analysis such as moisture content identification model based on ANNs (Hoła & Sadowski, 2019) and automatic superficial surface damage of efflorescence model based on CNNs (Wang et al., 2019).

STUDY AREA
Herodium, located south to Jerusalem on the edge of the Judaean Desert, was founded by Herod (37-4 BC). Herodium was an administrative centre and a summer palace as well. Recent excavations at northern slope of Herodium ( Fig. 1) have discovered a small royal theater with a glorious royal box. The room was decorated with magnificent wall paintings and stucco decorations (Netzer et al., 2009).

Description of the royal box and its decorations
The central and most impressive chamber of a series of chambers at the top of the theater was termed the royal box. The Western, Southern and Eastern walls of the royal box are splendidly adorned with wall paintings and stucco decorations, preserved to a height of 6 meters (Fig. 2). The wall paintings were not made by a common fresco but in secco, painting with a binder on a dry surface, rather than on a wet surface (Netzer et al., 2010).

The conservation process
The Conservation process had been conducted by two teams: since the reveal in 2008 until 2011 by The Israel Museum; and from 2012 until today by the Israel Antiquities Authority. Since the exposure, the decorations have been severely damaged ( Fig.  3) by salts weathering caused by structural faults and improper treatment after the exposure (Neguer and davidoff, 2015).

Figure 3.
Efflorescence on the surface of the wall paintings at the Royal box, Herodium (Neguer & Davidoff, 2015) Since 2012, important steps were executed to preserve and treat the sever condition the wall paintings were at. First, documentation of the present state of the decoration was conducted. A series of laboratory testings had been to analysis the pigment origin, moisture content at problematic area and the salts content at the walls. Water access to the structure has been reduced excavating areas collection the hill to the chambers, building a proper shed and applying drainage solutions. After these steps, conservation treatment of the wall paintings was preformed and included lacunas fill, plaster edges sealing, salts extraction, grouting and consolidation of plasters and pigments (Neguer and davidoff, 2015).

Present state
Since the beginning of 2021, the chamber is now open to the public (Fig. 4). The interior climate is now stable and controlled by air conditioning. However, according to a recent water penetration report (Sep. 2020) The enclosure contains un balances moisture areas in the chamber walls caused by construction faults (Cahana, 2020).

LABORATORY SIMULATION
Prior to the conservation treatment of the wall painting in Herodium, a salt testing (Wasserman, 2012) was performed on falling fragments of the painting. The results of the salt testing showed that chlorides and sulphates have been detected in each fragment in various amounts. In order to study the thermochemical behaviour of Chlorides and sulphates in porous material similar to the case study laboratory simulation has been conducted.
Samples (diameter 10 cm and 7 cm height) were created with similar substance that was used in the construction of Herodium. The mimic source mortar was created with a mixture of gravel, Arad sand, quarry sand, crushed ceramics, calcium oxide (burnt lime) in a ratio of 2:1:1:0.25:1.5 respectively. The first plaster layer mimics the arriccio layer consists of Arad sand, quarry sand and calcium oxide (burnt lime) in a ratio of 1:1:1 respectively. The top plaster layer which mimics intonaco (whitewash) was created with a mixture of stone powder and calcium oxide (burnt lime) in a ratio of 1:1 respectively (Fig. 5). The samples were constructed in three composition groupslimestone (A), Mortar (B) and Complex samples consist of limestone with mortar and on top two layers of plaster as described above. To avoid uneven lay in the construction the complex group was created with three repetition groups (C-E).

Figure 5. Creation process of the study Samples
The chlorides and sulphates were examined in a set of treatments. The samples were once again divided into four groups, three treatment groups (1,2,3) and a control group (4). The samples in the three treatment groups were impregnated at the bottom of the sample with three water base salt solutions, (1) 0.01% sulphates (SO 4 -2), (2) 0.01% chlorides (Cl -) and (3) 0.01% chlorides (Cl -) and 0.01% sulphates (SO4 -2 ) (Fig. 6). The samples were marked according to their composition and the given treatment group (Table 1).  After the impregnation, the samples were set to dry for a month. The stones sample were suspected to be contaminated and were discarded. Once the Samples had dried, each treatment group was heated with external source to 45⁰C degrees and their cooling process was recorded with a sequence of thermal images taken every 10 sec for 5 min and every minute for 3 minutes more. The data was collected by NDTherm camera, by Opgal. The camera is a broad band system (7.5 -14μm) with resolution of 384 x 288 pixels. The data was stored in the memory of the android and was processed with MATLAB 2020b. The data of each sample in the group treatment was filtered with a center circle mask that defined the region of interest (30 x 30 pixels) (Fig. 7).

Figure 7. Data recording with Therm-App
The interest region of each sample was investigated, to obtain information about the thermochemical characterization of the porous Herodium wall substance and chlorides and sulphates, salts that were found present in the fragments of the wall painting (Wasserman, 2012). The temperature values at the region of interest were collected along colling time of 8 minutes. First, for each data frame the mean was computed. The coefficient of variance at the region of interest was also computed to test the creation process of the samples or the impregnation process both were conducted manually. To examine inaccuracy the coefficient of variance was considered as validation factor.
In order to investigate subtle patterns and behavior of the sample's substance and soluble salts within, a first derivative of the temperature mean over time was computed for each sample. To sharpen the investigation outputs an absolute log value of the first derivative output was computed and presented as a factor to the cooling time. The outputs have been compared according to the composition groups to detect a subtle pattern caused by the nature of the sample substance. The outputs have also been compared according to the different treatment groups to detect subtle patterns caused by soluble salts.

PROPOSED INSITU METHODOLOGY DESIGN
The proposed methodology design is based on five stages: (1) NDT field measurements, (2) Preprocessingcalibration, image registration and super-resolution CNN model (SRCNN), (3) Processing -deep learning model base CNNs architecture, (4) Postprocessing -Validation, (5) Change detection over time. The first four stages complete the deep learning-based soluble salts mapping system (Fig. 8), while the fifth stage is performed on two sets of the system outputs data taken at different seasons or after severe climate change to investigate shift of salt at the sub surface via change detection.

a b
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China

Dataset -Multiplatform field measurements
The measurements will be focused on a segment of the southern wall of the royal box. The framework will cover an area of 2 meters width and full-length long (about 6 meters). The chosen area will include known problematic faults such as areas suffering from capillary motion and outer constructional cracks according to a recent water penetration Report at the royal box (Cahana, 2020). The measurement campaign will be composed of two NDT methods and will be performed pointwise. The campaign will be carried out on a selected framework and on the calibration model. The model will be created in accordance with the walls structure and materials of Herodium. Another campaign will be carried out after seasons change or severe climate change at the interior since the first campaign. The data acquisition with Active IRT will be carried out as a sequence over time starting with external stimulus being applied to the inspected subject until the end of the cooling process. The external stimulus will be based on an optical stimulation of halogen lamps. The data collected will be a sequence of images with the time factor equivalent to the depth factor. Two protocols will be performed: Conventional thermography (continuous heating) and Pulsed thermography (short heating). Each measurement will cover an area of approximately 40cm*40cm until full coverage of the framework.
With EFD Radar, the acquisition will be performed with a direct contact, keeping the probe head in contact with the surface. The measurement will be conduct with repetition set microwave frequency (1-1.5 GHz) and the time factor will be equivalent to the sub surface region depth. Measurements will be carried out in a grid creating a point matrix across the framework. Each point covers an area corresponding with the area of the probe head.

Preprocessing
The acquisition data, consisting of point matrix will be integrated into a 3D data set. The input data will be normalized to the calibration neutral sample values, and a vignetting reduction will be conducted. The multi-source data will be integrated via superresolution CNN (SRCNN) model: The size of convolution filters of a layer in the SRCNN and thus the size of tensors is defined by the spatial-spectral extent of the filter within a single input feature map and the number of input feature maps in this layer. Each filter computes a single output feature map by moving over fixed locations. At any precise location, the filter inputs multiple feature values and outputs a single value, which is then subjected to the nonlinearity ReLU.

Processing
system will consist of three stages, i.e., two separated CNN steps and Salinity Index (SI) classification.
First, the image selection CNN will be adapted from LeNet-5. This CNN will have two sets of convolutional layers, which will be followed by two fully connected layers and then a SoftMax classifier. Based on comparison with the calibration sample, two output classes will be defined: one class for image sections that resemble the calibration sample and one class for image sections that do not resemble the calibration sample, which is suspected as containing soluble salts.
Next, the image sections, suspectedly containing soluble salts, will be used as an input for the salt isolation CNN, adapted from You Only Look Once (YOLO) v5. This CNN, which demonstrated top performance in many image object detection challenges, has 24 convolutional layers followed by two fully connected layers. The fully connected layers then will be used to extract image features to predict the coordinates of a 3D box, defining the soluble salts regions.
Finally, at the third stage, the cropped images of the predefined box will be used as input for the classification. The output will be processed with a well-known classification called Salinity Index (SI) (Marie-Victoire et al., 2018). The index will be used for detailed evaluation of the isolated region and detection of salt concentration levels in the selected segments pixel by pixel.

Validation
The results will pass several validation examinations with error detection and quality testing of the output of processing stage. First, the results will be reviewed by Mr. Jacques Neguer, Director of Art Conservation Department at Israel Antiquities Authority. Mr. Neguer led the conservation project of Herodium wall paintings. Next, the result will be crossed with the conservation analysis maps of the royal box preformed in 2012 before the intervention of the IAA team. These maps are part of documentation of the conservation project of the royal box (Neguer & Davidoff, 2015). A resemblance will be expected to be shown in the locations found defected back in 2012 to the areas found in the processing stage marked as high levels according to the Salinity Index (SI). Finally, a similar process will be performed by crossing the results with the constructure faults according to a water penetration report from September 2020 (Cahana, 2020) mapping the faults seen in the interior and exterior of the modern structure suspectedly considered as the cause of penetration water to the royal box.

Change detection over time
The integrated NDT data sources by cascaded SRCNN and endto-end CNNs systems will enable dynamic monitoring of rapid The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China changes at the surface and in-depth salt detection and identification at a high spatial resolution. The detected areas with salt solution will be projected onto series of maps and systematically monitored after seasonal change or extreme climatic changes. To perform a change detection, the system (stages 1-4) will be applied once more at different seasons or after severe climate change.

PRELIMINARY RESULTS
The laboratory simulation stage in the study has been conducted prior to the main stage of the study in order to collect information in supervised conditions of the thermochemical behaviour of soluble salts, chlorides and sulphates, within different subset. The data collected by the Therm-APP was used for that purpose. The range of the coefficient of variance at region of interest, the core of the samples, along the cooling time was found to be between 19.8 to 17.7. The low coefficient of variance values is indicating that samples creation and treatment processes were comparable and there is a low variance among the mean temperature at the region of interest along the time sequence. This allows us to focus on the mean of the record data.
While comparing the samples temperature mean over a cooling time of 5 minutes with samples at the same group treatment (Fig. 9a) and with samples with the same substance composition no significant pattern has been shown. Furthermore, the results of computing the first derivative to the temperature mean over a cooling time showed mild patterns but not enough to understand any repetitive behaviour (Fig. 9b). at the aim of sharpening the data at the search of the different thermochemical behaviour among the samples, a log of the first derivative was computed. The results were compared according to the four treatment group. In contrast to the previous comparisons, the results have showed more significant patterns in samples at the same treatment groups by peaking at similar times. The result of treatment group 1 (Fig. 10b), 0.01% sulphates, have showed a set of peaks at time of 210 sec, 240 sec and 270 sec, while the middle peak is the higher among them. The result of treatment group 2 (Fig. 10a), 0.01% chlorides, have showed a set of peaks at time of 120 sec and 270 sec. The result of treatment group 3 (Fig. 10c), 0.01% chlorides and 0.01% Sulphates, have showed a set of peaks at time of 150 sec and 260-280 sec. Unlike the rest of the peaks the last peak pattern did not align at the same time and showed same shift. The control group 4 (Fig. 10d) also shown a match set of peaks at time of 90 sec and 150 sec. The different thermochemical behaviour of chlorides and Sulphates can be detected comparing the patterns. While the samples that have been impregnated with chlorides peaks at an early stage of the cooling process, at the first 120 seconds, the samples that have been impregnated with sulphates peaks only at The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China a later stage of the cooling process, at 210-270 seconds since the recording began. In contrast, a similarity pattern can be detected though all the treatment groups the around time of 270 seconds, a pattern that is not shown in the control group. While both treatment group 1, 0.01% sulphates, and treatment group 2, 0.01% chlorides, showed aligned peak at 270 seconds, treatment group 3, the mixed group, show a stronger peak that spread between 260-280 seconds. Another similarity can be drawn between the control group, group 4, and treatment group 2, 0.01% chlorides. Both peak at the same time of 150 seconds.

CONCLUSIONS
The results obtained during the laboratory simulation can be view as a library of thermochemical behaviour of chlorides, sulphates, and their mixture. Each sample in the same impregnation treatment group have shown similar behaviour. While the treatments have presented a strong similar influence on all the samples in the same treatment group, the base subsets of the samples (limestone, mortar and combination of limestone, mortar and plaster) have been hardly noticeable. For instance, similarity could be drawn between the samples at treatment group 2, impregnated with chlorides, to the control group which have some peaks that align at the same time. Though this connection in partial and does not appear in the other treatment groups. Apart from this, it seems that the samples from control group have also shown similar behaviour among themself. The cause to that could be explained by common characters of the base subset, but further investigation is required to determine the nature of this similarity.
In addition, the two types of salt examined in the simulation are very different from one another and possess distinctive contrasting characters. The sulfates are less hygroscopic than chlorides and soluble salts and while evaporating become supersaturated and lingers at a certain location. In contrast, soluble salts like chlorides will spread farther, accumulate and form solutions characters with high hygroscopic while evaporated (Arnold & Zehnder, 1991). This could be representative in the two different characteristic pattern of sulfates and the chlorides. While the samples impregnated with sulfates consist multiple peaks, which appear only at progressive stage of the cooling period, the samples with chlorides present fewer peaks with a significant one at an early stage.
Most importantly, there is a particular behavior that repeat though all the samples impregnated with salts, treatment group 1-3, that does not show in the control group, group 4. This could be an indicator for presence of salts (sulphates, chlorides or their mixture) in uncontrolled conditions areas such as the case study of this research.
It should be noted that additional measurement in different time gaps and a larger samples amount is needed to establish a universal of chlorides, sulphates library of thermochemical behaviour of the tested salts.

ACKNOWLEDGEMENTS
This laboratory simulation was conducted with the help of OPGAL. The company have supplied the Therm-App, Thermal camera, that was used for collecting data.
The study is also conducted with the help and the guidance of the of art conservation department at the Israel Antiquities authority (IAA), Responsible for the conservation project of the decoration at Herodium. The department has created the calibration model and the laboratory samples while the head of the department will take part in validation stage the study. We would like to extend my sincere thanks to Mr. Jacques Neguer, The Formal head of the art conservation department at the Israel Antiquities authority, for his support and guidance in this proposed study design. we would also like to extend my gratitude to Mr. Fouad Abu Taa'a, conservation expert at the Israel Antiquities authority, for sharing his knowledge of the Herodian architecture, characteristic materials, and construction techniques of this era and for his help with the design and creation of the study samples and calibration model.