DOCUMENTATION AND DETECTION OF COLOUR CHANGES OF BAS RELIEVES USING CLOSE RANGE PHOTOGRAMMETRY

The digitization of complex buildings, findings or bas relieves can strongly facilitate the work of archaeologists, mainly for in depth analysis tasks. Notwithstanding, whether new visualization techniques ease the study phase, a classical naked-eye approach for determining changes or surface alteration could bring towards several drawbacks. The research work described in these pages is aimed at providing experts with a workflow for the evaluation of alterations (e.g. color decay or surface alterations), allowing a more rapid and objective monitoring of monuments. More in deep, a pipeline of work has been tested in order to evaluate the color variation between surfaces acquired at different époques. The introduction of reliable tools of change detection in the archaeological domain is needful; in fact, the most widespread practice, among archaeologists and practitioners, is to perform a traditional monitoring of surfaces that is made of three main steps: production of a hand-made map based on a subjective analysis, selection of a sub-set of regions of interest, removal of small portion of surface for in depth analysis conducted in laboratory. To overcome this risky and time consuming process, digital automatic change detection procedure represents a turning point. To do so, automatic classification has been carried out according to two approaches: a pixel-based and an object-based method. Pixel-based classification aims to identify the classes by means of the spectral information provided by each pixel belonging to the original bands. The object-based approach operates on sets of pixels (objects/regions) grouped together by means of an image segmentation technique. The methodology was tested by studying the bas-relieves of a temple located in Peru, named Huaca de la Luna. Despite the data sources were collected with unplanned surveys, the workflow proved to be a valuable solution useful to understand which are the main changes over time.


INTRODUCTION
Close range photogrammetry proved to be the most powerful technique for the documentation of Cultural Heritage (CH) goods.Its attractiveness is mainly due to the ease of data collection, the low cost of both hardware and software components and the existence of well-established pipelines for the 3D reconstruction of real objects.The benefits of virtual reconstructions, based on Multi View Stereo (MVS) and Structure from Motion (SfM) algorithms, range from the documentation till their virtualization for dissemination purposes.As a matter of facts, the output products from these techniques are digital representations of the real artifacts, which depict the starting point for almost every kind of application.3D models have the twofold advantage of providing a faithful representation of the reality and, at the same time, of representing a valuable tool for in depth analysis or advanced inquiries of a good (Cabrelles et al., 2010).Recent research trends demonstrate that the backbone for the preservation of mankind's heritage is, nowadays, the photogrammetric acquisition.Devices like commercial cameras are becoming more and more handy, making them preferable with respect to other cumbersome instruments like drones or Terrestrial Laser Scanners (TLS).Despite the accuracy achieved with the latter is higher, the quality of the sensors and the possibility to perform a data fusion from different techniques, make MVS approaches more suitable for the archaeological domain (Pierdicca et al., 2016).In the last decade in fact, the use of this approach has become paramount if one is dealing with archaeological areas or findings.This, for many reasons.Archaeological sites are almost always in emergence conditions: the ruins are often in abandon, the state of conservation is precarious, the materials are perishable and they are very difficult to be reached.Their digital preservation is thus entrusted in more agile and friendly procedures of both acquisition and processing.It is not unusual to face with conditions of unplanned surveys, performed during an excavation or with strict timing and low budget, though they are capable to produce useful results for the research community (Pierdicca and Malinverni, 2017).The literature provides a plethora of case studies, demonstrating how such technology is able to cover a wide spectrum of possible applications that can be developed using the resulting 3D data as the starting point.
In this light, every kind of spatial data can be useful even for restoration purposes.In fact, the automatic extraction of information may reveal huge potential for insiders performing in depth analysis over ancient artefacts.In archaeology, where the condition of surveying and the shapes of the artifacts are impervious, all the advantages described above become fundamental.The digitization of complex buildings, findings or bas relieves can strongly facilitate the work of archaeologists, mainly for in depth analysis tasks.In the field of CH research, the characterization and classification of the conservation state of the materials that constitute the surface of the artefacts are essential to study their damage.Notwithstanding, even if new visualization techniques facilitate the study phase, a classical naked-eye approach for determining changes or surface alteration could bring towards several drawbacks.
The research presented in this paper is aimed at providing experts with a workflow (detailed in Section 3) for the evaluation of alterations (e.g.colour decay or surface alterations), allowing a more rapid and objective monitoring of monuments.More in deep, a pipeline of work has been tested in order to evaluate the colour variation between surfaces acquired at different époques.For testing the methodology were chosen the bas-relieves of a temple located in Peru, named Huaca de la Luna.This case study (described in Section 2.2) is suitable for two reasons: first, the buildings within the archaeological site are made of adobe, a particular kind of raw clay that is perishable and in a danger because of the weather conditions.Second, the bas-relieves are under the monitoring of experts that make sporadic intervention of colour re-filling, based on naked eye observation without the support of technological tools.The most widespread practice, among archaeologists and practitioners, is to perform a traditional monitoring of surfaces that is made of three main steps: production of a hand-made map based on a subjective analysis, selection of a sub-set of regions of interest, removal of small portion of surface for in depth analysis conducted in laboratory.To overcome this risky and time consuming process, digital automatic change detection procedure represents a turning point.From the above, the need to document and detect changes within a scene through automatic procedures has become essential.
In this paper, the whole process of extracting changes along time is outlined, describing in detail the steps of the processing in section 4Section 4. It will be demonstrated in Section 5 that the output maps from the change detection process are GIS ready, suitable for decision making and for further investigation by the archaeologists and restorers.The application of the proposed methodology, adopted within the case study of the ortho-rectified images of bas-relieves, proved to be suitable to recognize the major classes of defects and to evaluate the conservative state of the bas-relieves, in a more reliable and objective way.

OPERATIVE BACKGROUND
As stated in the introduction section, the main purpose of this work is to perform a change detection procedure able to provide information about the changes of a surface in different époques of acquisition.However, from the literature study performed by the authors, emerged a lack of well established pipelines of work suitable to be used in the archaeological domain and, deeply, performed with bas-relieves surfaces.While change detection is a common practice to extract information at a geographical scale (for instance with LU/LC thematic maps (Mancini et al., 2015)), its use at a smaller scale for detecting changes of archaeological findings has not been experienced so far.For this reason, it is useful to share with the readers some related works dealing with change detection in the CH domain and to briefly describe the study area of the present work.In the following both aspects are outlined.

Related works
The topic of understanding the dynamics of changing is not recent itself.The work described in (Burroni et al., 2002) for example, introduced a method to study the degree of change affecting a prehistoric context as the result of environmental processes.However, it was based on the direct examination of a representative sample of stone tool by-products, and on the identification of all surface alteration features.The main issue arose by the authors was concerned with the automation of the diagnosis process.Towards this direction, it is worth to mention the work by Appollonia et.al (Appolonia et al., 2006), where image processing tools have been used to improve and make semi-automatic the study of chemical decay causing visible changes in colour of some regions.This study can be considered a turning point in this field, since it makes a large adoption of geomatic techniques for in depth analysis of ancient artefacts.As a demonstration of this, in (Atkinson et al., 2010) geomatic techniques were broadly adopted to extract information like magnitudes (linear dimensions and surface areas) of the pathologies of both monuments (cracks, collapses, damp and deterioration of stonework).Advanced techniques for the radiometric and geometric analysis have been experienced in (Barazzetti et al., 2010).The work describes the analysis of radiometric contents using multi-spectral images and the 3D surveying of a painting for its geometric deformation analysis along time.The results demonstrate how noninvasive surveying techniques and evaluation methods for CH objects are fundamental since, by adopting these common technologies and techniques, museums and insiders can improve interventions policies.In (Crespo et al., 2010), a methodology for the documentation and analysis of historical buildings is presented.By integrating TLS data with image based analysis, allowed obtaining thematic maps with the size and position of damages.A similar experience can also be found in (Appolonia et al., 2015).
The literature also provide some experiences more closely related to the archaeological domain.In (Levin and Filin, 2010), a three-dimensional image-based change detection model that integrates the stochastic nature of the observations into the analysis is described.In the proposed method, the detection of changes is based on outlier analysis which, according to the authors, overcomes the limitations of a point-to-point differencing or of segmentation and classification of the data.In some other cases, the aid of TLS was adopted to understand the changes over time of rocks (Alba and Scaioni, 2010).However, further sensors like Near Infrared (NIR) Cameras were needed to filter out vegetation from the captured data.Dating back to more recent findings, it is worth to cite the work of Cerra et.al. (Cerra et al., 2016) since it points out the necessity of using change detection techniques in this field.In this paper WorldView2 images have been used to detect the changes occurred during the civil war in Syria, underlining the importance of analysing valuable heritage sites with all the techniques available.
According to (Lu et al., 2004), good change detection research should provide the following information: i) area change and change rate; ii)spatial distribution of changed types; iii) change trajectories of land-cover types; and iv) accuracy assessment of change detection results.When implementing a change detection project, three major steps are involved: i) image preprocessing including geometrical rectification and image registration, radiometric and atmospheric correction, and topographic correction if the study area is in mountainous regions; ii) selection of suitable techniques to implement change detection analyses; and iii) accuracy assessment.The accuracy of change detection results de- Given the above and in line with the recent research trends, here below we propose our approach of change detection, not before having described the case study in which the procedure has been tested and validated.

The case study
In South America, the north coast of Peru offers interesting archaeological testimonies, of the ancient kingdom Moche, which lived in the area near the city of Trujillo between 100 a.c. and 700 d.c., located in Moche's Valley in Peru (see Figure 1).The Huaca de la Moon is a masterpiece of human creative genius.Its 12,000 square meters of polychrome murals are a prime example of the raw earth constructive technology typical of Moche cultural.It has no reference in the world in terms of the design and complexity of its murals.Its iconographic and aesthetic wealth is of an exceptional universal value that responds to a pattern that was handled during seven centuries.It is also an exceptional testimony to the Moche cultural tradition because it is the embodiment of political power through religious ideas, where the temple, the most sacred space in the communion between man and his gods, is associated with the sacred mountain (named Cerro Blanco), to reach the world of divinities and divinity itself.The Huaca de la Moon is actually an architectural complex composed of two sacred temples with the shape of a truncated pyramid, called Old Temple and New Temple, built entirely of raw sand, where only privileged people entered, either to worship their gods or to give them their lives (Benson, 2012).The geographical position and the detailed aerial view of the of the location of the architectural findings is reported in Figure 1.
The methodology have been tested in one of the well conserved bas-relieves that can be found within the temple, representing the god of the mountain.The bas relieves belong to the Moche culture (as it's visible from the Figure 2), were richly decorated and painted with different colours, using pigment of animal and vegetable origins.The temple was widely painted with various colour themes: black, red, sky blue, white and yellow.The action of the sun and the weather condition facilitated the decay of the painted walls.
The research activities conducted to analyse the state of conservation of these invaluable masterpieces are in line with Archaeological Project of the Huacas del Sol and de la Luna proposed as its fundamental pillars the investigation, conservation, conditioning and social use of the monument, respecting the authenticity of the cultural context.Although this persistent and rigorous work may not be evident at first glance, it is present in everything, from architecture with its extraordinary reliefs and mural painting, to the various artifacts and tiny objects found in archaeological excavations.The achievements of the Project have not only transcended the academic and scientific world.Their contributions are also significant in the local and national community, arousing a special interest in the values and conservation of their cultural heritage.Most of the Project workers are neighbors of the traditional Moche countryside and with their hard work take pride in being actors in the discovery of the majestic monuments and artistic achievements of their ancestors.Why is it investigated?To expand our knowledge of the archaeological site and Moche culture, learning about past societies, publishing information, creating replicable management models in other archaeological projects, creating social awareness and strengthening local and regional identity.Why is it preserved?To preserve, display and perpetuate the archaeological site, as part of our cultural heritage, and its surroundings.The Archaeological Project applies an interdisciplinary research policy and methodology to ensure the correct recovery of scientific information and the immediate conservation of archaeological remains.

CHANGE DETECTION WORKFLOW
Detecting regions of change in images of the same scene taken at different times is of widespread interest due to a large number of applications in diverse disciplines (Malinverni and Fangi, 2009).Despite the diversity of applications, change detection researchers employ many common processing steps and core algorithms (Radke et al., 2005).The goal is to identify the set of pixels that are "significantly different" between the images of   (Radke et al., 2005).Prior to go in deep with the technical details, the readers might found useful the overall description of the workflow (testes within the framework of the described study area) stated in this section and described in Figure 3.
The first step of the methodology consists on the creation of the point clouds of the bas-relieves, processing two sets of images acquired in two different époques: 2006 and 2012.For each époques, once the Digital Surface Model (DSM) was obtained, the ortho-rectified image was produced, exploiting the advantages of, respectively, the third dimension and the high resolution.Both the products have been submitted to an image pre-process in order to make the datasets of the different époques comparable.More in deep, image alignment and resampling were performed.
For achieving a reliable extraction of the information about the colour changes over time, a supervised classification based on the common maximum likelihood algorithm was applied.Thus, the images were divided into different classes thanks to whom a first change detection between the two époques was done.Thanks to the hard thematic map, obtained by means of the pixel based classification comparison, we had in output a general overview about the most degraded areas.Anyhow, the hard thematic map gives only a general overview of the degradation but it doesn't analyze in detail the colour decay of each pigment in different areas.In order to determine the region in the images where one can localize and quantify the degradation, the process of segmentation has been performed using the ortho-rectified images and the DSM (Sturari et al., 2017).The BAATZ segmentation algorithm divides the image into regions with common characteristics based on key features extracted from the dataset (e.g., geometrical or colorimetric).Following, we can perform an hybrid comparison assigning representative classes from the per pixel classification by the Winner-Takes-All (WTA) approach.It is used to exploit the pixel-based results in classifying the objects coming from the previous image segmentation, thus providing a refined object-based classification.The approach sorts out meaningful segmented regions into thematic classes in terms of membership class percentages.A classification according to the predominant class attribute can provide a "object" thematic map.This process avoids the residual "salt and pepper" effect of a pixel-based classification and improves the result in terms of spatial consistency, semantic representation and GIS ready production, more comprehensible for the end user.

Input Images Processing
The data set of images has been acquired in different époques (in 2006 and 2012) and in both cases the surveys were unprogrammed and low cost.Hence, the acquisitions are not well-structured, since they arise from different digital camera with various characteristics.In the 2006 the images of 3648 x 2736 pixels were shot by a Panasonic Lumix DMC FZ50, focal length 7.4 mm, 10.1 megapixel high resolution CCD.On the contrary, in the 2012 the set of images came from a Canon Power Shot SX30IS digital camera with focal length of 6,614 mm, providing a maximum resolution of about 14.1 megapixel and images of 4320 x 3240 pixels.These variability required a preprocessing stage to create two DSM and the relative orthoimages with comparable characteristic (Ground Sample Distance (GSD) 2.428 mm) to perform the following change detection analysis, using some reference points, given from archaeological books and well distributed on the images, so putting in real dimension the investigated area (11,64 m 2 ).Unfortunately, the radiometric pre-processing did not avoid completely the variability (light and shadow) inside the image itself.Moreover, this problem does not give in some part (lateral) of the orthoimage a reliable DSM, which depended on a low automatic cloud points.Some gaps remain and make less robust the results.For this reason and to perform a reliable comparison we selected only the central area (3.843701 m 2 ) of the orthoimage to apply the procedure (see Figure 2).

Classification, Segmentation and Change Detection
Automatic classification has been carried out according to two approaches: a pixel-based and an object-based method.Pixel-based classification aims to identify the classes by means of the spectral information provided by each pixel belonging to the original bands.The object-based approach operates on sets of pixels (objects/regions) grouped together by means of an image segmentation technique.At the first step to perform a supervised hard classification we have provided a training data set referred to 5 classes in relation to the colours present on the bas relief (red, white, yellow, blue, gray).Applying a Maximum Likelihood algorithm on the visible bands of each orthoimage we obtained, at first, a per pixel thematic map, then improved adding the surface information by the DSM (Figure 4) during the classification process.
Subsequently, after a salt and pepper filter to reduce the noise of the sparse pixel, the crossing of these maps with the control data set produced the confusion matrices used to assess the per pixel classification accuracy.The per pixel classification gave good results in terms of Overall Accuracy (OA) without DSM (96,55 % in 2006, 89,33 % in 2012).These values can be found in Tables 1  and 2. But the improvement of the comparison between 2006 and 2012 data classification comes from the use of a classification that includes the depth information, that allowed us to reach an OA of 96,70 % in 2006 and 93,29 % in 2012 (details are reported in Tables 3 and 4).A detailed class assessment can be provided also by comparing the Accuracy (Producer Accuracy) per class in row and the Reliability (User Accuracy) per class in column, where the "gray" class is confused with the other ones and it could be indicate the colour decay of the pigments.
At this stage it is already possible to infer some information about the color changes among the years.As it's visible in Table 5, the  The per pixel classification represents the input to the following test, that is WTA procedure (interested readers can rely on the procedure explained in (Malinverni et al., 2011)).The results of pixel based comparison were thus reported to the corresponding region to highlight the percentage of changing within every area (see Figure 5).
The object attribute assignment gave new thematic maps in vector form with the correlated tables that provide information for each classified polygon about its membership class percentages, which is useful in evaluating the accuracy in terms of stability assessment.This approach is required, in order to perform a correctly segmentation algorithm which defines the entities corresponding to real part of the objects.Another major contribution of this

RESULTS AND DISCUSSION
The hybrid classification (both pixel and object) described in the previous section, even if performed on a limited portion of the bas relieve, can help to get some interesting outcomes.As it's visible in Figure 6, in green are outlined regions where happened the changes, mainly concentrated on the lower part of the bas relieve.Furthermore, the stability maps is obtained by putting the  class that doesn't change in Yellow, while the Red polygons are the unstable or changed classes.The stability map supports the reliability of the hybrid classification procedure and also helps to perform the visual interpretation approach, assigning nonhomogeneous classes in an automatic way (Figure 7).Thus, it becomes a valid instrument that drives the user in the degradation map consultation, indicating which polygons are critical areas in terms of heterogeneity and providing the segments candidate for future evaluation.
The differences between the two classifications could be explained by the different data input despite the pre-processing performances on the spectral aspects of the raw images.Sometimes, the polygon not correctly partitioned can produced a mistake in the class assignment when the dominant class percentage is derived by the pixel presence inside the polygon too.If there is confusion (presence of different covers) and the polygon includes more areas, this will be a problem for the correct definition of the class.Both approaches have some disadvantages: the object-based approach is heavily influenced by the quality of segmentation results, while the pixel-based approach, which can exploit only spectral features, might result in errors when it is difficult to separate spectral response of similar colour (white vs gray).
Following the workflow we achieved a classification that can provide an useful tools, a starting point for the restorers.

CONCLUSION
The research work described in this paper demonstrates how digital products arising from photogrammetric acquisitions allow one to extract useful information about the decay and degradation of bass relieves in archaeological settings.Despite the data sources were collected with unplanned surveys, the classification, with the different methodologies described, proved to be a valuable solution useful to understand which are the main changes over time.Obviously, the more the information is frequent and precise, the more the classification is accurate and reliable.
The use of the depth information improved the classification quality; it is hence required the production of a reliable DSM to assess a more reliable classification of the surfaces.Working on an object classification can be considered a step forward with respect to a per pixel approach, since it allows one to work on single classes or zones of the bas relieve.Working only of the pixel based classification in fact, mainly due to the fact that the input images were not perfectly aligned and calibrated, the noise effect produced by isolated pixel could affect the quality of the maps.With the WTA procedure, this effect is avoided, and the output product is more readable.The only drawback with this latter approach is that, in the detailed portion of the surface, we might loose some precise information.
The procedure described in the paper represents a first step of a research that could take into consideration the intervention of experts (like archaeologists and restorers) to understand the potential of such approach in the restoration domain.Up to now, in the literature there are few examples and available data sets to compare this workflow and validate it's effectiveness for archaeological purposes.By the way, in the future we are planning to extend the pipeline of work also for other datasets of nearby bas relieves, providing the archaeologists and the research community with up to date datasets that can help to deepen the understanding of color changes or decay of archaeological goods.

Figure 1 .
Figure 1.Geographical location of the site.Top image represents the territorial framework, the bottom image represents the ortophoto with the two main Huacas in the area.pend on many factors, including: precise geometric registration between multi-temporal images, calibration or normalization between multi-temporal images, availability of quality ground truth data, the complexity of landscape and environments of the study area, change detection methods or algorithms used, classification and change detection schemes, analyst's skills and experience, knowledge and familiarity of the study area, and time and cost restrictions.

Figure 2 .
Figure 2. The image represents the orto-rectified image of the bas-relieve using the set of images of 2006.The dashed blue line depicts the portion of the entire surface that has been used for the experiments.

Figure 3 .
Figure 3. General workflow performed to achieve change detection information about the bas-relieves analysed different époques; these pixels comprise the change mask.Estimating the change mask is often a first step toward the more ambitious goal of change understanding: deciding and detecting which changes are considered significant and which are considered unimportant are difficult problems and vary by application(Radke et al., 2005).Prior to go in deep with the technical details, the readers might found useful the overall description of the workflow (testes within the framework of the described study area) stated in this section and described in Figure3.

Figure 4 .
Figure 4. Per pixel thematic map obtained from the combined use of ortho-rectified image and DSM of the 2006 image.

Figure 5 .
Figure 5. Object thematic map coming from the selection of per pixel classification into the segmented regions

Figure 6 .
Figure 6.The hybrid classification comparison by green region map between 2006 and 2012 displayed on ortho-image.

Figure 7 .
Figure 7. Stability map based on WTA classification: yellow polygons are stable while red ones are unstable.

Table 1 .
Per pixel classification confusion matrix based on the orto rectified image of 2006 acquisition.

Table 2 .
Per pixel classification confusion matrix based on the orto rectified image of 2012 acquisition.

Table 3 .
Per pixel classification confusion matrix based on the ortho-rectified image of 2006 acquisition, including the depth information (DSM)

Table 4 .
Per pixel classification confusion matrix based on the orto rectified image of 2012 acquisition, including the depth information (DSM)

Table 6 .
Region based comparison between 2006 and 2012 acquisitions.