Multitemporal 3D modelling for cultural heritage emergency during seismic events: Damage assesment of S. Agostino church in Amatrice (RI)

One of the challenging purposes that must be undertaken by applied geomatics, is the need of monitoring by documenting continuously over time the evolution of urban spaces. Nowadays, this is a subject of great interest and study, mainly in case of sudden emergency events that implicate urban areas and specific historical buildings of our heritage. The newest Geomatics technique solutions must enable the demands of damage documentation, risk assessment, management and data sharing as efficiently as possible, in relation to the danger condition, to the accessibility constraints of areas and to the tight deadlines needs. In August 24 2016, the first earthquake hit the area of central Italy with a magnitude of 6.0; since then, the earth never stop shaking in a wide area in the middle of Italy. On 26 and 30 of October, two other big seismic events were recorded (magnitude 5.9 and 6.5) and the already damaged built heritage were struck again. Since the beginning of the emergency all the available resources (human and material) were deployed and the world of researchers is trying to furnish an effective contribute as well. Politecnico di Torino, in coordination with the national institutions, is deploying people, expertise and resources. The geomatics research group and the connected Disaster Recovery team (DIRECT http://areeweb.polito.it/direct/) is part of this process and is working in deep contact and collaboration with the Remotely Piloted Aircraft Systems (RPAS) group of the Italian Firefighter. Starting from the first earthquake the late medieval religious complex of S. Agostino has been carefully monitored and detected, using a multi-perspective oblique imagery strategy with the aim to achieve 3D aerial and terrestrial models, in a multi-temporal perspective concerning three different time situation.


Geomatics for Cultural Heritage damage documentation
The newest Geomatics technique solutions must enable the demands of damage documentation, risk assessment, management and data sharing as efficiently as possible, in relation to the danger condition, to the accessibility constraints of areas and to the tight deadlines needs. Usually multi-temporal analysis for slow evolution of the phenomena characterizing the built environment is monitored by analysing satellite images with high resolution and/or photographic cartography and orthoimages with not very tight time intervals, for example, fiveten years. However, when unpredictable and overwhelming phenomena occurred, the first aid must have available sustainable operational solutions and effective tools to solve the first data acquisition phase, in order to ensure ready-to-use accurate models rich in metric and radiometric information to help technicians in damaged sites in their heavy time-consuming operation of damage assessment. The handling of low-cost sensors with their top efficiency for aerial and terrestrial photogrammetry based on digital imagematching algorithms and with the use of viable post processing times is the best possible compromise between timeliness and accuracy. Thanks to the exponential growth of RPAS (Remotely Piloted Aircraft System) associated with the continuous improvement of computer vision based digital photogrammetry * Corresponding author software and algorithms, in recent years the geomatics solutions provided a great active support to the environment documentation as well as the very close-range documentation of built heritage Georgopoulos et al., 2016;Ruiz Sabina et al. 2015;Remondino et al., 2011;Lerma et al., 2012). In fact, the use of drones equipped with cameras and/or highdefinition video devices, along with on-board GPS systems, have growing its recognized role for the low-cost and non-hazardous aerial documentation techniques for metric survey purposes. They are proving increasingly crucial in emergency situations, not only for remote observation of sites and emergency action planning for the collection of qualitative information, but also for production of very large scale metric data (Wang et al., 2013;Boccardo et al., 2015;Hirokawa et al., 2007). Nowadays that are 3D models from which is possible to extract highly accurate textured data in post processing and to share them on open source web platform for planning collaborative operation. In case of multi-temporal models, in a context of seismic danger, the structural behaviour of buildings can be monitored and many reflections about potential collapse mechanisms can be generated, together with the planning of focused intervention reasoned with technical expertise as structural and geotechnical engineers, restorers, construction managers and directors of conservation.

The seismic wave in central Italy
Between August and November 2016, three major earthquake events occurred in Central Italy. The first event, with M6.1, took place on 24 August, the second (M5.9) on 26 October, and the third (M6.5) on 30 October. This earthquake sequence occurred in a gap between two earlier damaging events, the 1997 M6.1 Umbria-Marche earthquake to the north-west and the 2009 M6.1 L'Aquila earthquake to the south-east. These events occurred along the spine of the Apennine Mountain range on normal faults and had rake angles ranging from -80 to -100 deg, which corresponds to normal faulting. Each of these events produced substantial damage to local towns and villages. The 24 August event caused massive damages to the following villages: Arquata del Tronto, Accumoli, Amatrice, and Pescara del Tronto. In total, there were 299 fatalities, generally from collapses of unreinforced masonry dwellings. The October events caused significant new damage in the villages of Visso, Ussita, and Norcia, although they did not produce fatalities, since the area had largely been evacuated. The area struck by the earthquake sequence is characterised by active fault systems already described in the geological literature published since the 90s of the past century (Galadini & Galli, 2000, 2003. In particular, the Apennine sector between Campotosto (south) and Colfiorito (north) presents normal fault systems trending NW-SE to NNW-SSE, with surface expression longer 20-30 km, made of minor fault sections 5-10 km long. The fault systems and the related fault sections are supposed to represent the superficial expression of seismogenic sources potentially responsible for shocks with M ranging between 5.5-7.0.

The S. Agostino Church in Amatrice
The S. Agostino church is located in the south-east corner of Amatrice and, like the major part of the buildings in the village, was involved in the series of earthquakes that hit the regions of central Italy. During these catastrophic events, a large part of the church has been destroyed (including the entire roof, the right wall and the bell tower). The church was composed by a sandstone façade (currently only partially preserved) in which is inserted a marble portal with Late Gothic decorative patterns (Figure 2 left). The two decorative elements that adorned the two sides of the portal and supported the base of the pinnacles (a bear on the left) and a caryatid (on the right) survived the earthquake. On the portal lintel is reported the year of the foundation of the church (1428) and the coat of arms of the town (Ruggeri, 1995). The rectangular bell tower is located on the right side of the church and a pair of oblong windows with archlets and reinforcing lanceted lesenes is present on each side of the structure. The building, attributed to the architect Giovanni dell'Amatrice, was built next the old city walls by the Augustinians monks. Originally the church was dedicated to San Nicola da Bari, in the first half of the 15th century it was then dedicated to S. Agostino. Inside the church two valuable frescoes (both of uncertain authorship) representing the Annunciation  In 1580 and then in 1781 the building was damaged by a fire and the interior, especially the antique apse, had to be partially rebuilt. In 1703, an earthquake damaged the façade, during the works of renovation a big rectangular window was added.
In the mid-19th Century the vault was entirely rebuilt and also the bell tower underwent renovations to be consolidated. Between 1933 and 1935, during Fascism, the façade was partially altered. It was increased by 4 meters and a rose window was added in place of the big rectangular window of the XVIII century. Moreover, the interior of the building was altered in the second half of the 20th century (between 1963 and 1967). Four lateral apses, two plaster statues next to the altar and the choir were destroyed; furthermore, the valuable organ dating back to 1790 and attributed to Domenico Antonio D'Onofrio of Sulmona was dismantled and removed.

The acquisition campaigns
The first reconnaissance (5 th -9 th of September) performed after the earthquake of Amatrice (24 th of August) allowed to acquire a terrain dataset of the S. Agostino church with some topographic measurements as well. The images were acquired by a Nikon D800E reflex digital camera equipped with a 24x36mm frame. The camera has a 36 Mpx full-frame CMOS sensor with a pixel size of 4.89μm. The images (7360 x 4912 pixel) have been acquired from almost 8 m, then were post processed for balancing light and colours and masked for the next processing steps. The access to the area was, during this first campaign, very short since the area was in the red zone. Only images of the main façade (south), west and belt tower were recorded since in the other parts the entrance was forbidden according to the instability of the church structures.  Just after the first reconnaissance, the Geomatics group of Politecnico di Torino started a very interesting collaboration with the RPAS (Remotely Piloted Aircraft System) group of the Italian Firefighters. According to the fruitful cooperation with the RPAS group, the data acquired with the UAV's of the firefighters were used in order to complete the documentation of Sant'Agostino. The first employed dataset was acquired on the 10 th of September with the well-known INSPIRE 1 (Figure 4) realized by the DJI company (Dà-Jiāng Innovations Science and Technology Co., Ltd Technologies). The system is equipped with the ZENMUSE X5 camera with a focal lenght of 15mm F/1.7-F/16 and a Field of View (FOV) of 72°. The system is able to acquire 4K videos and 16 MP (4608x3456) images as well. For the area of Sant' Agostino a manual flight was performed using the DJGO application. More than 600 UAV images were acquired around the church using traditional nadiral and oblique setting of the camera as well, in order to improve the visibility of the different parts.
After the first event, the church was shore up using the traditional provisional structures realized by the firefighters in emergency areas. As many other structures in the area the building was declared unsafe and controlled with ad hoc analysis and surveys.
The new main event of the 30 October 2016 (6.5 M) increase the damages of the church especially in the belt tower (the top collapsed), in the main façade and in the east part. After this event a new reconnaissance was realized for the 29 th of November up to the 4 th of December and new data were collected (terrestrial and aerial). For the terrestrial acquisition the afore mentioned camera was used again while for the aerial one in this case the Phantom 4 was employed. The Phantom 4 is equipped with a 4K video camera that has a 1/2.3" CMOS sensor, 94-degree field of view, 12.4 MP images. The flight was performed again in manual mode according to the shape of the church in order to acquire sufficient number of images for a correct realization of a complete documentation of the area for photogrammetric purposes. The firefighters RPAS group acquired a new dataset on the 19 th of January, with the INSPIRE 1 platform in manual mode in the same way of the previous acquisition activities in order to obtain nadiral and oblique images. As it is possible to deduce from the description reported above two main aspects need to be highlight. The first issue is connected to the GCPs: the coordinates of some points were measured only during the first acquisition campaign. The reason are connected to the lack of time during the different survey operation that didn't allow to measure new and common GCPs. Actually those GCPs were used for processing all the different flights, nevertheless the Church was seriously damaged, the accuracy on the used points and a first comparison on the achieved 3D models allow to realize an interesting multitemporal analysis. The second aspect is connected to the flight acquisition strategy: unfortunately, a standard procedure were not set up during those surveys. All the flights were realized in manual mode in order to better calibrate the image exposure and focus. According to the achieved results is possible to state the average flights elevation was set up at about 40 m above the terrain. Probably in order to work in the direction of a complete and fruitful multi-temporal application, we are working to realize a standardization of the flight acquisition strategy in order to optimize the nadiral and oblique acquisition and to better define the number and position of GCPs in large blocks of UAV images.

Data Processing
The three dataset were processed in the well-known commercial software Photoscan Pro by Agisoft (http://www.agisoft.com) (Aicardi et al., 2016;Themistocleous et al., 2015;Nex & Remondino, 2014); for point cloud management, optimization, 3D modelling and analysis, commercial 3DReshaper (http://www.3dreshaper.com/) by Technodigit-Hexagon and open source Cloud Compare (http://www.danielgm.net/cc) were employed. In the traditional Photoscan workflow, for each dataset high settings have been used for orientation, bundle block adjustment and densification phases. For the georeferencing phase a set of measured points by topographic techniques were employed in block adjustment. The achieved accuracy (Table1) allow considering the products suitable for a representation at a scale 1:100. In Figure 6 the achieved 3D models according to the different acquired dataset are reported. The main objective of the process was the realization of the high detailed 3D models for documenting, visualizing, mapping and measurement the damages occurred to the structure. Moreover, the orthophotos of the main and the East façade, and those of the bell tower have been produced in order to analyse more in detail the structural condition that affected the church after each earthquake.
As is possible to deduce from the orthophotos of the west façade realized according to the three employed dataset (t1, t2 and t3) showed in the next Figure 7, the influence of the sunlight seriously affect the radiometry of the final output. Despite an image pre-processing that as usual was performed for the acquired images, especially in the DJI sensor is very difficult to correct those effects. Some analysis are ongoing in the research group according to the national and international research on this topic (He et al., 2013;Milanfar, 2013;González-Aguilera et al., 2016) in order to define the better strategy for image preprocessing. Table 1. Dataset, parameters and results for 3 datasets. The t3 (*) dataset, as expected, was affected by some problems related to both georeferencing and serious structural subsidences.

DAMAGE ASSESMENT ON 3D MODELS
The main challenge of this work is to provide and evaluate a metrically controlled 3D model with very high detail. As shown in Figure 8, useful for direct damage assessment and emergency structural intervention for safety condition, produced with rapid solutions based on aerial acquisition by drone and, wherever possible, with terrestrial close-range images. Dense clouds and 3D mesh for each dataset, since georeferenced, have been optimized for 3D managing and analysis, by filtering and segmentation. This has led to a series of multi-temporal analysis on the structure evolution and behaviour and, in particular, on progression of collapse mechanism (Valente & Milani, 2016;Preciado, 2015). The first analysis on 3D models was the estimation of types, amount and accuracy of metric information to be extracted from clouds and triangulated mesh.
To reach this aims, proper metric data and graphic products will be extracted and specific analysis will be performed on: East and West wall; on the main façade in its connection; on belfry connected to East wall.

The East wall
The connection in the South-East corner is maybe one of the most significant enough to be deeply analysed in the global structural behaviour of this ancient masonry. Thanks to the possibility of multi-temporal 3D models comparison (Figure 8) an interesting analysis by orthoimages ( Figure 9) and point clouds comparison expressed in range colours classification ( Figure 10) can be processed. The orthoimages display great displacements on the masonries connection deformed by shear stress (up to 60cm); here the surface assessment was shown, together with the variation from t1 represented in range scale colours between blue and green. The scale values with the order of magnitude of some centimetres distributed over a mean height of 12m, confirms the serious structural damage that hit the building. Nevertheless, this wall is the only one to be still standing together with a part of the main façade.

The West wall
Another focus on the geometric evaluation by multi-temporal comparison that we chose to explore to support masonries stability assessment and contribute to the metric basis for structural diagnosis, is the West wall that, despite the East one, in T3 is not left standing after the subsequent seismic events. On the t1 model it can be traced a widespread lack of planarity of masonry surface (Figure 11a) on the based on a mean plane fit at the basis on wall. 75% of the surface differs from the vertical plan of about -0.06m<v<+0.08m. In Figure11b comparison of t2 on t1 after another seismic event and the collapse of the central area of the wall, in correspondence with the window opening. A series of shear stress traces can me observed, identified and measured with high scale detail on the 3D mesh. A preliminary evaluation of the collapsed volume can be performed too on these kind of models.

The main façade
Damages are consistent on the main façade and are well easy to read, detectable, and quantifiable too on multi-temporal 3D models. With t2 model the collapses occurred in t3 were easily maybe predictable and can be located and measured in relation to the global structural assessment. The key point of the dynamic behavior of the façade can be clearly identified in the lateral connection with transversal walls surrounding the church nave. In this specific focus on the left connection with the West wall ( Figure 14) the progressive evolution of collapse a series of break lines traces were detected in t1 in relation to the ones identified in t2 before the decisive collapse. Figure 14. Orthoimages of the main façade in t1, t2, t3.
Figure 15. 3D model breaklines detection on t1 and t2; the surface comparison between t1-t2 with appreciable differences of about 0.1-0.3m (more or less 55% on the surface)

The East belfry
In the 3D metric model of the bell tower, a significant displacement of the north side can be identified, quantifiable in a mean range value of 0.2-0.3m, clearly pinpointed from the slight sliding of bricks layers, in the masonry texture. Figure 16. Belfry temporal transformation after reiterated seismic shocks: t1, t2, t3. In range colors comparison between t1-t2 with values 0<v<0.50m (grey for values >0.5m)

CROWDSOURCING FOR CULTURAL HERITAGE
The first modern definition of the term "crowdsourcing" can be attributed to Jeff Howe (Howe, 2006). According to this author, "crowdsourcing" represents the act of a company or an institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. Nowadays, it can be identified in the practice of involving many people in small portion of projects using internet.
Crowdsourcing is a consolidated instrument to enhance the efficiency of a project with the use of few traditional resources. It can also provide the contribution of a dynamic community of clients that facilitate the progress of the project. In the last years, the involvement of the users has become necessary and fundamental also in the field of Cultural Heritage. The Convention on the Value of Cultural Heritage for Society, written in Faro in 2005 and ratify by the Italian government in 2013, has focussed the attention in this direction. In Europe, different experiences already exist where the involvement of people was crucial for the digital reconstruction of existing or lost cultural artefacts. The high performances of SFM systems and the availability of digital images from different sources through the web allow to achieve encouraging results. The Project Mosul, for example, was created in 2015 by M. Vincent and C. Coughenour in order to digitally reconstruct destroyed monuments (Vincent et al., 2015. https://projectmosul.org/). The project HeritageTogether (Karl et al., 2014. http://heritagetogether.org/) was created to produce heritage data using the support of the local communities. Many other projects and experiences can be found in the literature (Stathopoulou et al., 2015;Bashar, 2016;Wahbeh, 2016).

Reconstructing the original state of the S. Agostino
In March 2017, we started a new project to evaluate if it would be possible to create a 3D model of the church of S. Agostino through an online campaign of crowdsourcing. In the first phase of the project we wanted to evaluate the type of images that were online available for the church and the quality of the results we can expect. We performed some researches on the major online platforms for the storage and sharing of images and we selected a set of around 20 images. The images were selected depending on size, quality and presence of metadata and were all focussed on the decorated portal of the church. The same solution for SfM (Agisoft Photoscan) was used in this first phase and some encouraging results have been reached (Figure 17).

Some issues, actual results and future progress
The first, expected, issue was related to the nature of the selected images. These photos were acquired mainly by tourists and not for photogrammetric purposes. To overcome this factor a bigger number of images was necessary. The second issue is related with the calibration of the camera: the images were acquired with different sensors and the presence of the metadata is fundamental to perform an automated rough calibration. Moreover, no topographic measurements are available for this original state of the church and we are trying to use portions of the other models to scale and georeferenced this preliminary project. Finally, the images were acquired in different period of the year and in different times of the day: the conditions of light were different for every image and a radiometric correction was needed.
Preliminary results of this test are encouraging. We were able to obtain a good model of the original state of the main portal of the church, before the damages of the earthquakes. The results are showed in Figure 17. We decided to continue the research in this direction and to aim at a complete reconstruction of S. Agostino. For this purpose we'll need a bigger collection of images of the subject. In next months, we will try to create an online platform to facilitate the collection of data and we want to start a communication campaign on the social media to reach more people that could possess images of the church and we will contact the local associations and institutions to see if we can have access to their archives.

Perspectives on the use of these online shared models
The models of the three states of evolution were prepared to be shared on a web visualizer through the 3D HOP package. The online visualizer, showed in Figure 18, was customized thanks to the library of available JavaScript functions. The first customization was the addition of a selection tool that allows the users to choose which model or models visualize. In our case we decided to use some tools that allows to perform measurements on the model (distances and coordinates), some tools for the activation of different modalities of visualizations and a tool that enable the creation and management of section planes on different directions. Figure 18. The online 3D HOP visualizer of the multi-temporal model (left, September; right, February)

CONCLUSION AND FUTURE PERSPECTIVES
One of the next steps of the research can be related to the standardization of the operative workflow for the management of these kind of emergency scenarios. Both for the integrated terrestrial and aerial technical approach on the field survey and in the phases of data post processing and 3D geometrical analysis each step has to be finalized to the Direct support, in view of a multidisciplinary approach to the reading and interpreting, of masonries structural behaviours. On the other hand, the setting up of a clear and rapid workflow for model generation and damage assessment can help the operators that are supposed to work in the first phases of people and buildings safeguard. Moreover, in the next future, it will be possible to further implement the online visualizer and add the model of the original state of the church (when available). The main targets for this part of the project are: the local communities and the experts. We want to provide a digital reconstruction of the damaged heritage to the people living in these territories and, in the meantime, we want to create a tool for the operators that will be involved in the process of restoration and reconstruction of such heritage.