FUSION OF TLS AND UAV PHOTOGRAMMETRY DATA FOR POST-EARTHQUAKE 3 D MODELING OF A CULTURAL HERITAGE CHURCH

Nowadays, the necessity of heritage documentation is essential for monitoring, maintenance, and understanding needed for conservation. The survey phase has been considerably improved using cutting-edge technologies such as Unmanned Aerial Vehicles (UAV) and Terrestrial Laser Scanners (TLS). Both of these technologies have been applied in heritage documentation individually or combined. Heritage documentation in a post-natural disaster is a situation that requires rapid data acquisition on a hazardous field. On 12th of June 2017 an earthquake (Mw=6.3), south of Lesvos island, Greece occurred, which was devastating for the Vrisa village destroying, among many other buildings the main church. The Greek State decided from the first moment to restore the whole village, which was proclaimed as a “traditional settlement” since 2002, in its original place starting from the church and the school due to the symbolic meaning that those have to a local community. For this purpose, a 3D model of the church was requested by the authorities for damage assessment. In this paper TLS and UAV photogrammetry has been used in an integrated design to rapidly facilitate the acquisition of the whole church, eliminating all possible occlusions. The TLS was exploited for the acquisition of the facades while the UAV was used for the acquisition of the roof. The recent improvement of the post-processing algorithms provided the ability to implement the fusion of TLS and UAV models and deliver an accurate 3D model of the whole church the same day that the survey was conducted. * Corresponding author


INTRODUCTION
The necessity of recording cultural heritage buildings is extremely important (Letellier et al., 2007), especially after strong earthquakes that cause damages to significant constructions such as monuments, heritage buildings, and landmarks.Traditional methods for damage assessment are time-consuming, and many times the authorities are unable to access due to extensive damages.New 3D documentation technologies can help to achieve accurate and precise results while surveying complicated constructions and significant cultural monuments destructed by an earthquake.
Terrestrial Laser Scanners (TLSs) can provide high-quality 3D models of damaged buildings after an earthquake.TLS's benefits are the collection of precise data, in decreased field time, while there is no need to touch the measured object, which in some cases is not possible or desirable.Additionally, Unmanned Aerial Vehicles (UAVs) can provide very high spatial resolution and high-quality images of the area after an earthquake, using a mounted high-resolution digital camera.Aerial images can be shoot vertical or oblique during low height flights with increased safety for the operator.The processing of these images with specific algorithms can provide very accurate 3D models.In particular, the Structure from Motion (SfM) algorithm is frequently used in terrestrial and aerial photogrammetry when the results must be produced quickly (Westoby et al., 2012).
The emergence and evolution of new technologies such as TLSs and UAVs, help to acquire data that, after processing lead to accurate and precise 3D models of complex objects.However, no single sensor can acquire complete information of a large object even if several multi-surveys are applied (Xu et al., 2014).The integration of TLS and UAV photogrammetry techniques allows obtaining comprehensive models of complex objects by using each technique in contexts, presenting the optimal operating conditions.The methodological development for data acquisition, along with the ever-increasing computational power that has reduced processing times, offers great potential for the rapid modeling, even of whole areas.The raw data derived from the above technologies can be combined for the creation of 3D models that are dense enough comparing to the representation of the real object.The two technologies interact and complete each other on covering the possible occlusions and can be applied in emergency situations such as an earthquake, where the rapid and efficient record of the current state is crucial.This paper focuses on the integration of TLS and UAV photogrammetry to rapidly facilitate the acquisition of a cultural heritage church, eliminating all possible occlusions, after an earthquake in order to help the authorities for its damage assessment.The TLS was used for the acquisition of the facades while the UAV was used for the acquisition of the roof of an 18th-century church at Vrisa village in Lesvos island that was severely damaged during the 12 th June 2017 earthquake.In a small traditional settlement like Vrisa, that was hit by a natural disaster, beyond the material damages, there was an impact on the psychology of the local community.Landmarks like churches and schools have symbolic meaning for the residents.Those landmarks have priority at the restoration and reconstruction of a settlement, so as to give the message of a return to the old -before the earthquake-everyday life of the community.et al. (2016) mention that for many applications the combined use of data taken from terrestrial an aerial sensors seems interesting.From the airborne perspective, the upper parts of objects including roofs can be observed, while the terrestrial perspective can complement the data from lateral views to retrieve a complete visualization of interesting areas.In the past decade, several methods have been developed for the geometrical fusion of the two types of datasets for better 3D mapping in various applications (Wu and Tang, 2015).Tong et al. (2015) proposed a practical framework for the integration of UAV photogrammetry and TLS for 3D mapping and monitoring of open-pit mine areas.Documentation based on integrated ground-based survey methods and UAV photogrammetry have confirmed its leading role for analyses and further interpretations in archaeology (Balletti et al., 2015).In another project, the integration of both techniques delivered products such as maps, orthophotos and 3D models of the archaeological park of São Miguel Arcanjo, and the ruins that are there (Reiss et al., 2016).Those products are essential information to help the stakeholders act efficiently on preserving the archaeological artifacts, natural resources and other around the park area.Chiabrando et al. (2016) showed the usefulness of the combination of TLS and UAV applied on landscape and architectural assets.Wu and Tang (2015) suggest that the fusion of data derived from TLS and UAV photogrammetry provide with more potential in various applications than that provided using a single type of data, based on the respective advantages and disadvantages of the two types of datasets.

Post-earthquake response
It is evident that in a typical survey the fusion of UAV imagery and TLS data offers only advantages, in cases where both needed.However, for emergency surveys, where there are constraints such as time and security, a different situation occurs (Zaragoza et al., 2017).There, for example after an earthquake, it is necessary to collect data for damage assessment, in the most unfavorable conditions, where many areas may be inaccessible, and the crucial concern is safety (Dominici et al., 2017).During emergency surveys, due to safety issues, it may sometimes is required the application of TLS and UAV methodologies beyond best accuracy boundaries, in order to obtain a basic 3D model as comprehensive as possible of the scene (Zaragoza et al., 2017).
On 12 th of June 2017 (UTC 12:28:38.26)a magnitude Mw 6.3 earthquake occurred offshore Lesvos Island in SE Aegean Sea, caused one fatality, and partially ruined the village of Vrisa on the south-eastern coast of the island (Kiratzi, 2018).Vrisa village proclaimed by the Greek State as a "traditional settlement" according to the provisions of the Presidential Decree 208/2002, because, apart from its overall architectural interest, it had remarkable architectural and morphological features and was an excellent example of local folk architecture.
Immediate has been the mobilization of all civil protection agencies to address and manage the consequences of the disastrous phenomenon.Amongst other actions, was the association to a team to undertake a 3D visualization of the whole village, in order to help the local and national authorities for the damage assessment.It was essential to emphasize on buildings that have symbolic meaning for the society of the village.One of those buildings that suffered from severe damage, due to the earthquake, was the main church of the village.
The main goal of the present study was to create a highresolution 3D model of the church.For that purpose, the combination of UAV photogrammetry and TLS was necessary to utilize the complementary characteristics of both methods.There was a need for a rapid survey in order to give a full visualization of it to the relevant bodies so they can assess the damages and manage its restoration.The crucial part of the process was to create a methodology that would lead to optimum 3D visualization results, rapidly but at the same time taking into consideration factors like accessibility and safety.

Study Area
The study area of the present paper is the church of Zoodochos Pigi in Vrisa village (Figure 1).In 1723, the first part of the church erected in the place where the central church is today, which was built in 1803.The style of the church is a typical lesbian architectural style of the 19th century (Metropolitanate of Mytilene, 2018).Due to the geographical location and orientation of the church, recording only by terrestrial methods was insufficient because of many blind spots, especially on the roof.The use of both TLS and UAV photogrammetry, was necessary, due to the volume of the church, so as the whole study area was fully covered.

Workflow
The study area was characterized as high-risk area due to readyto-collapse walls.The first step of the workflow followed was to determine the specifications of the survey.Thus TLS used to scan the facades of the church from a distance, while the UAV used to acquire photos that will eliminate the occlusions from the terrestrial survey such as the roof of the church.In order to achieve a rapid way to collect data, the relevant flight plans and the scanner positions was chosen so that with the minimum number of scanner positions and the minimum number of aerial photos the entire study area was covered.Subsequently, the raw data of the two methods were processed, to lead to two individual point clouds.Then the contour of the roof was extracted from both point clouds.The contours used firstly for the comparison of two point clouds and then to assist on the fusion (registration) of them.The complete workflow that was followed is shown in Figure 2.

Equipment:
For the terrestrial scanning of the church, a FARO Focus 3D TLS was employed.The scanner uses laser beams to produce detailed 3D representations of complex environments and geometries in only a few minutes.It is based on the phase-shift principle where the measurement of distance relies on the difference in the signal between the emitted and returning laser beam.Focus 3D has a range of 120 meters, a field of view of 360º horizontal and 305º vertical and measurement speed up to 976k points/sec.Scanner's main characteristics are the high accuracy, high resolution, high speed and also the small size and lightweight.Furthermore, Focus 3D is integrated with a digital color camera, dual axis compensator, digital compass, and altimeter.

Survey:
The survey for the acquisition of data took place on 21th of June 2017 (Figure 3), few days after the earthquake.First of all, in a TLS survey, it is essential to establish an optimal distribution of scanner positions to cover the threedimensional space or object to be scanned fully.In particular, positions should be well distributed to cover the entire desired area, and no obstructions should be placed in the line of sight of the object of interest.It is also necessary to control the range of the scanner with respect to accuracy, since the further the distance from the object, the lower the resolution and the accuracy of the final product will be (Quintero et al., 2008).In emergency surveys, there is a need for avoidance of dangerous places (e.g., ready-to-collapse walls).In all surveys the fewer the scanner positions, the better will be regarding time needed for fieldwork.
After selecting the scanner positions, an important step in the measurement methodology is to determine the location of control points used to merge the scans and/or the georeference of the three-dimensional model.These points are usually circular or spherical targets, which are automatically recognized by point cloud processing software.The control points can be used either for the registration of the individual scans or as points for quality assessment.
The method used for scanning the building was open traverse, which starts at first point proceeds to another and ending at an unknown position.The traverse techniques are used for the control of small sites.As Andrews et al. (2009) mention, the loop passes around the perimeter of the site with stations sited so that radial detail shots are also possible or so that spurs can be set out to cover detail work with the minimum number of additional stations.Usually, the open traverse involves linking each scan to the previous scan by moving artificial targets such as spheres or planar checkerboards.
A total of thirteen scan positions distributed around the site (Figure 4) and the open traverse method employed in order to avoid some obstructions like debris, trees or anything that limited the field of view.Two of the chosen scanner positions were on the rooftops of small buildings located around the church.Also some black and white checkerboards targets were placed, of 18x18 cm dimension, so as to be used as control points.These targets were placed around the building in such way that they were visible in multiple scans.Focus 3D was set on a tripod, then it was leveled and centered at all positions of scanning.The field of view of the scanner was set at all positions (Figure 4), considering that there is enough overlap (more than 30%) between two consecutive scans.An overlap of 30%-50% between adjacent scans ensures to generate a very accurate 3D laser model (Guarnieri et al., 2017).The resolution that was employed was ¼ which means that the point density (spatial resolution) is 6.13mm at 10m.As we move further away from the object the point distance increase, so at 20m the distance between each point will increase to 12.26mm.The selected quality was set to 3x which means that every point was fired three times by the laser beam for better measurement.Finally, the total time for scanning was around seven minutes for each scan and about two hours for the whole survey.

Equipment:
For the aerial survey a Phantom 4 manufactured by DJI was used.The quadcopter has an integrated camera with a CMOS (1/2.3 inch) 6.17mm width and 3.477mm length sensor and 12Mpx of resolution.The camera lens is a prime f/2.8 which focus at ∞ while it creates a FOV of 940.The camera is mounted onto a 3-axis gimbal stabilization system with adjustable sensitivity and accuracy of ±0.02°.The Phantom 4 has GPS, GLONASS plus and high precision ultrasound system combined with an optical recognition system.This combination allows high precision altitude measuring up to 0.1m till 10m.The optical recognition sensors are also responsible for the obstacle avoidance system.The drone accuracy position is ±0.5m vertical and ±1.5m horizontal by only using the GPS and GLONASS.

Survey:
The gimbal system allowed the change at the inclination of the camera from 0° to 90° during the automated flights.The speed of flight was predetermined at 1.5m/sec and the maximum flight time was 8 min.The complexity and the state of the structure of the building led to the combination of two different types of flight plans (Chiabrando et al., 2017).
The first on with a nadiral configuration of the camera and the second with an oblique configuration.The flights were autonomous where the autopilot was in charge of following the plans as designed.In order to acquire the most stable photos, the flight plan was programmed to hover at a waypoint and capture the photo (Eisenbeiss, 2009).This method extended the flight time and the waypoint density.For the nadir images, the operator created a polygon (Figure 5-right) at the application basemap surrounding the church building.For the acquisition of the oblique type of images a point of interest was set to the center of the roof (Aicardi et al., 2016) of the church and then the course of the drone was onto the radius of 15m (Figure 5 left).Because there were trees close to the church and the narrow path on the east-north side the same course was applied to an 18m radius so as to have a clear view of the object.The flight parameters of all three flights are shown in Table 1.No markers for tie points on the building were placed before the flight because of the unstable state of the church.The total flight time was 18 minutes, and 240 photos were acquired, sufficient number for that kind of structure (Fiorillo et al., 2013).

DATA PROCESSING
The processing stage was divided into two major parts, the point clouds creation from the UAV photos and TLS and the part where the fusion of the point clouds was processed.
The data collected from the thirteen positions using TLS processed with FARO SCENE software (FARO, 2018).Registration is the first step of processing in order to allow these multiple scans to be brought together into their correct position and relative to each other.The technique used was the known as cloud-to-cloud registration.Cloud-to-cloud registration does not require any use of targets but clouds are aligned to each other by extracted features within the overlap area of each cloud.This method of registration needs proper geometry and overlap between scans (Liscio et al., 2016).Thus, the thirteen point clouds aligned to each other by applying an algorithm based on the Iterative Closest Point (ICP) algorithm (Besl and McKay, 1992).The ICP algorithm checks the distances between all the points of the point clouds and estimates the transformation to align all sets thus resulting in a minimal error.After the registration, the points were colored using the photos from the scanner's integrated digital camera.
For the photogrammetric process, the commercial software ContextCapture (Bentley Systems, 2018) was used.The software processing algorithm is mainly based on the Structure from Motion (SfM) algorithm that make use of Scale-Invariant Feature Transform (SIFT) extracted features to create 3D models from a series of overlapping photos (Strach et al., 2017).At first, the photos were aligned, and the type of photo set was parameterized as an aerial structural data set, then the alignment calculates the location of each photo in 3D space (Figure 6).The calculation is done using geolocation, orientation, and visual algorithm of point matching between images (Tariq et al., 2017) Also the internal and external orientation of the photos was computed due to the GPS metadata on the photos.Afterwards, a colored point cloud was created which represents the geometry of the building (Turner et al., 2012).The spatial reference system was the same as the drone's GPS which is WGS84.
Figure 6.Projection centers as a result of low height circular flight and the tie points identified.
On the two individual point clouds two filters were applied: a) a Statistical Outlier Removal, which rejects points based on two parameters: firstly, it considers k nearest neighbors for each -k and it computes that way the average distance of each point to its neighbors and then it rejects the points that are farther than the average distance, and secondly a number of times the standard deviation (PCL, 2018) and b) a Noise filter, which locally fits a plane around each point of the cloud then removes the point if it's too far away from the fitted plane (CloudCompare Wiki, 2018).
The next step was to crop the two datasets to the desired study area, removing on the same time unwanted objects like vegetation, nearby buildings and other objects which could affect the alignment process.After that procedure, the contour of the perimeter of the roof extracted from both datasets (Figure 7), using the algorithm proposed by Hackel et al. (2016).The contour derived from the TLS point cloud was used as a reference and the Hausdorff Distance (Aspert et al., 2002) compared to the one derived from the UAV cloud was calculated.For each pixel in the segmentation result, the minimum distance to the reference contour was calculated.
Then the maximum of this set of minimum distances was used as a comparison metric for the quality assessment of the two point clouds.Another quality assessment undertaken was the comparison of the dimensions measured of the checkerboards with the real dimensions of them.Next and last step of the data processing phase was the registration of both datasets.Firstly a match procedure of the two datasets took place using the extracted contours, in order to assist the registration algorithm.Secondly the ICP algorithm registration algorithm was used for the registration and thus the fusion of the two datasets.The open source software CloudCompare (Girardeau-Montaut, 2018) was used.A cohesive point cloud was created that represented the geometry of the study area.

RESULTS
The registration process of TLS data, due to enough overlap between pairwise scans (>27,7%), produced a very accurate dense point cloud of 114M points.The Mean Registration Point Error was 1.9 mm, and the Maximum was 3.9 mm.The two filters cleaned the point cloud, and along with the cropping procedure, reduced the number of points to 85M.
For the UAV photogrammetry process, 103,713 tie points (Figure 6) were automatically found, which helped the alignment of the 240 photos.The Median reprojection error was 0.36 pixel and the Root Mean Square (RMS) of reprojection errors was 0.51 pixel.The RMS of distances to rays was 0.006m.The point cloud that was created from the UAV photogrammetry consisted of 89M points.The two filters cleaned the point cloud, and along with the cropping procedure The fusion of the two datasets conducted by registering them with the ICP algorithm.The was based on 50K random sampling points, and the RMS was 2.04 cm.The final product was a dense 3D point cloud of the church (Figure 8).Obviously, the results were visually expected: UAV had a few blind spots on the roof, especially due to vegetation and many on the facades where on the contrary TLS had blind spots viceversa.
Figure 8.The final 3D model after the fusion of the two datasets.

CONCLUSIONS
In this paper all the steps and challenges addressed, and solutions adopted concerning the survey, and solid modeling from the fusion of TLS and UAV photogrammetry data of a heritage church.The combination of two different techniques has been presented in order to generate an accurate 3D model of a cultural heritage church rapidly, in order to help for its damage assessment, due to an earthquake.The main idea was to merge in a unique result data derived from UAV photogrammetry and from terrestrial laser scanning in such a way that the final model could be impeccably explored.
The results described above need to be verified at a different site.They clearly show a considerable advantage in UAV photogrammetry over TLS at this kind of site, where a large part is inaccessible by terrestrial methods.On the other hand, applying TLS to the surveying of civil engineering sites and the high accuracy of point positioning offered by this method suggest it has an advantage over UAV in terms of accuracy.However, in the posed task, accuracy related only to the surveying method is less important than the position from which observations are made.Given the satisfactory results achieved with the adopted hybrid approach, this paper further demonstrates that these two technologies, can be successfully combined in order to create high-quality 3D recordings and presentations in a short period of time, when they are appropriately employed.The proposed workflow for data acquisition, along with the ever-increasing computational power that has reduced processing times, offers excellent potential for the rapid 3D modeling of buildings after a natural disaster such as an earthquake.

Figure 1 .
Figure 1.NorthEast aerial view of the study area.

Figure 3 .
Figure 3. TLS team members in the study area.

Figure 4 .
Figure 4. Sketch of the church with scanner positions and fieldof-view of each scan 3.2 UAV

Figure 5 .
Figure 5. Flight plan for the oblique photos (left) and the nadir photos (right).Other parameters such as exposure, camera yaw, White Balance, and speed were set.Each type of flight ensured an overlap between consecutive photos at least of 80%.Two software packages were used to design the flight plans: DJI GS pro (DJI, 2018) for the nadiral configuration and Litchi flight app (LITCHI, 2018) for the oblique.No markers for tie points on the building were placed before the flight because of the unstable state of the church.The total flight time was 18 minutes, and 240 photos were acquired, sufficient number for that kind of structure(Fiorillo et al., 2013).

Table 1 .
UAV flight specificationsThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-3/W4, 2018GeoInformation For Disaster Management (Gi4DM), 18-21 March 2018, Istanbul, Turkey The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-3/W4, 2018GeoInformation For Disaster Management (Gi4DM), 18-21 March 2018, Istanbul, Turkey reduced the number of points to 58M.The contour of the perimeter of the roof extracted from both datasets and the comparison of the two contours gave a Hausdorff error 4,32% between them.The comparison of checkerboards' dimensions gave an RMS of 1.4cm.