IMPROVING PERFORMANCE OF FEATURE EXTRACTION IN SFM ALGORITHMS FOR 3D SPARSE POINT CLOUD

The use of Structure-from-Motion algorithms is a common practice to obtain a rapid photogrammetric reconstruction. However, the performance of these algorithms is limited by the fact that in some conditions the resulting point clouds present low density. This is the case when processing materials from historical archives, such as photographs and videos, which generates only sparse point clouds due to the lack of necessary information in the photogrammetric reconstruction. This paper explores ways to improve the performance of open source SfM algorithms in order to guarantee the presence of strategic feature points in the resulting point cloud, even if sparse. To reach this objective, a photogrammetric workflow is proposed to process historical images. The first part of the workflow presents a method that allows the manual selection of feature points during the photogrammetric process. The second part evaluates the metric quality of the reconstruction on the basis of a comparison with a point cloud that has a different density from the sparse point cloud. The workflow was applied to two different case studies. Transformations of wall paintings of the Karanlık church in Cappadocia were analysed thanks to the comparison of 3D model resulting from archive photographs and a recent survey. Then a comparison was performed between the state of the Komise building in Japan, before and after restoration. The findings show that the method applied allows the metric scale and evaluation of the model also in bad condition and when only low-density point clouds are available. Moreover, this tool should be of great use for both art and architecture historians and geomatics experts, to study the evolution of Cultural Heritage. * Corresponding author


INTRODUCTION
Point cloud comparison is a method sometimes used to evaluate the metric quality of the 3D reconstruction process in photogrammetry. Finding new ways to perform this method is great of interest when dense point clouds are not available. In fact, point clouds, coming from different primary data and/or techniques (e.g. the ones coming out from photogrammetric survey and the ones coming out from a laser scanning process), could vary greatly in their point densities and their accuracies. This is due to the intrinsic characteristics of the instruments, the sensor size and the distance between sensor and object (Bracci et al, 2018). Especially in photogrammetry working with sparse point cloud happens very often, for example when the surface of the object or scene is difficult to reconstruct because is shiny (Delis et al., 2017), texture-less (Hafeez et al., 2017) or curved (Wong & Chan, 2010). The result of point clouds is also affected by bad illumination conditions (Girardeau-Montaut et al., 2005), the thickness of the object and its transparency. Above all, the way in which the data is acquired can cause noisy results and blunders especially when different platforms or lowcost sensors (Byrne et al., 2017) are employed, due to scale and illumination changes or quality and quantity of single sources (Farella et al., 2019).
Another issue in comparison of sparse point clouds concerns the processing of photos and videos from historical archives to extract metric information. In the field of architecture and Cultural Heritage it is necessary to use these sources for the virtual reconstruction of buildings that no longer exist or have been transformed over time to compare with the current state for documentation and restoration purposes. Photographs and videos stored in national and private archives provide a huge amount of cultural and historic information of great value because they testify the state of buildings, parts of a city and urban environment at a specific time and show famous heritage monuments which were destroyed or damaged. However, these materials have characteristics that make it difficult to implement photogrammetry: lack of basic information that is useful for a photogrammetric process (e.g. type of the used camera), low radiometric quality and resolutions, suboptimal light conditions, and damages due to improper conservation. When the photogrammetric process is successfully completed, the point clouds obtained are not necessarily very dense. Nevertheless they are precious and sometimes unique resource to be exploited metrically for Cultural Heritage documentation. Also in these applications the development of original ways to improve the performance of feature extraction for the assessment of metric quality of the sparse point clouds is of great importance.
The method presented in this study extracts feature points chosen by the operator in one image and after automatically extracted from the other images, as an improvement of the automatic feature extraction processes used in the photogrammetric processing. This procedure could be also useful in case of lack of point clouds to be compared: the presence of some specific known points, selected by the human operator, will allow the correct scaling of the obtained point cloud. This paper is divided into four parts. The first part is concerned with the methodology used for this study and the innovation introduced with the proposed workflow. The second part deals with two different case studies, focusing on two different situations in which the workflow can be applied. The third section presents the findings of the research and finally the fourth part analyses and discuss the results giving some conclusions.

Open issues in sparse point clouds comparison
The metric comparison between point clouds with different density is limited by the fact that the 3D model is constituted by incomplete parts. Consequently, the quality assessment of point clouds is a challenging problem since this 3D representation format is unstructured (Javaheri et al., 2017). To solve this issue, different approaches have been proposed in previous studies: outliers filtering (Hu et al., 2019) and noise smoothing (Wang et al, 2013); automatic filtering procedure based on some geometric features computed on the sparse point cloud created within the bundle adjustment phase (Farella et al., 2019). These studies have focused on finding automatic solutions to the problem. However, in the case of photogrammetric processing of historical archive material, the difficulty of sparse point clouds comparison lies in the fact that there is no direct correspondence between each point in the two clouds (Tazir et al., 2018). It represents a major problem when it is necessary to scale the model for its metric quality assessment. If point clouds from a recent survey are available, the comparison between this point cloud and the one resulting from the photogrammetric processing of historical photographs could fail due to fact that the few points of the sparse cloud do not correspond with the ones of the dense point cloud. Even more difficult is posed when there is no current survey but only historical drawings and archive projects are available from which distances to scale the model can be extrapolated.

Proposed workflow
Considering the open issues previously identified, in this research a photogrammetric workflow is presented to process historical images and to assess the metric quality of the reconstruction on the base of a comparison with a different 3D model. The photogrammetric pipeline chosen as reference in this workflow is the COLMAP (Schönberger et al., 2016) opensource Structure-from-Motion and Multi-View Stereo (MVS) algorithm implementation, developed by ETH of Zurich, (COLMAP, Johannes L. Schoenberger, 2019). Two main blocks compose the proposed workflow. The first one is the standard photogrammetric pipeline in which an additional step of "Feature point selection" (highlighted in red in Figure 1) was added, after the "Feature detection and extraction" phase in order to manually select the tie points to use during the subsequent "Feature matching and geometric verification" step. The second block consists in evaluating the metric quality of the results obtained from the previous photogrammetric process. In order to reach this objective, firstly the cloud-to-cloud distance and then the residuals were estimated. Figure 1. Flowchart of the proposed workflow.

Feature point selection
The detection and extraction of new features from images in COLMAP can be performed in two different ways. The first way is to consider known feature (e.g. single points), importing a text file in which the image coordinates (x, y) expressed in pixel and the scale and orientation information are indicated, writing one line per feature. The second way is the manual choose of the feature points within all the tie points extracted from the automatic "Feature detection and extraction" step in the workflow. In fact, data processed in COLMAP are stored in a customized database that could be easily managed. The previously detected key points are stored as row-major float32 binary blobs, (binary large object that is a collection of binary data stored as a value in the database) where the first two columns are the x and y image coordinates in pixel. Thanks to SQLite it is possible to query the database to choose the best keypoints to use for the matching and the incremental reconstruction. This assisted processing, in both ways, allows the choice and the filter of highlight points such as corners and outline feature. The matching algorithm searches for the selected feature point in each image to estimate the epipolar line in the other images. The result of the matching is a point cloud that, even if sparse, contains the corresponding point to the dense point cloud to compare.

Metric quality assessment
The assessment of the metric quality of the photogrammetric reconstruction was added as final step of the workflow. This evaluation consists in two main procedures: a global comparison between a sparse and a dense point clouds, and a punctual comparison between specific feature points selected in the two point clouds.
The first comparison was performed thanks to the use of the software CloudCompare (https://www.danielgm.net/cc/, October 2019). This open source software allows the comparison of point clouds by estimating their distances through the Multiscale Model to Model Cloud Comparison (M3C2) plugin that uses directions normal to one of the two surfaces to compute local distances and provides estimations of confidence intervals for each measurement (Lague et al., 2013). The advantage of this algorithm is that avoids the problems raised by the low density of the point cloud resulting from the photogrammetric process and rather performs a direct comparison of the two point clouds. In fact, thanks to the selection of the feature points in the previous step, the problem of the lack of points in some parts of the point cloud is avoided because the presence of these points in the point cloud is guaranteed. For each point of the sparse cloud, a closer point can be defined in the dense cloud and the algorithm can estimate the surface change as the distance between the two points.
The second comparison was performed firstly by scaling the 3D model obtained from the photogrammetric process on the basis of the feature points on the dense point cloud, which are the same of the feature points previously selected in the photogrammetric process. Finally, the estimation of the residuals between the coordinates of the feature points in the two point clouds concludes the metric evaluation.

CASE STUDIES
In order to test the algorithm, two case studies were chosen with two different purposes.
The first one was to study the transformations from the historian and architectural point of view of Karanlık church in Göreme National Park (Cappadocia, Turkey) by comparing the reconstruction from historical archives photographs (by G. de Jerphanion's, 1925-42) and the actual state of the church, with particular focus on the wall paintings.
The second one was to measure the difference of the Former Matsuno-Yu (Public Bath) building in "Naka-machi Komise Street" (Japan) before and after the restoration by comparing the point cloud obtained from local archive images (2013) and a recent survey of the building (2018).

Churches in Göreme National Park and the Rock Sites of Cappadocia
"The Göreme National Park and the Rock Sites of Cappadocia, registered UNESCO's World Heritage list in 1985, is located in Nevşehir Province, 280 km southeast of Ankara. It is in this spectacular landscape, wherein rock-cut churches provide unique evidence of Byzantine art from the post-iconoclastic period, especially during the 9th and 10th centuries (criteria for selection (i)). The valley marks the center of Cappadocia and has more than 100 churches from the end of 9th century to the 11th century. It was only after the publication of G. de Jerphanion's work (1925-42) that the importance of the Cappadocian Churches was put into evidence. The French Jesuit professor had explored the eastern part of Cappadocia and left great documentation of the rock-cut churches with many drawings and photographs. But the condition of the rockcut churches is not the same as Jerphanion's period. The spectacular landscape of Cappadocia depends on the fact that most of the rock here is tuff, which is easily eroded. In other words, the rocks from which the churches were hollowed out of are gradually disappearing. And extended abandonment of the churches also brought along artificial damages, such as malicious scars and graffiti. In addition, the rapidly increasing number of tourists harms the environment of the preservation of the monuments" (Higuchi et al., 2019).
One of the famous masterpieces of this area is the Karanlık Kilise (11 th century) and it was chosen as case study.

Nakamachi Komise Street in Japan
Kuroishi is a city which locates on the northern edge of Honshu, the largest main island of Japan. The city has a traditional townscape from the Japanese feudal era (Edo period, from the 17th century). Here, each traditional townhouse has a wooden arcade, Komise, in front of the building and the arcades continue without gaps (Shimazu and Nasu, 2019;Kuroishi, 2005). Komise protects pedestrians from snow, rain and heavy sunlight during summer. Because of its characteristic townscape, the central part of the city, Naka-machi, is designated as an important preservation districts for groups of traditional buildings in 2005. As Naka-machi has such an attractive street, the street of Naka-machi is also designated one of Japan's hundred greatest streets. The case study analysed is a building along such street with Komise, the Former Matsuno-Yu building. It was a traditional Japanese public bath but close in 1993. After Naka-machi became the important preservation districts for groups of traditional buildings, the building became a community center with supports of locals in 2015. For the processing, pictures before restoration were taken by locals in 2013 and pictures after restoration were taken by prof. Nasu and his students during their on-site survey in 2018.

RESULTS
Following the results of the implementation of the workflow on the two case studies are discussed. With the "Feature detection and extraction" step some keypoints were detected, but, as shown in Figure 5, some important radiometric corners in the photograph, that appear also in other images, are missing.    Thanks to the presence of these points, it was possible to perform the distance comparison in CloudCompare between the sparse and the dense point clouds for both case studies. In the following Figure 9 it is shown that the distance between the two point clouds for the first case study are less than 0.1 m. For the second case study the distance between the two point clouds present a greater value, until 1 m. It is a demonstration that the restoration works widely modified the shape of the building. Figure 10. Cloud to cloud distance for the Former Matsuno-Yu building.
For this reason, it was chosen to calculate the residual values only for the first case study. In the following Table 1

CONCLUSIONS AND PERSPECTIVES
The purpose of the current study was to determine a method to manual extract feature points in photogrammetric processing of historical images in order to use them in the matching step of the reconstruction and to guarantee their presence in the resulting point cloud, even if sparse. This allowed the metric quality assessment of the results through the comparison of point clouds with different density and resolution.
The results of the method applied on the first case studies have demonstrated that there were not evident variations in shape of the represented object between the reconstruction from the historical photographs and from the recent survey. However, the 3D model obtained from historical images presented results of accuracy less than 10 cm that are acceptable also in the cases in which the object reconstructed from historical photographs no longer exists.
Instead, for the second case study the metric differences in terms of distance are very high due to the fact the building was extensively transformed during the restoration.
The experimental work presented here is one of the first investigations into the level of quality of results it is possible to reach when processing historical photographs from archives. Further research studies need to be carried out in order to evaluate the effectiveness of the experimental methodology and to extend the application to other case-studies.
The findings should make an important contribution to the field of Cultural Heritage studies because they provide a new tool for the metric quality assessment of historical material in archives.