SPHERICAL IMAGES FOR CULTURAL HERITAGE: SURVEY AND DOCUMENTATION WITH THE NIKON KM360

The work presented here focuses on the analysis of the potential of spherical images acquired with specific cameras for documentation and three-dimensional reconstruction of Cultural Heritage. Nowadays, thanks to the introduction of cameras able to generate panoramic images automatically, without the requirement of a stitching software to join together different photos, spherical images allow the documentation of spaces in an extremely fast and efficient way. In this particular case, the Nikon Key Mission 360 spherical camera was tested on the Tolentini’s cloister, which used to be part of the convent of the close church and now location of the Iuav University of Venice. The aim of the research is based on testing the acquisition of spherical images with the KM360 and comparing the obtained photogrammetric models with data acquired from a laser scanning survey in order to test the metric accuracy and the level of detail achievable with this particular camera. This work is part of a wider research project that the Photogrammetry Laboratory of the Iuav University of Venice has been dealing with in the last few months; the final aim of this research project will be not only the comparison between 3D models obtained from spherical images and laser scanning survey’s techniques, but also the examination of their reliability and accuracy with respect to the previous methods of generating spherical panoramas. At the end of the research work, we would like to obtain an operational procedure for spherical cameras applied to metric survey and documentation of Cultural Heritage.


INTRODUCTION
In the last few years, there has been a rapid development of testing panoramic images and spherical photogrammetry applied to Cultural Heritage, both for purely commercial purposes and for tourism promotion, but also for threedimensional metric documentation, especially in the Architectural and Archaeological field (Kwiatek et al., 2015;Paris, 2015;Kwiatek et al., 2014;Cannella, 2013;Fangi, 2012;D'Annibale et al., 2009;Zara, 2004).The metric documentation of Cultural Heritage with multiimage spherical panorama has already achieved good and accurate results in the past (Fangi, 2017;Fangi, 2015;Barazzetti et al., 2010;Fangi, 2010;Luhmann, 2010;Fangi, 2009;Fangi, 2007;Schneider et al., 2005).Spherical photogrammetry is based on acquisition of partly overlapped images acquired with the same camera from a unique point of view and then projected on a virtual sphere: the sphere is mapped on a cartographic plane according to the equirectangular projection, also known as latitude-longitude projection.The introduction of low-cost action cameras has further allowed a research improvement in this specific field of interest, providing a new starting point for comparative analysis with traditional survey methods (Barazzetti et al., 2017;Pérez Ramos et al., 2016).Nowadays there are many spherical cameras with different performances and costs available on the market; some examples are the Samsung New Gear 360, the Rico Theta S, the 360fly 4K, the Vuze K360 and, among the more expensive models, the Panono Camera and the Insta360 Pro 8K.Moreover, in the last few years, the software tools have had a considerable increase in the field of spherical photogrammetry and, although with some limits, there are some software based on Structure from Motion algorithms able to manage and process panoramic images for three-dimensional representations.
In this paper, we want to analyse the potential of spherical images for the reconstruction of three-dimensional models in Cultural Heritage, starting from the comparison between point clouds obtained respectively from a photogrammetric survey with spherical cameras and from a laser scanning survey.Specifically, in this work, the Nikon Key Mission 360 spherical camera will be tested: this camera automatically combines two images obtained from lenses with a 180° field of view and it allows to generate images with dimensions up to 7744x3872 and 4K Ultra HD videos.The KM360 has two 21MP CMOS sensors with integrated f/2.0 lenses characterised by a 1.6mm focal length.This camera (Fig. 1) allows to take pictures both independently and through the SnapBridge application managed by smartphones through a Wi-Fi and Bluetooth connection.From this application it's possible to remotely control the camera and change some photo's settings, such as ISO and exposure.The images are subsequently downloaded and displayed, already stitched together, directly in the equirectangular projection.In this projection (Snyder et al., 1994;Snyder, 1993;Snyder, 1987), meridians and parallels are represented as straight lines as well as the poles that are equal in length to the equator.This kind of cartographic representation is neither conformal nor equivalent; moreover, the height of the map is equal to a meridian's development and the relationship between parallels and meridians is equal to 2:1.Nowadays, three-dimensional survey techniques are widely employed in Archaeology and Architecture to document and digitize Cultural Heritage: the advantage of using spherical images for the documentation of Cultural Heritage mainly consists in the completeness of information that can be obtained thanks to the 360° field of view.Moreover, each panoramic image includes or replaces a high number of photos acquired from different positions and with a traditional acquisition geometry: this entails a significant decrease in terms of time acquisition.In addition, thanks to their low-cost system, small size and lightness, the spherical cameras find in Cultural Heritage the perfect application field in order to preserve and record an inestimable value documentation.This work is part of a wider research project that the Photogrammetry Laboratory of the Iuav University of Venice has been dealing with in the last few months; the final aim of this project will be not only the comparison between 3D models obtained from spherical images and laser scanning survey's techniques, but also the examination of their reliability and accuracy with respect to the previous methods of generating spherical panoramas.At the end of the research work, we would like to obtain an operational procedure for spherical cameras applied to metric survey and documentation of Cultural Heritage: the work here presented is therefore intended as a preliminary approach in the field of spherical photogrammetry with low-cost acquisition systems.In this paper, the work has been divided into four parts: in the second section it will be described the acquisition phase of spherical images, with a particular focus on the different acquisition geometry applied, the topographic network and the laser scanning scans surveyed for comparison.The case study is the Tolentini cloister in Venice, which used to be part of the convent of the close church and now location of the Iuav University of Venice and of the Photogrammetry Laboratory.The reference model for the analysis on the achieved accuracy will be a point cloud obtained by a laser scanning survey performed with the phase based laser scanner CAM2 Faro Focus 3D S120.In the third section, it will be described the processing phase of spherical images and the creation of photogrammetric models in Agisoft Photoscan Professional, analysing the errors obtained.In the fourth section, the photogrammetric point cloud will be compared with that obtained from the laser scanning survey using the open source software CloudCompare.At the end of the procedure some final considerations on the metric accuracy achieved will be presented and discussed.

THE TEST FIELD
The Tolentini's cloister in Venice has been part of a larger project concerning the convent and the nearby church both planned by Vincenzo Scamozzi, who worked for the Teatini's order until 1599.The monastery is easily recognizable by some typical architectonic elements: a cloister with portico on pillars, cells on three sides of the first floor, refectory placed at the cloister's edge.The second floor is characterised by wider spaces and a rare example of a sixteenth century air route, maybe planned by Scamozzi himself.The monastery was closed in 1810 due to the Napoleonic laws and since then it was used for several functions, including a barracks in the nineteenth century and a military district after the First World War.In 1958, the State Property Office decided to use the building for the University of Architecture and it started some restoration works involving various architects, including, in 1985, Carlo Scarpa with the project of the University entrance.

Spherical images acquisition with the KM360
The acquisition of spherical images with the Nikon KM360 was performed using different acquisition geometries.It was decided to acquire images following both a more traditional photogrammetric scheme and a freer geometry with less photograms.In fact, these new digital sensors able to acquire with a 360° field of view allow even the less expert in photogrammetry to acquire an object maintaining a sufficient overlap between photos: the final accuracy of the photogrammetric products achievable in this way has to be verified in any case.In addition to the previous two acquisition methods, in which the camera was positioned on a pole held directly by the operator, the Nikon was also placed on two topographic tripods to acquire a further set of images with the known position of the projection centre, in order to do some future metric analysis (Fig. 2).In this way, a complex test field was planned on which to perform various metric controls and to test different acquisition geometries of spherical images.For this work, the photograms were acquired directly by the operator without using the smartphone application: at the end of the procedure, a total of 86 images were acquired.
The traditional scheme applied in the cloister is typical for the acquisition of indoor environments, which follows the shape of the walls and recalls the geometric figure of a "square".The second scheme, instead, developed through the two symmetry axes of the cloister, identifying a "cross" through the medians of the square (Fig. 3).
Figure 3.The three acquisition geometries: the "square" scheme, the "cross" scheme and the position of the two tripods.
In the first case, 22 images were acquired whereas 18 in the second one; instead, 8 images were acquired on the tripods for each of the two positions, applying small rotations to the camera.Finally, some photograms were also acquired inside the portico characterized by cross vaults.
Once acquired, the images have been downloaded and displayed directly in equirectangular projection: the images' dimension are 7744x3872 pixels (Fig. 4).
Figure 4. Spherical image acquired with the Nikon KM360.

Topographic survey
For the topographic survey, 28 black and white checkerboard targets were placed on the colonnade of the Tolentini cloister: all the targets were positioned at the same altitude thanks to the Topcon RL 25 laser level.The reason of this choice is due to the fact that the cloister was chosen as a wider test field on which to perform further analyses with spherical images, as already mentioned previously.The topographic support was performed using the Leica TCR 1103 total station.Two points were materialized in the cloister from which all the measures have been observed and from which all 28 targets positioned on the colonnade were collimated with the forward intersection method (Fig. 5).
The observations were adjusted with the least squares method implemented in the MicroSurvey StarNet software.At the end of the procedure, a standard deviation on the coordinates of about ± 2mm were obtained.
Figure 5.The topographic measures (on the left) and the used instruments (on the right).

Laser scanning survey
At the same time, a laser scanning survey was carried out with the phase based laser scanner CAM2 Faro focus 3D S120; in this way, it was possible to obtain a metric data comparable to models obtained from the spherical images.
As already known, this laser scanner is particularly suited for fast data acquisition (up to 976.000 points per second) maintaining a high level of precision (±2 mm in a range from 0.6 to 120 m).Specifically, the scans have been acquired with angular increments of 0.035°, in order to have a point every 6 mm at a distance of 10 meters.Two scans were acquired from the centre of the cloister, obtaining a cloud of more than 55 million points (Fig. 6).
Within the Faro Scene software, the scans were oriented and rototranslated in the topographic reference system thanks to the 28 targets positioned on the colonnade and collimated with the total station, obtaining an accuracy suitable for architectural representations.
Figure 6.The laser scanning point cloud with the scans' position highlighted.

3D MODELLING WITH THE NIKON KM360
The acquired images have been processed with Agisoft Photoscan Professional.As it has been highlighted in recent researches (Grasso, 2016), the use of panoramic images with this particular software has many advantages: for example, orientation procedures are not so heavy both from the computational point of view and in terms of time, especially if compared with previous generation methods of spherical images.
The next section describes the procedures applied for the creation of photogrammetric models, comparing the different acquisition schemes and with a particular attention to the errors obtained.

Image processing
The first photogrammetric model was created with all the 86 photograms acquired in the cloister: the photos on the two tripods, those with the pole following the previously described "square" and "cross" scheme and, finally, those under the portico were elaborated together in the same project.
In this work, two different orientation procedures were applied (Nex et al., 2014): in the first case (Fig. 7), the GCPs were identified directly in the solution of the bundle adjustment, in order to treat them as weighted observations inside the least square minimization.This approach is the most rigorous since it minimizes possible deformations of the photogrammetric block and any systematic errors.On the other hand, in the second model it was used a free-network approach in the bundle adjustment and only at the end of the process a Helmert's transformation was applied to georeference the obtained model in the topographic reference system.
Figure 7.The first photogrammetric model created with the GCPs in the bundle adjustment solution.
Since the topographic targets were all at the same altitude, in order to avoid any systematic trend of the model, nine further control points were identified in the upper part of the cloister's façades, obtaining the coordinates directly from the laser scanning point cloud.
In the first case, the software oriented 76 photos with respect to the initial 86 while, in the second case, it was able to process all the photos together.In both cases, the models were very noisy, although some masks have been created for each photo in order to exclude some portions from the model computation (such as the sky or some overexposed areas).
The following table (Table 1) shows the comparison between the RMS errors obtained with both the procedures.As expected, the first case, although still very noisy, has a smaller error if compared to the free-network solution; for this reason, it will be used as a model for comparison with the following elaborations of the "square" and "cross" acquisition geometries.

Control
Therefore, we also wanted to test the accuracy achievable following the two described acquisition geometries (Fig. 8): for both cases, these models were elaborated identifying the GCPs directly in the bundle adjustment solution.
Figure 8.The two acquisition geometry acquired: the "square" (on the left) and the "cross" scheme (on the right).
The model with the typical acquisition scheme of indoor environments was created after the orientation of 22 photograms: at the end of the procedure, it had a smaller error (Tab.2) with respect to the "cross" scheme (18 photos).Also in this case, the models have much more noise if compared to photogrammetric models created with a frame camera.
As shown, both the first model with all the images and the one obtained with a "square" acquisition scheme, have similar errors, while the "cross" scheme shows bigger ones.Therefore, even in the case of low-cost systems acquiring spherical images is necessary to follow the traditional photogrammetric acquisition scheme for an accurate model's creation.
From the results obtained at the end of the procedure, the two models with the lower errors were the first one, created with all the acquired photos, and the one with the "square" acquisition scheme.Finally, it was decided to use the first model, which also included the acquisition of one side of the portico, to perform a metric comparison analysis with the laser scanning point cloud.

Control Points (23) RMSE X (m) RMSE Y (m) RMSE Z (m) Error
In fact, this model was the most complete also from the qualitative point of view of the surveyed environment and it presented a similar accuracy both on the GCPs and on the Check Points, approximated to 3cm.

EVALUATION OF METRIC ACCURACY
To verify the metric accuracy of the photogrammetric survey with the Nikon KM360 spherical camera, it was decided to compare the obtained model with the laser scanning point cloud, used as reference.
The evaluation was carried out with the open source software CloudCompare: the comparison between the two models have interested the internal façades of the cloister.Only one of the four front has no upper floors: in this case, however, the images were also acquired in the portico area and this allowed to plan some comparisons also for this side of the cloister.Before proceeding with the analysis, it was necessary to decimate the laser scanning point cloud, which had twice the amount of points compared to the photogrammetric one.In fact, the first one has about 55 million points, while the second one has only about 30 million points.
At the end of the decimation, a cloud of about 27 million points was obtained in order to be used as a reference for the metric analysis.The previous images (Fig. 9) show the analyses carried out with the four photogrammetric façades compared with the respective laser scanning fronts.The comparison range has been set from 0 to 20cm: the metric result is based on calculation of the minimum distance between each point of the two models using the Nearest Neighbor algorithm.
The software also allows to calculate some statistical parameters, such as average and standard deviation.The following Table 3 shows a numerical comparison between the mean and standard deviation values calculated at the end of the analyses.As it can be seen first from the images and then also analysing the statistical values, the façade that has the greater deviation is the one with only the part of the colonnade and the portico behind, the fourth front in the upper table.Maybe this is due to the fact that the point cloud of was generated without any control point; probably the photogrammetric model had errors already in the elaboration process.

Test
The comparison between the first three shows that about 60% of the analysed area has differences between 0 and 3cm with respect to the reference surface, while if we extend the area to 70% we get differences up to 4cm.The case of the fourth façade is quite different: in fact, here, we have differences from 0 to 5cm (for 60% of the area) and up to 7.6cm (for 70% of the area) with respect to the reference surface.

CONCLUSIONS
This first analysis on spherical images has allowed the evaluation of some advantages and some limitations concerning the use of the Nikon KM360 spherical camera.
The fast acquisition phase of the images and the quite easily camera's conditions of use are only two of the reasons that make these new low-cost sensors useful for many users, even if not strictly specialized in survey and Geomatics field.Furthermore, obtaining a complete 360° documentation of an architecture or an indoor environment has a great importance and effectiveness, especially if there is no enough time available to carry out the survey.However, as demonstrated in this preliminary test, the achievable photogrammetric elaborations are not yet comparable with the metric accuracy obtained with a laser scanning survey or with traditional photogrammetry.Concerning the errors of the photogrammetric models and the comparative analysis with the laser scanning point cloud, the data obtained with the KM360 can not yet be used for a detailed scale representation, but may nevertheless be useful for a qualitative general description of an architecture's geometry or for a volume's evaluation of an environment in a larger scale (1: 100 -1: 200).Moreover, an aspect that has to be considered in the future analyses concerns the estimation of camera calibration parameters and possible distortions introduced by the lens system.At the moment, the software used for processing photogrammetric data (Agisoft Photoscan Professional) does not allow to compute the calibration parameters of spherical cameras.The information concerning the geometric characteristics of the camera is normally applied during the stitching process which, in the case of the KM360, is an automatic procedure managed directly inside the camera itself.
As already described previously, the work here presented is therefore intended as a preliminary approach in the field of spherical photogrammetry with specific cameras: a future research development will therefore consist in analysing the algorithms for calculating the calibration parameters and any distortions introduced by the optical system of the camera.Furthermore, we also want to investigate the possible applications achievable from the photogrammetric procedure concerning a specific field that does not require high metric accuracy, as the use of spherical images for documentation and description of Cultural Heritage through Virtual Reality applications.At the end of the research work, we would like to obtain an operational procedure for spherical cameras applied to metric survey and documentation of Cultural Heritage.

Figure 2 .
Figure 2. The acquisition phase with the KM360 on the topographic tripod.
A comparison between the 3D models created: RMS error of the GCPs (on the top) and Check points (above).

Figure 9 .
Figure 9.The comparison between the photogrammetric and the laser scanning point cloud of the four façades of the cloister.

Table 1 .
A comparison between RMS errors of the two photogrammetric models with all the acquired photos.

Table 3 .
The statistical parameters of the four analysed façades.