IMPROVING SPHERICAL PHOTOGRAMMETRY USING 360 � OMNI-CAMERAS : USE CASES AND NEW APPLICATIONS

During the last few years, there has been a growing exploitation of consumer-grade cameras allowing one to capture 360 images. Each device has different features and the choice should be entrusted on the use and the expected final output. The interest on such technology within the research community is related to its use versatility, enabling the user to capture the world with an omnidirectional view with just one shot. The potential is huge and the literature presents many use cases in several research domains, spanning from retail to construction, from tourism to immersive virtual reality solutions. However, the domain that could the most benefit is Cultural Heritage (CH), since these sensors are particularly suitable for documenting a real scene with architectural detail. Following the previous researches conducted by Fangi, which introduced its own methodology called Spherical Photogrammetry (SP), the aim of this paper is to present some tests conducted with the omni-camera Panono 360 which reach a final resolution comparable with a traditional camera and to validate, after almost ten years from the first experiment, its reliability for architectural surveying purposes. Tests have been conducted choosing as study cases Santa Maria della Piazza and San Francesco alle scale Churches in Ancona, Italy, since they were previously surveyed and documented with SP methodology. In this way, it has been possible to validate the accuracy of the new survey, performed by means an omni-camera, compared with the previous one for both outdoor and indoor scenario. The core idea behind this work is to validate if this new sensor can replace the standard image collection phase, speeding up the process, assuring at the same time the final accuracy of the survey. The experiment conducted demonstrate that, w.r.t. the SP methodology developed so far, the main advantage in using 360 omni-directional cameras lies on increasing the rapidity of acquisition and panorama creation phases. Moreover, in order to foresee the implications that a wide adoption of fast and agile tools of acquisition could bring within the CH domain, points cloud have been generated with the same panoramas and visualized in a WEB application, to allow a result dissemination between the users.


INTRODUCTION
Spherical Photogrammetry (SP), introduced by Fangi more than one decade ago (Fangi, 2007), (Fangi, 2010), (Fangi, 2011) proved to be still a powerful photogrammetric technique to be used for Architectural Heritage (AH) surveys and documentation.It allows one to collect priceless images that, once transformed into panoramas, enable the surveyor to re-built the geometries of a building with a "point by point" plotting.This is not trivial, since, weather in case of complex shapes (like sculptures, decorations, mouldings and so on) points clouds are the winning solution, on the contrary, for the representation of more simple geometries (e.g. the fac ¸ades of a Roman Church), a "point by point" plotting is preferable.Besides, the collection of panoramic images of ancient sites might represent a valuable source of documentation in such cases where, due to the war or natural hazards, architectures are seriously damaged or even lost (dAnnibale et al., 2011), (Fangi, 2015).Of course, SP still have some drawbacks; for instance, the plotting phase, as well as the creation of panoramas, is very time consuming.However, even if the first one is far from being solved and needs further investigation in order to automatise the detection of homologous points, nowadays the technological progress opens up new possibilities to speed up the latter.During the last few years in fact, there has been a grow-⇤ Corresponding author ing exploitation of consumer-grade cameras allowing one to capture 360 images.Among the others, the most affordable are the Ricoh Theta S, LG 360 CAM, Samsung Gear 360 , Nikon Key mission 360 .Considering more professional devices instead, it is worth to mention the GOPRO OMNI with spherical-head, Sphericam V2 and the Panono 360 Camera.In this regard, in 2009 a survey test conducted with the Ladybug 3 was presented during the ASITA conference hold in Bari, Italy (Fangi and Schiavoni, 2009).The tool was composed of 6 cameras, 5 of which with horizontal axis and the sixth with vertical axis.During this experience, it was demonstrated that, despite the low resolution and the offset among shooting and camera centers, it is possible to perform a survey, at the expenses of accuracy.Thanks to the technological enhancement of new tools, the resolution of the final panorama has been strongly increased, reaching 104 MPx for the Panono 360 camera1 .It is composed of 36 cameras, displaced over a sphere of about 15 cm of diameter, covering the complete field of view of 360 degrees.W.r.t. the SP methodology developed so far, the main advantage in using 360 omnidirectional cameras lies on increasing the rapidity of the following three steps of the procedure: • Acquisition: there is not the need to shoot multiple images with a spherical head, representing a great benefit, especially for indoor scenario.Since the camera has a fixed focal length, it is better to avoid shoots with close a foreground and far backgrounds.Besides, since lighting conditions in indoor environments can be impervious, it is suggested to use bracketing or HDR techniques in order to compensate the light exposure.
• Panoramas creation: there is not the need to perform the stitching between multiple images, representing a very time consuming step.A simple vertical straightening is enough for making the panorama suitable for the orientation.
• Verticality: Panono 360 embeds a verticality sensor which allows one to avoid the time-consuming process of forcing the verticality; in fact, SP works properly just in case panoramas are quasi-horizontal.
The use of this kind of sensors is thus particularly appropriate in those situations where a fast acquisition is compulsory, i.e. in case of emergency documentation (earthquakes, war, accidents, forensic applications and so on).This research paves the way towards new frontiers of SP.Mobile mapping applications can be easily obtained with a simple walk-around the area where the Cultural Heritage stands.In this way, realizing 360 panoramic view with of the surroundings, it is possible to achieve a complete metrical reconstruction of the objects.With SP, the surveyor can perform a 'point by point" plotting getting in output the wireframe drawing; moreover, exploiting the latest improvement of commercial suites of digital photogrammetry, a Structure from Motion Multi-View Stereo (SfM-MVS) workflow can be performed, creating a dense point cloud using the same panoramas.By this way, the two approaches can be combined and augment the results.A similar application can be found in (Pierdicca et al., 2016).It can be thus summarized that the main contributions of the article are the following: i) improving the rapidity of acquisition and panorama creation which are fundamental steps to perform SP methodology; ii) exploiting omni-camera for documentation purposes, assessing the reliability of the device for emergency architecture; iii) outlining a pipeline of work which brings from a simple acquisition of 360 scenes to the web visualization of point clouds.

RELATED WORK
Exploiting 3D and geo-spatial information for CH related applications is increasingly among researchers.However, given the needs of collecting images in a quick and affordable way, from this field is becoming paramount to adopt cheaper and more flexible means of both data collection and 3D reconstruction (Vergauwen and Van Gool, 2006).The answer to these issue is nowadays represented by 360 images, which enable one to capture the surrounding world.In addition, the derivation of metric results from spherical images for interactive exploration, accurate documentation and realistic 3D modeling is receiving great attention due to high-resolution contents, large field-of-view, low-cost, easiness, rapidity and completeness of the technique (Barazzetti et al., 2010).To date, the most common practice for creating cylindrical or spherical panoramic images is generally entrusted on collecting linear arrays and rotating camera, with very high metric performances, on a panoramic head (Luhmann and Tecklenburg, 2004).In other words, a set of partly overlapped images are shot from a unique point of view with a camera which is rotating around its perspective centre.3D reconstructions can be then achieved once the panoramas are oriented.In the literature several examples of complete 3D modelling projects based on this mathematical formulation were described in (Barazzetti et al., 2010), (Fangi and Pierdicca, 2012).In these papers, equirectangular projection were generated from a set of images stitched with a software for panoramic photography (such as PTGui, Autopano, etc.).However, the above mentioned scientific works, highlight that the procedure is very time consuming; besides, the stitching phase lead to the creation of some misalignments that affect the orientation.The solution to such issues is nowadays represented by new sensors that, at the expenses of resolution, enable to collect spherical images with just one shot.In (Kwiatek and Tokarczyk, 2015) and (Kwiatek and Tokarczyk, 2014) the topic of exploiting 360 images for photogrammetric purposes is faced.The articles attempt to describe a procedure for using images coming from a video sequences, highlighting that the images have a relatively low resolution.Hence, this factor reduces the accuracy of photogrammetric orientations.Another negative factor is the way in which the immersive video frame is created.It is not a perfect spherical panorama, because images that comprise it do not have a common projection centre.However, the advantage of stitched full-spherical images is their number that make the video and the way in which they are recorded densely along a trajectory.Central-perspective original images have limited field of view which causes problems in finding tie points between images, especially in rooms without much details (e.g.walls painted in one colour).The research on the photogrammetric potential of the immersive video sequence, described in this paper, is focused on the following factors that affected the modelling quality: • the way in which images are stitched into panoramas; • the choice of the sphere radius which was declared when creating video frames; • the density of video sequence images.
In (Remondino, 2011), a review over the 3D measurement sensors and 3D modeling techniques, with their limitations and potentialities, requirements and specifications, is discussed, together with examples of 3D surveying and modeling of heritage sites and objects.In the literature, the work that is closest to the research presented in these pages can be found in (Barazzetti et al., 2017).In this paper, authors perform some experiments with the Samsung Gear 360 .Even if the sensor is very cheap, resolution does not allow one to achieve the desired accuracy.Moreover, this kind of cameras can shoot a panoramic image exploiting two fisheye lenses that have to be, in any case, stitched one to another.Considering the researches conducted so far, 360 imaging sensors have huge potential for photogrammetric measurements, even if further studies need to be undertaken, especially on the optimisation of measurement accuracy and its economics.In the following, in line with the most recent research trends, our approach is proposed.

Spherical Photogrammetry
The photogrammetric outputs based on SP are the result of a first CAD drawing which require further steps in order to achieve the final 3D model.In fact, there are many variables in the workflows due to the possibilities to integrate various techniques.The advantages are the high resolution, the FOV up to 360 , the low cost, the completeness of the information and the high speed of takings photos.On the contrary, the creation of the panoramas, as well as the plotting and the orientation are, up to now, fully manual.Nowadays more accurate and efficient tools and instruments are available for CH recording, such as laser scanning, SLAM and dense multi-view 3D reconstruction, but they are still very expensive.SP was mainly conceived and designed for cultural and architectural metric documentation for low cost solutions and hazard conditions.Following the classical photogrammetric procedure, performing a bundle block adjustment (Fangi and Nardinocchi, 2013) it is possible to start from the SP orientation and to use the 3D modeling tools to create the 3D model basing on the rules of the projective geometry with a method called panoramic image-based interactive modeling.This technique is suitable for the architectural representation because it is a "point by point" survey (conversely to dense matching techniques which produces point clouds) and it exploits the geometrical constraints of the architecture's geometry to simplify the 3D modeling process.In other words, this method is appropriate when dealing with well-defined objects that can be reconstructed in the basic geometries on distinct planes; conversely, dealing with complex shapes and surfaces that can not be represented with well defined objects (i.e.archaeological findings), the approximation for finite elements shall be replaced with more complex meshes.Therefore, the surveyor has to comprehend the geometry of the architecture before modelling it.In this approach, the modeling methodology is based on the use of texture mapping techniques in a generic modelling software as virtual projector of an image, and thus to be used to model an architectural object.If the projection centre and the orientation are fixed in the 3D virtual space, objects could be created, moved and modified to match the projections (Wahbeh, 2016).

Data collection and processing
As stated in the previous section (Section 2), image acquisition based on 360 is not so spread, hence the data acquisition phase deserve a brief explanation.The Panono 360 camera is a ballshaped camera containing 36 camera modules, a processor, memory and an accelerometer.It can be used in the following ways: trigger it via a built-in shutter button when holding it by hand or mount it on a tripod or pole and trigger it remotely through a mobile phone application.Once the images have been captured, they are transferred via Wi-Fi or Bluetooth to the connected smartphone where a first low-resolution preview image is stitched.However, to get the final stitched panorama, the original images data has to be sent to Panono's cloud servers, where the stitching of the final output panorama is performed.After the stitching process, the output panorama comes back, being compatible to be visualized with Google Photosphere or similar 360 visualization tools.Users will also have access to the original images that were captured by the Panono's individual 36 camera modules.By default the entire process is fully automated.However, from the application is possible to control key parameters such as shutter speed, ISO and white balance.The most important feature, fundamental for indoor environment, is the HDR mode.
Without any doubt, the Panono 360 is the fastest way possible to collect 360 images existing so far, and this represent the main improvement for SP.Moreover, the fact that all individual frames of the sphere are captured at the same time also means that there are no problems with moving subjects.From our test, there are any ghosting, disappearing limbs, multiple versions of the same person in the image or other artifacts, that are quite usual in most stitched panorama images.The equirectangular panorama has a size of 16384x8192 Px.Each pixel corresponds to 400/16384=0.02gon.
Example of panorama, for both indoor (in HDR mode) and outdoor scenario, are reported in Figure 2.

EVALUATION OF METRIC ACCURACY
Since no previous tests have been conducted for accuracy for 3D surveying purposes, we set up an accuracy evaluation test conducted inside the laboratories of Universitá Politecnica delle Marche.A set of 5 panoramic images of a wall was acquired with the Panono 360 placed on a tripod.Besides, in order to evaluate if the imposition of the verticality might affect the final quality of the orientation, the bundle adjustment have been calculated using two sub-set of images: one without the correction of verticality, and one with the correction of verticality, performed using PTGui stitching software.The estimated calibration parameters were then assumed as constant values for a 3D reconstruction project of a straight wall, on which a set of targets was installed and previously measured with a total station.Images were oriented with Sphera (developed by Fangi), a photogrammetric suite specifically designed for SP, using 55 targets points of known coordinates.At a second stage, the orientation was then performed using only 4 fixed points to be used as GCP, and using the remaining ones as check points.In Figure 3 1 and reveal an Sigma-zero of about 3 Px and 2 Px respectively for the study tests previously described.Residuals of observation can be thus considered equivalent, even if the error is reduced when performing the bundle adjustment with only 4 points of known coordinates.This aspect can be explained with the degree of freedom of the calculation when reducing the number of constrains.In both cases, the results confirm a good metric accuracy for the project.
It was then calculated an F-Fisher test for validating the quality of the statistical sample.F-Fisher test with a significance level ↵ = 0.01 shows that is valid the alternative hypothesis, say the two sigma-zeros are significantly different.In any case, the typical error is 2 pixels at best.Results of the LSBF (Least Squares Best Fitting) are depicted in Figure 4, representing the differences from the fixed GCP and the free block.The mean difference average in module is 0.015 m.
For the sake of completeness, it is worth to note that the capturing scenario of the validation test was not performed in its optimal configuration.In fact, the network of GPS constrained the acquisition in only one direction, even if a full 360 plotting might assure a better quality of the test.The reason because of we used this configuration is due to the fact that the points used for the test where set for camera calibration.Besides, it is even worth to say that, in order to avoid parallax error, panoramas have to be taken at a safety distance from one another; in our case, the distance is acceptable (3 m at worst).

RESULTS: 3D MODELLING AND WEB VISUALIZATION TOOL
Results obtained with the described calibration procedure are more than satisfactory and discrepancies between the different tests are negligible.In fact, the residuals of observation between The the rectified/not rectified configuration are similar, and performing the bundle adjustment using 4 versus using all points of know coordinates gives almost the same results.At this stage, the surveyor has two possibilities, namely performing a "point by point" plotting or creating a point cloud of the architecture by using an MVS photogrammetric tool.As stated in the introduction section, the two churches have been already plotted by using SP; in Figure 5 the wireframe drawing of the two Churches is reported.Some points, coming from the previously SP survey, are used to perform the bundle adjustment of the dense point clouds, as described in the following.To avoid redundancy, the procedure of the experiment will be described using as example only one of the two churches (specifically the San Francesco one); indeed, the data collection phase for outdoor and indoor scenario (apart from the activation of HDR mode) is equal and, in terms of accuracy, no discrepancies have been noticed.

Orientation and surface extraction from panoramas obtained with Panono 360
Image acquisition with the Panono 360 was carried out by placing the camera on a pole and taking the pictures with the mobile phone application.This way the acquisition is very fast and do not need further post-processing steps.The camera allows even non expert users to collect a sufficient amount of information to retrieve a geometrical reconstruction of a building.It is worth to note that, for indoor scenario, illumination conditions are extremely important, due to the common issue of having uniform lighting conditions in the whole scene.The collected dataset is Table 2. Survey data and processing data.
Number of images 14 Average Flying altitude 25,3 m Tie points 18,789 Reprojection error 9 pix GSD 1cm/px composed of 14 panoramas, well distributed among the plan of the church (see Figure 6).
The final goal of this phase was to test the performances of MVS tools dealing with spherical images.In fact nowadays it is possible to achieve acceptable results in terms of accuracy from the point clouds extracted with the spherical camera model.Given the good resolution of the panoramas, textures allowed to use dense matching techniques for surface modelling.Image processing was carried out in Agisoft Photoscan c enabling the panorama mode.Image orientation with SfM procedure took about 10 minutes, whereas dense matching for point cloud extraction took more than 3 hours, given the high quality of the images.Before to proceed with the creation of the mesh and the subsequent texturing phase, we performed an accuracy assessment of the point cloud by using 30 control points of known coordinates well distributed among the church, computed with SP.This step allowed the scaling of the model in its real scale.Some images of the meshes generated from a dense point cloud of about 1.5 billions points are reported in Figure 7. Statistics about the point cloud computation are reported in Table 2, whilst residuals and total error are reported in Table 3.
The results achieved with this procedure deserve some comments.First of all, residuals are quite high (about 8 cm); the reason should be attributed to the resolution of the panoramas.
In fact, even if the image quality is sufficient to perform MVS procedure, the recognition of omologous points in some parts of the church was impervious (especially for those points far from the shooting centre more than 20 m).Moreover, the quality of the mesh is not convincing at all, being a bit rough.Hence, post processing steps like cleaning and decimation are necessary.

WEB tool for the visualization of point clouds
As the methodology described above demonstrated to be valuable for CH purposes, it is worth to wonder which is the more reliable way to share the information arising from 3D metrical restitution.In fact, whether for insiders it is important to use stand alone tools in order to perform in-depth analysis, it is even important to convey the results of 3D reconstructions using those channels which allow to reach the vast majority of the population, being the CH belonging to the whole mankind.In this regard, the use of WEB solutions should become the paramount media for disseminating CH knowledge.In the last few years, the WebGL framework allowed to overcome the proprietary plug-in issue, and three-dimensional content is slowly becoming a usual type of content for web pages.Let's take, for example, the experience from mobile mapping applications; the documentation with spherical or panoramic photography is getting a very common practice for many kinds of visual applications (e.g.Google Street View, 1001 Wonders, etc.).With the new potential of WEB visualization tools, it is allowed to foresee that in the upcoming years we might assist to a similar experience even for point clouds.The developed tool moves towards this direction, paving the way for a thorough exploitation of 3D metrical information to be spread via WEB.
From Photoscan, the computed dense cloud can be directly exported in Potree 2 binary format.This data structure can be easily visualised on a web platform based on Potree, a free open-source WebGL based point cloud renderer for large point clouds.This visualization is optimized to be accessed by web browsers, with mobile responsive interfaces, thanks to the progressive loading and rendering of two-dimensional height profiles, and point-wise 2 http://potree.orgadaptive point sizes, which adjusts the size of each point to the level of detail as additional points are streamed in over time.Main visualization features are: • Data navigation in the space with pan, tilt and zooming functions; • Layer annotation and selection on point cloud data; • Metric dimensional measurement in distances, surfaces and volumes.
Images of the web tool developed are depicted in Figure 8.For the purpose of this dissertation, we report only some pictures of the Santa Maria della Piazza church, referring interested reader to the link at the bottom of the page for the visualization of both Churches 34 .

DISCUSSION AND CONCLUSION
The research experiments described in this article are promising.The Panono 360 proved to be suitable for metric reconstruction; the accuracy values and the error tests described in Section 4, can be considered comparable with a traditional panorama obtained with a standard camera mounted upon a panoramic head.It can be thus stated that, for the upcoming future surveys, this sensor can be a valuable alternative to speed up the process of data collection, no expenses the accuracy.The camera is very simple to use, and acquisition can be performed even by non expert users.Being the illumination fundamental for indoor scenario, the camera allows a proper acquisition with the HDR mode.As a conclusion, we can state that the pipeline of work is promising and it is possible to draw some points in favor, listed below: • the acquisition phase is dramatically speeded up and is very easy, since it can be performed via mobile phone; • there is not the need to perform the stitching procedure.This aspect lead to a couple of advantages.First of all, stitching is very time consuming.Second, panoramas are ready to use, being geometrically more correct than a panorama created with the dedicated software; • there is not the need to impose the verticality, so panoramas can be used suddenly once received back from the Panono cloud service; • it is the optimal solution for architectural heritage, especially for indoor environments; • the number of Ground Control Points needed is very low, as proved by the laboratory test conducted.
The methodology, however, presents some drawbacks.In fact, the accuracy values from the point cloud cannot be acceptable.
In fact, even if the residuals from the GCP are suitable for architectural scales of representation, the reprojection error is quite high, meaning that the point cloud is a bit coarse; hence, it cannot be used from scratch for modelling purposes but needs further operation of post-processing.This aspect will be investigated in the future.Moreover, it would be interesting to evaluate if increasing the number of panoramas will increase the quality of the point cloud.Even this aspect will be further investigated.Finally, the web visualization deserve a brief comment.Regardless the quality of the point cloud, this research paves the way for foreseeing that this kind of fast and agile tools of data collection will be used to extensively collect entire historical centres with the twofold purpose of documenting the CH and of sharing its knowledge with the mankind, not only using the images but also having tools to interrogate the models.

Figure 1 .
Figure 1.The Panono 360 with a schematic representation of the workflow used to create panoramas.

Figure 3 .
Figure 3. Testing environment used for the evaluation of metric accuracy.(a): One of the 5 stations, where are visible the markers of known coordinates.(b): Orientation network of the photogrammetric model.

Figure 5 .
Figure 5. Wireframe drawing of San Francesco (a) and Santa Maria della Piazza (b), plotted by using Spherical Photogrammetry.

Figure 6 .
Figure 6.The bottom view of San Francesco church with the positions of the panoramic images used to create the dense point clouds.

Figure 7 .
Figure 7.The interior of the Church after the creation of the cloud and 3D mesh, with the of textures.

Figure 8 .
Figure 8. Web visualization of the point clouds.The tool allows to interact with the points in terms of features and appearance, while a dedicated tool allows the user to infer metric information about the building and the surrounding environment in case of outdoor scenario(a): San Francesco alle Scale.(b): Santa Maria della Piazza.

Table 1 .
Results of the test field

Table 3 .
This table reports the overall estimated error for the GCP with ellipse shape in X, Y and Z.