360° IMAGES FOR UAV MULTISENSOR DATA FUSION: FIRST TESTS AND RESULTS

A relevant research topic in the photogrammetry field is related with experimenting, at different levels, data fusion and sensors integration, aiming at the development of rapid mapping systems, capable of quickly delivering accurate data, for a wide range of applications. The presented contribute aims at exploiting the potentialities of spherical images and videos acquired using a 360° camera mounted on board of a medium-sized Unmanned Aerial Vehicle (UAV). More specifically, the focus has been the development and testing of a rapid-mapping hybrid system, capable of fast acquisition of data from both the on-board sensors. The reduction of the data acquisition time has been, and still is, a hot topic faced by the researchers in the Geomatics community, especially linked to some hazardous operative scenarios, where the reduction of the time on the field is crucial for operators’ safety reasons (like in post-earthquake early damage assessment surveys). After a deep analysis of the available scientific literature, it turned out that researchers have been more focused on terrestrial applications of these emerging sensors (360° cameras), and no significant studies for aerial application have been conducted yet. First of all, some laboratory tests have been carried out, in order to evaluate the metric accuracy of the 3D models generated using the employed 360° sensor; thereafter, a solution for acquiring spherical images from 360° camera mounted on a light UAV has been designed. Problems and issues have been addressed and discussed, and results and improvements are, at the end of the paper, evaluated and proposed. * Corresponding author


INTRODUCTION
During the last decade, the market of low-cost and consumer grade cameras has exponentially grown, and a wide range of new sensors is nowadays available. The possibility of exploiting the inherent metric information derivable from spherical and cylindrical images, obtained from non-metric cameras, has been one of the most recent and prolific research topics within the researchers in the photogrammetric community (Barazzettiet al., 2018;Fangi, 2006;Fangi et al., 2018). At the same time, the market of Unmanned Aerial Vehicles (UAVs) faced a sensible growth, allowing on one hand more affordable prices and, on the other hand, the possibility of deploying different types of payloads. The use of UAVs for photogrammetric purposes is a consolidated practice (Colomina & Molina, 2014;Nex & Remondino, 2014) and has been applied in a wide range of situations and scenarios (Hartley, 2017;Skarlatos & Vamvakousis, 2017;Sutheerakul er al., 2017). The recent tests and experiments with Commercial Off The Shelf (COTS) and low-cost sensors in geomatics-related application are, more and more, oriented towards a multi-sensor data fusion approach (Chiabrando et al., 2018;Herrero-Huerta et al., 2016). Sensor integration allows to combine data derived from two or more devices with the aim of reducing the uncertainty of the observations gathered separately from each source.
Moreover, it is possible to use different kind of sensors in order to obtain heterogeneous data to be used for multiple analysis (multispectral, thermal, RGB, etc.), exploiting the advantages of the known relative position of the sensors. The research presented hereafter is part of the above-mentioned scenario; testing the possibility of integrating the data acquired with a commercial 360° camera that has been mounted on-board of a commercial grade UAV ( Figure 1) and has been used in combination with the RGB camera already embedded on the platform. In the following sections the advantages and disadvantages of the proposed approach will be presented and discussed. Figure 1. The UAVs/360° camera set up adopted in this research to perform a double sensor acquisition.

Use of 360° cameras for metric purposes: state of the art
In the recent years the market sector related with spherical images and videos was subjected to a rapid growth; spherical images are commonly known as panoramas (Luhmann, 2004) and apart from their use for documentation, visualization and sharing purposes (Clini et al., 2018;Kwiatek, 2012), like 360° videos, AR/VR applications, etc., these products may offer some interesting advantages from the photogrammetric point of view, not fully explored yet. The first researches on the approaches connected with the photogrammetric use of these kinds of images can be traced back in 2007 (Fangi, 2007), and were connected with the generation of spherical images through the stitching of different images acquired around a nodal point from the same camera, following the Computer Vision approach developed by Szeliski and Shum at the end of the '90 (Szeliski & Shum, 1997). In the following years, and till today, different tests and experiences on the use of these systems for photogrammetric application were further extended, in particular in the field of CH documentation (Barazzetti et al., 2018;Fangi, 2015;Fangi et al., 2018;Kwiatek & Tokarczyk, 2014) and more specifically toward the direction of developing Structure from Motion (SfM) approaches in the field of spherical photogrammetry. The development of these researches relates to different elements: the increasing availability of COTS 360° multi-cameras that allows to acquire in a rapid way spherical images, the implementation of more refined algorithms and solutions for the stitching of images and the enhancement of the photogrammetric solutions for the processing of these images. Compared to the traditional photogrammetric approaches, the use of these kind of sensors creates new issues that need to be analysed and solved and it is therefore crucial to properly study these cameras and experiment them under different conditions, to better exploit the metric information inherent their products. According to the typical photogrammetric approach, one of the main problems to consider, in order to reach an acceptable metric integrity of the generated 3D products, is to overcome some of the issues related with the calibration of these new devices. However, in the photogrammetric processing of 360° images the solution of the Interior Orientation (I.O.) is not achieved during the phase of image matching and camera orientation but is already considered solved in the previous phase of image stitching. This is due to the fact that spherical images are considered distortion free, up to a scale factor, and the parameters of I.O. are retrieved during the stitching phase (Fangi & Nardinocchi, 2013). However, during the development of this research it appeared crucial to define, a-priori, a trustworthy set of the estimated distortion parameters for each of the used sensors. As already reported, modern COTS 360° cameras are, in fact, usually composed by two (or more) sensors assembled into one body or mounted onto a common rig. It is therefore important to separately consider and calibrate each camera, in order to guarantee that the estimation of the Interior Orientation Parameters (IOPs) of each camera is performed apart according to the approach followed in Teppati Losè et al., 2018. Two different solutions for the processing of the data collected with a 360° system can be followed, and the correct estimation of the IOPs allows to improve both these approaches. One approach implies that the collected data are processed as single images; the correct estimation of the IOPs allows to separately consider the different cameras embedded in the system and to retrieve a good calibration for each of the employed cameras, enhancing the metric accuracy of the generated models. The other approach consists in process the collected images as stitched panoramas; in doing so the IOPs can be used to improve the stitching phase and thus the overall quality of the generated images. As spherical cameras are not originally designed for metric purpose, similar principles adopted for other low-cost and COTS cameras have to be considered, in order to study the mathematical models that allow to estimate the distortion parameters of the employed lens (Arfaoui & Thibault, 2013;Schwalbe, 2005); moreover different considerations and solutions must be formulated when spherical images are used in combination with frame images obtained from other sensors (e.g. georeferentiation and acquisition strategies).

HARDWARE SET-UP
For the purposes of this research, two devices have been used: One medium-sized commercial grade UAV (DJI Phantom 4 PRO Obsidian) and a low-cost and COTS 360° camera (GoPro Fusion).

DJI Phantom 4 PRO Obsidian
Phantom 4 PRO Obsidian is a lightweight UAV produced by DJI. It has a 1" 20 Mpx sensor, capable of recording 4K 60 fps videos. Compared with other UAVs of the same manufacturer, Phantom 4 PRO Obsidian's camera has a global shutter that could be preferred for photogrammetric data acquisition, compared to rolling shutter (Vautherin et al., 2016). The main specifications of this system are reported in the following

GoPro Fusion
GoPro Fusion is a lightweight low-cost 360° camera, released on the market on November 2017. The camera is capable of recording 5.2K videos and shooting panoramas at 18 Mpx (being 9 Mpx each of the two hemispheres). The system is indeed composed from two cameras coupled on the opposite sides of the device's body. The main specifications of this system are reported in the following Table 2.
Among the wide availability of 360° cameras on the market, the choice to use and test this device has been supported by several evidences that can be summarized as follows: • Reduced weight of the device allows the camera to be easily carried on board of a medium sized UAV.
• The camera is a low-cost solution, if compared to other professional 360° cameras on the market.
• Compared with other sensors, the GoPro Fusion offers a good proprietary software solution for the processing of the collected data. It allows both to have access to the single images collected by the two cameras, both to guarantee an overall good quality of the stitched panoramas, making easier and faster the pre-processing operations.  Table 2. GoPro Fusion main specs.

UAV-Camera connection and stability tests
An aspect that has been investigated in this work is related with the practical issues connected with the on-board installation of the 360° camera and the related problems; these are especially linked to the stability of the platform during the flight and the vibrations transferred to the sensor itself, that can influence the quality of the acquired data. The versatility and autonomy of modern UAVs guarantee a good grade of freedom when there is the need to customize the on-board payload, also in case of COTS systems; this could happen in all the cases where two or more sensors can operate on the same platform simultaneously. The concurrent data acquisition from different sources offers, in fact, some advantages in terms of cost (time and budget) required for carrying out survey activities; as the fully interoperability between different payloads on the same platform can't be always provided by the producer (due to the non-standardization of the available payload), some Do It Yourself (DIY) solutions have, however, to be sometimes adopted. In this case, after some preliminary tests with 3D printed solution, that weren't satisfying due to the fact that a good rigidity of the overall set up wasn't guarantee, it was decided to adopt a commercial solution that was available on the market; a support designed for the use with the Phantom 4 that allows to lift different lightweight payloads, commercialised from the Polar Pro company. A technical drawing of the employed support ( Figure 2) and its installation on the UAV platform ( Figure 3) are showed in the following figures, and the main characteristics of the adopted set up are reported in Table 3.    Table 3. Configuration of the employed UAV with the 360° camera attached and information regarding the change in performance.

EXPERIMENTAL TEST IN CONTROLLED FLIGHT FIELD AREA
A first phase of the research conducted on this 360° system, that won't be reported in this contribute, was dedicated to a deep analysis of the above-mentioned camera. The IOPs of both the fisheye sensors that compose the 360° system were retrieved thanks to the creation of a 3D calibration test field that was acquired and used to complete the calibration phase of these two cameras. Moreover, the potentialities on the use of this camera for photogrammetric purposes were tested and analysed as well, in order to stress the reachable accuracy and to study the issues connected with the deployment on the field of these systems. The test previously reported were achieved on terrestrial acquired dataset; the second step of the research, described in this paper, was connected with the idea of moving the use of this system from a terrestrial to an aerial application, in combination with the sensor already embedded in a commercial UAV platform employed for the tests that will be further described. Different workflows to complete the image orientation phase by combining images acquired from the 360° camera and traditional frame camera have been stressed, and the performance of the data fusion for improving the BBA accuracy has been evaluated. A demonstration of the proposed workflow has been applied on a test site (a flight field in an isolated and safe area), making possible to observe and evaluate the application of the joint sensors under several conditions; during the achievement of this test, the different practical issues connected with the flight of the UAV system with the 360° camera installed were as well evaluated. An overview of the different acquisitions carried out on the test area and of the processing phase of the data following different approaches are reported in Figure 4. While the on-board camera acquired oblique images, the 360° camera acquired spherical images in simple grid configuration. Processing has been performed for both the dataset separately and then in a third dataset composed joining the previous two. In general terms, oblique images are achieving the best performances from a photogrammetric point of view when working in combination with nadir datasets (Aicardi et al., 2016;Rossi et al., 2017). Moreover, it is not uncommon that the orientation of sole oblique images usually lacks enough overlap and thus require the use of a coherent nadir dataset. For these reasons and to overcome these issues, the aim of the test performed was to employ the data collected from the 360° camera (acquired simultaneously with the oblique dataset of the Phantom camera) to enhance the orientation phase of the photogrammetric processing, providing a higher number of tie points and trying, at the same time, to improve the overall image matching phase. The time component is often a crucial element during the fieldwork, and so the need of performing, at least, two flights is limiting when an operation is conducted under time constrain condition. The combination of the two different datasets, acquired at the same time with different sensors mounted on the same UAV platform, can represent a good solution to enhance the quality of the survey and to reduce the time spent on the field. It is then also interesting to evaluate how, and with which limits and advantages, sensors with so different characteristics can be combined in a photogrammetric approach.
The acquisition with the Phantom 4 Pro, and with the GoPro Fusion mounted on it, were performed in a selected area of the flying field. A flight plan was designed using the Pix4D Capture app: the flight was performed at an altitude of 30 m from the ground and setting up the camera orientation of the UAV at 45°; the images from the 360° cameras were instead acquired setting up a shooting interval of 1 second. The main characteristics of the planned and performed flight in this test area are reported in the following A total of 87 oblique images were acquired by the UAV's camera; on the other side, the images acquired with the 360° camera were redundant and were thus down sampled to a total of 53 spherical images (after having completed the stitching phase with the GoPro software solution).
To assess the metric accuracy of the test, a set of 10 artificial 50x50 cm targets were homogenously positioned on the area of interest before the acquisition and were measured with a Network Real Time Kinematic (NRTK) approach. A Global Navigation Satellite System (GNSS) receiver, in this case a Geomax Zenith 35, was employed using the real time corrections derived from the HxGN SmartNet GNSS permanent stations network, allowing to reach an accuracy between 2 and 3 centimetres on each of the measured points.

Data processing and results
The processing phase led to the achievement of different tests and to the realization of several models: using only oblique images (project A), using only spherical panoramas (project B) and merging both the datasets (project C). As already reported in the previous section, a first operation was related to the selection of a proper number of spherical images. Only 1 out of 4 acquired images have been used (equivalent to around 1 image every 3 seconds). The standard photogrammetric workflow was adopted for processing all the three datasets, and in each dataset, after importing the measured coordinates of the targets, they were marked on the single images. The software Agisoft Photoscan Pro v. 1.4.0 build 5650 has been used for processing all the projects, running the alignment phase with no tie points limit. All the results reported are the ones printed after marking the GCPs on all the images and running a refinement of the alignment step.
In project A, where only the oblique images were processed, the software was unable to successfully align all the images of the dataset; an overview of the camera estimated positions is showed in Figure 5. Aside of that, the orientation of the dataset with only spherical images (project B) has been successfully achieved, even if with a relative higher error on the Ground Control Points (GCPs), probably due to the distortions introduced by the fisheye lenses that were not totally solved during the stitching phase. An overview of the estimated position of the cameras is showed in Figure 6. Once completed the processing of these two dataset and after the valuation of their main characteristics and metric accuracy, it was decided to merge the two sets of images and to process them together (project C) in order to evaluate strength and weakness of this approach, and also the quality of the deliverable results; an overview of the camera stations of the two dataset combined together is showed in Figure 7. A first enhancement derived from this approach is related to the fact that in this case the entire dataset of oblique images has been aligned, and all the camera positions and assets have been estimated. In order to investigate the possibility to further improve the control on the GCPs error, a copy of project C has been created (project C I ); here the GCPs have been marked only in the oblique images, as doing so was expected to reduce the high reprojection error introduced by the GCPs observed on spherical images. The main processing parameters of these three different projects are reported in Table 5. As this was a preliminary test, only GCPs have been used for assessing the estimated errors. The results reported in this table show that the use of a combination of the acquired data from the two on-board sensors allows an improvement of the overall photogrammetric processing.
Even with a slightly higher error on the GCPs, the geometrical reconstruction of the surveyed area has been improved, as it is possible to observe in some details of the different generated models that are presented in Figure 8. It is important, however, to note that the test area was too flat to appreciate the improvement in the reconstruction of the elevation of the objects, and thus an application on a real case study appeared to be necessary. Considering the obtained results, the proposed method of data fusion and sensor integration proved to be fast and reliable, compared with the time required, both for acquiring and processing the data.

TESTING THE PROPOSED APPROACH ON A REAL CH CASE STUDY: THE VALENTINO CASTLE
After completing the experimental tests in a safe environment (flight field) the proposed approach was tested on a real scenario: the UAV/360° camera configuration has been deployed on the site of the Valentino Castle in Turin (Italy). Several acquisitions have been performed, allowing both the generation and the evaluation of products and analyses to enhance the study and the knowledge of the surveyed CH artefact. Thanks to the experience gained from the tests performed in the flight filed, a different configuration of the double sensors acquisition was adopted in this case: the simultaneous acquisition of semi-nadiral images, in combination with spherical images (as showed in Figure 9). As will be further reported, this different approach has been tested both for enhancing the quality of the obtainable orthoimages, both for experimenting a complementary configuration in respect with the one already tested on the flight field. Aside for the orientation of the UAV's camera, the acquisitions were performed adopting the same methodology implemented for the tests performed in the flight field, using the Pix4D capture app and setting all the parameters connected with the flight planning; the main characteristics of this acquisition are reported in the following Table 6.
Overlap Sidelap Flight Time Flight Altitude 70% 70% 13 min 58 s 40 m Table 6. Characteristics of the flight plan of the Castello del Valentino's dataset.
As for the previously described strategy adopted, the data acquired in the Valentino Castle test site have been processed following the modalities described in Figure 10.

Data fusion approach for 3D object reconstruction enhancement
The first dataset (project D) was processed using only the nadiral images acquired with the UAV's frame camera, and thanks to the relatively low Root Mean Square error (RMSe) achieved both on GCPs and Check Points (CPs), it was used as ground truth dataset (Table 8).
As expected, and as it is known from other works (Grenzdörffer et al, 2008), even if the nadiral acquisition allows the generation of high quality 2D products, there are limits where there is the need to assess the 3D characteristics of the object. On parallel, the dataset processed using only spherical images (project E) allowed to solve some issue related with the 3D reconstruction of the surveyed architecture, completing the reconstruction of areas that presented evident gaps in the project D. In this dataset the main limits are related with the difficulty on identifying and marking the control points, as they are not easily detectable in the spherical images, especially due to the large acquisition distance; moreover, the model is lacking on details. Comparing the two dataset it is possible to observe that the sparse cloud generated from only nadiral images appears to be denser than the one generated by spherical images, as showed in Table 7. It needs to be reported that in the processing of these dataset spherical images have been masked in order to exclude the part of the images displaying the body of the UAV, in order to avoid the detection of wrong tie points that might negatively affect the alignment or the 3D reconstruction phase.
The third dataset (project F) was processed combining nadiral and spherical images, in order to verify the potentialities of the data fusion and sensor integration derived from this approach. After having considered the issues emerged from the previously performed tests and the problems that occurred when images are processed separately, it appeared necessary to stress on overcoming problems of both sensors and acquisition geometry, aiming at integrating strengths and solving issues of the two methods.
As it is possible to observe, even if the combined dataset presents higher RMSe than the nadiral dataset (Table 8), part of the object that were not reconstructed using only nadiral images were successfully reconstructed in the combined dataset ( Figure 11).  As spherical images provide a lower radiometric quality than the one obtainable by the UAV's camera (mainly due to the smaller sensors dimension), it was interesting to generate and compare the orthophoto produced from project F first using all the images and then using only nadir images ( Figure 12). The last results seem to be better in terms of quality of the final product, overcoming the issues of having different part of the orthophoto with radiometric discontinuity and quality loss, as showed in detail in Figure 13.  Figure 13. Details of the radiometric and resolution discontinuity in orthophoto generated using images from both sensors.

Deliverable products and analysis
Apart from the documentation of the surveyed object, that can be valorised with the production of architectural drawings, like section and elevations ( Figure 14) Different analysis can be produced: In this case the DSM (Digital Surface Model) has been used to inspect the slope of the courtyard's surface, as shown in Figure 15. Further graphical representations of the same analysis can be delivered in different ways, as axonometric views with isolines ( Figure 16).

CONCLUSIONS AND FUTURE PERSPECTIVES
Considering this experience, it is possible to state that the use of spherical cameras installed on UAVs and the subsequent multisensory integration and data fusion is a procedure compatible with the 3D reconstruction of surveyed object using a photogrammetric approach. The adopted methodology is therefore to be preferred where only one or few flights can be performed, being faster than combining nadiral images with a double grid oblique acquisition to achieve a proper object reconstruction. Especially in those situations where a rapid mapping solution is preferred (like disaster management and all those situations in which the rapidity of the survey is crucial) the combination of the on-board frame camera with the 360° systems can represent an added value. One of the issues that was addressed in this work is that, to achieve a more accurate 3D reconstruction, it is strongly advised to identify and mark the control points only on the frame images, in order to mitigate the error introduced by the distortion present in the spherical images, not fully corrected in the stitching processing. Moreover, it is important to don't acquire images at a distance higher than 30-40 m from the object, as doing so will reduce the size of the object in the spherical images, decreasing consequently the quality reconstructed 3D model. The employed methodology appears to be fully applicable in disaster management and damage assessment rapid mapping application and emergency scenarios, especially given the possibility to retrieve not only metric information in minutes, but also other products useful for the management of the emergency; the use of 360° videos could be, for example, used for disemboguing the real level of damage during visual classification in damage assessment operations, or where there is the need of improving the capabilities of the visual observation right from above hot spots or points of interest. However, as stressed in this paper, the proposed approach can also be successfully deployed for the documentation of CH, allowing to reach satisfying levels of details and accuracy. Moreover, widely considering the documenting possibilities offered by the obtainable products (3D models, DSM, elevation, drawings, fisheye images, 360° videos, spherical panoramas, etc.), Virtual Reality application can be implemented, allowing an immersive navigation of the investigated object and offering new and interesting possibilities for the dissemination and fruition of the surveyed objects. This work must be considered as a first step towards these new kinds of application, and the encouraging outcomes of this research foreshadow the possibility of furtherly deepen this topic. One of the interesting features to study is to evaluate the accuracy of the integrated GNSS antenna of the 360° camera, and to analyse how it affects the orientation of the images. Studying more advanced solutions, not only for low-cost but also for higher resolution cameras, is another interesting aspect to cover in future researches, aiming at a better improvement of the results obtained so far.