DEVELOPMENT OF PANORAMIC VIRTUAL TOURS SYSTEM BASED ON LOW COST DEVICES

: The virtual visit is a simulation of reality that allows virtually changing time or placing. Unlike this, virtual reality offers the possibility to associate the real world with the virtual world, while ensuring interactivity and an improvement in prospecting the reality . One of the most used form of virtual reality is the 360° panoramic imagery and video, currently in the market we can find many providers of panoramic cameras, going from costly professional cameras to the low cost consumer grade cameras. In addition, many software providers offers the virtual tours creation and visualisation, to our knowledge, only google street view offers full mapping functionalities, but the data capture and collection is an opaque operation. Generally, smartphones are equipped with positioning capabilities, based on sensors like gyroscopes, accelerometers, magnetometer used as compass and GPS, the integration of this data allow the estimation of the pose. This paper discusses the creation of an interactive virtual reality with 360 ° panoramic images with the possibility of automating its acquisition and integration in mapping environment by using smartphone positioning methods, while the panoramic images are taken using costumer grade camera, which composes a low cost system for acquiring and diffusing panoramic virtual tours map. The developed solution is a dynamic web interface using several open source libraries and programming tools .


INTRODUCTION
Human beings by their curiosity love traveling and discovery, it is the lifestyle for some but it can be a dream for others. It may be difficult or impossible to do this for multiple reasons (money, lack of time, physical incapacity…). For this reason, it is possible to offer a solution to overcome these difficulties, offering the possibility of traveling with a single click! With the advent of digital technology as well as the enormous advancements in the computer field, it is now possible to promote tourism while projecting oneself into another sphere, which is that of the virtual or augmented reality. Indeed, synthetic images made us travel in a virtual world made of a reconstruction of objects more or less resembling reality. However, currently, these synthetic images are being replaced by live omnidirectional shots (photos, videos) of objects. The virtual visit is a simulation of reality that allows virtually changing time or placing. Unlike this, virtual reality offers the possibility to associate the real world with the virtual world, while ensuring interactivity and an improvement in prospecting the reality. Since longtime, panoramic imagery has been used for the topographic and mapping purposes where Mr. Bagley has proved that the panoramic camera is extremely valuable in certain kinds of topographic work (Bagley, 1917). The technologic advancement in optics and mechanics, and recently, in electronics has widely impacted this type of camera and enlarged their field of use (Fangi et al., 2018;Fangi and Nardinocchi, 2013;Luhmann, 2004;Wahbeh et al., 2016). Nowadays, many panoramic cameras are present in the market; these cameras are generally grouped in two categories. The first one is composed of a single camera equipped with a rotation mechanism, this type of panoramic systems can be composed of an ordinary camera mounted on a panorama adapter, the panorama image is generated using a multiple image panorama mosaic (Kaimaris et al., 2014;Luhmann, 2010). In this category we find also specialized line-scanning cameras such KST EyeScan M3 and SpheroCam, in these cameras the detector is composed of a single line sensor (Luhmann, 2010;Parian and Gruen, 2005). The disadvantage of this category cameras is that the camera system must be steady during the panorama capture (rotation), this means that these cameras can't be used to acquire panoramic videos, but the most interesting option in these cameras is theirs very high resolution, leading to image size achieving 1.8Go (Luhmann, 2010). The second category regroups panoramic acquisition systems (camears) that are composed by an assembling of many cameras. The creation of a panoramic image is done using specific software allowing the transformation of the images from perspective projection to a panoramic image projection such the equirectangular projection that is widely used (Fangi et al., 2018). In this category, we find a large variety of consumer grade cameras such Ricoh Theta S, LG 360°CAM, Samsung Gear 360°, or professional devices like Panono 360° and Ladybug 5 (Fangi et al., 2018;Kwiatek and Tokarczyk, 2014). The major advantage of this category of cameras is the possibility to record immersive videos, Immersive video involves thecapture of a live-action scene that presents a 360° field of view (Kwiatek and Tokarczyk, 2014). The second category of panoramic cameras is widely used in mobile mapping systems (MMS) (Blaser et al., 2017), this system can be defined as a photogrammetric mapping agent that is usually defined as a set of navigation (global navigation satellite system (GNSS) and inertial measurement unit (IMU)) and remote sensors such as cameras, LIDAR, and odometer sensors integrated in a common moving platform (Khoramshahi et al., 2019). These systems can capture the environment by locating the data collected by the sensors based on the position and attitude data collected by the navigation sensors. The environment capture is a prerequisite for the virtual reality generation and the use of MMS will allow offering a correct positioning of the virtual environment w.r.t the real geographic coordinate frame. The system proposed in this paper consider using a combination of a low cost panoramic video camera as a sensor with a smart phone used as a navigation sensor through a developed application to record the position provided by the GNSS sensor and the attitude given by the set of orientation sensors in the smart phone. The addition of a web mapping touch in the generation of virtual reality will facilitate works and save time by automating the positioning of these panoramic views of the site to be visited (Koeva et al., 2017). Many opensource libraries are available offering these possibilities; we can cite OpenLayers.js and leaflet.js for mapping and basic GIS functionalities and Three.js or Panolens.js to visualise panoramas.

OVERVIEW
Many Virtual tours online softwares are present in the market, we categorize them into three main groups the first one contains a kind of platforms dedicated to expose panoramas as artistic views where the users share their images in this category we can find Kuula.co, 360cities.net or Airpano.com which is specialized in exposing aerial panoramas that are generally taken by drones. In the second category we find those that are specialized in presenting panoramas based on 3D architectural models such modelo.io or openspace3d.com, they proposes the integration of panoramas as secondary functionality, this type of softwares are generally used by architects or in real estate advertising. The third category is the type of sites that offer the creation, the storage and the visualisation of the panoramic virtual tours, the offer a large variety of navigation tools and enhanced visualisation environment, we can cite here: marzipano.net, paneek.net, makevt.com, kolor panotool or vtility.net. These softwares doesn't offer the possibility to attach the panoramas to a geographical position. The main soft that provide fully this functionality is google street map, it offers a full linkage between the map and the panoramas but in visualisation mode only. The acquisition is mainly done by google using special equipment (see figure1), while users can share independent panoramas only or as part of private business advertisement(google maps, 2021). The google street view coverage map indicate that the capture is mostly focused on developed countries since it is commercially more interesting, but the developing and underdeveloped countries the coverage is very low. Therefore, it is important to develop a low cost, easy to setup system based on open source software allowing governments or small business in these countries to offer similar service that is very helpful for cultural and touristic sectors.

The panoramic image types
In the general case of simple photographs (non-panoramic images) where the field of view is less than 180 °, perspective projection is used. In the case of a panorama, this projection reveals significant deformations (image stretching) as soon as the opening angle becomes large and approaches the theoretical limit of 180 °. (Pierre Grussenmeyer and Élise Meyer, 2021) For this, other types of projections are used, and the type of projection used to generate it defines each type of panorama. The first type is the spherical panorama, it has the advantage of being able to represent the entire surrounding space (from -90 ° to 90 ° vertically and from -180 ° to 180 ° horizontally) homogeneously (except at the poles). Equirectangular projection is the transformation that allows the generation of this type of panorama, it allows passing from 3D space to a unit sphere of coordinates ρ, θ, φ (Aghayari et al., 2017). The spherical type of panoramas is used in this work, the environment is projected into the unite sphere using the following formula: Where ρ, θ, φ are the spherical coordinates of a given point The second type is the cylindrical one, the same equirectangular projection is used, but unlike the spherical, this type of panorama does not cover the entire vertical field of view (less than 180 °). The mathematical model is based on a relation between the pixel coordinates system (x', y') and the ground coordinates system (X, Y, Z) (Luhmann, 2010).
The last one is the cubic panorama, here the directions of space are projected (in perspective projection) on the six faces of a cube representing the six directions of three-dimensional space.
The graphic representation of a panoramic image in cubic projection is equivalent to a "development" of the faces of a cube. Cubic projection offers the advantage of large coverage of the surrounding space, but it introduces discontinuities at each edge of the projection cube.

The experimental system
In this paper, we present an experimental system composed of two parts. The first part is a low cost acquisition system. This system is composed of first, a smartphone allowing, through a developed android software, the collection of the phone's onboard navigation information provided by sensors such as accelerometer, gyroscope, magnetometer and GPS positions, the smartphone is used to capture the positions and attitude tagged with time. The developed android application called "Road Recorder" allow to record the GPS position in one file with a frequency of 1hz. In a second file we record a set of motion sensors readings at frequency of 100hz, these sensors can be physical (hardware) one like accelerometer, gyroscope and magnetometer or softsensors like orientation and rotation vector sensors, these softsensors represent a virtual sensor which combines data from different physical sensors and proceed to fusion in order to provide more accurate data rather than using raw data. Also, the application records a camera video stream from the phone camera with same time stamp as the previous sensors allowing for the future versions to proceed to the data fusion in SLAM fashion using the phone camera as "positioning" sensor.
The second component is a Samsung 360-gear (2017) camera to capture panoramic videos over trajectories (figure 5), this camera has a dual 8.4 megapixels sensor with integrated fisheye lenses, offering dual cam video resolution of 4096 x 2048 pixels at 24 fps, and dual cam photo resolution of 5472 x 2736 pixels. Similar camera has been used for 3D reconstruction and points cloud generation in a photogrammetric process (Barazzetti et al., 2018(Barazzetti et al., , 2017Perfetti et al., 2018).

Software description
In addition to the hardware part of this system, we have a set of software allowing the analysis, the organization and the visualization of panoramic sets to augment the mapping experience over open sky sites or roads and the navigation into the interior of buildings.
The used Samsung 360-gear camera allow the acquisition of two fisheye images covering more than 180 degrees of field of view, the software provided with camera was used to assemble the acquired images and videos and the composition of the spherical images into an equirectangular projected image.
The proposed system is considered as low-cost based solution using a camera that is considered as a costumer grade camera that costs about 140$, offering a resolution of up to 4096x2048 @ 24fps 360° Video which is quite sufficient for this application.
The smartphone is tightly attached to the camera as in (figure 5-6) then the transformation from the camera to the phone is a constant transformation therefore the attitude and position readings will allow the passage from the ground and the image coordinates frame and vice-versa. In order to locate the spherical image in the terrain we have to consider a series of transformations as fellow: 1. From image to camera using The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) Where x,y are pixel coordinates / λ, Φ : Longitude and latitude in radian wrt image unite sphere / m,n are the image size 2. From the camera to the smartphone This passage is ensured by a rigid spatial transformation expressed as follow: Since the camera and the smartphone are tied together, as shown in the next figure, the parameters are fixed as Tx=5cm, Ty=3cm, Tz=0 / r=0,p=90°,y=0° Figure 6: Transformation from the camera to the smartphone.

From smartphone to the ENU
After reading the attitude from the smartphone devices, it is possible to pass from the phone frame to the local ENU (East-North-Up) frame using a rotation matrix.

From ENU to geographic
This transformation is based on the reading of the position provided by the GPS embedded in the smartphone.

The synchronisation
The integration of the collected navigation data and the video will allow the geotagging of the panoramic images stream, this is done using a developed software to permit the synchronization between them since the smartphone data and the video are collected independently.
To obtain some point in time allowing the synchronization we proceed as follow: after the beginning of the capture, the user shall stay stable in a movement clear environment for few second, after that we start the movement on the wanted road or region. After the initialisation stage, the user is free to move and capture the desired area.
To detect the movement start moment on the navigation data we can analyse the velocity from GPS for a one-second accuracy or the accelerations from the accelerometer since it has a higher frequency. To detect this moment on the video we proceed by calculating the absolute value of the difference between the first image and the current image, an augmentation of the difference is observed at the start of the movement. Then each frame from the video can be attached to position/attitude data through time, this video is then subsampled into independent frames each 10 meters based on GPS data and the time stamp.
In figure 5, the 10 th second in the smartphone data corresponds to the 4 th second in the video.

The main web application
The created image/position/attitude dataset are then fed into a developed web platform allowing the integration and the visualization of the panoramas on maps (Figure 7). The main architecture of this application is shown in the next figure The positions of the two next and previous images are located on the current panorama automatically throw their back projection to easily navigate throw the collected data.
The International Archives of the Photogrammetry, Remote Sensing andSpatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) To deal with the indoor case where the GPS signal is missing therefore the automatic localization of the images is impossible, another interface was developed allowing the manual relative positioning of the panoramic photos in a way to visualize them on a single point (orange point on Fig. 9) on the map through which the site can be visited (Figure 10-11). The data of this type is usable for the touristic promotion, both for commercial case such hotels or cultural case such for museum, heritage sites ( Figure 12).

CONCLUSION
This work presents a low cost mobile mapping system composed of a consumer-grad panoramic video camera as sensor and a smartphone used for pose estimation.
The proposed system allow the acquisition of immersive video and its automatic analysis for geographic based subsampling and the location of selected panoramas on the ground to automatically create virtual tours with mapping context. Data collected and analysed is then fed into a developed web application allowing the visualization. This web platform is developed using exclusively open source software, where we used python flask for the backend and vue.js, Openlayers and Three.js for the frontend development. The proposed approach, as it is at very low cost, is very interesting for virtual tour introduction in the developing countries permitting the promotion of cultural site or even town presentation with a low budget allocation.