DEVELOPING A STRATEGY FOR PRECISE 3D MODELLING OF LARGE-SCALE SCENES FOR VR

In this work, we present a methodology for precise 3D modelling and multi-source geospatial data blending for the purposes of Virtual Reality immersive and interactive experiences. We evaluate it on the volcanic island of Santorini due to its formidable geological terrain and the interest it poses for scientific and touristic purposes. The methodology developed here consists of three main steps. Initially, bathymetric and SRTM data are scaled down to match the smallest resolution of our dataset (LIDAR). Afterwards, the resulted elevations are combined based on the slope of the relief, while considering a buffer area to enforce a smoother terrain. As a final step, the orthophotos are combined with the estimated Digital Terrain Model, via applying a nearest neighbour matching schema leading to the final terrain background. In addition to this, both onshore and offshore points-of-interest were modelled via image-based 3D reconstruction and added to the virtual scene. The overall geospatial data that need to be visualized in applications demanding phototextured hyper-realistic models pose a significant challenge. The 3D models are treated via a mesh optimization workflow, suitable for efficient and fast visualization in virtual reality engines, through mesh simplification, physically based rendering texture maps baking, and level-of-details.


INTRODUCTION
During the last decade, Virtual Reality (VR), existing in different forms, content, and devices, has emerged as a disruptive technology for engaging spaces and objects that would otherwise be physically inaccessible and unknown. High quality visual perception is the key for VR to achieve immersive experiences since realistic reproduction of sound and haptic senses are still at an early stage. There is a variety of technology innovations that made VR vastly available to the public; the development of low cost sensors for real-time tracking and position / orientation estimation, new 3D data capturing devices, the vast adoption of web services, the continuous increase in hardware capabilities and the existence of accessible high quality headsets have all pushed VR to become a key mean of interaction for a variety of applications. Furthermore, the frameworks that support Virtual Reality, such as Unity3D and Unreal Engine, have matured enough to support all main commercial devices and give access to non-experts and creators that are not software engineers to the VR world for developing applications in all sort of domains. VR has become the section of different academic and industrial fields. Architecture, engineering, construction (AEC), Training, Cultural Heritage (museums and sites), Entertainment (Games and Films), Healthcare and Digital Twins, are only few of the sectors that vastly benefit from the VR. For example, key software for AEC begins to integrate in real-time with VR engines. Moreover, VR plays an important role in critical applications (e.g. disaster management) by increasing situation awareness. Spaces of interest like archaeological sites, geoparks and natural habitats are some characteristic cases that could benefit from representing them in VR. A key element of VR advances is the abundance of captured data. Thus, accurate geometric 3D representations based on actual measurements can support incredibly immersive experiences, comparing to artificial designs, or mathematical models' generations. On the other hand, VR is an exceptional way of presenting reality when it is not accessible. The future of VR for public and experts looks promising with the recent graphics processing units supporting real time raytracing and gradually boosting the development of even more realistic scenes.

PREVIOUS WORK
Developing an application for VR includes the selection of a modelling approach for the scene. 3D surveying is an increasingly adopted approach as it ensures a high level of realism, but also elevates the potentials of VR, as discussed in literature (Zhao, 2009). Accurate 3D information, such as imagebased and range scanner-based point clouds are rapidly becoming more available to the users, easy and low-cost to acquire. Hence, it is an open topic to discover novel and effective ways to make good use of them and increase the realism of VR experiences (Virtanen et al., 2020). Moreover, extended reality (xR) applications usually require the geographical reference of any kind of data, either this is the base map, some detailed 3D models, or any other layer of information. The complexity of these geospatial data often demands software design decisions that affect the xR experience (Richardson et al., 2019).

Geospatial data in VR
Geographic Information System (GIS) and Virtual Reality (VR) are met in the 90s (Haklay, 1998), but the last few years the cross section of the two fields (VRGIS), along with 3D modelling, has evolved rapidly due to the advances in hardware for mobile VR, sensors and desktop processing and the big data that allow storing, processing and serving huge amounts of geo data. VRGIS properties initially set two decades ago, are realism, free navigation, and full GIS capabilities, such as spatial queries (Faust, 1995). Although VR is not yet mainstream capability, except perhaps some specific fields, such as airspace industry, research projects and market analysis in VRGIS indicate that it can be used for educational purposes, training, and planning. At the same time, augmented reality, which can be helpful in field operations, either this is an industry, or a city-scale interaction, demands the exploitation of geospatial data. Also, VRGIS is gaining momentum along with the rise of digital twins and internet-of-things, as these technologies bridge the gap between digital and physical worlds. The fusion of VR and GIS is grabbing increasing importance and applicability in a variety of domains, such as urban planning, environmental planning and impact assessment, security and risk management, visualization of complex data, cultural heritage modelling, education, and training (Fisher and Unwin, 2002). Risk assessment in difficult and hazardous situations can also vastly benefit from VRGIS, thus VR based on real 3D data is a key enabling technology for first responders, such as fire fighters, in order to train them in a realistic environment, but also to simulate the evolving conditions of an event during its progress (Moreno et al., 2012). Professionals on any many application fields rely on geographic information to make real-time decisions. In Wagner (2019), the concept of situational awareness is raised, even in the context of virtual meetings in a 3D environment.
A high-impact application for VRGIS is city management; 3D modelling of cities is gradually converging with GIS and BIM to give access to landscape, building information, traffic, infrastructure, residents, to name a few, to public and authorities. (Li et al., 2016(Li et al., , 2015. 3D models derived from different methods can fuel a number of city-scale application when exploited in VRGIS (Biljecki et al., 2015). In Chandler et al. (2018), the applicability of VR and geospatial information is demonstrated in monitoring and operating different cases of built environment across an infrastructure life-cycle. Educational exploitation includes the development of tools for immersive VR for geosciences presenting accurately reconstructed 3D models (range scanners and images), such as a volcano and modelling the progress of an eruption (Zhao et al., 2017) . This will allow nonexperts in modelling and VR to conduct field work without the difficulties these trips have by the safety of their office.
Data and information manipulation on the actual 3D space is gaining importance. Marriott et al. (2018) pinpoint the value of 3d information visualization. Atmospheric and environmental data can also be presented in VR to facilitate not only for interactive visualization of complex data, but also for better analysis via the spatial component. Thus, Helbig et al. (2014) show that VRGIS can assist the climate change understanding via the increased capabilities for data analysis. In (Li et al., 2011), geodata from harsh environments, such as marine environment in Antarctica, are visualized, interpreted and analysed in VR. More generally speaking, nowadays, large volumes of data with many dimensions tend to be the typical scenario for many use cases and many of them are georeferenced. Hence, the visualization of such, in a wider sense, geospatial data is a demanding field, which can actively assist in the interpretation of complex data (Andrienko et al., 2010). VR can offer a lot since in already offers a 3 rd dimension. (Zhong et al., 2012) analyse the spatiotemporal visualisation of georeferenced data. The advances to the area of data manipulation are moving to the direction of utilizing an immersive environment (immersive analytics), to have better collaboration among experts from different domains and datadriven decisions.
Constructing large VR volumes aggregate difficulties which exceed conventional 3D display visualization. In our case, a reverse problem statement workflow was conducted, hence, the technical specifications needed for VR, along with the user requirements, set the needs of the data capturing, processing and blending. Moreover, data integrity and quality assessment are prerequisites for a valid integration of geospatial data in a VR world.

DATA COLLECTION
The onshore data consist of drone and satellite (Sentinel 2) RGB imagery, as well as Shuttle Radar Topography Mission (SRTM) and LIght Detection And Ranging (LIDAR) data for Digital Elevation Model (DEM) generation. The underwater data of Santorini island consist of a bathymetric digital terrain model, but additional data from open DEM (EMODnet) were fused to the final seabed 3D model. Moreover, ROV missions took place in certain Points-Of-Interest (POI), leading to prominent 3D reconstructed models and highlighting Santorini's underwater environment. Regarding the ROV images, further research was conducted to address the colour effects in underwater environments. Finally, GNSS measurements provide the necessary georeference of the 3D models.

Onshore data
In recent years, UAVs became a common platform for data acquisition due to their low-cost and their adaptability to different needs and requirements. Moreover, satellite images with great spatial resolution like Sentinel-2 provide with adequate multispectral data to serve a wide range of applications. In our case, Sentinel's RGB orthoimage was used as a texture-map on the 2.5D terrain. Finally, SRTM data are exploited as the base elevation model of the area in study.

UAV data
Image datasets from drones were collected in such a way as to contain vertical and oblique imagery, describing parts of Oia and Fira settlements, as well as the Kammeni island. Thirty two flights, containing a total amount of 10000 images ( Figure 1) were performed using a DJI Phantom 4 Pro, while the total area covered is around 160 hectares. In this work, we propose an effective method for drone image collection in steep and rough terrains similar to Santorini's Caldera. Four flight directions for Fira and Oia were planned, one set of two oblique and one of nadir acquisition directions, while for each set, flights went alongside contour lines and across them. Furthermore, an overlap of 80-90% on each direction, within flight strips is essential. Capturing oblique images with different inclination setups lead to a more precise and reliable 3D model reconstruction. Also, ground sampling distance varies in oblique images and this can become a critical failure in ground sampling in steep terrains, thus, the oblique flights were approximately 20% lower than vertical. The final ground sampling distance is around 1.5-2 cm and the results are considered to be exceptional. summarizes the usual process of defining the flight parameters, and the various additional parameters such as ground control points distribution. Also, aggregates the data capturing process with the data processing and the photogrammetric products.  The flights were carried out manually, semi-automatically or with autopilot, depending on the environmental conditions and the risk assessment that took place in situ. In total, 120 Ground control points were surveyed by a GNSS receiver. Considering the enormous height differences and the dense urban terrain, ground control points were distributed along landslide so that a GCP would exist every 30 meters in 3D world space. Due to local legislation, only physical points were measured such as manhole cover and sidewalk edges.

Satellite data
The RGB data for Santorini's terrain derive from a Sentinel-2 image (Figure 3Figure 3). The spatial resolution of the texture map created from the ortho-image is 10 meters. Furthermore, color adjustments were applied to improve the vividness of the colours of the satellite images.

DEM data
The digital elevation model (DEM) from SRTM (Shuttle Radar Topography Mission) data with spatial resolution of 30 meters was used (Figure 4). SRTM employs interferometry, a wellknown radar data capturing process in geodesy and remote sensing.

Offshore data
Bathymetric data are becoming increasingly important for scientists and experts in fields such us maritime transportation, environmental studies, oceanography and seabed modelling. A typical multibeam bathymetric mission can scan hundreds of square kms. The bathymetric model, which is provided by NKUA, was combined with the onshore DEM leading to a complete 2.5D terrain model of Santorini island. ROV missions The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B4-2020, 2020 XXIV ISPRS Congress (2020 edition) were conducted in certain points-of-interest, while preprocessing image enhancements were applied on the raw images.

Bathymetric data
Multibeam bathymetric surveys were carried out by R/V AEGAEO of the Hellenic Centre for Marine Research (HCMR), during three cruises carried out in 2001, 2006 and 2012, covering an area of 2.480 km 2 over the Santorini volcanic field (Nomikou et al., 2014(Nomikou et al., , 2013(Nomikou et al., , 2012. The surveys utilized a 20 kHz, hullmounted SEABEAM 2120 swath system, suitable for operation in water depths between 20 m and 6000 m and at speeds up to 11 knots. The system forms 149 beams over a maximum angular coverage of 150°, covering a swath width up to 7.5 times the water depth. The typical water depth in the survey area is 500 m, corresponding to a swath width of 3.75 km. The average position of the ship was determined to within ±10 m by GPS navigation (Trimble 4000). The multibeam data processing included georeferencing using navigation data, removal of erroneous beams, noise filtering, interpolation of missing data and removal of rogue points (e.g., Caress et al. 2008).
Furthermore, bathymetric data were acquired on-board the R/V Marcus Langseth's using the Simrad Kongsberg EM122 12kHz multibeam echo sounder, in 2015 (Hooft et al., 2017). Data were processed using the MB-SYSTEM open-source software and merged with pre-existing data (Nomikou et al., 2013(Nomikou et al., , 2012. The EM122 multibeam bathymetry system has a maximum swath width up to 6 times the water depth depending on the sea state. The multibeam data processing included georeferencing using navigation data, removal of erroneous beams, noise filtering, interpolation of missing data and removal of rogue data points. Particular attention was paid to abrupt changes in seafloor topography due to steep relief at fault zones and volcanic edifices, these features were inspected and manually edited. Santorini's caldera walls, rise to over 300 m above sea level, while the maximum depth of the caldera seafloor is about 390 m below sea level. The present configuration of the caldera consists of three distinct basins that form separate depositional environments (Nomikou et al., 2014(Nomikou et al., , 2013. The North Basin is the largest and the deepest (389 m) developed between the Kameni islands, Thirasia and the northern part of the Santorini caldera. It is connected by a narrow steep-sided channel with a depth of 300 m to a scallop-shaped ENE-WSW aligned feature that lies outside Santorini caldera, NW of Oia Village.
The smaller West Basin is encompassed by Aspronisi islet, Palea Kameni and Southern Thirasia with a moderate maximum depth up to 325 m. The flanks of the basin are gentle in the western part and steepen close to Thirasia and Aspronisi. The South Basin is bounded by the Kameni islands (to the north) and the southern part of the Santorini caldera (to the south). It covers a larger area and is shallower by ~ 28 m than the western basin. The seafloor morphology suggests that the southern basin has been separated from the western and northern basins by the development of a series of subaerial and submarine volcanic domes, aligned in a NE-SW direction. Apart from the subaerial Kameni islands, the most well-known submarine extrusion is the reef close to Fira Port, which has grown from 300 m b.s.l. up to 40 m b.s.l.

ROV data
ROV missions took place in different POI. Due to normal underwater light conditions, initial images preserve haze, noise and lack in vividness. Thus, pre-processing image enhancement filtering is most of the times, essential. In Figure 5, initial and filtered image are shown. A typical ROV ( Figure 6) survey can be held similarly to a UAV mapping mission, i.e. keeping constant distance from objects, preserving the overlaps etc. A really important task before mission is to perform a detailed study of the area of interest, so that various determinants are known in advance, such as the morphology of the area both underwater and in the coastal zone, the possible access points , the distance from the shore, water conditions such as clarity, underwater currents, etc. Depending on the area of study and the desired product characteristics, some general guidance is introduced. Initially, high overlap 80-90% is compulsory, mainly to address the miss-manoeuvring effects and the height differences appeared, when the distance from the object in survey cannot be consistent. Scale bars, with size, respective to the capture distance are also necessary.

3D RESONSTRUCTION
3D reconstructed models from images are increasingly being exploited in VR environments, although there are certain limitations and difficulties that need to be addressed. In this work, we integrate Meshroom and Colmap in our methodology, for the process of ROV and UAV imagery. Colmap () is capable of inserting ground control points in aerotriangulation solution and exploits GPS data while Meshroom () facilitates the 3D reconstruction process leading to minimum user interaction.

ROV data
Enforcing image enhancements (Figure 7) led to finer 3D reconstructed models. Feature extraction and dense matching results show that color corrections play a significant role in underwater environments. Two datasets are exploited (Figure 8), one contains images from part of Dykes (Santorini's Caldera) and another one containing images of a shipwreck.

UAV data
UAV captured images were processed in Colmap (Figure 9). The main reason is that in Colmap it is possible to integrate GPS and GCP data in the solution. Over and above this, a main difference between underwater and urban models created in this work, is that modelling areas such Oia and Fira is far more challenging and requires a lot of images. Thus, it is advisable to preserve the control of the 3D reconstruction process. Figure 10 and Figure 11 show the results for Fira, Oia and Kammeni island.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B4-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 11. 3D Reconstruction results of Kameni island.

GEOSPATIAL DATA BLENDING
In many applications, e.g. study of evolution and prediction of geological phenomena, it is necessary to blend data that have been collected in the past with more recent ones. This process raises a number of questions, such as how the data projection systems can be connected, how their different spatial resolution is connected, i.e. different pixel sizes, how the geospatial accuracy of each set affects the end result and so on. Bathymetry data most of the times is carried out independently of the topographic data and normally, for different purposes. This makes their union quite difficult due to their georeference mismatch e.g. It is also known that a coastal area is a dynamic environment that constantly change shape, either due to human intervention or natural causes. Particularly in Santorini, morphology of coastal areas may change radically, due to the geological phenomena observed on the island. These are some of the main issues in depictions of large digital geospatial backgrounds or as it is addressed in literature, digital worlds. A common process in Bathymetry and topography data is to place one digital model on top of another and merge their overlapping sections using simple statistical methods. The methodology proposed in this work was conducted in "global mapper". Bathymetric (10m) and SRTM (30m) data were scaled down to meet the smallest resolution of our dataset (LIDAR) leading to an underwater elevation model. Afterwards, the elevation data are combined based on the slope of the relief which is applied within a buffer zone of 51 meters. Although this minimizes the spikes and the steep effects on the terrain, it can lead to ambiguities, so some basic manual treatment may occur. Finally, the digital elevation model data and the Sentinel 2 imagery are combined via applying a simple nearest neighbour matching schema leading to the final 2.5D terrain background. This led to better and more descriptive results.

Methodology proposed
Initially, a contour line based on sea level (altitude 0m) is created. Additionally, two buffer zones, for both directions (inside and outside) of the contour line, 50 meters and 1 meter respectively, are constructed automatically. Retrieving the union of both polygons lead to a final strip polygon (Figure 12) of 51 meters width, covering 50 meters horizontally offshore and 1 meter onshore. The objective of these values (50m-1m) is mostly to preserve the onshore details and to map the RGB data only to the onshore elevation model. Figure 12. The automatically generated strip polygon of 51 meters width.
Afterwards, using the slope of the reliefs of the on shore DTM model and the bathymetric height model, a simple linear interpolation is held. Figure 13 shows a small part of the generated model. Figure 13. DEM construction by using simple interpolation based on the slope of the two reliefs.
A 2.5D model of Santorini combining Bathymetry, LIDAR and SRTM data is created (Figure 14, Figure 15). Finally, a texture map is constructed by simple applying a nearest neighbour color match ( Figure 14). Figure 16 shows the result in 3D view in Global Mapper.

3D MODEL OPTIMIZATION FOR VR
Although latest GPUs are capable of handling tens of trillions floating-point calculations every second, 3D assets for VR environments make a huge impact in performance. Moreover, it is essential to have a smooth and simple model without losing the details though. Triangular meshes are widely used as a representation tool for 3D objects and basically, include the geometry of the object which specifies the vertices, their colors and in some cases, additional properties such us normal vectors etc. In this work, a mesh retopology and simplification is applied on the 3D models (Figure 17), both tools given by Instant Meshes (Jakob et al., 2015), a powerful and open source tool for 3D mesh optimization. Simplified meshes are then re-textured in Meshroom. Additional products are created base on PBR approach. Normal map, Height Map and Ambient Occlusion Map are applied on the simplified model. "xNormal" uses the geometry of the high poly model to estimate the Normal Map of the low poly model. As a final step, LODs are generated automatically by employing "Simplygon" software.

GEOSPATIAL DATA IN VR
Nowadays, loading a VR-ready model in Unity is rather a simple procedure. Depending on application requirements additional post-processing camera effects as well as further lightning can be applied on the scene. In Figure 18, the resulted models are been explored in VR mode.

CONCLUSION
To conclude with, large scenes that have a lot of details and accurately depict reality, can be efficiently presented in VR environments and this will become more and more relevant to the near future, as the 3D content (from reality) will increase and the capacity of VR devices will improve.