LOW-COST DEVELOPMENT OF AN INTERACTIVE, IMMERSIVE VIRTUAL REALITY EXPERIENCE OF THE HISTORIC CITY MODEL STADE 1620

: As virtual reality and 3D documentation and modelling technologies become increasingly powerful and affordable tools for architecture, planning, and cultural heritage preservation and communication, it has become increasingly important to develop low-cost methodologies for the creation of 3D immersive virtual environments and interactive experiences. Doing so makes this technology more viable for institutions such as museums and other cultural institutions, who often work within strict budgets. In this paper, we describe a workflow used to build an interactive, immersive virtual reality experience around a virtual city model of the town of Stade (Germany) in the year 1620. This virtual city model is based on a physical 3D model of the town, exhibited in the Stade town hall. The workflow begins with the digitization of this model using digital photogrammetry, followed by the subsequent low-and high-polygon modelling of the individual architectural assets in Autodesk Maya, texture mapping in Substance Painter and finally visualisation within Unreal Engine 4. The results of this workflow are a detailed 3D historical environment with a high degree of realism and in which interactivity can easily be added. In addition, the workflow takes a highly iterative approach that allows the performance of the virtual environments in the game engine to be monitored at each stage of the process, and that allows adjustments to be made quickly. To increase the potential of the virtual environment as a tool for education and communication, interactive elements and simple game mechanics are currently being integrated.


INTRODUCTION
In recent years virtual reality technology has become cheap and powerful enough to reach a much broader public.This development presents exciting new opportunities for cultural heritage institutions, such as museums, seeking to engage new audiences through their collections and archives.In particular, coupled with digital 3D reconstruction techniques, virtual reality can allow audiences to experience historical spaces at a 1-to-1 scale in a spatially immersive visual and auditory environment.Furthermore, the use of interaction within these virtual worlds encourages a degree of sensory-motoric immersion (Holopainen, Björk, 2005), and is thus a very powerful tool for learning.As a result, this has inspired an increased interest in the gamification of cultural heritage education, bringing it under the banner of 'edutainment'.In this paper, we show that these technologies can be employed at low-cost in order to create an interactive, immersive virtual reality environment.In particular, we focus on the case study of the digital 3D reconstruction of a historic city model of the town of Stade in the year 1620.This project was conceived in cooperation with the Schwedenspeicher Museum in Stade, where the VR experience will be exhibited as an educational tool for museum visitors and school groups to present a brief look into the history of Stade.

RELATED WORK
There exists a small but growing corpus of digital 3D historical city models that has emerged in recent years thanks to the increasing availability of the necessary software and hardware.Physical 3D scale terrain and city models gained popularity * Corresponding author from the 16th-century onwards, primarily as a tool for military planning (Warmoes, 2018).Indeed, there are several examples of digital 3D models that take an existing physical 3D city model as their starting point.These include the city of Solothurn (Switzerland) by Nebiker et al. (2009), Duisburg (Germany) by Jedrzejas, Przybilla (2009), Prague (Czech Republic) by Sedlacek, Zara (2009), Toul (France) by Chevrier et al. (2010), Hamburg in 4D by Kersten et al. (2012), Liege (Belgium) by Pfeiffer et al. (2013), and Geneva (Switzerland) by the Auguste Magnin Relief Project, coordinated by the city of Geneva (http://makingof.geneve1850.ch/3D/FR/#Geneve_1850-3D).The digital models produced by these projects have a wide range of applications.While some have been built for the purposes of documenting, preserving and communicating museum collections (e.g.Chevrier et al., 2010), others were conceived for their potential as research tools (e.g.Pfeiffer et al., 2013).Needless to say, these two aims are not mutually exclusive.The use of virtual reality as a way of exploring these models at a human scale is, however, comparatively rarer.This is perhaps due to the challenge of modelling cities with sufficient detail to appear convincingly realistic within the virtual environment, while at the same time adhering to a stringent set of technical criteria that puts strict limits on the amount of data that can be loaded.The potential of virtual reality city models that can be explored at a 1-to-1 scale is nonetheless great, not only for presentation and planning, but also for education and the diffusion of knowledge (Thompson et al., 2006).Virtual reality city models have the power to simplify extremely complex information for people with different levels of expertise through immersion and interactivity.
In particular, historical architecture and cultural heritage are ideal subjects for interactive visualisation in virtual reality as they already tend to be landmarks in their own right and are often exhibited in museums or as tourism attractions.Because of this, such sites already have large, interested audiences that may be engaged through VR experiences.Several examples of VR visualisations of historical geo-data are published in the literature.The Old Segeberg town house was introduced as one of the first virtual museums using the HTC Vive as a Head Mounted Display (HMD (Kersten et al., 2017b).In addition, two historic towns (including the surrounding landscapes) were developed as VR applications for visits in the past: Duisburg in the year 1566 (Tschirschwitz et al., 2019) and Segeberg in the year 1600 (Deggim et al. 2017b).Two religious cultural monuments are also available as VR experiences: the Selimiye mosque in Edirne, Turkey (Kersten at al., 2017a) and the wooden model of Solomon ś temple (Kersten et al., 2018).Furthermore, Edler et al. (2018) present a workflow for constructing an interactive cartographic VR environment to explore urban landscapes.

THE HISTORIC CITY MODEL OF STADE
The Hanseatic City of Stade is located 45 km to the west of Hamburg, in the state of Lower Saxony in northern Germany.Connected to the Elbe by the Schwinge tributary, the town of Stade was once a powerful member of the Hanseatic League of city-states that dominated trade in northern Europe and the Low Countries from the 12th through the 16th centuries.Excluded from the league in 1601, the town was under Swedish rule from 1645 onwards, and then subsequently under Danish rule from 1712.The Schwedenspeicher Museum in Stade holds numerous town plans and maps from the period that give some idea of the extent of the city and its surrounding area.In the foyer of the town hall there is a physical 3D model of the town, based on its layout in the year 1620, which gives information about the numerous churches in Stade.Many of these existed up until the great fire of 1659 and bear witness to the rich religious and cultural life of the medieval town.In its heyday, the city had five districts, each with its own church: the monastery quarter with the church of St. George; the merchant quarter with St. Cosmas et Damiani; the bishop's quarter with St. Wilhadi; the fishermen's quarter with St. Nicolai; and the castle quarter with St. Pankratii.The physical 3D model of the town (Figure 1) has dimensions of 2.1 m × 1.3 m, represented at a scale of ca.1:550.
Unfortunately, little is known of the history of the physical model itself.It was commissioned for the original opening of the Schwedenspeicher Museum in 1977, and subsequently moved to the foyer of the city hall in 2010 when the museum was under refurbishment.The name of the artist or indeed the exact historical sources on which the reconstruction are based are not known.Given that it was originally commissioned by the museum, however, it may be safe to assume that the model was intended as an interpretative aid for visitors.

THE UNREAL GAME ENGINE AND HTC VIVE PRO VR SYSTEM
A game engine is a software-development environment designed to create video games for consoles, mobile devices, and personal computers.Its main functionality is typically comprised of a rendering engine for 2D or 3D graphics to visualize textured 3D models, a physics engine or a collision detection to interact with objects, an audio system to emit sound, scripting, animation, artificial intelligence, streaming, and performance management.A game engine is a powerful platform to display 3D terrain data in VR, because it provides the functionalities necessary for a user to become immersed in a virtual world.
At present, several game engines offer free access for developers, such as the Unity engine from Unity Technologies, the CryEngine from the German development studio Crytek, and the Unreal Engine 4 (UE4) from Epic Games.A current overview and comparison of different game engines can be found in GameSparks (2019).Selection of the appropriate engine for a project is based on the components mentioned above and its adaptability to the existing work process.For this project, UE4 was chosen because it implements -opposed to other engines -a visual scripting language known as Blueprints.Visual programming with Blueprints does not require the writing of machine-compliant source code and enables a much easier entry into the development environment for users with little programming knowledge.In addition, Unreal offers sufficient access to documentations, tutorials, and asset stores.Due to limited time for this project, no other engine was tested with the proposed workflow.
HTC Vive Pro (www.vive.com) is a virtual reality headset (Figure 2) for room-scale VR.It was developed by HTC and Valve Corporation, first version released on 5 April 2016, and is currently available on the market for EUR 879 (October 2019).Basic components are the headset for the immersive experience, two controllers for user interactions, and two "Lighthouse" base stations for tracking the user's movement (Figure 3).The device uses a gyroscope, accelerometer, and laser position sensor to track the head's movements as precisely as one-tenth of a degree (Painter, 2015).Wireless controllers in each hand, with precise SteamVR-tracking, enable the user to freely explore and interact with virtual objects, people, and environments.The VIVE-controller is specifically designed for VR with intuitive control and realistic haptic feedback.The Lighthouse system uses simple photo sensors on any object that needs to be captured.By sweeping a line laser over an object, the time difference of arrival is calculated over various photo sensors to estimate a 6-DoF pose.To avoid occlusion problems by non-penetrative laser light and to increase tracking accuracy, this is carried out with two diagonally positioned Lighthouse stations that light the space.Figure 3 shows the setup of the VR system HTC Vive Pro, including the interaction area (blue).

WORKFLOW
The following workflow was developed to create the virtual city model: (1) data acquisition by digital photogrammetry, (2) image orientation, dense point cloud generation, and meshing using the software Agisoft PhotoScan/Metashape, (3) production of height map from photogrammetric meshed model in Autodesk Maya, (4) loading of height map into UE4 and block-out of scene using low-poly meshes, (5) 3D solid modelling of high-poly building meshes with Maya guided by photogrammetric mesh and reference material, (6) UV texture mapping of polygon meshes using Maya and Substance Painter, (7) high-poly meshes integrated into scene in UE4, (8) modelling and painting of landscape using native UE4 tools, (9) integration of motion controls and interactions in UE4, and (10) immersive and interactive visualisation of the city model in the VR system HTC Vive Pro using Steam VR 2.0 as an interface between the game engine and the HMD.

DATA ACQUISITION
In

3D MODELLING, TEXTURING AND VR INTEGRATION WORKFLOW
For this project, the modelling and texturing workflow was optimised in order to minimise iteration time between different versions of the virtual model, as well as to facilitate performance testing of the final VR experience.
The photogrammetry mesh itself was too heavy to integrate directly into the game engine.Therefore, a height map was generated from the photogrammetry mesh using Autodesk Maya, then subsequently brought into UE4 as a displacement map to generate a high-definition (4K) landscape mesh.Using the native modelling tools within UE4, low-poly buildings were modelled and placed on the landscape in the locations of the architectural structures (Figure 5).In order to reduce future modelling time, while at the same time retaining sufficient variability between the structures, a total of 13 different lowpoly townhouse variations were created in order to represent all 974 townhouses in the city.These low-poly variations were designed in order to represent as accurately as possible the variation of house types found in the physical model, differing according to variables such as footprint size and shape, building height, number of floors, and roof type.In addition to the houses and other domestic structures, additional low-poly meshes were modelled for the city's defensive and harbour Next, these low-poly meshes were exported to Autodesk Maya to use as templates for the creation of high-poly architectural models.In order to recreate accurate architectural details for the high-poly models, it was necessary to gather a significant amount of additional reference material.The detail of the physical model itself was insufficient to create a convincing and realistic model when scaled up to 1:1.Furthermore, it was decided that the digitisation of the model would provide a valuable opportunity to supplement the model with additional historical details.In addition to architectural reference photographs taken during research trips to Stade and Lübeck (the latter another former Hanseatic League city with surviving 17th-century architecture), reference material was gathered from painters of the Dutch Golden Age (e.g.Jan van der Heyden, Cornelis Springer), online databases (e.g.http://www.fotografie-architektur.de/index.html),as well as other 3D digital reconstructions of north German and Low Countries architecture.Archival material, including old city plans and views from the 16th and 17th-century city, were provided by the Schwedenspeicher Museum (Figure 6).The museum exhibits themselves also provide valuable additional information on architectural styles present in Stade during the 17th century.

A B C
Figure 6.View of the town of Stade in the year 1550, after a woodcut by Martin Weigel (Wirtgen, Bohmbach, 1974).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W17, 2019 6th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 2-3 December 2019, Strasbourg, France A modular workflow was adopted to model the townhouses and defensive structures of the town (including the wall and defensive towers).This workflow involves breaking down architectural structures into smaller, repeatable elements, which can then be recombined in a number of different ways to create multiple unique models.Once the individual elements have been modelled, different iterations for complete buildings can be quickly tested and finalised.The modelling of the special buildings, listed above, was done on an individual basis, with limited use of modular parts, and instead a focus on the particular idiosyncrasies of each structure.Gradually, each of these high-poly meshes was reimported into UE4 and replaced the low-poly placeholder meshes in the scene.At each step of the high-poly modelling, care was taken to ensure that the VR scene continued to comply with performance benchmarks outlined in the UE4 documentation (https://docs.unrealengine.com/en-US/Platforms/VR/DevelopVR/Profiling/Overview/index.html).
Once the high-poly models had been integrated into the scene, the models were UV-unwrapped in Maya and textured in Substance Painter, a 3D model painting program.Substance Painter allows for the combination of image textures and procedural design algorithms in order to create more complex texture sets.All textures for each structure, including diffuse (Figure 7a), normal (Figure 7b), ambient occlusion, metallic, and roughness maps (here combined in one texture atlas -Figure 7c), were packed together onto 4K texture atlases to increase run-time efficiency within UE4.Each architectural model in the scene received one material (in UE4, a material is used to refer to the combination of 2D textures and the shader code that tells the game engine how to render the textures) including the three texture atlases shown above.In areas where repeating textures were obvious, vertex painting was used to add an additional level of variation and contribute to the realism of the environment.For each mesh, four Level of Detail (LOD) models were generated using UE4 native tools, in order to boost performance by reducing the number of triangles in those models that were further away from the camera in a particular frame.Figure 8 shows four different screenshots taken within the Unreal Editor window, representing the state of modelling and texturing of the various architectural elements at the time of writing.
The original 4K resolution landscape that was used for the placement of the low-poly models in the scene was replaced by a 2K resolution landscape for the final scene in order to increase run-time efficiency.This landscape was hand-painted to create the streets, paths, and open areas of vegetation, using the native UE4 landscaping tools shipped with the engine.Additional plant meshes, including deciduous trees and meadow grasses, were also placed in the landscape by hand using the foliage tool in UE4.A second, low-poly landscape was created using SRTM 1-arc second DEM data to serve as a low-poly backdrop to the city.
In order to improve run-time efficiency, the lighting within the engine was baked into individual light maps for each building, reducing the render calculation load on the engine.
Finally, a navigation mesh was created in the scene and basic motion controls and interactions programmed using the blueprints node-based programming interface in UE4.These allow the user to move around the virtual city using a simple point-and-click teleportation function linked to the HTC Vive controller.
With the basic visualisation and interactions completed, the next step in this project is to develop further interactions, bringing the city model closer to an immersive and interactive world that can be used in education and for serious games.One aspect currently being prototyped is the addition of 360panorama photographs of the modern city of Stade that can be viewed in virtual reality by clicking on information points at particular locations in the virtual city.This juxtaposition of the historic and modern city invites users to directly compare the contrasting urban landscapes as an entry point to exploring the historical processes that shaped the town.
In order to test-implement this feature, a total of eight 360degree photographs were taken on 09.09.19 with a Samsung Gear 360 camera, at different locations in the town (Figure 9).These were geotagged to aid later placement in the virtual environment.The photos were stitched together using the native Samsung Gear app for iOS, and imported into UE4 as panoramas.These panoramas were then projected as textures onto the inside of eight UV-unwrapped spheres, and placed within the environment at the appropriate places.At each of these locations was also placed a 3D animated widget (in this case, an animated circle on the ground) that serves to draw attention of the user to the area and indicate the presence of a panorama (Figure 10).
Figure 10.3D animated widget notifying user of the presence of a panoramic photo, in location corresponding to panorama photo in Figure 9.
The interactive elements of this feature were implemented using the UE4 Blueprints visual coding interface.Once users enter the immediate area of the widget, a trigger is activated allowing the panorama to be switched on.A text prompt accompanies this widget and encourages the user to press the correct button to enter the panorama.Once these conditions are fulfilled (user in correct area and pressing correct button), the panorama sphere appears and the user is fixed inside the sphere in such a way that ensures they are able to view it from the correct position.Pressing the button again allows them to leave the panorama sphere, and once more hides the sphere.
With this basic prototype implemented, in the future these relatively low-quality photographs can be replaced with higher-quality versions.
In addition to this, a number of further extensions may be envisioned: -Information points to give supplementary information about different historic structures in the town.-A transportation feature to ease travel around the virtual space, which is large in size.-An introductory tutorial to help new users navigate the virtual world for the first time.-Physical information signage about the virtual city within the museum exhibit to help transition users from real to virtual world.-Simple game mechanics that provide challenge and immersion within the virtual world.
The precise implementation of these and other features depends on further discussions with and requirements of the Schwedenspeicher Museum in Stade.

CONCLUSION AND OUTLOOK
The VR experience is planned to be included in the Schwedenspeicher Museum in Stade in 2020 for the 400 year anniversary using the HTC Vive Pro as a head-mounteddisplay and running on the Steam VR 2.0 platform.
This project has been important in developing a workflow that allows for high-detail visualisations of virtual city models.The high level of detail achieved is crucial to the immersion and realism of the virtual environment, but comes at the cost of increased time spent on the modelling and texturing stages.At the time of writing, around 450 hours of work have gone into the project, of which 275 have been dedicated to high-poly modelling and texturing.Emphasis was placed on the hand modelling of certain individual architectural features in order to better reflect the variety and diversity of shapes and forms that would have been found in the historical city of Stade in 1620.
In addition, the high level of detail in the virtual environment had to be constantly balanced against the performance benchmarks of the hardware and software used to exhibit the VR experience.Epic Games, who publish UE4, recommend a frame rate of 90 Frames per Second (FPS) during real-time rendering using the HTC Vive Pro with Steam VR, in order not to cause discomfort for the user while wearing the headset.Although this high frame rate is not critical for VR experiences that include fewer dynamic elements in the environment, it is nonetheless important to get as close to this target as possible in order to create a smooth user experience.In order to do this, it was necessary to limit building textures to a single 4K resolution texture atlas, generate four LOD models for each mesh, and design the environment so as to reduce the number of buildings in view in any one frame.
It is hoped that this workflow may be reapplied and further developed for similar projects in the future.

Figure 1 .
Figure 1.The historic 3D city model of Stade in the year 1620 and a part of the physical model in the town hall.

Figure 2 .
Figure 2. HTC Vive Pro setup including two controller and lighthouse tracking system (www.vive.com) 2011, the Photogrammetry & Laser Scanning Lab of the HafenCity University Hamburg digitised the physical 3D city model of Stade using photogrammetry.106 photos were taken using a SLR camera Nikon D90 with Nikkor lens 20mm and a zoom lens (focal length 85mm).All photos were processed in Agisoft PhotoScan to generate a dense point cloud of 20,7 million points with high quality (Figure4).For the scaling of the point cloud the distances of three scale bars were measured and determined in PhotoScan with an accuracy of 0.3mm.Finally, a textured 3D mesh model with 4,1 million faces was generated as a basis for further 3D modelling.

Figure 4 .
Figure 4. Top: 3D dense point cloud of the physical city model of Stade in the year 1620; Bottom: Perspective view of textured 3D meshed model.
commercial structures, such as the city walls, bridges, defensive towers, and harbour cranes.Furthermore, a number of special buildings in the town were selected for individual modelling.These include the churches of St. Pancratii, St. Nicolai, St. Cosmae, and St. Wilhadi, St. Mary's Abbey, the monastery of St. John, the Stade town hall, an historic merchant's house, and the ruins of St. George's Abbey, still extent in the city's landscape in 1620.

Figure 5 .
Figure 5. Screenshots from the Unreal Engine editor window showing the first stage of the process: the placement of lowpoly meshes into the scene, guided by height map generated from photogrammetry mesh (top).The different coloured meshes each represent different variations of townhouse (bottom).

Figure 8 .
Figure 8. Screenshots showing the current state of the virtual environment, as of writing.(A) Looking onto the Pferdemarktplatz; (B) View down the Hökerstraße with town hall on right; (C) The Stade harbour; (D) City skyline.