SEGEBERG 1600 – RECONSTRUCTING A HISTORIC TOWN FOR VIRTUAL REALITY VISUALISATION AS AN IMMERSIVE EXPERIENCE

: The 3D reconstruction of historic buildings and cities offers an opportunity to experience the history of relevant objects and their development over the centuries. Digital visualisations of such historic objects allow for a more natural view of history as well as showing information that is not possible in a real world setting. New presentation forms, such as the virtual reality (VR) system HTC Vive, can be used to disseminate information in another dimension and simplify the access by changing the user’s viewpoint from a listener and viewer into being an integrated part of an interactive situation. In general, this approach is a combination of education and entertainment, also known as “edutainment” or “gamification”, a term used in the education sector as describing where motivation to learn is encouraged through adding a competitive element. It is thus a step away from simple consumption of information towards experiencing information and a more literal interpretation of “living history”. In this contribution, we present the development of a 3D reconstruction of the two towns Segeberg and Gieschenhagen (today: Bad Segeberg) in Schleswig-Holstein, Germany in the Early Modern Age around 1600. The historic landscape and its conversion from a reconstructed virtual town model into an interactive VR application is also described. The reconstruction is based on a recent digital terrain model as well as survey data of surviving buildings, historic visual information based on historic drawings and written accounts from that era. All datasets are combined to a single walkable virtual world that spans approximately 3 km².


INTRODUCTION
Recent advances in contemporary Virtual Reality (VR) technologies are going to have a significant impact on everyday life.VR typically refers to computer technologies that use a combination of hardware and software to generate realistic images, sounds and interactions that replicate a real environment and simulate a user's physical presence in this environment.VR has been defined as a realistic and immersive simulation of a three-dimensional environment, experienced or controlled by movement of the user´s head and body, or as an immersive, interactive experience generated by a computer (Reality Technologies, 2016).Through VR it is possible to virtually explore a computer-generated environment as a different reality, and to immerse oneself into the past or in a virtual museum without leaving the current real-life situation.Cultural heritage monuments or historic environments are ideally suited both for thorough multi-dimensional geometric documentation and for realistic interactive visualisation in immersive VR applications.Thus, VR offers an attractive opportunity to visit objects in the past (Gaitatzes et al., 2001) or places that are not easily accessible, including from positions that are not possible in real life.Rua & Alvito (2011) described living in the past for historical research, presenting specifically a study of architectural history of the Roman villa of Casal de Freiria as a case-study using game engines and virtual reality as supporting tools.The architectural development of a building's history, including the presentation of the six construction phases of the building from 1541 until today, has been presented in the first virtual museum (Deggim 2016).The virtual museum was later extended to a VR program using the VR system HTC Vive as a In the current contribution, the reconstruction of the historic towns Segeberg and Gieschenhagen, North-Germany in the year 1600 as virtual 3D models, and their processing for data integration into the game engine Unreal, are presented.The project has been conducted in co-operation between the Photogrammetry & Laser Scanning Lab of the HafenCity University Hamburg and the museum Old-Segeberg Town House, Bad Segeberg, Germany.One of the major parts of the project was the reconstruction of the Segeberg castle (Siegesburg), which was destroyed 1644.The virtual return (reconstruction) of the Siegesburg has been published by Deggim et al. (2017).The workflow from data acquisition via 3D modelling to VR visualisation using the VR system HTC Vive, including the necessary programming for navigation, is described in the paper.Furthermore, the use (including simultaneous multiple user environments) of such a VR visualisation for historic towns is discussed in this contribution.

HISTORY OF BAD SEGEBERG
Bad Segeberg is a small town in the state of Schleswig-Holstein, North-Germany, about 50 km north-east of Hamburg and 25 km west of Lübeck.Today, it is mostly known for the Karl May Festival in its open-air theatre located at the Kalkberg, a small gypsum mountain near the town centre.However, the history of the city dates back to the 12 th century when the first castle (Siegesburg) was built on top of the Kalkberg.The town, which at that time was only called Segeberg, developed around this mountain.This town grew together with a nearby settlement (Gieschenhagen) and faced various destructions and reconstructions of the castle and the town itself due to war, sieges and fires.From the 17 th century on, the castle fell into ruins and the mining of the mountain began.Today, only part of the Kalkberg remains, while the only remnants of the castle are the lower parts of a well shaft.In recent decades, some of the historic buildings in Bad Segeberg have been demolished, with the Alt-Segeberger Bürgerhaus (Old-Segeberg Town House) now being the oldest house in the town.Fig. 1 illustrates the two towns Segeberg and Gieschenhagen in an engraving by Braun-Hogenberg in the year 1588 (Braun & Hogenberg, 1612-1618).

Reconstruction of the Kalkberg
The historic Kalkberg was reconstructed both digitally and physically for an earlier project.Its main purpose was the creation of a miniature of the historic Segeberg in the museum.
First, the modelling of the Kalkberg was carried out in butter, using historic sources such as paintings, isometric maps and descriptions, in order to derive only the shape of the mountain without any scale.Afterwards, the existing landscape was modelled by meshing airborne laser scanning (ALS) data from the national survey of Schleswig-Holstein.The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W8, 2017 5th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 28-29 November 2017, Hamburg, Germany

Previous 3D Models of the Siegesburg
The first virtual 3D model of the Siegesburg (Fig. 4

Previous 3D Models of the Alt-Segeberger Bürgerhaus
A cooperation was formed between the museum Alt-Segeberger Bürgerhaus and the Hafencity University Hamburg (HCU) in 2011.Students of the university performed several surveys of the historic building, using photogrammetry and laser scanning (Kersten et al., 2014).The resulting 3D model was the basis for several projects in the following years, including the reconstruction of the six construction phases of the building and the animation of video sequences of the modern and historic building stages.The latest step was the development of a virtual museum for the Alt-Segeberger Bürgerhaus (Fig. 5) (Deggim 2016), consisting of interactive menus for every part of the exhibition, focusing on the building history, both as desktop and virtual reality versions (Kersten et al., 2017b).

Concept and how to deal with historic sources
The project Segeberg 1600 aims for a complete 3D reconstruction of Segeberg and Gieschenhagen in the year 1600, when the town was in the transition between late medieval and early modern age.1600 was chosen because it is close to the date of origin of the main historic source: an engraving from Braun-Hogenberg from 1588, showing major parts of both towns from the north (Fig. 1).Alongside this and other minor historic visualisations, some written sources have been used (e.g.land register).It was typical for that era that for several reasons illustrations of towns or landscapes did not accurately depict the reality.In this case, the town shown on the engraving is a much distorted version of the actual landscape.In order to show more parts of the town, buildings were placed next to the Kalkberg which would have been (from the artist's point of view) behind it or hidden by other buildings.Also, some buildings are shown bigger than they probably were, likely due to the intention of the artist to depict them in a more wealthy and powerful state to satisfy the bailiff of Segeberg, who was also the client for the engraving.Furthermore, every object on the engraving has significant vertical exaggeration.
The interpretation of these sources was carried out by a local historian.An iterative workflow has been established in close collaboration with the historian to ensure the historically most likely appearance of the landscape and the buildings.The concept also includes several steps for data and creation time reduction and improving processing time, e.g.recycling of building parts, textures and making use of as many of the earlier models as possible, as long as they fit in the data reduction concept.

Town Houses
The modelling was carried out with the CAD program Auto-CAD by Autodesk.The main building type in Segeberg and Gieschenhagen during that era were the so called "Bürgerhäuser" (town houses), where most of the people lived and worked.The museum Alt-Segeberger Bürgerhaus is the last remaining building of this type and it was thus used as a basis for the models.
To ensure a variety in the townscape, several versions of the townhouse models with small adjustments in geometry, textures and size were made (Fig. 6).
Figure 6.Two different versions of a town house with variations in size, windows and framework In total, 12 different versions of town houses were modelled and duplicated, resulting in 230 models of town houses in the 3D city model.

Unique buildings
Several buildings visible in the sources differ significantly from the town houses.The most obvious cases are the large complexes of the Siegesburg on the Kalkberg and the church with the adjacent monastery in Gieschenhagen (Fig. 1 and Fig. 7).
The castle is largely based on the model of Herzberg ( 2015) with some elements (e.g. the well house) of Deggim's model (2015).Further adjustments were made for this project, including a rescaling of the whole castle when the wrong scale became apparent during VR testing of the program.Landscape and details were added later in the Unreal Engine (see section 5.3).
The church and the monastery were the first models created specifically for this project.Since most of the medieval church still exists today, it could be used to validate the dimensions of the vanished parts of the monastery.However, the comparison between today's church and the depiction on the engraving (as well as between today's Alt-Segeberger Bürgerhaus and its depiction) added to the conclusion that the historic source is not very reliable.Other unique buildings are mostly somewhere between the church and the town houses in size and illustrate a certain status or function, like the town hall, the Rantzau-Palais (palais of the bailiff), town gates, barns or administrative buildings.
Figure 7.The reconstructed church and the monastery in Gieschenhagen, view from northeast

Texture Mapping
For the texture mapping, the program 3ds Max was used.Almost every building in Segeberg and Gieschenhagen had a timberframe and/or brick architecture.Thus, several textures with old bricks and wood were used.Most of the textures were downloaded from an online database (textures.com)but for some buildings, actual photos from the remaining church and town house were taken, processed and used as textures.To reduce the data size and processing time impact due to texture streaming, most textures did not exceed a size of 1024 x 1024 (2 10 x 2 10 ) pixels.

Unreal Engine
A game engine is a software framework that provides a platform for the creation of interactive software.Game engines are typically used for the creation of video games for PCs, consoles or mobile devices.The engines differ in functionality and content, but most include a rendering engine for 2D or 3D graphics, engines for the detection of physical interactions like collisions or overlapping (physics engine) and a scripting tool to describe and control the behavior off all elements within the video game as well as handling the player input.Further tools may include animation, audio systems, networking (e.g.multiplayer) and optimization features for graphics and performance.A game engine controls the course of the game and is responsible for the visual appearance of the game rules.For the development of virtual museums or in general interactive programs presenting 3D data, game engines offer many necessary concepts with much functionality so that users can interact with the virtual environment.
In the past, the development of game engines was mostly based on the development of a specific game with paid licensing to external game developers.In recent years, however, most of the large engine providers have focused more on the advancement of engines and additionally offer free access for developers.Some of the game engines with free potential use are the engine  (2016).The selection of the appropriate engine for a project is based on the provided components mentioned above, the adaptability in the existing work processes as well as special preferences of the (game) developer.In the framework of this project, the game engine Unreal was selected due to the opportunity to develop application and interaction logics using a visual programming language called Blueprints.Visual programming with Blueprints does not require the writing of machine-compliant source code.Thus, it provides opportunities for non-computer scientists to program all functions for an interactive program using graphic elements.The saving in time associated with this method of software development allows for the generation of additional scenarios and for more intensive user testing.Game engines are, therefore, very well-suited to the development of virtual museums, virtual showrooms of reconstructed environments and alike.

Data Import and Data Reduction
Though most of the buildings have already been combined to a town model in 3ds Max, every building type was imported as an individual object into the game engine to allow for further data reduction by instancing.If the same object is placed more than once in a scene, it is better to create an instance of this object than to copy the whole object.Thus, only the new location, orientation and other location-specific information has to be saved for each individual building.Geometry and textures stay the same for all instances and thus have to be saved only once.
The improvement of performance is a major task when creating real-time applications.In contrast to video renderings, every frame has to be computed "on the fly".For the VR system HTC Vive, 90 frames per second (fps) are recommended -for each eye, resulting in twice the resolution (2160 x 1080 in total).If this threshold is not met, the frames are not perceived anymore as a continuous flow and it is likely that the user starts to suffer from motion sickness.To ensure the compliance of this threshold, a compromise has to be found between the quantity and quality of objects and visual effects on one side and low performance costs on the other side.Next to the already mentioned steps this has been achieved by low-poly-modelling, different texture resolutions and object LoDs (depending on the viewer's distance from the object), precomputation of static shadows as well as an adjustable screen resolution within the program to adapt to the performance power of different PC systems.

Landscape and Details
The 3D model of the Kalkberg was the basis for the landscape modelling in the Unreal Engine.The engine provides a comprehensive landscape editor that allows for a variety of terrain manipulations.Ground textures, plants, rocks and trees can be imported and placed on the terrain with a brush feature that allows for a quick placement of many objects.As the past 400 years extensively changed the landscape, only the rough pattern of the present appearance can be used.The historic sources, on the other hand, are known to be inaccurate in some locations (see 4.1), e.g. the distribution of hills around the village Segeberg.Fig. 8 shows the development from recent and historic sources to the final landscape in the model.Fields and pastures, separated by a network of fences, single trees and small forests, bushes, lakes and a swamp were placed according to a landscape sketch from a historian (Fig. 8 centre).To block the visual axis in the distance, a slight fog was added, not only providing a realistic feeling of distance but also limiting the area, which had to be filled with landscape elements and 3D objects.For the same purpose, the borders of the 3D model were declared inaccessible and filled mostly with trees.
Figure 8. Aerial orthophoto superimposed with historic buildings (top), raw 3D model with landscape and property sketch (centre), and textured 3D model in the Unreal Game Engine (bottom, green lines represent fences for property areas).
For better comparison with the engraving, south is up.
For additional variety, small objects like chests, buckets, boxes, tables, benches, cannons and many other small constructions such as a wooden shelter and a cart were modelled and placed in the scene (Fig. 11).These details are important for the model to appear alive.The 3D model thus provides the impression of depicting the everyday life at that time.

Virtual Reality
Virtual Reality (VR) means the replacement of the real world with a digital one, letting the user believe that they are moving through this artificial environment by tracking his movements and providing a corresponding visual output to a headset.For this project, the HTC Vive was used.It is a virtual reality headset for room-scale virtual reality.It was developed by HTC and the Valve Corporation, was released on 5 April 2016, and is currently available on the market for EUR 699.Basic components are the headset for the immersive experience, two controllers for user interactions and two "Lighthouse" base stations for tracking the user's movement (Fig. 9).By means of VR, the user can dive much deeper into the 3D model than would be possible with a desktop application.Having the impression of actually standing in the model is a totally different experience than just looking at a miniature from the outside.Tests with the VR version of the virtual museum of the Alt-Segeberger Bürgerhaus revealed that this immersive approach has a large positive impact on the user experience and her interest in the presented topic (Deggim et al., 2017;Kersten et al., 2017b).It has therefore been decided that the 3D model of Segeberg 1600 shall be available in VR as well.

Game Logic
The user can move around within the physical limits of the VR system (e.g.cable length, distance from tracking stations).The position and orientation of the headset correspond to the position and orientation of the avatar in the virtual environment.This allows for an area of movement of typically a few square meters.
To reach locations outside of this area, a teleportation system has been implemented.By using a hand controller, which is tracked the same way as the headset, the user can point towards the location he wants to teleport to.When clicking, the endpoint of the teleportation parabola becomes the new current location of the user.He can thus jump through the entire 3D model.Since this model is very large, further quick links have been programmed to enable the user to go directly to the most important locations of the model.
The accessible area was determined both automatically (e.g.minimal distance from objects, inaccessibility of steep slopes or roofs) and manually (e.g.limit area to parts with buildings and / or relevant landscape, inaccessibility of lakes) (Fig. 10).The starting point in the 3D model is at the monastery, approximately in the middle of the model.A little introduction text near this starting point provides the necessary explanations to navigate through the scene.
Figure 10.Overview of the 3D model historic town Segeberg and Gieschenhagen including the accessible area of the model (coloured in light green) and the area borders which follow obstacles in the model such as fences, treelines or steep slopes

CONCLUSIONS & OUTLOOK
The virtual reality application Segeberg 1600 has been successfully implemented using the reconstructed historic towns Segeberg including the castle and Gieschenhagen including the church and the monastery.The dimensions of the final 3D model are 1.8 kilometers from east to west and 1.6 kilometers from north to south.It includes the historic environments of both towns Segeberg and Gieschenhagen with more than 300 buildings or major structures in total, the mountain (the historic Kalkberg) with the Siegesburg and the surrounding landscape with pastures, forests and lakes.The model has been filled with very detailed vegetation and scenery objects for the exploration and experience of historic life in the year 1600.Additionally, the scene has been expanded in all directions to create a realistic feeling of distance and to put the historic town in the context of the surrounding landscape.The user will be able to navigate through the scene with a teleportation function, which can be operated with the handheld motion controllers of the VR system.The 1.0 version of the program has a size of 3.1 GB and runs on 64bit Windows systems with Win 7 or higher.The costs of the technical equipment including the VR system HTC Vive and a powerful gaming desktop PC are approximately 3000 EURO, which corresponds to low-cost.However, the generation of the data including the textured 3D model in the game engine Unreal costs in total 30.000EURO, which corresponds to a factor 10 of the system costs.
The VR application will be included in the museum exhibition in 2018.Test demonstrations with this new way of information dissemination and similar VR programs showed a high acceptance of this new form of visualisation, increasing the interest of the audience both in the city history and in the 3D technology.Fig. 11 shows some impressions of the finished historic 3D model of Segeberg and Gieschenhagen.It is planned to expand the content of the program in the near future.It is also possible to extend the functionality of the program and include further interactive elements, such as information menus, videos or -following the gamification concept -little tasks for the user that he has to solve in order to obtain some information about an object.A combination of this model with Segeberg models of other eras is also possible and would provide a 4D experience about the urban development of Bad Segeberg over the centuries.Further ideas include a multiplayer system and the improvement of visual quality by higher texture resolutions, additional texture features like normal and displacement maps as well as further performance improvements by the means of both better hardware and optimized programming.

Figure 1 .
Figure 1.Historic engraving of the two towns Segeberg and Gieschenhagen in the year 1588 from Braun-Hogenberg Fig. 2 (left)  illustrates the ALS data as a mesh, including the current Kalkberg and the open-air stage of the Karl May Festival in Bad Segeberg.From this DEM, sections with a spacing of 10 m were generated and plotted at a scale of 1:330.These sections were transferred to cardboard, cut out and then fixed upright on a planar platform (Fig.2centre) for further modelling using gypsum (Fig.2 right).Next, the shape of the modelled butter mountain was visually transferred to the physical 3D model by manual modelling.The finished physical 3D model of the historic Kalkberg (Fig.3left) was photographed with a Nikon D800 for digitisation using dense image matching.The final meshed model is illustrated in Fig.3(centre), which was combined with the meshed model from ALS data after scaling (Fig.3right).The combined 3D model was the base for the modelling of the Siegesburg.

Figure 2 .
Figure 2. The reconstruction of the Kalkberg from airborne laser scanning data: digital elevation model (original data, left), printed scaled sections of the DEM (centre) and physical 3D model of the Kalkberg at it exists today (right) left) was created in Cinema 4D by Dipl.-Ing.Uwe Oswald, lecturer for 3D design at HTK -Academy of Design in Hamburg, some years ago.However, this virtual object includes some small false assumptions about the shape of the castle and it had a low level of detail.Thus, a second virtual model was created based on the reconstructed digital terrain model of the Kalkberg in Cinema 4D using the expertise of a historian.This was used by a model builder to create a physical castle model on top of the Kalkberg for the museum exhibition.All these sources were used as a basis for two visualisation projects of the Siegesburg, carried out in 2015: a) reconstruction with 3dsMax (Fig.4top)(Herzberg,  2015), b) reconstruction with Lumion 3D (Fig.4bottom)(Deggim, 2015).Both visualisations are publicly available as video sequences on the YouTube channel of the Geomatics Department of the Hafencity University Hamburg.1

Figure 5 .
Figure 5. Information menu in the Virtual Museum of the Alt-Segeberger Bürgerhaus demonstrating the six construction phases in an animation 4. MODELLING

Figure 9 .
Figure 9.The components of the VR System HTC Vive (top) and system setup of the HTC Vive (bottom) (HTC Corporation 2017)

Figure 11 .
Figure 11.Impressions of the historic life in Segeberg and Gieschenhagen in the year 1600 -from top to bottom: Lübecker Straße, street in front of the monastery court, west entrance to the monastery, well house in the castle court The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W8, 2017 5th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 28-29 November 2017, Hamburg, Germany The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W8, 2017 5th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 28-29 November 2017, Hamburg, Germany Unity from Unity Technologies, the CryEngine of the German development studio Crytek, and the engine Unreal from Epic Games.A current overview and comparison of different game engines can be found e.g. in O'Flanagan (2014) and Lawson The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W8, 2017 5th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 28-29 November 2017, Hamburg, Germany