VIRTUALISING AN OTTOMAN FORTRESS – LASER SCANNING AND 3D MODELLING FOR THE DEVELOPMENT OF AN INTERACTIVE, IMMERSIVE VIRTUAL REALITY APPLICATION

: “A picture is worth a thousand words”: a famous quote about knowledge dissemination but also literally true. The documentation of cultural heritage (CH) monuments is carried out by measurements and photos and stored in 3d models - not by textual information alone. So what could be a more straightforward way to inform the public about CH than visual information? This approach can be extended not only by providing static images or videos from predefined angles but by giving the user the opportunity to interactively explore the virtual representation and interact with the scene. Recent advances in contemporary Virtual Reality (VR) have made it available to more people as prices have dropped. New devices have entered the market so that VR is not limited to VR labs, but is available even at home. With modern head-mounted displays the user can immerse himself in the virtual CH monument to explore and interact with it. Game engines offer tools for rapid development of interactions and help to produce visually appealing worlds. In this paper is presented the generation of a virtual 3D model of Rumeli Hisarı, an Ottoman fortress at the Bosporus in Istanbul, Turkey (Fig. 1) and its processing for data integration into the game engine Unity. The project has been carried out as a co-operation between BİMTAŞ, a company of the Greater Municipality of Istanbul, Turkey and the Photogrammetry & Laser Scanning Lab of the HafenCity University Hamburg, Germany with the aim of a VR application for an immersive and interactive visualisation of the fortress using the VR system HTC Vive. The workflow from data acquisition to VR visualisation, including the necessary programming for navigation, is described. Furthermore, the possible use (including simultaneous multiple users environments) of such a VR visualisation for a CH monument is discussed.


INTRODUCTION
Virtual Reality (VR) is defined as "a realistic and immersive simulation of a three-dimensional environment, created using interactive software and hardware, and experienced or controlled by movement of the body" (Random House Unabridged Dictionary, 2018).The term was introduced the first time by the author Damien Broderick in his science fiction novel "The Judas Mandala" published in 1982.Other recent definitions are available in the literature from Dörner et al. (2014), Freina & Ott (2015) and Portman et al. (2015), which describe VR mostly in a broader sense, though some also explicitly include non-interactive content.This technology will have a great impact of our future working and everyday life, with applications already existing in medicine, engineering, architecture, cultural heritage (CH) and many other fields.Since VR allows the user to dive into virtual worlds and simulate a user's physical presence in this environment, not only the present, but also the past and the future can be visualised and viewed.VR offers an attractive opportunity to visit objects in the past (Deggim et al. 2017a) or places, which are not easily accessible, often from positions which are not possible in real life.Moreover, these fundamental options are increasingly being implemented today through so-called "serious games", which embed information in a virtual world and create an entertaining experience (edutainment) through the flow of and interaction with the application (Anderson et al. 2010;Mortara et al. 2014;Liarokapis et al. 2017).
Historical architecture and CH objects are ideal subjects for realistic interactive visualisation in immersive Virtual Reality applications as they are already landmarks and exhibited in museums as tourism attractions.Several examples of VR visualisations of historical geo-data are published in the literature.Gaitatzes et al. (2001) present visits to objects in the past as an attractive opportunity using CAVEs.The Old-Segeberg town house was introduced as one of the first virtual museums using the HTC Vive as a Head Mounted Display (HMD) for immersive experiences (Kersten et al. 2017b).Two historic towns (including the landscape around these towns) were developed as VR applications for a visit in the past: Duisburg in the year 1566 (Tschirschwitz et al. 2018) and Segeberg in the year 1600 (Deggim et al. 2017b).Two religious cultural monuments are available as VR applications: the Selimiye mosque in Edirne, Turkey (Kersten at al. 2017a) and the wooden model of Solomon´s temple (Kersten et al. 2018).Edler et al. (2018) present a workflow for constructing an interactive cartographic VR environment to explore urban landscapes.A typical example for an interactive VR visualisation of a large landscape with the dimensions of 160km × 80km is the artic Clyde inlet in Canada (Lütjens et al. 2018).
BİMTAŞ, a company of the Greater Municipality of Istanbul, Turkey, generated a virtual 3D model of the fortress Rumeli Hisarı for the Istanbul City Government to promote the monument for tourism purposes.Furthermore, this project was related to the generation of 3D architectural drawings, restitution and restoration plans of the fortress.This 3D model has been processed in co-operation with the Photogrammetry & Laser Scanning Lab of the HafenCity University Hamburg, Germany, to develop a VR application for the immersive and interactive visualisation of the historical fortress in Istanbul using the new VR system HTC Vive as a new kind of knowledge dissemination.

THE FORTRESS RUMELI HISARI, TURKEY
Rumeli Hisarı is an Ottoman fortress (Fig. 1), housing a museum, located on the European side of the Bosporus in the Istanbul district Sarıyer opposite the Anadolu Hisarı.Rumeli Hisarı was built in 1452 by Fatih Sultan Mehmed II (the Conqueror) in order to control the passage of the ships entering through the Bosporus, where the strait is only 700 m wide.The geographic location of the fortress is ideal for controlling the Rumeli coast of the waterway as it is located at the narrowest point of the strait.The Rumeli Hisarı was built on the west side of the strait based on the topographic condition of the terrain and constructed as a separate unit.On a sloping terrain, the fortress has an east-west width of 120 meters and a north-south length of 250 meters.On the land side, there are two towers (Saruca Pasha, Zagnos Pasha) with the shape of cylinders in both corners on the western side and a third tower (Halil Pasha) with 12 sides (facades) on the sea side.Until 1830 the three large towers, named after the three viziers, were still covered with leaded pointed roofs.The walls connect these towers in an architectural form consisting of small secondary towers.The interior layouts of the towers forming the fortress are similar to each other.None of the three towers have a passage open to the fortress body walls, but rather are self-contained.The entrance to each tower is only accessible from the inside of the fortress.
The large gates of the fortress open to the outside and are very close to each tower.There are three gates to the fortress's interior; the north-facing Mountain Gate, a second one facing the city Hisarpece Gate, and the last one the Sel Gate with iron windows and is always closed.At the eastern sea gate there are still several cannons beaded.It is learned from the historic sources that a mosque and some wooden structures were available in the castle.After the storming of Constantinople, the fortress served as a prison.In 1953 the fortress Rumeli Hisarı was restored on the occasion of the 500th anniversary of the capture of Constantinople.Since then the castle has served as a museum.

WORKFLOW
The following workflow has been carried out to generate a detailed virtual 3D model of the complete fortress: (1) data acquisition using terrestrial laser scanning with three Zoller + Fröhlich IMAGER scanner, ( 2

DATA ACQUISITION AND 3D MODELLING
The scanning of the entire fortress and its environment has been carried out in February and March 2017 in almost four weeks using the three terrestrial laser scanning system IMAGER 5010c, IMAGER 5010x und IMAGER 5006h of Zoller + Fröhlich by the staff members of Solvotek company which was the sub-contractor of BİMTAŞ for the scanning of the fortress (Fig. 2).For each scanner, two persons were available for carrying out the scanning.In total, 1480 scans of the fortress were acquired with the scan resolution high (6mm @ 10m) and super high depending on the location in the fortress.Each scan took 13-15 minutes due to additional acquisition of colour photographs.B/w targets, which were well distributed in object space, were used for geo-referencing the point clouds.The 3D coordinates of all targets were determined in a superordinate coordinate system by geodetic surveying using GNSS.For later colorizing of the laser scanning point clouds the Z+F IMAGER 5010x and 5010c used the integrated i-Cam for parallax-free 360° HDR colour photography, while a DLSR camera Canon Rebel T6 with a Canon 8mm fisheye lens was placed on the nodal Ninja MKII panoramic head after each scan of the IMAGER 5006.Four images per 360 o view were acquired per scan station.All scans were registered in LaserControl software with a precision of approximately 5 mm.This registration processing took almost 30 days for two operators.In total a data volume of 2.5 billion laser points with a point spacing of 5 mm was scanned, which corresponds to a data volume of 856 GByte including the image data.The registered point cloud of the fortress was provided in 600 tiles, but these tiles were merged to two point clouds (western and eastern part of the fortress) in ReCap.Two parts of the coloured point clouds are illustrated in Fig. 3.
The 3D modelling of the fortress took 3.5 months for two operators to generate the final model as illustrated in Fig. 5  For later texture mapping of the solid 3D model of the fortress additional photographs were taken using a Canon EOS 50D camera with zoom lens 18-55mm during the field campaign.800 photos were taken by two persons in one week.These images were radiometrically enhanced in Photoshop before texture mapping.The final 3D model of the fortress comprises 659,000 polygons (Fig. 4).This amount of data is acceptable for a sufficient performance of the VR application in the game engine Unity.

GENERATION OF A VIDEO SEQUENCE
To generate a video sequence for tourist applications the solid 3D model of the fortress (Fig. 5 and 6) was exported as FBX file format for use in the visualisation software Lumion from Act-3D B.V. in Warmond, The Netherlands.Lumion enables users to create videos and images without any prior training by editing the generated environment in real-time and by extremely fast rendering using GPU rendering technology.All atmospheric and light effects were prepared in this software before rendering.The complete rendering took two days using four computers with GTX 970 graphics card, 16 GB RAM and Intel i7 4700K processor.Finally, a video sequence of six minutes with the resolution of 1920×1080 pixel was produced in the format AVI, which corresponds to a data volume of 1.0 GByte 1 .

THE GAME ENGINE UNITY
For the virtual reality application the game engine Unity was used.A game engine is a software framework designed for the creation and development of video games for consoles, mobile 1 http://bimtas.istanbul/3d-modelleme.aspx devices and personal computers.The core functionality typically provided by a game engine includes a rendering engine for 2D or 3D graphics to display textured 3D models (spatial data), a physics engine or collision detection (and collision response) for the interaction of objects, an audio system to emit sound, scripting, animation and networking.A game engine offers the ability to control the course of the game or simulation and is responsible for the visual appearance of the virtual world.For the development of a virtual reality application, game engines offer many necessary concepts with much functionality so that users can interact with the virtual object.The solid 3D model of the fortress was transferred to Unity using the FBX file format.The data volume of the model had 659,000 polygons, which guarantee sufficient computer performance during VR visualisation due to the small number of polygons.At the end of processing in Unity, an executable program for the VR system HTC Vive was developed.

THE VR SYSTEM HTC VIVE
HTC Vive (www.vive.com) is a virtual reality headset (Fig. 7) for room-scale virtual reality.It was developed by HTC and Valve Corporation, was released on 5 April 2016, and is currently available on the market for EUR 599.Basic components are the headset for the immersive experience, two controllers for user interactions and two "Lighthouse" base stations for tracking the user's movement (Fig. 8).To bridge long distances in the virtual object, teleportation or fly-through functions can be developed for navigation of the user.The device uses a gyroscope, accelerometer, and laser position sensor to track the head's movements as precisely as one-tenth of a degree.Wireless controllers in each hand, with precise SteamVR-tracking, enable the user to freely explore virtual objects, people and environments, and to interact with them.
The VIVE-controller is specifically designed for VR with intuitive control and realistic haptic feedback.The Lighthouse system can also track various everyday objects by combining them with VIVE-trackers.To avoid occlusion problems and increase tracking accuracy, tracking is carried out with two lighthouse stations that sweep structured light lasers within the space.Fig. 9 shows the Virtual Reality System HTC Vive in action, while a user navigates through the fortress model.

THE VR IMPLEMENTATION
For a virtual walk within a real environment, the dimensions of the object must be to scale.The basis for the VR visualisation was therefore the textured CAD model generated in 3ds Max.
After data import into the Unity engine some material parameters had to be adjusted because even after many years of game design the transformation of 3D models into game engines is still a task to be solved completely.In an interactive world the user's view can't easily be restrained to the direct proximity.Therefore the real surroundings of the fortress were visualized based on the Istanbul 3D city model.The digital terrain model with a resolution of 1 m was used for the landscape in a TIN representation with about 430,000 faces.The terrain is textured with an orthoimage of 0.25 m GSD and the dimensions of 8192 pixel squared.For the surrounding buildings, LOD 2 data was placed into the scenery as a block model with standard roof structure.To contrast the highly detailed model of the fortress with the surrounding city, the LOD 2 dataset was only colorized in grey for the buildings and orange for the roofs.Specific buildings like the Fatih Sultan Mehmet Bridge are clearly recognisable to the informed user of the VR application (Fig. 12).
Figure 9.The Virtual Reality System HTC Vive in action using the controllers for navigation and interacting with the environment Virtual reality utilizing a HMD is a solitary task by definition, because all outside senses are dampened to enhance the immersion and, therefore, the experience.To create a more social experience, multiple users can join the experience to explore the virtual object together as avatars while each user is at a different real world location using a HMD.The independent networking solution Photon (Photon Unity Networking) is used to implement the synchronisation of multiuser movements.Voice is acquired through the included microphone in the HMD and played back for all users with respect to the speaker's position (spatialized audio).Photon also provides the server infrastructure and is free to use for a moderate user count.

CONCLUSION AND OUTLOOK
In this contribution, the digital workflow from 3D data acquisition to the immersive visualisation of the fortress Rumeli Hisarı located at the Bosporus in Istanbul, Turkey has been presented.Based on the modelled and textured 3D data the game engine Unity, an immersive virtual reality visit was developed utilizing the Virtual Reality System HTC Vive.This VR system provides a truly immersive experience that allows the users to step inside the virtual fortress environment without having ever seen the fortress in real life.CH monumentshistoric, non-existing, or remote -are ideal objects for immersive VR application due to the general attractiveness and interesting history for potential users.The integration of additional information such as text, photos, video, sound, animations and user interactions including the implementation of serious games allow a new kind of knowledge dissemination, which could be defined as edutainment.If multiple user functionality is implemented in the VR application, new capabilities for interested parties can be offered to hold interactive discussions in virtual project meetings about aspects of architecture, structural analysis, fortress history and virtual restoration of the CH monument.As an ancillary effect, VR applications enables precise geometric quality checking of the modelled 3D data during the VR visualisation.
However, comprehensive and time-consuming data recording, detailed modelling and photo-realistic texture mapping of the real environment is an essential requirement for the creation of an immersive visualisation in virtual reality.In addition to this, a smooth visualisation can only be guaranteed if the data volume, i.e. the number of polygons and texture resolution of the environment to be visualised has been reduced and adapted respective to the available computer performance.

Figure 1 .
Figure 1.Rumeli Hisarı, the historic Ottoman fortress at the Bosporus in Istanbul, Turkey (Courtesy of Şirin Akıncı Mimarlık Bürosu) ) Registration and georeferencing of scans using LaserControl, (3) segmentation of point clouds in tiles in LaserControl, (4) re-organisation of point cloud tiles in ReCap as preparation for modelling in 3ds Max, (5) 3D solid modelling with 3ds Max using segmented point clouds, (6) texture mapping of polygon models with 3ds Max, (7) data conversion for the import into the game engine Unity, (8) composition of the highly detailed model into the surrounding DEM and 3D city model, (9) development of motion control and interactions in Unity, (10) implementation into the HTC Vive, and (11) immersive and interactive visualisation of the fortress in the VR system HTC Vive using Steam VR as an interface between game engine and HTC Vive.

Figure 2 .
Figure 2. The terrestrial laser scanning system Z+F IMAGER 5010 at the fortress Rumeli Hisarı (Courtesy of Solvotek Mühendislik) and 6.In this step, geo-referenced point clouds were imported to Autodesk Recap and Geomagic studio (for smaller details) for the generation of 3D meshes.These meshes were used as the basis for solid modelling in the program 3ds Max to derive CAD drawings and the 3D model.Similar workflows were already presented byDursun et al. (2008),Kersten et al. (2009) andKersten et al. (2017).

Figure 3 .
Figure 3. Parts of the coloured point cloud of Rumeli Hisarı

Figure 7 .
Figure 7.The components of the Virtual Reality System HTC Vive scheme was utilized that is only limited to the ground plane of the fortress and the close surrounding.The movement feature was implemented using the SteamVR package from the Unity asset store.Fig.10and 11 illustrated the implementation of the fortress model in the game engine unity.

Figure 10 .
Figure 10.Outside of the fortress in the VR application (Unity) with the controller (top) and perspective view of the fortress inside (bottom)

Figure 11 .
Figure 11.Perspective view of the fortress in the VR application (Unity) with the 3D city model of Istanbul in the right background