THE DIGITAL RESTITUTION OF LOT 3317: USING UNDERWATER IMAGE BASED MODELLING TO GENERATE VALUE IN VIRTUAL HERITAGE EXPERIENCES

The intent of this paper is to demonstrate the cultural value that Virtual Heritage experiences can generate. As a proof of concept, game resolution assets were optimized from high polygon image based models (IBMs) and rendered in real-time, within a Head Mounted Display (HMD). The case study of lot 3317 was chosen to facilitate heuristic explorations pertinent to the history of the St. Lawrence Seaway, Canada. This case study explores methods in creating and validating heritage values, while valourizing assets as a cultural resource for use in the future. While at the same time, challenging the current spatial dynamics by conceptually reducing the distance between the viewer and the flooded landscape. In addition to demonstrating the cultural values generated through the veneration of lot 3317, the paper aims to provide a model for optimizing and post processing meshes produced through Under-Water Image-Base Modelling.


Figure 1. Image of St.Laurence
Early Canadian economic and national development derived primarily from natural resources in the major metropolitan centres along the St. Lawrence River and Upper Canada. As a result, the shoreline of the St. Lawrence was spotted with early loyalist settlements, farms, hamlets and villages. Geographically, the International Rapids section proved to be one of the most difficult portions of the river to navigate. (Macfarlane) To allow uninterrupted passage from the Atlantic Ocean to the Great Lakes, more than 35 thousand acres of land between Canada and The United States were flooded. Lake St. Lawrence was the resulting creation, a 100 square mile headpond approximately 60km long.
Lake St. Lawrence extended over 20,000 acres in Canada and 18,000 in the United States. The entire project displaced 65km of bi-directional Canadian National railway, 56km of the King's Highway and 225 farms. On a more personal level, the flood displaced 18 cemeteries, 1,000 cottages and entire communities, many that predate the 20th century. In total, the entire project relocated over 6,500 Canadian citizens and 1,100 American citizens.
Many flooded sites are extremely close to popular tourist destinations. Some of which include The Long Sault Parkway, The McLaren Campground, Milles Roches Campground, Woodlands Campground, Farran Park Campground, Upper Canada Camp Ground, Riverside-Cedar Campground as well as Upper Canada Village and The Lost Villages Museum (Ministry of Culture). The unfortunate reality is that even with many destinations relatively close, these sites go unnoticed. This is due to the fact that each site is not publicly recognized and they are only accessible by boat, making them extremely difficult to find.

Intent
The intent of this project is to promote heuristic explorations surrounding the history of the St. Lawrence Seaway, and the villages displaced during construction of passageways that was completed in 1958. As a general aspiration, it was intended that this project would encourage continuing research pertaining to The Lost Villages and enhance awareness of their heritage value as a cultural asset. More importantly, the content produced will demonstrate that there are benefits in maintaining a link between these assets and the historical lore of the community.

VIRTUAL HERITAGE TOURISM
Heritage tourism pertains to people visiting sites where objects are venerated for their cultural, aesthetic and/or historic value. Objects of interest are 'consumed' by the visitor as part of the tourism experience (Fisher & Smiley, 2015). Today, there is a growing demand for first person involvement in heritage sites, especially if they wish to be successful tourist destinations (Pine & Gilmore, 1998). If this is true, then it can also be said that in heritage tourism, the value of both physical objects and intangible heritage is directly related to the tourist's need for a stimulating experience.
Unfortunately, the area chosen for this particular case study is underwater. In addition to this, the preservation and/or implementation of traditional procedures in retrofitting the archaeological site with the intent of generating tourism is not possible. To facilitate or stimulate any level of interaction, a hybridization of current dissemination techniques was considered as an alternative. As per the suggestions in the Canadian Registry of Historic Places, the medium in which the participant experiences the site would benefit from being a firsthand experience. On an additional note, it must not only satisfy the need for tourist consumption, but also promote awareness of the site and contribute to its veneration.

Heritage Visualizations
Mediums and methods in visualization have always been an important aspect of disseminating spatial information and heritage data. As an installation, this study will enable individuals to explore an otherwise inaccessible site, and invite visitors to make judgments based off of their own experiences. It will also enable a reflection on the political realities which greatly influenced the construction of the Seaway. Most importantly however, it will demonstrate the historical significance of the cultural assets lost during the flood and allow for physical representations in a cultural context, specific to Canadian History.

Virtual Heritage (VH)
The term 'Virtual Heritage' (VH) can be used to describe the digital products that use virtual-reality (VR) to disseminate cultural-heritage. For the purpose of this study, cultural heritage refers to "objects, properties and sites with archaeological, aesthetic or historical value" and 'virtual heritage' refers to instances of these objects, properties and sites represented within a digital context (Marinos, 2012). In this definition, Virtual Heritage provides heritage conservationists with an alternative medium of disseminating heritage values, opposed to the traditional techniques such as pre-formulated external observations.

Virtual Tourism
As digital representations of real spaces, Virtual Heritage environments provide the illusion of presence. In doing so, they offer participants the chance to 'digitally' visit places that are normally inaccessible (Marinos, 2012). As an interactive and immersive experience, these synthetic environments provide a surrogate to reality, but in a medium that is simulated and autonomous (Stanislav, 2016). The benefits of such experiences can be defined in their ability to create meaningful connections between the real world and virtual visitors, while also enabling participants to interpret objects and environmental qualities on their own accord.

The Campbell's Barn
As the primary focus for this study, the remains of buildings on either side of the Kings Highway No.2, in the area formerly known as Santa Cruz, Ontario, were surveyed. Aerial photographs visualize part of a submerged foundation on the south side of the road, and a concrete floor on the north side. After careful research in collaboration with the Interpretive Training Programs & Historical Research Officers at Upper Canada Village, the foundation was speculated to be a barn, once owned by The Campbell's (lot 3317).
Once the lot was identified, a series of real-time, interactive representations of lot 3317 were expressed through VH experiences. The digital assets identified in the following chapters were utilized to provide virtual visitors with real-time interactive graphic representations, viewed within a Head Mounted Display (HMD). This allows for the participant to become fully immersed within the re-constructed environment, and the augmented geometry within it.

Virtual Reality + Image Based Models
Computational and advances in the last decade have made it possible to render more complex geometry in real-time game environments. In conjunction, Image Based Models (IBMs) are becoming a popular starting point for generating game assets and realistic environments. As valuable assets, these models are also beginning to appear more frequently in VR experiences as well. This in part, is due to their interesting textures and proficient modelling techniques. Using IBMs as the base mesh, in combination with materials rendered with physically based properties (PBR), developers are now able to fully recreate accurate representations of heritage in VR. A promising example of this technique is 'Nefertari: Journey to Eternity by Reality Virtual', a recreation of the tomb of Queen Nefertari in Egypt (Curiosity Stream et al, 2018).
Virtual Heritage experiences can be rendered in either 3 directions of motion (3DoF) or 6 directions of motion (6DoF). In 3DoF VR systems, the user's movement is limited to rotational head movements. In 6DoF VR system, the user's head and body movements are tracked in a 3D space. (Google Developers, 2018). Visualizing IBMs through a VH experience is possible through both 3Dof systems (passive) and 6DoF systems (active). An example of this would be the use of static panorama in a 3DoF system or rendered real-time experience with six degrees of freedom (6DoF). Each system allows the user to explore a site, while creating an interface that bridges the gap between the physical site and the potential visitor.

Image Based Models
IBMs and most models generated using structure from motion (SFM) pipelines are commonly used as communication devices in heritage conservation. Their highly detailed topologies and photo realistic textures make them valuable assets for static rendering visualizations (Hayden, 2018). As a result, the majority 3Dof VH systems can use unprocessed IBMs straight from standard SFM pipelines. However, most 6Dof VH systems are rendered in real time, and as a result mesh surfaces generated from typical SFM pipelines are not ideal assets for these types of experiences. 3 major aspects that limit IBM mesh surfaces from being game resolution assets are their polygon face distribution (polygon flow), overall polygon count and in some cases their unoptimized UV maps. Rendering high polygon meshes requires large amounts of processing power when the VH experience is playing (Hellman et al. 2018). With an optimized version of the model, similar visual quality is achieved with less processing power. The following section outlines a potential workflow for appeasing these limitations.

Workflow Overview
The workflow proposed in this study for optimizing underwater IBMs has been derived from standard protocols and procedures commonly used by the Computer Graphics (CG) industry in producing game resolution assets. Similar to the gaming industry, highly detailed models are created with varying levels of detail (LOD). This workflow enables the artist to select appropriate LODs that are optimal for VR experiences.
Although specific projects may require slightly alternative workflows, the generation of game resolution assets through SFM applications will consist of three basic stages. These stages can be defined as pre-production, production and finally post production (Vaughan 2012, 22-23.). Pre-production involves the planning and required research leading to the capturing of data, production refers to typical image based modelling techniques, and post production refers to the optimization of these models.
Real-time experiences rendered within HMDs require all visible meshes to be rendered at a high frame rate to prevent the experience from making the participant nauseous. Photogrammetric meshes are typically very detailed, and their high polygon count is often problematic. Typically Virtual Reality simulations must be rendered at a minimum of 90 frames per second (fps). If the polygon count of a mesh is too high, the PC may have difficulty maintaining the required frame rate, while rendering the scene. As a result, a lower number of polygons are required for the experience to be successful. Juxtapose to this statement, surface details are also vital to the fidelity of the experience. To maintain surface details while lowering polygon counts, the mesh must be optimized with techniques that transfer surface details to a low poly mesh in the form of texture maps. In order to optimize a densely triangulated mesh exported from Agisoft Photoscan, digital sculpting software is used to retopologize it's surface topology. In Z-brush, the exported IBM is used as a reference for the creation of an optimized mesh. This new mesh is created with less polygons and a more efficient quadrilateral facet based topology. The mesh optimization was then further carried out by replacing unnecessary mesh faces with custom texture maps, which allows for the preservation of surface details while reducing the polycount. This was facilitated through texture maps such as normal maps (parallax maps). This technique simulates the removed surface details on the optimized mesh, by altering its surface normals during light calculations when it is viewed in the HMD. For simplicity, the workflow outlined in this paper will focus primarily on the south-west corner or the submerged foundation as seen in figure 6.

Figure 7. Coded Targets
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W10, 2019 Underwater 3D Recording and Modelling "A Tool for Modern Applications and CH Recording", 2-3 May 2019, Limassol, Cyprus In order to capture the required data, coded targets were used on site to establish a network of local coordinate systems known as ground control points (GCPs). On site, as the water current is quite strong, weighted targets were used as GCPs to prevent movement during the acquisition phase. In total, 54 vinyl targets with a low specularity were adhered to 6mm x 105mm x 105mm metal plates. Due to the depth of the water and the fixed angle of the lens, the working distance was quite short. As a result, a larger number of photographs were required to capture the entire site. The resulting data was split into multiple chunks to reduce processing time and re-merged later in the process.
Ensuring the consistent placement of each marker was extremely important, as the markers were used during camera calibration to confirm matches between images throughout the photo alignment. They are also used to confirm measured distances within the scene and perform marker based chunk alignments to the final meshes.

Data Acquisition
The digital restitution of this particular submerged asset was completed using under-water photogrammetry, as it is the most cost effective and reliable form of capturing and reconstructing underwater data in the field of marine archaeology. It was also effective in mitigating environmental influences and allowing for data to be captured in conditions where strong currents, vegetation and wave motion were present. The procedures carried out during the acquisition were comprised of five main stages: Capturing Data, Camera Alignment, Building Dense Point Clouds, Building Mesh Topologies and the Texturing of Mesh Topologies.

Image Based Models
! Figure 8. Image Based Models of The Cambell's Barn

Polygon Count + Mesh topology
Most IBMs are generated with moderate to high quality settings and produce meshes between 200,000 -600,000 polygons, (in many cases these numbers can be even higher). Depending on the intended purpose for the mesh object, the number of polygons can be reduced through a typical decimation processes. If the IBM is to be used in the creation of a game resolution asset, the mesh must be processed in an external software. In practice, this means introducing a number of 3D modelling and visualization techniques that will reduce the overall polycount, while preserving the meshes appearance when rendered.
The task of reducing detailed models with 'bad' mesh topologies, while also preserving its visual fidelity, is common amongst most CG artists. Fundamentally, CG artists are concerned with the model's polygon count and it's topology.
Generally, the quality of a game asset is relative to the mesh topology for performance reasons and for workability. Typically, most game resolution assets consist of four-sided polygons called quads, whereas most IBMs exported through SFM pipelines contain three sided polygons that are unevenly distributed. Similar to most models created in digital sculpting software, models can have millions of polygons and lack optimized surface topologies (Vaughan 2012, 128). In most cases, CG artist use specific tools to re-create the topology of their 'sculpt', using the high poly model as a reference. This technique can be applied directly to the workflow proposed in this paper.

Sculpting + Re-topology
To produce high quality models with a specific number of polygons, CG artists commonly use a freeform modelling software specific to digital sculpting. The process of creating an optimized topology from a less organized mesh is called retopology. This shape building toolset is particularly useful because it has no topological constraints and allows the artist to create dynamic levels of polygon resolution (LODs), all within the same model. Both Autodesk Mudbox and Pixologic ZBrush have similar toolsets regarding retopology features. The tools and techniques found in typical CG workflows, (i.e. retopology, UV Mapping and the use of specialized texture maps) are used to create low polygon versions of one asset, while maintaining visual fidelity. In essence, highly detailed models with large numbers of polygons can be optimized for real-time rendering in a game engine similarly to that of a digital sculpt (Vaughan 2012, 128-130).

Dynamesh Tool
Two well noted digital sculpting softwares are Autodesk Mudbox and Pixologic ZBrush (Vaughan 2012, 282). For the workflow presented in this study, Pixologic ZBrush was used to retopologize the model. The DynaMesh tool is ZBrush's newest base mesh generation tool. When the Dynamesh tool is used, the base geometry is re-ordered and it's topology is selectively adjusted into a quad based topology. The DynaMesh tool is designed to create low and middle resolution sculpting stages for CG artists, however it is also an ideal way to retopologize imported IBMs. The LOD of the converted mesh will be defined by the DynaMesh Resolution slider. Once the geometry is converted into a DynaMesh, ZBrush entirely re-creates the geometry with evenly distributed quad based polygons across the mesh surface. This is ideal for cleaning and optimizing the polygons of the IBM, as it removes all pre-existing topology layouts from the original model.

UV Mapping
The DynaMesh tool works exceptionally well for retopologizing meshes created in most SFM applications. However, the geometry it produces lacks UV texture coordinates. If the model does not have UV texture coordinates, the mesh textures will not appear correctly when its materials are applied (Vaughan 2012, 314.). This can be corrected through surface parametrization, otherwise known as UV mapping.
Surface parameterizations, like UV Mapping, generate 2D layouts of the mesh's surface topology. Thus establishing a relationship between the 3D model and it's respective 2D textures, via the texture map. It does this by essentially unfolding or stretching the 3D mesh onto a 2D plane, generating an identical number of mesh faces. This 2D representation is known as a texture map (Paquette, 2013;Ganovelli et al., 2015). The corresponding mesh faces in this map then determine how the texture is displayed over the 3D mesh surface. (Autodesk, 2016.)

Manual UV Mapping
Most IBM softwares can generate UV maps automatically, however the UV maps created in these softwares aren't necessarily optimized for post production. In many cases, to create optimized UV maps that are ideal for post production, the mesh must be 'Unwrapped' manually. Depending on the geometry, manually unwrapping the mesh topology will achieve more uniform distribution of polygons over the texture space. Professional CG artists typically unwrap mesh topologies in a way that will create the least amount of distortion, while optimizing the UV map to use as much of the texture as possible (Angel and Shreiner, 2015). In practice, this is done to create optimized texture layouts based off of predetermined polygroups that will minimize stretching, deformations or overlapping areas in the texture map. In addition to wasting less UV map texture pixels (texels), this method creates UV maps that are more easily edited in Adobe Photoshop. For the purpose of this case study, the desired texture resolution is a predetermined value of 4096 x 4096. In real time rendering, texture resolution also has an impact on the processing power required to deliver the VH experience. As a result the texture resolution should be only as large as needed. As seen in figure  9, the mesh topology of the automatically generated UV map (B) is less efficient than the optimized UV map (A). In the automatic UV map, the mesh topology is divided into 'islands' creating large voids of unused space. Therefore the pixels space that is purely devoted to mesh textures covers a smaller percentage than the Texture than in the UV map (A). Assets rendered within the VH experience must look as detailed as possible, while topologically their polygon count should be as small as possible. Without sacrificing the fidelity of the experience, CG artists often simplify complex assets that may compromise the performance of real-time experiences through texture baking. Texture baking is an 'appearance preserving' technique that transfers surface detail from one mesh to another through 8-, 16-or 32-bit 2D textures. These details are then projected to the mesh via texture coordinates that are stored within the Texture Map. This technique allows surface details to be rendered in real time, while the geometry of the mesh can remain less complex (Vaughan 2012.).

Texture Map
Texture maps are components of the mesh material that enable the low polygon model to be rendered realistically in real time. In Physically Based Rendering (PBR), most materials consist of 5 main Texture Maps: The diffuse texture (or albedo colour map), the normal map (specialized bump map), the ambient occlusion map (shadow) , the roughness map (specularity), and the metallic map (reflection). These textures allow the high polygon mesh to mimic the way the material will respond to different lighting conditions. For this study, a diffuse texture was generated in PhotoScan. In order to effectively transfer the high quality scan texture to the optimized low poly IBM, the previously discussed texture baking techniques were employed.

Normal Map Texture
Tangent based normal maps are used to create the illusion of complex surface details in low polygon models. The texture is a 3 channel 32-bit image, where each channel corresponds to components that alter the terminal point of the surface normal at a given pixel. Red-Green-Blue (RGB) values are used to create surface normal deformations in 3D space relative to its tangent plane: red values affect the X-axis, while green and blue shades affect the Y-and Z-axis (Chopine, 2011). As a result, colour and shade values extracted from the texture determine how the game engine renders fine geometrical details embedded within its material. An example of how details are transferred via the normal map can be seen in Figure 11 below.

GAME ENGINE
The execution of creating a VH experience is similar to the workflow of creating video games. The developer/ conservationist will require software that facilitates the management of 3D environments, the ability to import assets (optimized IBMs) and the ability to narrate or curate the experience with additional imagery, sounds and video. The software must also package the VH experience with programmed interactivity and gameplay controls.
Currently, most VR Experiences are created using game engines. Choosing a software package for the novice designer having minimal knowledge was the primary deciding factor in this workflow. Unity3D and Unreal Engine (UE) (Epic Games) were the two most popular software considered. UE was primarily of more interest because the software package came complete with a suite of flexible creation tools designed for virtual reality. UE was also 'production ready' out of the box, which eliminates the need for manual scripting and additional plugins.

Generating a VH Experience
The VH experience of Lot 3317 was generated using Unreal Engine 4.17. After post production, each game resolution asset was imported into UE. Each asset in the scene should be imported in place, as they are registered to previously established coordinate systems. Once the game assets are loaded, their materials can be applied. If the materials aren't automatically generated on import, a basic material can be generated using the texture maps created during the post production phase.
To establish a datum and a reference to water levels, a script was loaded that simulates the undulating waves of the St. Lawrence. The movement of the water made the experience more interactive and enhances the fidelity of the notion of being under water. After the addition of this script, the model is lit with an environmental map and a directional light. To package the scene, VR controls are added. An example can be seen in Figure 12. showing the optimized IBM successfully rendered within the scene.

IMPROVEMENTS
This project was undertaken as a proof of concept to demonstrate the potential value in image base modelling techniques, specifically for Virtual Heritage productions. The overall pipeline and workflow was delivered without funding. If funding was provided to further the project, there would be an opportunity for improvement in production techniques, equipment and post production.
To begin, it would be extremely beneficial if the data was captured using a higher quality camera. A better camera would improve the texture's fidelity, as well as the topology of the IBM. In addition, a colour calibration chart, as the fidelity of colour plays an important role in VR immersion (Slater and Wilbur, 1997).
For Post Production techniques and workflow, the quality of the asset would benefit from further development of PBR material textures, such as: the diffuse texture (or albedo colour map), the ambient occlusion map, the roughness map, and the metallic map. In the future, capturing values for roughness and specularity of the surface would make the asset more realistic. It would also be useful to facilitate some form of delighting to ensure that environmental shadows from the data acquisition phase are removed. With more developed material textures, the optimized IBM would appear more realistic, lead to increased user immersion and a stronger connection with the site.

CONCLUSION
As an installation, the digital restitution of the submerged archaeology found on lot 3317 will enable participants to partake in a synthetic re-appropriation of space. In this action, each person is given the opportunity to understand, observe and explore the site. While at the same time, challenging the current spatial dynamics by conceptually reducing the distance between the viewer and the flooded landscape. By improving the understanding of how the political, economic and geographical changes challenged previous members of the community, it allows for the development of a new a discourse, further enhancing the value of the site.