AUGMENTED EXPERIENCE TO DISSEMINATE CULTURAL HERITAGE : HOUSE OF COMMONS WINDOWS , PARLIAMENT HILL NATIONAL HISTORIC SITE ( CANADA )

The use of photogrammetry and terrestrial laser scanning for building information modelling (BIM) in the documentation and conservation of Cultural Heritage (CH) is now well established. By combining BIM with the latest visualization technologies, powerful, semi-immersive experiences can be developed to enhance the dissemination of CH. In semi-immersive experiences such as Augmented Reality (AR), digital content can be overlapped on to physical spaces, providing a new way to interact with both the physical space and the digital content. This paper discusses the translation of a digital object created using BIM, into a physical object and the utilisation of this physical object as a trigger for an AR experience. The case study looks at one of the neoGothic window frames from the House of Commons in the Centre Block of the Parliament Hill National Historic Site, in Ottawa, Canada. The window frame is one in a series that represents a Canadian province or territory with a stained glass feature that includes floral emblems and heraldic symbols from the respective provincial or territorial shield. The frame in this case study corresponds to the stained glass window of five provinces. Using the replica frame as a target, the user can select which stained glass windows they would like to view in the AR application. Through these combined technologies, we argue that CH can be revealed in a more interactive way and therefore more engaging manner — making even inaccessible architectural details readily available to the public.


INTRODUCTION
The case study presented in this paper focuses on a series of symbolic windows in the House of Commonsone of the most significant spaces in the Centre Block.The Centre Block attracts thousands of visitors every year; however, due to a rehabilitation project it will be closed for several years and visitors to Parliament Hill will not be able to access the interior.The Carleton Immersive Media Studio (CIMS) is studying Augmented Reality (AR) technology as a medium to allow visitors to continue to access the heritage assets inside the Centre Block during the rehabilitation.
The AR app discussed below shows the stained glass design of five House of Commons windows.For context, there are four typologies of Gothic tracery that make up the series of twelve windows in the House of Commons.One of the typologies is host to five different stained glass designs, each of which corresponds to one Canadian province or territory.This is the frame we have focussed on.The virtual stained glass must be perfectly aligned with the milled frame in order for the augmented experience to be legible and successful.For this reason, it is very important to study the best tracking method.This study focuses on image-based recognition methods, where camera pose is calculated from a set of corresponding points between two images.

RELATED WORKS
Unlike Virtual Reality, Augmented Reality (AR) is not a totally immersive technology since virtual objects are superimposed on the real world.The user can see the real world with a layer of augmented objects (Azuma, 1997).The main value of AR technology is the possibility of displaying virtual information that the user cannot detect with their own senses.In this way, AR can improve understanding of real objects or even help the user to perform real-world tasks (Azuma, 1997).Due to the amount of possibilities that this technology offers, AR applications have been studied in several areas such as surgery (Bernhardt et al., 2017), agriculture (Huuskonen and Oksanen, 2018), education (Jamali et al., 2015) or cultural heritage (Gutierrez et al., 2015).The adoption of this visualization technology in cultural heritage improves the knowledge and satisfaction of the visitors (tom Dieck and Jung, 2017), and even achieves experiences that are more powerful (Younes et al., 2017).
Several studies investigate the use of AR applications in cultural organisations such as museums or archaeological sites with the aim of improving the visit without investing money in new hardware (Angelopoulou et al., 2012).By using AR as a mobile guide in museums, visitors spend more time appreciating the art exhibitions (Chang et al., 2014) and it also attracts visitors to revisit the site (Jung et al., 2016).
The goal of an AR app is to show virtual information in a specific location of the real world.This location can be calculated with smartphone sensors such as GPS and gyroscope or through homologue points between two images, being less and more accurate respectively.In some cases, is not necessary to have a very accurate location, such as the popular game Pokémon Go, but in other cases the virtual content has to be located in a specific position (Blanco-Pons et al., 2018).Most studies in the cultural heritage field apply AR systems for the recognition of 2D elements but there is a lack of AR studies focused on the recognition of complex 3D objects.

3D Modelling
A 3D model of the Gothic tracery was used in the production process of the milled frame.The model was initially created in Revit and highly detailed.It was exported as a solid body and further manipulated in Rhino.When imported into Rhino, the geometry at the back of the frame was consciously simplified to a plane so the milled frame could be mounted on a flat surface (Fig. 1a).The challenge was to simplify the model and create one solid polysurface that could be read by the 7-axis Kuka robot that milled the frame.Intersecting geometries were rebuilt until half the frame was fully closed and isolated.It was then mirrored and joined to produce a single solid as shown in Figure 1b.Other considerations that shaped the model included the size of the drill bit, the size of the bed on which the material was mounted, and the thickness of the chosen material for this frame.

Milling the 3D model
The complex detail of the Gothic tracery in the model required further scrutiny and simplification to maximize the efficiency and crispness of the milling.For example, the yellow highlighted area in Figure 2 was less than 1 mm higher than the surrounding tracery.It was lowered to be at the same plane as the adjacent elements.For the purposes of this AR experience and at this scale, the height difference is negligible.Miniscule gaps between surfaces are sometimes difficult to spot and they affect the tool path and break continuity.To optimize the tool path, these gaps were removed.
The material chosen for this prototype is low-density foam (240 kg/m 3 ).It is available in 25.4 mm thick sheets and so two pieces were glued together to create the required thickness of the frame.
The thickness of the material should also inform the thickness of the milled object.For this frame, the glue is slightly visible in the details of the tracery.Figure 3 shows the milling in process where the darker yellow is the layer of glue.Further prototypes will take this limitation of the material into account.

Augmented reality application
As mentioned above, AR technology integrates virtual elements into the real world through capable devices, in this case, mobile devices, which use Android as their operating system.To obtain good tracking based on image recognition in an augmented experience, pre-defined image detection and tracking is used to trigger pre-defined visualization of relevant information.Therefore, one of the key elements to consider is the recognition technology type that will be used.
Our approach uses the image tracking recognition algorithms of Vuforia which tracks the image by recognizing selected points of the 3D milled frame.Specifically, multiple image targets arranged in a specific position and illumination were designed and sent to the Target Manager of Vuforia (Fig. 4).Image Targets in Vuforia are images that Vuforia can detect and track (Vuforia, 2018).Vuforia detects features in the image targets and stores them in a database.These features are compared at run time with features extracted from the live camera image.With all this, Vuforia establishes a classification based on the viability of the image target and classifies it between 1 and 5 stars.Image targets with 4 or 5 stars are the better options to use since targets with lower ratings usually result in poor detection and tracking.It is important to note that each target must have high contrast, no repetitive patterns, and it must be rich in detail.
Figure 4: Image targets with different illuminations and positions.
Another important task is the treatment of the virtual content that we want to show in the app.In this case, the goal of this AR app is to enhance the user experience by overlaying information in the camera view while the user focuses the camera on the physical 3D milled frame.Thus, five different stained glass designs can be displayed within this frame typology.To display and align the different stained glass designs, 2D images of the windows, including the Gothic tracery, were integrated to simulate a complete window.Transparency was then added to the areas corresponding to the frame so they would not appear in the AR experience (Fig. 5).
Figure 5: Stained glass images of the five provincial designs hosted by the selected frame typology.
Afterwards it is necessary to develop the Android application.In this case, the System Development Kit (SDK) of Vuforia Engine for Unity was used to build the AR Android app and to create the scenes that are described above.The scene in Unity was configured by adding all the elements that the camera of an AR app like Vuforia requires to realize an AR experience including image targets, illumination, as well as the virtual content.Finally, a swipe gesture was added to the screen so the user can change the displayed stained glass window by swiping the screen from right to left. Figure 6 shows the final result of the AR app.

DISCUSSION
There were several factors that informed the modifications of the 3D model which prepared it for the milling process.First, the size of the model was considered.Either, a highly detailed Revit model can be used or a low polygonal version of the frame based on the existing Revit component can be modelled.In this case, the latter option was chosen.Although the detailed Revit model contained distinct features that would allow for high contrast when recognized as an AR target, the complexities of this model could not be efficiently read by the 7-axis Kuka robot used in the milling process.A simplified model allowed for the milling to proceed smoothly.Since the result would be a model of lower contrast, measures were taken to ensure it could still be recognized as an AR target.The frame was modelled as a single surface that contained enough detail for the AR technology to recognize the frame while still evoking the architectural qualities of the frame and allowing for the Robot drill bit to cut precisely without failure.
It is imperative to choose the right material for the intention of the prototype, which then informs the level of detail of the model.It is key to balance the level of detail required to achieve an effective AR target with the efficiency of milling the material.To ensure high enough contrast for the purposes of this prototype, milling foam was used which allowed for smaller details to be milled without much effort from the robot.Wood could have been chosen for the high contrast inherent in the grains of the material but it is denser and heavier.Also, low density foam was selected for its lightness to enhance portability.Although foam is matte in texture, the frame contained enough detail in its form to be recognized as an AR target without the need of a high contrast pattern.One limitation of the low-density foam was the 25.4 mm thickness of the sheet.The 3D model was thicker than the foam so two sheets were glued together for the milling process resulting in visible traces of glue in the detailed sections of the tracery.The thickness of material will be considered in future prototyping to gain the best result.Finally, the 7 th axis bed of the robot determined the 1:200 scale of the frame.If a larger bed or a different robot was used, higher details could be milled, thus enhancing the AR target recognition.
Recognizing the milled frame through the use of a single, unique image target was proven to be unsuccessful because the image stored in the application could not be reconciled with the image captured through the camera on a mobile device.Tracking of 3D objects, such as the milled frame, is difficult when using one image target due to the variations in light, depth and perspective and so adding more targets from different positions increases the success and recognition time of the app.
Another method of target detection supported by Vuforia is objection recognition with the use of a digital model rather than 2D, planar images.This recognition technique is less affected by environmental conditions such as illumination, shadow and camera perspective; however, this method requires additional computation time in the application, resulting in longer recognition time.In future work, it would be beneficial to test the implementation of 3D object detection.
With further testing of the AR app, some tracking issues became apparent with varying lighting conditions and background colours, which directly affected the contrast.A dark grey background was ultimately used to capture the target images (Fig. 4 and Fig. 7).Another challenge was ensuring that the stained glass layer, which is the augmented digital layer, would appear to lay behind the tracery as it is in reality.To achieve this, all parts of the window that were not part of the stained glass design were meticulously digitally removed.The resulting virtual content, the stained glass, was placed further into the background of the scene in Unity allowing the inner and outer edges of the milled frame to remain visible.Further, additional textual information was layered in front of the milled frame, set apart from the virtual stained glass, which also enhanced the perception of depth.
The layering of text, or any additional virtual information, poses another design challenge.In this case, the question was, how should the provincial affiliation of each stained glass window appear within the AR experience?The physical context of the milled frame was considered along with how the frame would be displayed for use with the AR app.For this prototype, the frame would be displayed vertically either mounted to a wall or standing on a flat surface.It was determined that the best location for the text would be at the bottom of the milled frame.Legibility was achieved by using black text on the cream coloured frame (Fig. 7).

CONCLUSIONS
The process outlined in this paper for a markerless, vision-based augmented reality applicationfrom the creation of the narrative and assets for 3D artefacts and virtual content, to developing an AR app and implementing user testingis a novel and engaging way to create Cultural Heritage assets for public dissemination.The application projects stained glass images of five provincial designs over a milled Gothic frame.As seen in this case study, the project was approached with a holistic view.The final product and many factors along the way must be considered during each part of the process.For example, the physical context of an experience must be considered when deciding what material is best to use, and a digital model originally designed for other purposes, such as BIM or visualization, must be analysed with scrutiny when used for fabrication.Furthermore, testing different augmented reality frameworks, such as ARCore and ARKit, is necessary to compare the results that could be obtained by the Unity application and to take advantage of their unique features such as real time surface detection and estimation of real life lighting conditions.To conclude, an AR application on a portable device that layers virtual information over a tangible, 3D artefact is a viable option as a tool for cultural dissemination, particularly in a situation where the original fabric is no longer accessible, like the temporary situation of the House of Commons which prompted this prototype.The experience is much more immersive when manoeuvring virtual data within a 3D object, in comparison to a similar experience using a 2-dimensional target.The perception of depth is more palatable and with user testing it is apparent that the 3D experience is more captivating and engaging.
The model was scaled to 1:200 to fit the constraints of the 7 th axis, the bed on which the material was mounted, and the deepest cutting depth of the model is no more than 76 mm to accommodate the 1.6 mm drill bit.It stands at 540 mm tall.a b Figure 1: a) Simplified plane of model.b) Model showing Gothic tracery.

Figure 2 :
Figure 2: Detail of tracery in Rhino model.

Figure 3 :
Figure 3: Milling process showing layer of glue.
a) AR experience.b) Screenshot of the AR app.

Figure 7 :
Figure 7: Image targets with dark background and text overlay.