MERGING PHOTOGRAMMETRY AND AUGMENTED REALITY: THE CANADIAN LIBRARY OF PARLIAMENT

: In recent years, Augmented Reality (AR) technology has experienced considerable progress and the combination of AR and 3D modeling opens up new opportunities regarding 3D data visualization and interaction. Consequently, the dissemination of cultural heritage can benefit from these technologies in order to display the cultural assets as realistically and interactively as possible. In this way, high-accuracy 3D models are integrated in the real world. Nevertheless, progress has also still been limited due to several factors. The paper presents a case study based on the recreation of the Queen Victoria sculpture in an AR application. Furthermore, the environment of the sculpture is simulated by panoramic images, inside the Library of Parliament in Ottawa, Canada. The main problems for the development of an AR smartphone application from panoramic images and photogrammetric 3D data are described in this paper. The characteristics of AR systems are explained in detail, analyzing all the steps involved and the available solutions considered.


INTRODUCTION
The case study presented focuses on the development of an augmented reality (AR) application to disseminate the Library of Parliament in an innovative way.The main feature of this building is its circular shape and the use of galleries.In the center of the room, is the white marble statue of Queen Victoria.The application simulates looking through a window inside the Library and users will be able to appreciate this singular building from anywhere to inside the historic building.
Parliament Hill receives thousands of visitors every year, being one of the main attractions of the city; however, in the coming years the Library cannot be visited due to rehabilitation work.In order to allow visitors to experience this building when they arrive at Parliament Hill, the developed app will be displayed on site, through a real-size image of the window, thus simulating the interior of the Library in the most realistic way by means of AR.AR is a variation of Virtual Reality (VR) technology; while VR completely immerses a user in a virtual world, AR superimposes virtual objects onto the real world (Carmigniani et al., 2011).This complements the real world with virtual information.Therefore, AR is very useful in real environments supplementing reality by adding information interactively in real time (Azuma, 1997).
Due to the advantages of AR, several studies have focused in this technology as a tool to disseminate Cultural Heritage.(Palma et al., 2019) study AR apps during on-site visits to display a wide range of virtual information and conclude that this methodology has not yet been widely discussed.(Carrión-Ruiz et al., 2019) develop an AR experience that superimposes different layers over a physical 3D object, displaying the twelve windows of the House of Commons of the Parliament Hill in Ottawa.When using a real object (a 3D prototype of the window) instead of a 2D image, the experience was more immersive.Another study on in-situ visualization for Cultural heritage sites is described in (Stricker et al., 2016), where the reconstruction of ruins is presented through an AR app.This paper describes the process followed to integrate the 3D model and the panoramic background to create the sensation of looking through a window.The virtually constructed Library was created in Unity, and the realistic 3D model of the queen was added in the center of a hemisphere with the panoramic texture of the Library.This scene was placed behind a window plane, which works as an image target to launch the AR app.When the mobile phone is targeting this window, the Unity scene with the Queen Victoria 3D model is displayed.

PHOTOGRAMMETRIC 3D MODELLING
3D modelling the statue of Queen Victoria was part of an ongoing documentation of the Canadian Parliament Buildings and the Library of Parliament.The main goal of the recording was for the creation of a building information model (BIM) for the Centre Block of Parliament.The Parliament BIM modelling is documented by Chow and Fai, (2017).Although the model of the statue was originally intended for BIM purposes, we were able to leverage the data for AR purposes.
The 3D model was captured using photogrammetry-a fast, low cost imaging technique for reconstructing geometry with color data.For reliable photogrammetric reconstruction, images are typically captured in a network of strips orthogonal to the surface of an object (Historic England, 2017).The statue of Queen Victoria is about 3.47 m tall (Young 1995).This poses a challenge when trying to orient the camera perpendicular to the higher areas of the statue.A u f L b ' architecture was necessary to ensure full photogrammetric capture.Before L b ' construction, the Librarian of the Legislative Assembly, Alpheus Todd u "T preferable shape for the Library Apartment is circular or polygonal, so that the most part of the contents of the Library u b w " Of u view is now occupied by our sculpture of Queen Victoria.We were able to use the circular shape of the Library with the varying levels of book shelves and galleries in order to create a sufficient photogrammetric image network.This technique is similar to using a turntable with object photogrammetry for a controlled circular network.This idea is illustrated by the object scanner built by Menna et al. (2017).Our circular control network is seen in Figure 1.It is also important to maintain consistent, diffuse and even lighting of the subject during the photogrammetric acquisition (Historic England, 2017).This became a challenge because at a certain time of the day, sunlight would be cast directly on to the statue, and was changing rapidly.Luckily, after letting some time pass, the sun changed its position and rendered the statue with diffuse, even lighting.Once captured, the images were then aligned together in Agisoft Photoscan using a self-calibrating bundle adjustment.From the alignment, a dense cloud, then a textured mesh was created.As a result of the process, a highly detailed, visually accurate 3D model is obtained (Figure 2).

AR APP DEVELOPMENT
The app presents a complete marker-less tracking solution (Uchiyama and Marchand, 2012).Our approach is based on the recognition of natural features points (Blanco-Pons et al., 2018;Sörös et al., 2011) which are visible in the environment and image tracking solution.We used our own image of the target as tracker (Figure 3) which fulfills the main conditions to be a good tracker, i.e. high complexity and good contrast.Thus, app users can scan our image and see inside the library of the parliament as if the window was opened (Figure 3).

Simulation of the Parliament Hill Library view
The following main elements were added to the AR scene to simulate looking through the Library Window:  Optimized 3D model of the Queen: a low resolution mesh for the AR scene needs to be created from the photogrammetric mesh.For this task, a retopology is applied to the mesh by Instant meshes software (Jakob et al., 2015). Panoramic background: the panorama image (Figure 4) of the Library was remapped in a hemisphere to get a frame background (Figure 5). Illumination: a directional light was located into the hemisphere to illuminate the whole scene and project shadows on the 3D mesh. Occlusion mask: The image target did not completely hide the unity scene, therefore, when the AR app was u f Qu ' w visible on the sides of the image target (Figure 6a).To create the illusion of looking through a window, the usage of an occlusion mask was crucial.This mask hides objects behind the image target, achieving a realistic result.In the case study presented, adding a plane with a DepthMask material, the objects behind the plane are not visible (Figure 6b).Finally, when any user targets their smartphone at the window, they will be able to appreciate the interior of the room from different positions with sense of depth (Figure 7).For this prototype, both the image target and the app is vertically displayed.In this case the image target works mounted to a wall or standing on a flat surface.Locating the best position for the image target was important, in order to achieve a realistic sense of depth.In addition, the user pose must be considered for the best use of the AR app and let the user enjoy the most of this experience.
Figure 7. AR app screenshots

CONCLUSION
The augmented reality application projects the Queen Victoria sculpture 3D model over an image of a window.A panoramic image of the interior of the library is shown behind the sculpture.The panoramic image is displayed so that the user experiences the sensation of looking from a window into the library.As it can be seen in this case study, many factors along the development must be bear in mind in order to create AR apps for indoor architectural applications.How immersive is the user experience would be one of the aspects to be taken into account.Regarding this, it would be interesting testing different AR frameworks to compare the results obtained, especially since new open source AR frameworks are constantly changing and evolving.Real time tracking detection or estimation of real illumination would be one of the aspects that can offer more realism and improve eventually the u ' experiences.The perception of depth is more captivating for the user and can be used like another way to visit the library.However, the cultural content that we are showing in this case could be insufficient.Therefore, the addition of information such as Historic building Information Modeling (HBIM) could be a good way to enrich the augmented experience.Finally an AR application can be a good option as a tool for cultural diffusion, and offers cultural information/experiences particularly when the cultural heritage is no longer accessible as in this case.

Figure 3 .
Figure 3. Architectural image target used for tracking.

Figure 5 .
Figure 5. Panoramic image remapped in the hemisphere.