3D Reconstruction and Presentation of Cultural Heritage: AR and VR Experiences at the Museo d’Arte Orientale di Torino

: For years, virtual reconstruction in the figurative arts, and sculpture, in particular, has been developing and consolidating. The workflow from the acquisition to three-dimensional modelling and to the integration of missing parts, has been optimized through processes entirely implemented in the digital dimension. The most recent developments in augmented reality and virtual reality technologies, together with the possibility of using low-cost and widely available devices, have made it possible to establish new links between the real and the virtual. The experiences presented in this paper comes up within the agreement between the Politecnico di Torino and the Museo d’Arte Orientale (MAO). The workflow set up for this research involves: structure from motion (SfM) survey, 3D modelling, and 3D philological reconstruction, then develops a proposal to implement augmented and virtual reality experiences aimed at the communication and fruition of the exhibits. The case study concerns two Japanese statues, and proposes their visualisation with the respective weapons virtually reconstructed, and through VR, involving the reconstruction of the interior space of a temple recognised as philologically compatible with the location of the statues within a statuary complex.


INTRODUCTION
For years, virtual reconstruction in the figurative arts, and sculpture, in particular, has been developing and consolidating. The workflow from the acquisition to three-dimensional modelling and to the integration of missing parts, has been optimized through processes entirely implemented in the digital dimension. In addition, reconstructive digital modelling offers the possibility of constructing philologically plausible architectural scenarios, recreating the ideal context in which the artworks, now held in museums, could have been originally located. The most recent developments in augmented reality (AR) and virtual reality (VR) technologies, together with the possibility of using low-cost and widely available devices such as tablets and smartphones, have made it possible to establish new links between the real and the virtual. The scope of the application of these technologies, therefore, has been broadened to heritage presentation, expanding the audience of users. The experiences presented in this paper comes up within the agreement between the Politecnico di Torino and the Museo d'Arte Orientale (MAO). The limited use of digital technologies and multimedia communication tools during the visit of MAO's permanent collections has been observed. For this reason, this proposal, thanks to the spread of AR and VR devices, aims to offer new, once unthinkable ways of visiting, with ever greater levels of involvement. Moreover, the possibility of using personal devices, in the current pandemic situation of Covid-19, is extremely interesting.

AR AND VR FOR THE PRESENTATION OF CULTURAL AND ARTISTIC HERITAGE
Computer Graphics (CG) is now commonplace in any field of human activity, from computer assisted design, to entertainment, scientific visualization, etc. CG always played a primary role also in the domain of cultural and artistic heritage, particularly concerning its restoration, conservation, presentation and communication worldwide. The incredible applications that have been requested over time by this domain challenged, at the same time, many researchers active in the field of CG, establishing a positive cycle that is stimulating innovation in many areas and are making technology advance at a fast pace. In this scenario it is worth to recall, for instance, activities pertaining the digital reconstruction of the real world through 2D/3D scanning and photogrammetry techniques on various scales, from small artworks to much larger structures and sites (Callieri 2004). One of the most common uses of digital models obtained with these techniques consists in the creation of new digital solutions for content fruition. Initially, there has been lot of activity regarding web sites developed as digital replicas of real exhibits and letting users experience distance visits choosing first the particular museum or gallery of interest, then selecting on a map or from a list the specific artworks to dig into by means of images or visual reconstructions characterized by various degrees of interactivity. Then, technological evolution and, in particular, the spread of VR and AR devices enabled the definition of new, unimaginable visit modalities, with ever greater levels of engagement. Today, products like Google Arts & Culture (https://artsandculture.google.com) let visitors remotely access more than 2000 museums and 10000 places of artistic and cultural interest not only via a common web browser, but also through a complete immersion into a totally virtual world using a headset made up, in its cheaper version, by a smartphone and a 5$ VR cardboard. Through the VR technology the visitors can not only live an experience that is much more similar to the one they would experiment in the corresponding physical place, but also be provided with additional content capable, e.g., to position a particular artwork in the specific spatial and temporal context it belongs to, to make them live its production stages, or to interact, through their voice, with a digital representation of the artist who made it (Walczak 2006). AR technology, on the other side, can change the way to intend the on-site visit (Barry 2012). In this case, through the screen of a smartphone or the lenses of a holographic display it is possible to add digital content to the real ones, "augmenting" them. The users can see, in the physical context of the visited exhibit or site, additional content, side by side or overlapped to the artwork of interest, e.g., to observe it in its original conditions or deepen the observation of aspects hard to see with naked eye.

DIGITAL COMMUNICATION OF MUSEUM HERITAGE AT MAO
The MAO Oriental Art Museum is a rather new institution in the field of Asian art preservation and presentation to the Italian public. It was opened at the end of 2008 as a joined effort of the City of Turin, the Region of Piedmont, and two banking foundations, and was intended as a way to spread a culture of mutual understanding between Italian and Asian citizens. In the years, however, it has also become the most active and visited museum of Asian art in Italy -approaching the number of 120,000 visitors in 2019 before the Covid-19 pandemic outbreak.
Originally, MAO had several touch-screen digital materials to be accessed by any visitor in the museum. Later on, after 2014, those screens were dismissed as obsolete and never replaced by new technologies whatsoever. As a result, MAO became a museum without any digital support to the visit, such as most museums set up before the new millennium. At present, visitors can only count on classical audio guides -some 80 entries altogether for the complete visit of permanent collections. While in-situ visits to MAO currently make limited use of multimedia, there is an undoubted focus on communicating heritage at a distance through the museum website, YouTube channel, and various social profiles. MAO joined the Google Arts & Culture project and is virtually visitable via Street View. It was the first museum in Italy to join the Google 3D pilot project in 2015 for 360 degrees visualization of artworks. Soon a selection of Tibetan thang-ka digitized by Google Art Camera will be made available too. Fondazione Torino Musei (FTM), the body managing MAO and two more museums of the City of Turin, has launched an online platform in 2020 for promoting education and workshops for schools -both remotely and in presence. MAO is of course part of this "InOnda" channel, which was conceived as a reaction to the physical limits imposed by the pandemic.
Recently this year and in the same perspective, the concessionaire of guided tours for FTM has built another platform in order to carry on digital guided tours of MAO and the other two museums. Thanks to this project with Politecnico, MAO is looking forward to introducing virtual reality and augmented reality in a broader frame of multimedia instruments for its visitors, both in person and remotely. Furthermore, is willing to make use of digital models to create tactile models of artworks, so as to increase its accessibility for everyone, the visually impaired ones included.

A MULTIDISCIPLINARY PROJECT FOR THE DIGITAL ENHANCEMENT OF MAO: BACKGROUND AND WORKFLOW
The complexity of the issues faced in this project needs a multidisciplinary approach. It involves scholars of survey and The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China representation, information processing systems, art history, archaeology, museography. Digital convergence facilitates the integration of knowledge data, while the virtual graphic visualization potential enables more effective communication between scholars from different disciplinary domains. Virtual reconstruction allows scholars to verify different hypotheses of philological restoration of the lacunae from different points of view: geometric, topological, formal, stylistic, material, even with the formulation of alternative hypotheses. The possibility of modifying architectural elements and spaces almost in real-time facilitates the heuristic process and the debate among scholars towards the most convincing interpretative hypotheses. At a later stage, virtual reconstruction allows to communicate to a broader, multicultural and multi-age audience the outcomes of the process of knowledge and choice of hypotheses as well as to create more involvement and interaction through in situ and immersive experiences. and what is mere conjecture. The second one is devoted to the specific field of virtual archaeology and defines virtual reconstruction as the use of the virtual model to visually recover an artefact from physical evidence, comparative inferences, and experts' studies. The workflow set up for this research involves: structure from motion (SfM) survey, 3D modelling, and 3D philological reconstruction, then develops a proposal to implement augmented and virtual reality experiences aimed at the communication and fruition of the exhibits. The multidisciplinary team developed an overall project aimed at the identification of a path that connects a series of artworks, realizing new heritage narratives through the museum halls. It aims to offer a series of multimedia experiences diversified according to the characteristics of the artworks on display and their position in relation to the hall and the visit path. Indeed, the artworks dimensions, morphological characteristics, materials, free-standing or along the walls of the room, as well as whether or not the work is kept in a display case or placed on a pedestal, affect not only the techniques for surveying the artworks but also the project of communication.
The case study concerns two Japanese statues (Ni-Tennō, i. e. temple guardians), made of wood with a few traces of pigment, and approximately 1.20 m high (Fig. 1). They are freely placed side by side, each one resting on a pedestal about 1 m high, which allows the visitor to walk around and see them slightly from below. The exhibition choice inside the museum hall was to invert the statues position compared with the traditional iconography, which presents the guardians frontally with the heads turned towards the center and the arms raised to hold the weapon outwards. The weapons are missing today. This involves different strategies in communication through AR, which proposes the visualisation of the statues one by one with their respective weapons virtually reconstructed, and through VR, which refers to the ritual position of the statues. This last experience, of an immersive nature, involved the reconstruction of the interior space of the small hall in a temple recognised as philologically compatible with the location of the Ni-Tennō within a statuary complex, as described in par. 6.

DIGITAL SURVEY OF NI-TENNŌ STATUES
There are currently several three-dimensional digital survey techniques that make it possible to obtain extremely accurate models, composed of millions of points, meshes and highresolution images.
For the case study presented, to optimize the acquisition for subsequent data processing, it was decided to use the photogrammetric survey. The excellent photographic quality guarantees to obtain a very accurate digital facsimile that allows not only to enrich virtual reality experiences but also the possible fruition of tactile replicas. The work presented here, in fact, joins the broader one of the doctoral thesis of Francesca Ronco (Arquitectura, Edificación, Urbanística y Paisaje program of the Universidad Politècnica de València) that foresees the realization of accessible exhibition paths inside the MAO, including multi-sensory experiences in situ (tactile paths and AR experiences) and on-line (VR proposals). Within the MAO's permanent collection, some works were therefore identified. For each of them was created an evaluation form containing the following parameters: maneuverability and inspectionability; illuminability; roughness; perceptibility of details; opacity; chromatic richness. Two wooden Ni-Tennō statues from the Japanese collection are among the selected artworkse. The most critical feature was the inability to move the artwork. The room in which they are exhibited is characterized by scarce and not homogeneous lighting and by the presence of other artworks and showcases with their relative reflections. In addition, as said, each statue stands on a pedestal. These characteristics have complicated the acquisition phase, in terms of maneuvers and necessary times.
The equipment used consists of: a Canon digital camera model EOS 6D with Canon EF 50mm lens; a tripod, and a set of lights (two softboxes and two lamps coupled with reflective umbrellas). It has been chosen to guarantee to the user a good yield of the artefact, to leave open more possibilities of use of the 3D model. The idea is to allow fruition in a detailed scale. The scale depends on the distance of the camera from the artefact, on the focal length used and on the pixel size, therefore on the characteristics of the camera sensor. It is fundamental to determine the Ground Sample Distance (GSD) that is the distance between two consecutive pixel centres measured on the artefact (Table 1). After deciding the acquisition distance and the focal length to obtain a certain GSD, the next step is to decide the number of shots needed to cover the entire artefact and thus obtain a final 3D model complete in all its parts. Considering the position of the artefacts and the properties previously illustrated it was decided to set the ISO to 200 and the focal ratio to 5.6f, and the exposure time to 1/10 of a second, to make the background more blurred than the point of shooting, thus facilitating the recognition of the subject. Before proceeding with the survey, we studied the shooting mode, defining the distance of the camera, the number of turns to be made around the artworks and the approximate number of shots for each turn. The shots were taken around the statues at a distance of about 1.10 m at four different heights (Fig. 2), using a ladder to frame the highest part of the statues and always making sure to have at least a 60% overlap between consecutive shots. To move with more precision, a trajectory as circular as possible was traced on the ground, keeping constant the shooting distance from the vertical axis of the work.  For the Southern and East Ni-Tennō survey, knowing that an overlap of about 60% between two consecutive images is still necessary, the acquisition phase was realized through 158 and 152 shots, respectively, while maintaining the shooting scheme. For both surveys, special attention was paid to the concave and convex parts that could create shadow areas that need to be covered. Therefore, the vertical movement of the camera between successive revolutions around the same artefact must be chosen according to the morphology of the object. The acquired data were processed using Agisoft Metashape®, which allows obtaining three-dimensional virtual models through the following processing steps: image alignment, creation of the sparse and dense point clouds, surface/mesh and texture generation. The mesh model was finally exported in .obj for its implementation within Blender® environment (Fig. 3).

PHILOLOGICAL RECONSTRUCTIONS: STATUES LOCATION AND MISSING WEAPONS
The two statues were acquired in 2006 from a well-known international art dealer, but their provenance and original location are unknown. The search for any possible setting was necessarily highly hypothetical. Only a few original wooden architectures of the late Heian period (794-1185) survive today, while most of them have been constantly rebuilt according to original standards. We then decided to take inspiration from a peculiar miniature hall of the Chūson-jí temple in Hiraizumi, the famous Gold Colored Hall of Konjiki-dō, which dates to the 12th century just like the MAO sculptures. The most prominent of the three daises supporting Buddhist statues is guarded by two tennō that were added in 1157 (Soejima, 2008, pp.102-103). They are standing in a specular posture at the front corners of the dais. Although quite different in style and details from the MAO works -the last being less plastic and more essential, and without gildingthey constitute one of the few original examples of how our guardians might have looked like in their original context. The Ni-Tennō are often identified with Zōchōten, guardian of the South, and Jikokuten, guardian of the East. Their position in front of a Buddhist group and their attributes, however, are not rigidly prescribed: one of them usually holds one or two swords (like in the Konjiki-dō), the other one a three-pronged spear and eventually a sword. From the position and the holes of the MAO specimens' hands, we inferred that the tennō on the left was rising a sword, and the one on the right was holding a spear. The shape of our sword was inspired by the same curved, katana-like blade shown by the Konjiki-dō example. However, we deemed that the spear held there by the tennō on the right was too small and delicate -even showing a ribbon-like decoration -to be used as an inspiring model when considering the sturdiness and essentiality of the MAO statues. We then shifted toward another Heian example, the Four Heavenly Kings (Shi-Tennō) of the Eastern Golden Hall in the Kōfukuji temple at Nara. Here three of them hold a three-pronged sansageki spear, which we took as a model, while Jikokuten himself wields a curved sword similar to the two swords at Konjiki-dō.

3D MODELLING AND AR/VR EXPERIENCES
The digital representation of the Japanese Konjiki-dō temple, judged to be consistent with the contextualization of the two Ni-Tennō, was based on the search for a sufficiently detailed model of the temple to be used as a starting point. An excellent reconstruction has been found on the 3D Warehouse platform. First, it was correctly scaled and modified using Blender® based on the plans and the elevations found within the documentation, where the temple is represented as a square of 5.4 m side (Yiengruksawan 1993, p. 34). Then some simplifications have been made to clean the mesh and decrease the number of vertices using the decimate geometry tool. In this way, the computing power required by mobile devices for real-time rendering is optimized and allows a fluid display of the contents (Fig. 4). On the basis of the information and images taken as a reference, three-dimensional figures, similar to those currently present in the temple, were modelled and placed on the three altars together with the reconstructions of the two Ni-Tennō (Fig. 6). The weapons of the guardians, considered philologically plausible by comparison, have been modelled with the same software and correctly positioned in relation to the statues (Fig. 5). To give the sword and spear realistic shapes, the reticle tool was used, which allows the shape to be modified by managing a few control points. These reconstructions were used in prototype AR and VR applications, integrating the tools of the Unity® development platform with those provided by Vuforia™ and Google VR plug-in. The colours and lights needed to render a realistic scene and to focusing attention on the statues were set in Unity®. A metallic gold finish was applied to the temple and the figures inside to recall the real appearance of the Konjiki-dō. On the other hand, the texture generated with Agisoft Metashape® was used for the two Ni-Tennō to give them more prominence in the scene. For the AR experience, it was chosen to use the artworks themselves as targets, i.e. the activators of the experience, exploiting the potential of the Model Target technology proposed by Vuforia™. In particular, using the Model Target Generator, the three-dimensional models of the statues have been transformed into an object to be recognized and tracked through the use of mobile devices. So, simply framing the statues, people can access the proposed narrative path which integrates the explanatory contribution of a narrative voice. This is thanks to the Guide View, i.e. a simplified form of the statue shown initially on the screen of the device, which helps the visitor to position himself correctly in front of the artwork. The alignment between the virtual scene and the real one is crucial because experience predicts that by touching the highlighted parts of the statues on the screen, texts appear with the name of those areas displayed as if they were in real space.
At the end of the augmented reality narration, the temple is displayed all around the statues giving the viewer the impression of being inside it. By activating the extended tracking feature, it was also possible to show the temple at 360° around the point of view. This potentiality provided by the Vuforia™ plug-in allows continuing to visualize the virtual scene even if the target leaves the camera frame (Fig. 7). For the VR experience, a virtual tour of the Kōnjikidō was created using the tools of the Google VR plug-in and Unity®. Initially, the visitor is in front of the main entrance while a pointer allows interacting with some objects in the virtual space. By simply pointing at an arrow that leads to a new point of view, the observer can move inside the temple. Here he finds the statues exhibited in the museum, positioned on the central altar. If the visitor stops with the pointer long enough on one of the statues, he perceives its change of colour, which becomes more vivid, while an inscription with the name appears above it.
To realize these gaze interactions, two customized scripts were integrated. The first one manages the movements of the camera allowing to shift the point of view, while the second one records the interactions with the objects using a technology called raycasting. When the latter is executed, an imaginary ray is created and crosses the three-dimensional scene starting from the centre of the camera frame. Its scope is to intercepts the objects present, activating the desired reactions. In the case of the observer's teleportation, a timer also intervenes. It regulates the time that elapses between the moment the arrow is intercepted and the one when the script that makes the camera move is activated (Fig. 8).  The most suitable visor for the type of context and VR experience, from a cost-cutting and safety perspective, was the cardboard, also for safety and reusability issues. The use of these visors involves visitors and makes their mobile devices as display for reproducing the virtual scene. This is why the threedimensional objects that make up the scene are composed of a few vertices and do not involve the use of complex textures. In any case, cardboard was chosen for its accessibility and ease of use. Moreover, the possibility of providing personal cardboards produced in the MODLab Arch of the Politecnico was evaluated.

CONCLUSIONS
The workflow set up in this research realizes a low-cost project for the digital enhancement of museum heritage. In this particular period of the pandemic, the desire to bring the focus back to the actual visit of the museum and the objects on display, achieving greater involvement of visitors in a perspective of inclusiveness, sustainability and safety has informed each of the operations of analysis, interpretation and presentation of heritage. Ongoing developments of the research operations presented in this paper, besides the production of The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China tactile replicas, concern the study of new ways of presenting museum heritage through AR virtual assistants.