AUGMENTED REALITY TO PRESERVE HIDDEN VESTIGES IN HISTORICAL CITIES. A CASE STUDY

: Mobile devices provide an increasingly sophisticated support to enhanced experiences and understanding the remote past in an interactive way. The use of augmented reality technologies allows to develop mobile applications for indoor exploration of virtually reconstructed archaeological places. In our work we have built a virtual reconstruction of a Roman Villa with data arising from an urgent partial excavation which were performed in order to build a car parking in the historical city of Valladolid (Spain). In its current state, the archaeological site is covered by an urban garden. Localization and tracking are performed using a combination of GPS and inertial sensors of the mobile device. In this work we prove how to perform an interactive navigation around the 3D virtual model showing an interpretation of the way it was. The user experience is enhanced by answering some simple questions, performing minor tasks and puzzles which are presented with multimedia contents linked to key features of the archaeological site.


INTRODUCTION
Currently, most conventional approaches to display Cultural Heritage (CH in the successive) contents consist of showing image and video contents which are streamed from multimedia repositories.The design of interactive contents requires more complex interaction patterns.Interaction concerns to the subject and its relation with the environment through a Visualization tool, involving the capability to identify key features to generate smart answers such that users can improve their body of knowledge.The main problem is how to manage the huge amount of knowledge in an attractive way for most people, in order to provide a consciousness of users as citizens.This far-reaching goal concerns to more efficient ways of Edutaintment based in enhanced presentations of multimedia contents.
Digitization of physical resources contained in large public repositories (museums, archives, Foundations) is the first step for the valuation of CH goods.Their reuse can be performed through distributed networks which are organized by means of a specific Ontology which has been previously developed.The lowest level supporting this organization concerns to metadata which are included as a basic layer.Multiple articulations can be developed from specific CH metadata towards EU multimedia data.So, we intend to provide an Internet support to create new digital contents and constructing an European citizenship aware of CH richness as a reference for diversity, tolerance and common values sharing.However, neither of these technological solutions contributes to make more sustainable the CH tourism if there are no visitors.
Edutainment is a challenge for knowledge transmission on multimedia platforms with large implications in education, cultural * Corresponding author: franciscojavier.delgado@uva.estourism and Small & Medium Enterprises (SME) activities bound to content production.This area involves a combination of techniques from modeling (interactive or semi-automatic), educational resources (text, images, videos, 3D models) and smart multimodal interfaces (tactile, speech recognition, facial expressions).All these functions are currently available for tablets and smartphones.
Virtual reconstructions in cultural heritage are hypothetical recreations of destroyed sites which are projected towards the present.For this reason, virtual and augmented reality applications are implicitly based in an eye's wink: they do not show the reality in a literal way (obviously impossible).Instead, we must reinterpret facts and sites, taking into account the most meaningful aspects for the current state of our knowledge.The same knowledge gives the opportunity of a dialogue not only between different representations, but also between different possibilities inside the same discourse.In this way, every virtual reconstruction becomes a place to understand the same reality from different viewpoints, where each user reconstructs its own version on a common platform interacting with other public / private involved agents.
The modeling of virtual contents for cultural heritage can be started from documentation obtained from image-and/or range-based devices.A sustainable economic activity requires an industrial development with a well specified methodology for business model, able of providing an economic return; it is necessary to achieve cost-effective solutions according to currently available standards.These goals imply the development of basic models and strategies for information fusion which allow the generation of lowresolution geometric models as a first step; these solutions are refined in successive stages by adding more sophisticated rendering solutions according to technological advances.This approach is not efficient enough to achieve a wider audience from the conception to the content production.It is desirable to develop more advanced models and tools for augmented cognition technologies.Last ones are supported by three-dimensional models which can be obtained from images with a large baseline and other representations arising from sketches, gravures or other sources not always properly georeferenced.Nevertheless recent advances in all these domains, it is necessary to develop more flexible tools for knowledge management in an interactive way.This far-reaching goal implies a more structured approach by specifying the flowchart between different building blocks.An extended version of the proposed storyboard must be able to integrate educational resources, content creation by graphical artists and mobile technologies around an interactive reconstruction of cultural contents.
The rest of the paper has been organized as follows.Section 2. starts with a description of the current location of the urban environment and the archaeological site to provide a contextualization to the reader.Section 3. shows our proposal of reconstruction of a Roman Villa and its environment from archaeological vestiges, with special regard to the hypocaustum and some hypothetical interpretation of rooms over this structure.Section 4. is focused on the selection of key features which define the interaction between the user and the virtual environment on mobile devices.Section 5. discusses the results to reinforce those aspects which can be improved.Finally, Section 6. presents the conclusions and some open challenges to be solved in the near future.

SURVEYING THE HIDDEN ARCHAEOLOGICAL SITE
Valladolid is a city of Spain.It was built between several rivers (Pisuerga and several branches of Esgueva) which were populated by vaccei before the Roman domination over Hispania.The Roman Domus and its archaeological remains, which have been the object of study in this work, are located very close to a branch of the Esgueva River and away from an original medieval nucleus which was reduced later.
In the 11th century, the old village grew towards this zone and Medieval, Renaissance and more modern constructions destroyed most Roman vestiges in the zone.After some preliminary archaeological excavations performed between the last years of the 20th century and the beginning of the 21st century, the City Hall decided to perform an underground parking in the zone, which is plenty of archaeological vestiges.Figure 1 displays an aerial orthophoto of the surveyed area near of "La Antigua" at the moment of excavations (left), and the same site nowadays with the garden covering the archaeological vestiges (right).
Figure 1: Aerial image taken at the time of performing the archaeological excavations 1 (left) and a picture of the current state of the surveyed area (right).
The opposition and annoyance of citizenship aborted this initiative.The citizen movement requested to declare the zone as archaeological site, but the City Hall decided to cover the remains with an urban garden.Because of this actuation, almost nobody can imagine the historic richness in the underground of the modern city.All those hidden elements are extremely meaningful and rich for self-location, as much as the church of "La Antigua" is.
The church has been independently surveyed in its current state and modeled both in high and low polygonization using Blender, an open source 3D modeling software2 .Figure 2 displays a rendered image of the current state of "La Antigua".
Figure 2: The virtual reconstruction of the current state of "La Antigua" rendered with Blender.

AN ESSAY OF REINTERPRETATION
The archaeological excavation was performed in a limited zone according to the available documentation from (STRATO Gabinete de Estudios sobre Patrimonio Histórico y Arqueológico, 2009).This excavation only reveals some elements of the Roman vestiges so it is not possible to obtain a complete picture of Roman constructions, which in the best of the cases are currently hidden.Indeed, most documented vestiges belong to historical periods different of the Roman period.The list of documented vestiges of several periods mainly include, among others: 1. several walls, 2. pipes, 3. pavements, 4. culverts, 5. ceramic vessels, 6. mosaics, 7. two hypocaustums, 8. human rests in an ancient necropolis.
In fact, and perhaps due to the urgency of the actuation, the archaeological documentation was generated using only traditional methods; it is incomplete and displays metric inconsistencies between documents which make difficult to track the changes.The main experienced issue was the disagreement between the archaeological strata of the different excavations and the height of the found walls.We have solved these issues by making a coherent adjustment of the 3D position of the elements considering just a single global reference for the 3D model.Figure 3 shows how strata information has been fused in the 3D model.
Blender has been chosen to design the 3D model since it is a multipurpose software covering several stages of the content production pipeline.Using Blender as a common sink for the available graphic resources we can collaborate and share elements of the design such as geometry, materials and textures, at the same time we obtain quick renders and previews for a more realistic visualization (see Figure 4).
The virtual reconstruction of the documented vestiges has been performed using the following methodology: 1) the original documentation has been surveyed for each stratigraphic unity, which has been compared with the photographic documentation; 2) this information has been used as a blueprint in Blender to extrude the topology of the site, with special regard to the original dimensions, position and orientation of each element; 3) a collection of tiled textures, normal and specular maps has been designed taking into account the original collection of pictures taken during excavation.Pictures have been manipulated when necessary using the GNU Image Manipulation Program 3 ; 4) textures and different maps have been applied over the geometry to define the appearance of the model and to achieve a more realistic visualization in real time.The large number of available pictures of the outside of the hypocaustum (more than 30) allows us to recover metric information about the dimensions and location of this element at the same time that a high resolution texture using photogrammetry.The reconstruction has been performed using Agisoft PhotoScan to calibrate camera shots, building a sparse point cloud, and converting it into a dense point cloud, which has been finally triangulated and textured.The result is a single dense mesh composed by more than 70.000 faces with a high definition and realistic texture.
Unfortunately the model is too dense to be rendered with conventional mobile devices so we have baked normal and diffuse maps over a retopologized low polygonal 3D model of the hypocaustum.Thus, we have obtained a mesh composed only by quad faces with a quite realistic appearance, thanks to the use of conventional shader programs running in the GPU in real time.This mesh can be merged together with the rest of vestiges, improving the realism of the initial reconstruction.A similar strategy has been used previously in the literature such as in (Aparicio Resco et al., 2014) and (López Romero, 2014).Figure 5 displays two screenshots to illustrate this procedure which combines Agisoft PhotoScan with Blender to embed realistic photogrammetric models into virtual reconstructions.Nowadays it is increasingly common to find models reconstructed from multiple views in films and other kind of animations from computer graphics productions where there is no need to pay attention to the weight and complexity of the model.However, this is critical when the model is deployed in mobile devices whether the model is available in the own device or it has to be downloaded from remote repositories.The Villa has been modeled from the position of some evidences of Roman settlements, which were found along excavations that took place between 2006 and 2009.The boundary of the Villa to the west has been determined by a well-preserved hypocaustum chamber and its corresponding atarjea (see Figure 6).The other boundaries of the Villa have been established by portions of walls, an opus signinum ground, the interpretation of the path of the river and the current location of the church of "La Antigua".The orientation and position of the second opus signinum ground (located at the square H2 of dimensions 220x220 cm) have provided the basis for a hypothetical distribution of the other rooms of the Villa.Besides, the Roman vestiges previously discovered inside of "La Antigua" make coherent the hypothesis that the Villa could be extended reaching the church.Figure 7 displays the hypothetical location of the Villa over the place where the archaeological excavations were performed.
In summary, the archaeological evidences of the Roman period have made us capable of recreating a hypothetical location of the Villa which has been combined with a hypothetical distribution of its rooms taking into account other Roman villas and domus described in the literature (Fernández Vega, 2003).Some of the most relevant examples in Castilla y León which have been previously documented are the Roman Villa of Almenara de Adaja, Puras (Valladolid) in (González Gallero, 2008), and the Roman Villa of Quintanilla de la Cueza (Palencia) in (García Guinea, 2000).
The superimposition of different stratigraphic layers with the current situation makes difficult to interpret properly the original Roman vestiges.In order to solve this problem, Figure 6 includes information from several archaeological excavations performed in the area between the years 2006 and 2009 which are divided in numbered stratigraphic units (U.E.).This figure shows the excavated areas and the architectural vestiges of several historical periods found in them.Please, note that different colours are used to represent different materials instead of different archaeological periods as could be expected.Actually, in this region there have been successive constructions over the vestiges of the previous period without taking care of preserving them, which has created a mess in the plane which makes it difficult to understand.The hypothetical contour of the branch of the River "Esgueva" has been used to limit the Villa to the west while the church limits the Villa to the east, as Figure 7 shows.
Finally, we have designed the textured 3D model of the Roman Villa, including both rooms and ornamental objects, just following the same methodology as we used in the case of the archaeological remains found during the excavations.Results are illustrated by Figure 8 which shows images rendered from different camera locations of representative places of the Villa 4 .The steps to create the complete 3D model of the Roman Villa were the following ones: 1.The 2D designs of the Villa (see Figure 7) were used as blueprint in Blender.2. The walls of the main lounges, rooms, baths and halls were extruded to create a basic mesh for the model.3. Some openings were allocated on the walls making loop cuts in the mesh of the rooms to create doors and windows.4. The different parts of the roof which belongs to different wings of the Villa were connected, forming a special kind of cross hipped roof. 5. Several textures for Roman mosaics were created using resources publicly available with Creative Commons licences 5 .
4 A video preview of a camera travelling through the Roman Villa can be seen at http://vimeo.com/116531713.This video does not contain the latest improvements made in the Villa which will be available soon 5 Textures with Creative Commons licence have been collected from CGTextureshttp://www.cgtextures.comThese images have required minor improvements and manipulation for tiling.6.A complete collection of materials was defined by creating different kind of maps in form of textures including, at least, a diffuse map to give the main color to the surface of the material, a normal map to specify how shadows derived from the roughness of the material must be projected without adding extra geometry, and a specular map to determine the brightness level and color reflected at each point of the surface of the material.7. Ornamental details and furniture such as statues, pictures, bath pools, oil lamps, banks and columns were added to complete the inside of the model.8.A simplified environment was also created, including a terrain mesh for the ground and the path of the River Esgueva. 9. Several light and a global sun light were added to the scene to create realistic lightning conditions for rendering.10.Rendering parameters such as ambient occlusion, blurring and anti-aliasing were tweaked to improve the quality of the rendered images.
At each stage we have paid special attention to preserve a coherent topology in the 3D model using only quadrangular faces, avoiding triangular faces and maintaining the simplest mesh for representing each object.For instance, the oil lamps have been modeled using just a couple of planes for the flame and a few more for the chains and the dish.The flame material contains a short animation and alpha transparency.Material maps have been created at the lowest possible resolution with a power of two dimension for better performance at loading textures.Our goal was always to design the most realistic model which can be rendered in real time by modern mobile devices GPUs.This goal involves the use of several maps for materials and different levels of detail to improve the mobile experience of the user in augmented reality applications.

AUGMENTED REALITY IN MOBILE APPLICATIONS
The first applications using virtual 3D models for showing cultural heritage contents based on mobile devices were developed as Archaeological guides from the late nineties going from (Vlahakis et al., 2002) to more recent contributions such as (Eggert et al., 2014).The information about the user's current location provides a context to display specific contents.First applications provided only information on laptops extended with head mounted displays (HMD) devices.A HMD consists of an external web camera, a digital compass, a backpack with a GPS receiver, a laptop wireless communication equipment and a battery.This equipment can be incorporated to a wheelchair but is not practical from the user's viewpoint.It is necessary to advance to increase the portability and to improve the performance of low-cost devices able of supporting AR-based applications on mobile devices.Recent advances by Oculus VR are very promising along the same line, but the current cost and the scarce developments are dissuasive.
Some applications to Cultural Tourism can be found in (Fritz et al., 2005), where the authors use a binocular system to visualize enhanced environments to find additional contents by means of a virtual walk providing an immersive experience in an indoor scenario.Unfortunately, this solution is not very wearable, and poses several problems for its extension to outdoor environments.
An extension of methodology based in Archaeoguides to more advanced Mobile Augmented Reality solutions (MAR in the successive) can be found in (Stricker et al., 2010).In this work, the authors develop an application to integrate several tools arising from position-based tracking, computer vision, and interactive visualization.The mobile devices is given by an Ultra Mobile PC with 15 fps.We have developed a similar solution which is based on laptops embedded in a wheelchair which has been proved in the historical center of Segovia (Spain).More recent technological advances for smartphones and tablets have allowed us to extend their approach by incorporating efficient solutions on Android-based mobile devices.A first version was introduced in (Delgado et al., 2013).
A recent general approach to Augmented Reality on mobile devices can be seen in (Adhani and Awang, 2012).In regard to their applications to Cultural Heritage, authors emphasize their role to visualize the invisible or transparent contents.To warrant an efficient management, it is necessary to combine different aspects which involve content providers and user's needs by means of and appropriate choice of involved technologies.In this work, we are using a markerless approach, having in account only GPS to display the contents in an interactive way.
From the user's viewpoint, some relevant advantages of AR solutions for customers are linked to the wearabililty of portable devices, a friendly management and the capability for experiencing in situ different levels of interaction with very rich objects.This situation must be compatible with very complex technologies which must be transparent for users.Technologies are supplied by different kinds of stakeholders and, by adapting some items of (Adhani and Awang, 2012), they involve to different aspects such as: • computational platform to provide and manage virtual objects in an interactive way; • display mechanisms for showing multimedia contents which can be switched according to user's preferences; • registration and tracking for aligning contents depending on changes in relative location; • interaction technologies depending on the user's profile and enabling him/her the usual tasks in mobile devices; • wireless networking by minimizing adverse effects linked to inertial phenomena and other artifacts: • data storage and access technology to provide customized information on different platforms.
Currently, smartphones and tablets (generally called mobile devices) fit to our purposes as they include sensors to receive GPS position, digital compass for orientation, accelerometer for tracking small motions and wireless communication using different protocols.All of them are integrated in a convenient size and shape allowing to disseminate the cultural heritage value in an interactive way.However, the user interaction is still limited because the user needs both hands to interact with the mobile device.To solve this problem and provide the user with a true freehand device two authors of this paper have designed and implemented an eye-based tracking for interaction with any low cost camera (like the frontal cameras of most current mobile devices).
The incorporation of a video camera and additional systems to support user tracking allow us to provide meaningful information and a realistic model for position tracking at the same time.
An image-based solution for tracking consists of aligning views captured by the mobile device with other views of the environment which are already available (or with projections of a previous available 3D model.This approach extends the traditional vision-based tracking systems (using natural landmarks) and 3D reconstruction from multiple views.
The high resolution model has been designed for a desktop application which can be explored and navigated at home.In this way, the user can search key features to be applied for in situ interactions by using mobile devices.Obviously, models to be displayed in mobile devices requires a lower resolution, specially with respect to the graphic resources (images and the different kind of maps).However, they also support advanced real time rendering effects such as normal mapping and shadow casting thanks to the advances in mobile GPU computing (see Figure 9).A better looking model improves the visual perception of cultural heritage content by the user, at the same time it encourages him to interact with the environment.The application can be tailored for two different kind of users depending on their age, playing capabilities and the level of interaction they want to achieve.

EVALUATION AND DISCUSSION
Our software provides a support for different multimedia contents, which can be displayed from an approximate location in an interactive and ludic way.Disseminated remains and the lack of historical information about the archaeological site are two important issues to provide a reconstruction of the current state at the end of the Roman Empire.The interpretation is performed from similar but better preserved Roman villas in the neighborhood nearby.Thus, we are neither intending to provide the true history nor a faithful approach to a reality, which is unknown in most aspects.Our goal is to understand the city as an open museum which is evolving over time as a consequence of the interaction with social agents.The developed application shows an example of how to integrate implicit and explicit knowledge arising from different sources to reinterpret the reality in connection with some snapshots of historical past from multimedia contents.Currently available data are unstructured, and most data are not accessible.A challenge for the next future is how to discover and integrate the huge amount of information as different layers to be superimposed on an evolving environment.In this way, we intend to contribute to illustrate the changing nature of different relationships between agents from the past to the present.This implies the development of more pleasant tools for the Internet of Things, including tools for content creation which can provide storyboards with different customized paths depending on the profile, e.g.
We have demonstrated how to integrate mobile technologies inside a historical discourse based on a interpretation of hidden archaeological vestiges.For most people, the use of technology must be transparent and in fact they should not appear in an explicit way.However, this goal involves to integration and georeferenciation of data, processing and analysis of models, fusion of information in different LoD in a common framework, rendering capabilities from open-source tools, access and management from mobile devices, remote access to Cultural Heritage repositories and reuse of multimedia contents.All of them require a combination of photogrammetric techniques, information processing and analysis, manual and semi-automatic modeling and rendering, advanced visualization techniques for interactive navigation, and local management from mobile devices.
It is necessary to develop a flexible amenable software architecture and able to integrate services and processes linked to multimedia contents and advanced tools for interactive visualization.Some still as standing technological issues concern to a) a more efficient reuse of resources (reactive algorithms, real time performance) under more flexible environments and compatible with several rendering environments (such as HTML5 which provides support for WebGL), but which are not fully integrated with mobile devices; b) the development of more friendly software tools to integrate the above technologies; c) the design and implementation of models to warrant interoperability between tools and data repositories in the Web 3.0 framework; d) the capability of reusing web resources, including cloud computing for custom applications.
Neither of the above challenges can be achieved if there are no economic resources to support them.In view of decreasing resources for Cultural Heritage and the lack of a pan-European data portal, economic sustainability of this kind of applications has become a major issue to obtain a return for the performed work.The above integration is focused towards different kinds of agents which are not passive consumers, but active agents wishing to participate in the discovery of other realities, linking a reinterpretation of the past to the current reality.
Paths cannot be uniquely determined, but they must be discovered as part of the action performed by each agent; thus, the adopted profile is crucial for edutainment purposes.To achieve this goal it is necessary to develop business models for applications based on mobile devices which can incorporate sponsorship, subscriptions, advertising-support, custom contents to be played at different levels and with different business models (including premiums linked to the achievement of partial goals), and creation of communities of users and content-creators.In absence of a truly interest by the large operators of videogames companies, it is clear that there is not a unique solution, and more structured mash-ups communities involving developers and content creators are necessary.

CONCLUSIONS AND FUTURE WORK
In this work, we provide answers to several basic issues concerning to the communication of CH contents by following an interactive strategy based in games with CH contents in historic urban environments.To achieve this goal, we have designed and implemented a software application which includes web services, Mobile Augmented Reality and Serious Games to enhance cultural experiences through immersive interactions performed at smart mobile devices.
The prototype of the developed application is illustrated with some examples corresponding to a currently hidden archaeological zone of Valladolid (Spain).The augmented representation allows to understand the city as a superposition of layers of different periods along the History, where social agents interact between them.
In other words, the architectural space is reconstructed from the interaction between different social agents.Therefore, the knowledge of evolution in old times provides some keys to understand their projection towards the future in an active way at the same time users can receive historical education shaped in a game.
The archaeological and historical richness of the zone suggests different extensions of the performed work in the near future, which would be developed depending on additional economic resources.Next steps include the development of a true multiplatform solution and the development of virtual characters for providing user guidance in the area; some more advanced applications could include avatars to ease a more friendly interaction.
In the current state the prototype does not include interaction between avatars; in the next version the application would allow the user to select the avatar representing the player and interact with the other players through their avatars.

Figure 3 :
Figure 3: The 3D model of the archaeological strata retrieved from documentation which includes vestiges from different historical periods

Figure 4 :
Figure 4: UV mapping of the roof in Blender (top) and render of the reconstruction of the archaeological vestiges (bottom).

Figure 5 :
Figure 5: The exterior side of the hypocaustum reconstructed in Agisoft PhotoScan (top) and the same 3D model imported in Blender (bottom).

Figure 6 :
Figure 6: The archaeological remains and the current location of "La Antigua" church.

Figure 7 :
Figure 7: The outline of the Villa located over the archaeological remains which have been hidden for better visualization.

Figure 8 :
Figure 8: Rendered images of two rooms in the Roman Villa: the atrium (top) and the caldarium, placed over the hypocaustum (bottom).

Figure 9 :
Figure 9: A Nexus 5 viewing the vestiges from different periods discovered in the same area in the three archaeological excavations.