EXTENDING A MOBILE DEVICE WITH LOW-COST 3D MODELING AND BUILDING-SCALE MAPPING CAPABILITIES, FOR APPLICATION IN ARCHITECTURE AND ARCHAEOLOGY

: One of the most challenging problem in architecture is the automated construction of 3D (and 4D) digital models of cultural objects with the aim of implementing open data repositories, scientiﬁcally authenticated and responding to well accepted standards of validation, evaluation, preservation, publication, updating and dissemination. The realization of such an ambitious objective requires the adoption of special technological instruments. In this paper we plan to use portable devices (i.e. smartphones, tablets or PDAs eventually extended to wearable ones), extended with a small plug-in, for automatically extracting 3D models of single objects and building-scale mapping of the surrounding environment. At the same time, the device will provide the capability of inserting notes and observations. Where the instrument cannot be directly applied, for example for exploring the top of a complex building, we consider mounting our device, or using equivalent existing equipment, on a drone, in a modular approach for obtaining data de-facto interchangeable. The approach based on the expansion packs has the advantage of anticipating (or even promoting) future extensions of new mobile devices, when the spectrum of possible applications justify the corresponding increased costs. In order to experiment and verify this approach we plan to test it in two speciﬁc scenarios of the cultural heritage domain in which such devices seem particularly promising: Strada


INTRODUCTION
Three/four-dimensional (3D/4D) model is a digital virtual content geometric-aware and reality based.In the field of architectural or archaeological heritage the 3D model must guarantee accuracy and quality of representation, regarding geometry, colour and building rules.The 4D concept improves the digital copy of cultural heritage objects with awareness of modification along the time and semantic shape, becoming in a knowledge based system.
Archiving and linking our 3D cultural heritage objects ensures the preservation of Archaeological and Architectural Heritage (AAH).The preservation has two different aspects.The first one is the preservation of the real AAH object owing the capability to perform advanced analysis over linked data (e.g.change detection).The second aspect is the preservation of a digital copy of an AAH object; this is particularly suitable for virtual museums where a user can interactively interact with the AAH object.
Cultural heritage sites and artefacts get a significant added value from high-resolution 3D models.A large number of models is increasingly available due to evolutions in technology and higher integration of survey techniques such as laser scanning and computer vision.Several attempts have been made to address the problem of automation the workflow in 3D model construction, reducing integration of formats and procedures.
In the field of survey for cultural heritage, it is mandatory to cite London Charter conceived to enhance the rigour with which * Corresponding author.E-mail: massimo.ancona@unige.itcomputer-based visualisation methods and outcomes are used and evaluated in heritage contexts.The London Charter for the Computer based Visualisation of Cultural Heritage (Denard, 2009) was expanded to encompass computer-based visualisation embracing 2D, 3D, 4D and even hard-copy printouts or computer generated physical objects such as replicas of museum artefacts (Denard, 2012).The London Charter states that an evaluation of the goals to achieve should prove first of all if 3D visualisation is an appropriate method, and if so, which 3D visualisation method is the most adequate to reach the goals.The Charter has also given birth to a new charter, the Seville Charter, which will propose specific implementation guidelines, in the field of Virtual Archaeology (VA).
Progresses are obtained in data management of 1 and 2 dimensions and they are already shared thanks to Europeana and its linked project.The more complex issues of 3D and 4D data management are still open, but remarkable tools and operational procedures have been developed to facilitate the cultural institutions in the creation of integrated 3D digital collections and to make easy use of these resources in their daily work (Arnold, 2014).Some recent works for digital repositories bring to important goals: sustainable and centralized safeguarding and archiving of 2D/3D data; free access to metadata; secure access to data for the different actors involved in scientific projects and the support and advice for these actors in the 3D data production and exploration through the latest digital technologies, modeling tools and virtual reality systems.
Today ICT can guarantee a new audience for cultural places and a sustainable management for public institutions.The current need for museums to augment their own revenue and improve their performance coupled with the transformation of cultural models towards more experiential services has led many to adopt a consumer orientation in an effort to make museums and their collections increasingly accessible to visitors.
The digital process of visual improvement of the artefacts is currently carried out based on photographic or TLS documentation of some archetypes.The commercial framework for mobile devices enables development of Augmented Reality applications.Using a pattern that is recognized by the device, the virtual model is shown as it is in the real world.Some works have the major novelty in that the experts in the cultural heritage field as well as museum staff, could benefit of this work and implement such a system.

Architectural and Cultural Heritage framework description
Some authors developed in the past years a robust background in laser scanning acquisition and architectural analysis (Guidi andBianchini, 2007, Clini et al., 2014), other works for 3D digital virtual models of ancient heritage monuments from on-site data, especially focusing on three-dimensional data acquisition techniques and color processing methods (Remondino, 2011) or focusing on simulation of large architectural heritage complex, restoration or colour information starting from real world data (Apollonio et al., 2014).
Clini et al. highlighted the need of validation of easy and low cost methods for architectural survey (Clini et al., 2013) while other authors investigated urban surveys and their use in the interpretation and design about cities and the methods for their modeling.
From the point of view of the automatic survey of AAH outdoor objects, the adoption of unmanned ground and aerial robotic platform (RPAS/UGV) represents an increasing opportunity.Actually the unmanned/manned robotic platforms are suitable to host complex survey payload as high-resolution camera or LIDAR (EU project ROVINA).Robot aided monitoring or surveying is involved in some projects such as After the Dark, winner of 2014 IK prize, and the MuseBot (Cigola et al., 2014).The currently available tools for AAH are shown in Table 1.

Information Technology background
Fallon et al. (Fallon et al., 2012) describe a system enabling rapid multi-floor indoor map building using a body-worn sensor system fusing information from RGB-D cameras, LIDAR, inertial, and barometric sensors.While map building in real-time is fundamental for our proposal, several differences exist from our environment and the original applicative context adopted in (Fallon et al., 2012).Fallon's context is centered on the need of emergency responders, security and military personal to quickly develop knowledge about a given environments.In such an environment, the user main task is not map building: he takes advantage of the constructed maps but has little time to dedicate to their construction.
In our case the user is completely dedicated to the map building process and the linking to the constructed maps with references to digital objects reporting a 3D representation of all relevant entities located in the area represented in the constructed maps.The user has no other mandatory task to perform during the map construction and the use of a lightweight hand-held device seems more convenient giving to its user more flexibility and interaction capabilities.
Today, the problem of indoor map building is considered of primary importance, see (Apple Indoor Mapping Project and iBeacon, 2014) for example.Moreover, low-cost full-color holographic 3D displays for smartphones are expected in the very next future.Their coming will be certainly an event that will give a wide relevance to 3D reconstructions.The companies are moving in this direction and, in particular, HP has already developed a wideangle, glass-free display prototype (Lewin, 2014).Fattal et al. introduce diffractive backlight solutions that seem well suited to realizing a multi-view 3D display for mobile devices (Fattal et al., 2013).
Several authors have used mobile phones for 3D map reconstructions (Tanskanen et al., 2013, Pirchheim and Reitmayr, 2011, Pan et al., 2011, Martin et al., 2014) obtaining interesting results despite the limitations of the device used.devices (Martin et al., 2014).Nevertheless most AR Apps only use sensors such as IMU, compass and GPS and barely consider image-based localization approaches.Apart from the computational cost of such methods, one of the main reasons is that they usually rely on the use of a 3D model or assumption on either the structure of the scene (supposed to be planar) or the camera motion (supposed to be rotational).Nevertheless, since the introduction of vision-based AR on mobile devices (by using AR-Toolkit Markers) impressive progresses have been made.The authors' approach is based on the decoupling of the mapping process from the tracking step in order to save computational power for the end-user application, an approach successfully considered for vehicle localization and augmented reality.Their system has been tested on both Android and iOS (Samsung Galaxy S2, Samsung Galaxy S3 and iPhone 4S).On both the Galaxy S3 and the iPhone 4S the tracking and image-based re-localization are done in real-time for an image of size of 320x240 and 480x360 pixels, respectively.
In (Pirchheim and Reitmayr, 2011) the authors present a realtime camera pose tracking and mapping system which uses the assumption of a planar scene to implement a highly efficient mapping algorithm.Their light-weight mapping approach is based on keyframes and plane-induced homographies between them.They solve the planar reconstruction problem of estimating the keyframe poses with efficient image rectification algorithm.
Mapping of a-priori unknown scenes in full 3D and tracking with a monocular camera still remains a hard task, or, at least, on mobile phones.Classical EKF-based approaches known from the robotics community seem to be computationally infeasible.Only the well-known PTAM system (Pan et al., 2011) has been shown to run as a prototype on the iPhone 3G.The authors tested their systems scalability on the Nokia N900, a phone equipped with a 600 MHz single-core ARMv7 and 256 MB memory.They processed camera images with a resolution of 320x240 pixels.In order to achieve the goal of rapid 3D reconstruction of environments, authors of (Martin et al., 2014) propose a method using several wide field-of-view images instead of a large collection of images, by adapting a cheap online space carving approach based on Delaunay triangulation to obtain a dense, polygonal representation of the environment.
Finally, in (Tanskanen et al., 2013) the authors propose a complete on-device 3D reconstruction pipeline for mobile monocular hand-held devices, generating dense 3D models with absolute scale on-site and simultaneously supplying the user with realtime interactive feedback.The method fills a gap in current cloud-based mobile reconstruction services as it ensures at capture time that the acquired image set fulfills desired quality and completeness criteria.In contrast to existing systems, the developed framework offers explore innovative solutions such as the usability of the on-device inertial sensors to make the tracking and mapping process more resilient to rapid motions and to estimate the metric scale of the captured scene.

INSERTING RECONSTRUCTED 3D DIGITAL OBJECT INTO THE BUILD MAPS
For the exhaustive description of an AAH object, we need 3D-4D model, additional data and metadata.The object can be adapted in order to extract new models with different purposes and for different users (a sort of custom view), in order to produced different views on request.
Remote sensing techniques are evolving and improving at an incredible speed.The software and hardware required to process the acquired data is generally less available.Furthermore, there are very few packages able to simultaneously deal with different sensors (and data).The development and use of new sensors requires the study and test of innovative sensor models and the investigation of the related network structures and accuracy performance.Also UAV platforms need some improvements for more reliable and fast acquisition procedures.
The recently introduced technology of human portable devices seems particularly suitable for extracting 3D models from the surrounding environment during an inspection of a museum or of an historical building (Fallon et al., 2012, Ancona et al., 2012).Their use is not intrusive nor dangerous for goods such as paintings, ancient drawings, potteries etc. and renders their adoption acceptable also in museums and in buildings containing precious and prestigious artefacts.Moreover, the wearer of the device has the capability of inserting photos, notes and observations with the underlining smartphone or tablet by adding customized information to the automatically collected data, for pointing out the most relevant aspects.In the external part of buildings, that a human with an hand-held device cannot reach, the use of drones could be applied.In this case is important the unification of the two approaches in order to produce indistinguishable digital objects.Also the data model adopted should be unified for making acquired data completely integrable.
There is an increasing use of hybrid sensors and platforms for the collection of different types of heritage information.The combination of different data sources allows the creation of different geometric levels of detail (LoD) and the exploitation of the intrinsic advantages of each sensor.The integration so far is mainly done at model-level (i.e., at the end of the modelling pipeline) while it should be exploited also at data-level to overcome the weakness of each data source.We need an all-in-one instrument that allow the acquisition of each important characteristic of object: shape, color, material, state of conservation.

Device extension
The main objectives of this approach are: • quality/precision of the obtained model, • low costs, • conformity to standards and well accepted procedures, • simplicity and rapidity of use.
An objective of our research is the application of a new methodology supporting the digital reconstruction of a wide spectrum of cultural heritage ranging from buildings and museums to single paintings, sculptures or drawings through a set of procedures supporting automatic digital 3D reconstruction of real architectural artefacts.Also works of art designed, but never built, can be considered for a virtual reconstruction based on the remaining documentation.
The paradigm we are proposing will be open, i.e. integrable with other mechanisms and any format of information, essentially for: 1. Supporting the use of 3D models of an object as the reconstruction of the reality and as a metaphor of the object itself.
2. Adopting models (i.e., virtual objects) as a systematic architectural replacement of the reality with an approach that uses descriptive replacements as a collection of structured models, with a well defined naming mechanism, based on a specific Architectural dictionary, in order to naturally extend (maybe merge) the concept of 2D GIS into that of 3D GIS.
3. Implementing a scalable system supporting a common data set usable from each kind of users (Superintendents of Artistic and Cultural Heritage, scientific researchers, tourists etc.) Thus, we propose a systematic interactive and collaborative mechanism for the acquisition of data to be integrated in standard 3D models in order to construct an information framework for the scientific visualisation, complete integration and web-based presentation of sets of spatial and temporal data naturally heterogeneous in order to facilitate the interpretation exploration and analysis of large quantities of data for pointing out their Geo-spatial, temporal and semantic properties.
Automation in 3D data processing is one of the most important issues when it comes to efficiency or costs.Now, different research solution and commercial packages have turned towards semi-automated approaches, where the human capacity in data interpretation is paired with the speed and precision of computer algorithms.Indeed the success of fully automation in image understanding or point clouds processing depends on many factors and is still a hot topic of research.The progress is promising but the acceptance of fully automated procedures, judged in terms of handled datasets and accuracy of the final 3D results, depends on the quality specifications of the user and final use of the produced 3D model.
In some applications, fast processing is essential and thus requires new algorithmic implementation, sequential estimation, GPU and multi-core processing.The internet is also helping a great deal in this sector and web-based processing tools for image analysis and 3D model generation are available although limited to specific tasks and not ideal for collecting CAD data, metric information and accurate 3D models.
The extraction and the derivation of geometric structures and semantic information are fundamental tasks in modelling.The existing procedures require some improvements to speed up the entire 3D restitution pipeline from images or range data.Images theoretically allow an easier interpretation and extraction with respect to range data.An ideal solution is able to produce truly 3D geometries with attributes and topologies and it should have an internal quality control procedure.A true sensor and data integration at an early point in the processing chain makes the information extraction more reliable, precise and effective.
Although heritage 3D recording and documentation should be an interdisciplinary task, clear protocols and user-friendly packages should be released to facilitate the use of data processing techniques to non-technical users.This will result in an increased adoption of standards for data storage and data exchange as well as sensor characterization and semantic organization of architectural models.
Embracing the Linked Open Data Philosophy will open the use and re-use of the data for different purposes with different level of details.The user (expert or not) must be able to perform queries connecting different concepts for different purposes: visualisation, predictive maintenance of cultural heritage, changedetection, etc.
Therefore we will extend smartphones or tablets PCs in order to implement an automatic construction of 3D digital models of cultural objects ranging from statues and fronts/fac ¸ades to scale mapping of buildings.We are developing a prototype device by using a smartphone extended with a plug-in which contains on its back two color cameras, a Micro-LIDAR (Pulsed Light, Inc), MPUs and a supplementary battery in order to power supply these sensors.A scheme of our device is shown in Fig. 3.
Figure 3: Device prototype scheme in which we show the smartphone with its extension: the two color cameras, the Micro-LIDAR, the MPUs and the supplementary battery.

National gallery of Marche (indoor demonstrator)
The National Gallery of Marche in Palazzo Ducale (Ducal Palace) at Urbino is the indoor demonstrator in our research.Our research already demonstrated how useful is to carry out high resolution acquisitions, to store those and finally to perform mobile or portable application for tourism or insiders (Clini, 2014).We developed a very cost-effective workflow in National Gallery of Marche, based on the Città Ideale painting (Fig. 4).
Now it is possible to involve the whole building of Palazzo Ducale, thanks to the new proposed device.This kind of test-bench provides solutions both for management and preservation of cultural heritage objects in the museum both for amazing architectural and historical value of the Palazzo Ducale.For the first task, the indoor demonstrator leads to obtain and validate operational practices through: • acquisition/survey of exhibition space to solve the onerous and difficult survey of complex indoor spaces; • acquisition/survey of museum pieces at different LoD, strictly connected with the intrinsic characteristics, and validation of their accuracy in team of experts; The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-5/W4, 2015 3D Virtual Reconstruction andVisualization of Complex Architectures, 25-27 February 2015, Avila, Spain This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-5-W4-453-2015• 3D modelling of same pieces and semantic segmentation based on an ontological representation; • 3D tool for non-expert users addressing the lack of digital tools in documentation and management of Cultural heritage, overcoming non digital (on paper) cataloguing, already diffused.
For the second task, the indoor demonstrator develops the following issue: • Virtual exhibition or AR application using architectural models carried out in the same museum demonstrator and related to acquisition about paintings, statues or mobile heritage.

Strada Nuova in Genoa (outdoor demonstrator)
The example of the Renaissance "Strada Nuova" (nowadays Via Garibaldi, Fig. 5) in Genoa is so relevant that Mario Labò in 1956 (Labò, M., Roma, 1956) wrote: The interest of artists and historians, for Genoese historical city center is timeless, and even more perceived by the international literature than the Italian one.
A heritage that is not yet appropriately and comprehensively documented, especially with the most recent ICT technologies, suitable for widespread diffusion, concerning the different urbanarchitectural and artistic aspects as well as the acquisition and processing of the existing documentary materials (Falzone, 1985).1. the whole environmental landscape values of this organic urban space, from its origin to the late transformation period into a residential district for noble families (a sort of small court of the Renaissance Genoese Republic); 2. the morphological aspect of the site facing north toward the hills of Montalbano and limited by the hanging gardens of the palaces Brignole and Durazzo Grimaldi, with its unique access from Piazza Fonte Maroso and Piazza della Meridiana; 3. the stereometric rhythm 15th to 17th century noble palaces, know especially for their architectural decorations and painted fac ¸ades (Palazzo Bianco, Palazzo Rosso and Palazzo Tursi) as well as museum heritage.
The Genoese painted fac ¸ades represent an important characteristic of Palazzi dei Rolli system, which has no equal in Europe (Vagnetti, 1970, Poleggi, 1972).An issue worthy of being documented and that, for its exposure and its conservation requires a specific depth of documentation, to be acquired with 3D digital models that are particularly precise to visualize the chromatic fidelity.This requires designing specific software, which can be implemented by a GIS standard tool, able to confront all the existing drawings, surveys, literary documentation, archival documents and maps with the new acquired data in time.A software that is characterized by: simplicity and rapidity of the procedure, cost containment quality, accuracy of the image and pattern obtained.
In particular the most useful technical survey equipment known today are LIDAR (TLS, Laser scanner), GPS and total station (TS), GPR (Ground Penetration Radar -GPR), and multispectral imagers.Moreover, the new mobile acquisition technology (used for smartphone) here proposed, could replace or supplement largely these tools.The key elements on which the work is essentially structured are: • 3D digital models as a metaphor of the detected object for navigation through the data; • survey as a cognitive system, described as a collection of structured objects, identified by a precise architectural vocabulary that allows easily extension of the concept of 2D GIS to 3D GIS; • scalable applications with a common database for every kind of end users (asset management, scientific study, tourism, etc.) by using filtered data according to the requirements.
The research will work on the definition of best practices for the acquisition of the data and the realization of 3D models, in order to build a real information system by providing an uniform framework for a web-based scientific visualisation of data.The goal is to facilitate interpretation, exploration and analysis of large volumes of data with significant geo-spatial, temporal and semantic issues.

CONCLUSIONS
A relevant and time consuming activity in applied architecture is the 3D/4D indoor map building of historic buildings and of all internal 3D cultural objects like statues, potteries, gables decorated in relief etc.
In order to speed-up the data acquisition needed for supporting such an activity we plan to use a hand-held sensor system composed by a smartphone (or tablet computer) extended with few optical sensors for precise distance evaluation.In order to verify our device we choose two specific demonstrators of the cultural heritage domain: Palazzo Ducale in Urbino and Strada Nuova in Genoa, both located in Italy.
A final remark concerns 3D reconstruction performed on smartphones.It is a promising activity in view of the next advent (forecast in 2 or 3 years) of real 3D holographic displays.Threedimensional display-technologies offer a wide applicative potential ranging from GIS and cartography to 4D object visualization (Schratt and Riedl, 2005).Holographic 3D visualisation will revolution most of today applications centered on 2D displays.

Figure 4 :
Figure 4: Workflow from acquisition to outputs in AAH fields: the Palazzo Ducale at Urbino demonstrator.

Figure 5 :
Figure 5: A synthesis image of Strada Nuova compared to an historical illustration.