DIGITALIZATION CULTURE VS ARCHAEOLOGICAL VISUALIZATION : INTEGRATION OF PIPELINES AND OPEN ISSUES

Scholars with different backgrounds have carried out extensive surveys centred on how 3D digital models, data acquisition and processing have changed over the years in fields of archaeology and architecture and more in general in the Cultural Heritage panorama: the current framework focused on reality-based modelling is then split in several branches: acquisition, communication and analysis of buildings (Pintus et alii, 2014). Despite the wide set of well-structured and all-encompassing surveys on the IT application in Cultural Heritage, several open issues still seem to be present, in particular once the purpose of digital simulacra is the one to fit with the “pre-informatics" legacy of architectural/archaeological representation (historical drawings with their graphic codes and aesthetics). Starting from a series of heterogeneous matters that came up studying two Italian UNESCO sites, this paper aims at underlining the importance of integrating different pipelines from different technological fields, in order to achieve multipurpose models, capable to comply with graphic codes of traditional survey, as well as semantic enrichment, and last but not least, data compression/portability and texture reliability under different lighting simulation.


INTRODUCTION
Digitalization is a fundamental part of the current scenario of Cultural Heritage documentation and intervention; it grew up and become a protagonist in the script of preservation and valorisation of heritage at risk (Grün et alii, 2004) also supplying researchers with highly reliable 3D models enabling new reading keys to formerly in-depth researched monuments (Bianchini, 2013;Cipriani et alii, 2016, Adembri 2014).Nevertheless several basic issues concerning 3D model true exploitation are still source of debate, here in particular, two aspects of these problems will be debated: the first concerns the pairing of data reliability and the needs of dissemination/interaction, the second is about the use of digital models as bases for traditional drawings (plans, sections, elevations, etc.).The higher the achievable resolution and accuracy, the higher the amount of hardware resources, as well as software efficiency needed: for this reason, an investigation line centred on data scalability, portability and efficiency of interactive visualization (higher frame-rate in real-time applications) started some years ago with the aim to merge different experiences from the word of remote sensing and entertainment applications (Fantini, 2012;Merlo et alii, 2013, Guidi et alii, 2016).On the other side, a more complex issue deals with reality-based models: in particular, the inevitable presence of big voids and how to complete themfor the achievement of a Constructive Solid Geometries - (Fantini, 2009).This last aspect is deeply linked to another source of debate among scholars that is the role of semantics and segmentation of 3D models from active sensors: can, a not perfectly "sealed" shape, undergo an automatic or manual segmentation/partition? How to integrate missing elements (occlusions due to surveying limitations), as well as sequences of edges (Cipriani et alii, 2014) resulting from a semantic partition and attain a reasonable and reliable output?Another issue to comment is the conversion of apparent colour texture into diffuse colour texture: on the one hand SfM-MVS, once become widespread, facilitated texture mapping upon heavy meshes with complex shape.On the other, 3D digital models and the corresponding textures, once imported into common render engines present the so-called Double Shadow Effect (DSE) due to the excessive pieces of information encoded in apparent colour texture (namely cast shadows, selfshadows, etc.).
Here is proposed a pipeline aimed at defining a set of closely related steps for the achievement of semantically-enriched multi-purpose models.The fundamental steps are:  The achievement of topologically-optimized geometries, capable to support LOD and CSG.


Reliable and differentiated BRDF/shading behaviour of LOD or low-poly meshes (depending on the situation).

STATE OF THE ART
As general statement, in the field of Visual Effects (VFX) and Computer-generated imagery (CGI) meshes formed by triangles are considered very poor in terms of rendering and animation.
The main reason is due to the fact that animation/morphing of virtual characters has to be LOD representations, based on subdivision surfaces (subD) rather than blocked resolution meshes, or NURBS models (De Rose et alii, 1998).Thanks to the achievements in both the fields of mesh parameterization and compression by means of displaced subdivision surfaces, or DSS (Lee et alii, 2000), new standards could be achieved in the first decade of this century through the progressive implementation of this variable-level-of-detail representation in the main geometric modelling applications addressed to entertainment.The main advantages of the displaced subD application to the field of Cultural Heritage documentation are: compression of morphologic detail, high portability and last but not least different output scales (fundamental for the achievement of conventional 2D drawings and physical representations based on 3D printing solutions).A strong limitation to an extensive usein the field of reality-based models for documentationis that the main subdivision criteria/rule implemented in commercial applications, the Catmull-Clark (Catmull et alii, 1978), works more efficiently with quad-dominant meshes, that are far from being a standard in the field of remote sensing.Due to this reason, timeconsuming manual operations called "re-topology"in the CGI jargonwere necessary in order to re-mesh heavy meshes formed by triangles into coarse quad-meshes suitable for the conversion into displaced subD (figure 1).
This approach can be considered as to be equivalent to reversemodelling pipelines based on coating, by means of NURBS patches, high level of detail meshes from sensors.The need to escape, or limit, manual "retopology" (it also requires skilled and experienced polygonal modellers) led to the development of several researches based on the automatic and semi-automatic detection of features and curvature lines on non-structured triangular meshes with the aim to use these morphological hints to obtain quad-dominant meshes (Lai et alii, 2008;Jacob et alii, 2015).Isotropic low resolution quad-meshes can be easily converted into subD, as well as completed (hole filling) in order to form CSG (figure 2).These LOD models can then be converted into DSS (figure 3).Optimized and easy-to perceive 2D versions of meshes allow an easy interaction between the user and the texture (Cipriani et alii, 2015), enabling easier and more accurate editing aimed at 2D segmentation of apparent colour that can be used as a base for the definition of specific alpha images enabling different BRDF behaviours inside single meshes (figure 4).

AIMS OF THE STUDY
The research aim is focused on the achievement of flexible, semantic-aware, multi-purpose digital models.These topics embrace several areaswhat Pintus et alii (2014) defined as the three main fields of IT application in Cultural Heritage (CH) -: acquisition, communication and analysis of buildings.The final purpose is then to define a comprehensive pipelinebased on the current panorama of charts, best practices and recommendations (Brusaporci et alii, 2013)capable to facilitate and promote the integration of several professionals during the numerous steps of reality-based modelling.The considerations exposed here could give some clues to be developedthrough automatizationin further interdisciplinary research and, at the same time, promote a deeper integration between IT and the legacy of previous architectural/archaeological representations: in terms of aesthetic, visual reliability, completeness of environment representation.
Moreover, "how to convert" 3D high definition models into conventional 2D drawing, is still a debated matter since it requires time-consuming and complex strategies (Martosa et alii, 2015) in order to make fit virtual simulacra with standard graphic codes and long-time tradition of archaeological drawings from XVIII and XIX century architects (Jacques, 2002).Those last examples are still used since they supply scholars and restorers with relevant information about masonries before XX century interventions (figure 5), witnessing the accuracy and the ability to summarize in one drawing several aspects that in the current technological framework may require great effort and time.
In addition to these matters, it is safe to say that several opportunities may derive from additional functions given to parameter space; it could be used for storing and encoding additional reading keys to the monument (Merlo et alii, 2012).In the case of Maritime Theatre -the Hadrian's personal domus -and the great southern nymphaeum of Piazza d'Oro (a vast multifunctional tricliniar area) archaeologists and architects working on it needed "multi-purpose" 3D model that could be used for the production of drawings (orthographic projections, perspectives, axonometric views), to be compared to archival documentation and historical drawings.Models had to be specifically designed in order to store and facilitate visual outputs underlining anthropic alterations that occurred during the last centuries (ranging from plundering of archaeological finds to bad or not aware interventions) and, last but not least, interactive models for making more easy virtual anastylosis.The achievement of multi-scale 3D models, namely LOD representations to be used as tools for virtual anastylosis become than a fundamental issue of the research unit working on both the pavilions of Hadrian's Villa, and on scattered elements once belonging to those buildings, in particular marble entablatures (stored in museums and antiquaria).For late Antiquity and Byzantine monuments from Ravenna (Arian Baptistery and the Basilica of San Vitale), the main focus was the achievement of a methodology enabling a more efficient BRDF definition for digital models of mosaic surfaces obtained through SfM-DSM pipelines.For these last cases the problem of topology, as well as, image and mesh segmentation was crucial in order to supply render engines with a detailed modulation of reflectance values (figure 8).
Figure 8. Specular, highly reflective materials are quite difficult to simulate in render engines: stratification of special RGB and greyscale texture is needed to specify BRDF.

THE PROPOSED APPROACH
With the aim to solve the problem of how to achieve reliable, multi-purpose models in compliance with the focus of this research, a five-steps methodology was developed:  A first draft point cloud segmentation is carried out manually in order to export toward mesh-processing applications single structural elements of the comprehensive point cloud.Then they are converted into high resolution meshes slightly bigger than the architectural element they represent in order to facilitate cuts by means of planes or NURBS patches.The set of independent high definition watertight meshes are achieved through global remeshing (in 3D Systems Geomagic design X) in order to get rid of both topologic and geometric defects (Cipriani et alii, 2014).In case of Maritime Theatre or Piazza d'Oro, the list of categories is the following: columns (C), walls (M), entablatures (T) and sectors of floor split into original flour (OF) and simple soil (F).These meshes are aimed at forming a semantic partition on which the optimization and integration process will be carried out through techniques coming from entertainment applications (displaced subD, normal, displacement mapping, etc.).


After an interdisciplinary discussion phase (with archaeologists, restorers, experts of history of art) a priority list is defined: each part of a complex is evaluated in order to understand its relevance and reliability (figure 9).Semantics, in case of complex archaeological site/buildings, has to be considered as an allencompassing tool to deepen the understanding of a building: the model, as well as its parameter space (namely, the way the model is split into (u,v) reference space) have to encode and make explicit characters/issues that otherwise could be misinterpreted on the basis of classical treatises.In other terms: strictly formal readings, based on classic orders (genus), may be quite far from being complete and reliable.Moreover, elements like floors, as well as portions of masonries still include relevant formal features, enabling further interpretations, in particular for removed elements.
Figure 9. Semantic enrichment cannot be carried out properly without interdisciplinary contributions: in particular it is fundamental the identification of wrong restorations.


In order to improve model's portability, reliable and less reliable/erroneous elements are optimized in two different ways.On the one hand highly detailed models of a building's specific parts (flat elements as floors without specific decorations, ground, restored columns, etc.) have been converted into "low-poly" models (using CGI jargon), then they were parametrized and finally, in order to apparently re-establish the original detail, normal maps have been calculated using the typical render-to-texture solutions called baking (Cohen et alii, 1998).On the other hand entablatures, columns, marble elements characterized by refined decorations and bass-reliefs, opus caementicium and earthenware regular trails on the floor undergone a quad-dominant re-meshing (through automatic solutions or manual re-topology, figure 10) in order to convert them into DSS with OpenEXR texture applied (Lee et alii, 2000, Fantini 2012, Guidi et alii 2016).Both quad-dominant and low resolution triangle meshes are converted into Constructive Solid Geometries (CSG).In some cases, depending on special needs of the final model can be more advisable to carry out a semantic partition (Cipriani et alii, 2014): in that case it is the summation of a set of meshes that leads to a watertight mesh thanks to the specific modelling procedures aimed at supplying continuity (figure 11).


All models were then textured using SfM applications since laser scanner and photogrammetric campaigns were carried out at once using a common network formed by RAD coded targets (Cipriani et alii, 2015a).Multi-sensors data played a key-role for the achievement of reliable representations of optically not-cooperating materials: in particular, column shafts affected by a not homogeneous sub-surface scattering phaenomenon (Benedetti et alii, 2010).


The interaction inside render engines among different models, namely in virtual reconstructions of architectural decorations digitally placed in their original positions, does need two further steps.For Lambertian surfaces a shadow attenuation/removal is advisable, as well as for materials like conductors (e.g.gold tesserae of mosaic tiles) that need special alpha images in order to specify different BRDF behaviours upon the same mesh.The shadow removal technique (Cipriani et alii, 2015b) applied in the case studies from Hadrian's Villa are based on two main points: the first is the possibility to have at one's disposal both three-dimensional geometry and illumination of the scene (figure 12).This includes the sensor/cameras location, the light source direction and the observed object geometry, from which a priori knowledge of shadows projection is derived.Both direct and indirect illumination are "baked" and stored inside a grey scale image that can be used as a mask able to control the passage of pixels corresponding to a double colour calibration (Cipriani et alii, 2015b): the result is a texture where darker areas are tempered (figure 13).In case of mosaic areas, the colour mapachieved through previous stepshas to undergo a segmentation phase in order to make specific alpha images enabling different BRDF behaviours to mosaic tiles with different optic features (figure 14).   Figure 14.BRDF definition of the apse at San Vitale: stratification of textures, as well as specific shaders applied by means of alpha images.

CONCLUSIONS
The pipeline summarized in this paper leads to the achievement of highly compressed meshes while keeping all the acquired morphologic detail gathered with sensors of different nature.In case of quad-dominant meshes the integration of missing areas needed for the achievement of CSGis faster and easier; moreover, due to the high interest shown by VFX software houses toward automatic quads-remeshing it is safe to say in the years ahead, we shall see robust implementations of these algorithms in several commercial applications.Simplified meshes based on entertainment pipelines are then easier to split/divide and complete, rather than triangles and they also enable easier parameterizations.A 2D (u,v) reference system visually consistent with the corresponding 3D model in (x,y,z) can also open a door for a deeper exploitation of parameter space, seen as an auxiliary to semantic characterization.Another important aspect concerns texturing inside SfM applications of models achieved through standard file formats of CGI: subD, displaced subD, OpenSubdiv.These forms of representation are not supported by the majority of the tools for photogrammetry and reverse modelling capable to reproject frames upon a shape.With the only exception of some high-end program belonging to design/mechanical engineering, subdivision surfaces cannot "dialogue" with the current scenario of remote sensing, so they have to be "freezed", i.e. blocked at a proper resolution and exported in other formats.
Once converted into a high-resolution meshes made of four sides polygons they can be textured inside Agisoft Photoscan or similar applications with the only precaution to keep the same (u,v) layout used for displacement/normal mapping.
As previously stated an useful feature of DSS, is the one of facilitating the control on mesh completion.Quad-dominant meshes are indeed far more "ergonomic" to be integrated using common mesh modelling tools, as well as rendered.Moreover, once converted into CSG they support interactive Boolean operations that, in the majority of entertainment applications, facilitate the production of sections and ortho-images (figure 15) with features quite close to traditional drawings.Polygonal bridges, hole-filling, extensions of connected trees of edges are easier on quads, rather than triangular high-poly meshes, for several reasons: the first is the easy and almost automatic selection on edges.Indeed quads are generally structured meshes with regular and aligned sequences of edges (accord to curvature lines).Finally yet importantly, parameterization efficiency on meshes like these is then the second reason that makes such low-resolution representations more advisable for the application of unwrapping algorithms previously listed.Quad-dominant, as well as triangular meshes, from the case studies from Hadrian's Villa are accompanied by textures of three kinds: normal maps obtained by render-to-texture solutions (baking), displacement maps in OpenEXR format (also from baking) and apparent colour texture applied through re-projection of frames using sets of re-sectioned cameras aligned using SfM applications.All these texture have been created in order to share (u,v) layouts for further processing phase concerning colour texture enhancement.
For the achievement of comprehensive and coherent representations of different elements gathered and assembled into a 3D reconstructive sceneas for the case of capitals moved from the area of the Canopus/Serapeumanother issue has to be considered: the problem of double or not coherent shadows of elements surveyed and photographed in different locations.Finally, semantics performed on colour bitmaps allows the creation of appropriate alpha-images enabling a more consistent representation of specular and reflective materials and it present the convenience of a simple bitmap editing, rather than a time-consuming segmentation of the 3D mesh (figure 17).

Figure 2 .
Figure 2. Steps for automatic retopology of a capital high-poly mesh into a pure-quad mesh by means of Instant Meshes application.

Figure 3 .
Figure 3. Left: the capital converted into displaced subD, right the original high-poly mesh.Optimized pure-quad meshes also enable fast and enhanced parameterization, rather than dense triangular meshes.The general development of increasingly more accurate parameterization methods is another important topic for a correct and true exploitation of DSS and low-poly meshes.In this sense Cipriani et alii (2014) already underlined the increasing interest of automatic parameterization solutions also in the field of architectural survey: isometric (length preserving), conformal (angle preserving), equiareal (area preserving).The possibility to mix different parameterization criteria inside modelling applications allows a full use of the parameter space in which colour and other information are encoded (normals, displacement, global illumination solution, occlusion, etc.).

Figure 4 .
Figure 4. Bitmap segmentation starting from an optimized (u,v) layout: manual or semi-automatic editing of the bitmap leads to the possibility to improve on a single surface different BRDF behaviours.

Figure 5 .
Figure 5. Two details of the sections passing through the southern area of the Piazza d'Oro at Hadrian's Villa by Charles-Louis Girault (1885).These drawings are still an important documentation source for comparisons, analysis, study.

Figure 6 .
Figure 6.Test phase on marble elements.The restored segment in concrete present an homogeneous result.Masonries and different typologies of marble artefacts have been acquired in different times with different devices (figure6) based on phase shift (b), on ToF (c) and with photogrammetric campaigns (d).Acquired data have been evaluated, integrated and then processed in reverse modelling applications, as well as entertainment applications in order to let them coexist and interact for specific aims of study(Adembri at alii, 2014).

Figure 7 .
Figure 7. Inside architectural elements coexist different materials (restorations and original parts).

Figure 10 .
Figure 10.Maritime Theatre digital model: application of semantics and different optimization techniques in order to enrich the model, reach CSG, as well as texture reliability .

Figure 12 .
Figure 12.A capital, removed from its original position.The a priori knowledge of both lighting and environment geometry allows a reliable reconstruction of the lighting during the photogrammetric campaign.

Figure 13 .
Figure 13.Once de-shadowed the apparent colour texture gets more close to a conventional diffuse colour texture, enabling more flexibility and reliability of objects under different illumination.Capitals formerly belonged to the Serapeum of Hadrian's Villa then moved to the Poecile.

Figure 15 .
Figure 15.A comparison between the southern nymphaeum of the Piazza d'Oro represented using textured displaced subD and a traditional drawing made Charles-Louis Girault (1885).

Figure 16 .
Figure 16.The Maritime theatre digital model obtained through the pipeline exposed in the paper.It can be efficiently used as the base for virtual anastylosis and other studies.Every part of the model is a CSG and allows sections as well as the production of reliable high resolution ortho-image.