KNOWLEDGE-BASED FRAMEWORK FOR AUTOMATIC SEMANTISATION AND RECONSTRUCTION OF MILITARY ARCHITECTURE ON CITY-SCALE MODELS

: The scale models of fortified towns belonging to the Plans-Reliefs collection are exceptional witnesses of the formation of the French territory. The aim of the URBANIA project is the valorisation and the diffusion of this heritage through the creation of virtual models. The town scale model of Strasbourg at 1:600 currently exhibited in the Historical Museum of Strasbourg was selected as a case study. We develop and experiment an automatic procedure to identify and reconstruct military architecture works from point cloud digitisation of this fragile and bulky heritage. A priori knowledge formalized in a domain ontology informs the identification of the works - via geometrical feature comparison and consistency evaluation within the fortification system morphology - and their parametric 3D reconstruction refined by direct fit to the initial point cloud.


INTRODUCTION 1.1 Context
Over the past ten years, the MAP laboratory has carried out several projects aimed at producing virtual mock-ups from scale models of ancient cities.The last project has as a study case the Strasbourg plan-relief with a surface area of approximately 72 m² (12 m x 6 m) and which dates back to 1725 (Fig. 1).The Plans-Reliefs is a collection of city-scale models and the testimony of Vauban styled fortified architecture, the foremost military engineer of Louis XIV.The collection is one of the largest and most famous in Europe (Buisseret, 1998).
A renewed interest in this heritage has emerged since the 2012 exceptional exhibition as they contribute to a better understanding of the history of representation, fortification, territorial formation, etc.Also, many cities represented in planreliefs are seeking a digital version, in order to shed light on their urban fabric and territory, uncover a multi-scale cultural value stemming from a city to a regional, national and European scale and of course, to present its fascinating and attractive heritage property for tourism.Most recently, the Musée des Plans-Reliefs at the Invalides in Paris has unveiled an application in mixed reality using Hololens HMD and taking as its subject the plan-relief of the Mont-Saint-Michel (Microsoft Today in Technology, 2018).
For a long time, scale models have been the most effective mean of communication to represent architectures and cities.However, these objects suffer today from a lack of accessibility due to their size, state of conservation, number, etc.Over the decades, various ways have been adopted to improve the visualization of these objects.The use of endoscopic techniques on models gives rise to various model readers, direct vision devices, photographic or videographic, such as Robert Auzelle's maquettoscope, or Martin Vantreeck's relatoscope, which simulate the point of view that an observer could have at the scale of the model by reducing the perception of the scale.Despite these innovations, macrophotographic problems related to shallow depth of field and hard light persist.Digital technology will bring new solutions and new challenges.In addition to the work questioning semantic enrichment, a handful of model digitalisation research carried out in Europe aims to automate the reconstruction of the buildings from this data.
Figure 1.The plan-relief of Strasbourg (12 m x 6 m) is protected in a glass showcase that is difficult to access and in lighting conditions that are not suitable for photogrammetry

Overview of the paper
In this paper we present a knowledge-based framework for the 3D modelling of fortification sites.While section 2 focuses on the related works, section 3 introduces the general framework for the automatic semantisation and reconstruction of the military architecture of the plans-reliefs.Its subsections detail the four main operations of the workflow: geometric segmentation (section 3.1), knowledge modelling (section 3.2), automatic recognition (section 3.3) and 3D reconstruction (section 3.4).The latter presents an evaluation of the computation time and accuracy for the fitted parametric reconstruction.Finally, some concluding and opening remarks are made at the end of the article.

STATE OF THE ART
In most projects, either the geometric modelling uses the collected data as a support for a manual geometric reconstruction with standard CAD software or the workflow is not documented.Research to automate the geometric reconstruction of old city models is rare and is based on work carried out on a scale of real cities.These are top-down approaches using a fortiori data describing roofs to extrapolate the geometry of buildings.Even rarer, projects can be noticed by the use of procedural city generation software allowing the automatic creation of several thousand buildings responding to precise stylistic rules.Hence, the objective there is more the restitution of an atmosphere than a faithful reconstruction.Furthermore, the geometric and colorimetric quality of the produced models does not require a high level of detail due to the use of Google Earth for their dissemination (Christen et al., 2016 ;Dylla et al., 2009).Anyhow, none of these projects has sought to apply such methods to the fortifications.

KASTOR-II: A KNOWLEDGE BASED APPROACH FROM SCALE MODEL TO 3D REPLICA -II
Previous research on digital reconstruction of military architecture works on city scale models has been conducted and presented by Jacquot et al. (2013).Although the use of a visual programming language editor has demonstrated the relevance when applied to knowledge-based reconstruction, such tools are inadequate to work with the amount of data contained in the point cloud.Since then, work has been carried out on the city islets of various plans-reliefs (Chevrier, 2015 ;Macher et al., 2017), but the reconstruction of the bastioned parts has remained at a standstill.
To get closer to the automation state, we present a knowledgebased framework informing both automatic semantisation and reconstruction (Fig. 2).The sources are a priori knowledge concealed in the contemporary treatises of bastioned fortification written by military architecture theoreticians.Such a framework has been developed for this case study, namely KASToR.
Digital acquisition provides a point-cloud of the Strasbourg's plan-relief as input data, which is a low-level and undifferentiated data that records the XYZ position and RGB colour of several million points.Performing a geometric segmentation with the Canny algorithm applied to the pointcloud's heightmap extracts high-level geometry information enable further analysis.In a similar way formalizing a priori knowledge in an ontology enable their processes.Automatic recognition of fortification works on the point cloud is based on these high-level data.It computes a geometric similarity between known architectural works and the data, and a consistency evaluation of these works using belief propagation within the theoretical morphology of bastioned fortification systems.Each identified geometry is rebuilt using a theoretical parametric model and fitted close to the point cloud through the registration of the model and its Smooth Rotation Enhanced As-Rigid-As-Possible (SR-ARAP) deformation.
We focused our experimentations on feature design in the segmentation process, consistency evaluation within the fortification system and precision of the digital reconstruction.
Figure 2. Framework simplified scheme

Point Cloud Input and Geometric Segmentation
The state-of-the-art digitisation techniques such as lasergrammétrie and photogrammetry produce point clouds, which are undifferentiated data demanding a geometric segmentation before identification and reconstruction processes.
The erratic accuracy of the point cloud of the plan-relief stems mostly from the 1:600 scale of the model, its size (between one metre and 170 m²), its level of details (less than a millimetre), and the cumulated imprecision of conservation and model making itself.In URBANIA, we use photogrammetry to digitise the plan-relief with a pixel size on the object of approximately 0.2 mm with a distance to the object of about 2 metres.For more information on the acquisition campaign and preprocessing of the collected data, please consult Macher et al. (2017).
Previous analysis (Jacquot et al., 2013) underlined the saliency of the planar representation of military architecture works.A great deal of these are normalized and said to be à trajectoire, i.e., their sections are swept along a planar regulating line.Hence, the geometric segmentation of the point cloud data should be the planar view of each unidentified geometry figured in the point-cloud.We reached a significant gain of robustness and accuracy using computer vision techniques, Canny algorithm applied on the point cloud heightmap, in contrast to the two-dimensional contour graph of the meshed point-cloud used by the previous KASToR framework (Fig. 3).Interpolation through the output points produce vectorised curves of the plan views.The bastioned fortification is a complex of works responding to defensive functions whose dimensions and positions are calculated in order to keep a siege in check.Throughout the 250 years of the development of this military architecture, these defensive systems and their regulatory layouts have constantly adapted to the progress of the artillery that is opposed to them.As a result, the number of systems and works fortification elements designed is significant, as illustrated by the number of publications on this discipline (Mateus, 2006).The geometry of each of the objects composing the fortifications is precisely described by construction rules.A study is therefore necessary to select the fortification treaties most relevant to the realization of a knowledge model adapted to our project.These reference works bring an interpretation of Vauban's fortification system (Fig. 4).Consequently, we work with the assumption that this information is enough to inform a bottomup semantisation strategy.Therefore, the latter is carried out from one architectural work to the next, using local insights on vocabularies (future ontology concept) and syntactic links (i.e., object property in our designed ontology) existing between them in the fortification system.
The structured and controlled vocabulary of the military architecture thesaurus clearly defines the formal concepts throughout the framework.The thesaurus is used as a basis for the development of a domain ontology (Chatenet and al., 2000).A domain ontology is an explicit and formal specification of shared conceptualisation (Gruber, 1993).We used the OWL knowledge representation language for its definition.In the ontology design process, the relationships between the architectural works, understood as concepts, are defined by object properties.
The high conceptual abstraction degree of an ontology makes it suited for heavy automatization tasks (Andrews et al., 2012), and have the elegance of conforming its structure to the morphology of the theoretical system it embodies.The ontology nodes figure the fortification works, the edges figure their syntactic relations understood as conceptual basic spatial links (proximity, symmetry, flanking, etc.) in a shape-grammar logic (De Luca et al., 2006;De Luca et al., 2011).
In order to inform the further analysis, each node includes links to its relative item in a template planar representation library and a parametric model library.Development of those libraries follows the expansion of the ontology model.Different experts can feed the knowledge system following their expertise fields and related skills.

Automatic recognition from geometric comparison and consistency evaluation within the fortification system
Identification of the military architecture works composing the town fortification are ensured from a geometrical point of view and from a morphological one.Geometrical Similarities Evaluation produces the basic data for the Consistency Evaluation Heuristic to expand, while assessing the syntactic coherency of the potential architectural elements detected, interpreted as part of the theoretical fortification system.
As a knowledge-based framework, the data provided by the structured knowledge shown above are committed in these evaluation tasks.The template planar representation library is used to perform Geometric Similarities Evaluation and the ontology's concepts and relationships informs the Consistency Evaluation Heuristic.

Geometry Similarities Evaluation:
Each fortification work is adapted to the topology and strategical context, from the town borders to its country-like outskirts.The difference between the template representation and the built element makes geometric similarity evaluation a process that shall incorporate some fuzziness.
Current state-of-the art techniques using machine learning are set aside because of the large databases needed for the unsupervised or supervised learning of these algorithms.For our case study as well as for most architectural elements, there are not enough instances to compose them.
We have conducted few experimentations using the C++ computer graphic library OpenCV.First with the calculation and comparison of the Hu moments of two images, namely the segmented geometry against each of the template representation stored in the corresponding library.Then, with the FLANNbased matcher on SURF descriptors for 2d feature matching between the segmented geometry against the same set of data (Fig. 5).The results were lacking robustness to be fully implemented in the workflow, a promising solution is held by the computation of similarity matrices via deformed-template matching and optical character recognition-like techniques.
Figure 5. Geometric Similarities Evaluation Experiments green: matches between template and segmented geometry Besides, geometric similarity evaluation is set up for failure in three non-exclusive cases: (i) if the segmented geometry is too exotic regarding the reference geometry, (ii) if the name of the architectural work is dependent of its place within a corresponding architectural system and, (iii) if the data is sparse or lacks accuracy.Therefore, we associate to this pure computer vision problem that is geometric similarity evaluation an assessment process considering the place of each element in the morphology of an architectural system; namely, the Consistency Evaluation Heuristic presented hereunder.

Consistency Evaluation Heuristic:
Heuristic is an algorithmic strategy used to solve complex and difficult optimization problem such as identifying the right segmented geometries as the right fortification elements they represent while using the fuzzy outputs of a geometrical similarity evaluation.
The process of identification and semantisation uses a heuristic roaming in the segmented geometry abstracted as the nodes of a factor-graph (Fig. 7).It begins at the first line of defence, the front bastionné, then follows the organic development of the fortification system iteration per iteration (Fig. 7.

3).
A factor-graph is a bipartite probabilistic graph consisting in variable vertices (nodes), factor vertices and undirected edges.
Edges links exclusively a factor vertex to a variable vertex.
There are two kinds of factor vertices: unary and binary.The unary factor is linked to one and only one node (Fig. 7.5).The binary factor is linked to two nodes (Fig. 7.6).As the nodes represent the segmented geometry, the unary factors embed the precomputed geometric similarity and the binary factors ties two nodes together along with their spatial relations.
At each iteration, segmented geometries featuring one or many spatial relationships with the ones already integrated in the factor-graph are appended to it.Thereafter, the evaluation of the consistency of each abstracted geometry as a part of the fortification system is conducted via the computation of their marginal probabilities through message passing (Fig. 7.7).We used the Sum-Product Loopy Belief Propagation (LBP) implementation in the Python module py-factorgraph (Yedidia et al., 2003) for the message passing in our factor-graph.
The last iteration is reached when the outermost fortification elements are reached, and no new spatial relation could be detected.At the convergence of the last LBP within the factorgraph, each segmented geometry identified as a fortification work is labelled and fit for the reconstruction step (Fig. 7.9).
Consequently, this layer of abstraction, that is the probabilistic graph representation of an architectural system, enable the use of the spatial information held in the domain ontology (name of the element, relative place and spatial relationships) to perform inferences on a new dataset.

Reconstruction: 3D Parametric model fitting using SR-ARAP deformation
A three steps modelling method ensure the fitting of a low-poly mesh on the point cloud featuring the salient morphology element of the rebuilt architectural work.In contrast to remeshing based methods which are prone to reproduce all the smoothness and irregularities of the model, while using an excessive number of polygons in order to fit the point cloud.
First, each of the identified fortification elements is reconstructed on the point cloud using their parametric model.This process was outlined as a suitable outcome for fortification due to the variable nature of these architectural elements (Jacquot et al., 2013).The parametric process stems from their theoretical drawing sequence found in the related treatises.Thus, instantiating a theoretical model on the point cloud, adapted on the chosen anchor geometry: their planar representation.
Theoretical models are aligned on the point cloud using a meshto-cloud registration.This is a rigid deformation based on translation and rotation of a source geometry to a target geometry.From this operation, there is a general improvement of cloud-to-mesh average distance.
Registered models are fitted close to the point cloud using the SR-ARAP algorithm for mesh deformation (Levi and Gotsman, 2015).During this operation, mesh topology is conserved while its most salient points are attracted to their closest relatives on the point cloud, computed via k-Nearest Neighbours (KNN) search (Arya and Mount., 1993).We used the C++ Computer Graphic Algorithm Library (CGAL) implementation of these both algorithms.KNN search is the most time-consuming operation of them all (Tab.1).The outputs are semantised 3D fitted parametric meshes placed on the point cloud (Fig. 6).
Polygon composition of the rebuilt mesh can be tweaked using remeshing and mesh subdivision algorithm to tune the fitting and the weight of the output.We used the fortification work known as demi-lune as case study for our prototype.We then evaluated the computation time of the automatic reconstruction (Tab. 1) and the accuracy of the 3D parametric fitted modelling with manual reconstruction (Fig. 8).As expected, the parametric reconstruction lacks proximity to the point cloud, approximated average distance is 2.47 mm.Registration operation align the mesh thus lowering the max distance but increasing the average to 2.65 mm.At the end of the process, deformed-fitted model lowers it to 1.09 mm, a better result than the manual reconstruction whose averaged distance is 1.27 mm.Computation time on a standard office PC (≈16 s) far outrun the manual one (≈10 min), while the accuracy reached by manual reconstruction is satisfying.
Figure 8. Accuracy evaluation of the 3D parametric fitted modelling.Point cloud to mesh distance.
3.5 Summary of the advances and improvements from the previous KASToR framework: Among the advances of this new method is the direct use of the point cloud as an input to the reconstruction process.Although manual intervention is always present, it is limited to setting the threshold of the segmentation operation.Finally, unlike the 2013 method, the reconstructed object is a deformed 3D model that is as close as possible to the geometry of the point cloud (Tab.2).

CONCLUSION AND FUTURE WORKS
Plans-reliefs and old city models are heritage artefacts with characteristics (size, scale, multi-scale and multi-domain heritage values, etc.).The most prominent of them all is their unmatched capacity to illustrate the role and evolution of bastioned fortification in the history of Europe, along with the unfolding of these cities' urban fabric.
The focus of this article is the automation of the semantisation and geometric reconstruction of the fortifications based on point cloud.We present a knowledge-based framework, mobilizing a priori knowledge in geometric reconstruction of the works, disseminated in a web GIS application.This platform would easily integrate information related to the construction rules, knowledge that is structured in the domain ontology.
Thanks to the results obtained, it is now possible to push the interest of the digital model by developing it towards a truly digital mock-up allowing collaboration between all parties interested in the plans-relief, either through data exchanges or by allowing intervention on a single model.
The perspectives opened by our current research is to display in digital space lightweight models of the Plans-Reliefs collection which can hardly be presented to the public or the experts in a physical place due to their cumbersome scale.From the collection point of view, further studies can be carried out to verify the coherence of the theoretical texts with the built phenomena represented by these models.
As a further inquiry, we would like to apply this framework to the architectural scale, building whose design depends on a shape grammar system.In another field of expertise, we would like to test our reconstruction method as a possible approach for other analyses, such as FEM analyses, that require the same level of accuracy.

Figure 6 .
Figure 6.Reconstruction and point cloud fitting of bastioned fortification elements

Table 1 .
Computation time for automatic demi-lune reconstruction