ONTOLOGY-BASED WEB TOOLS FOR RETRIEVING PHOTOGRAMMETRIC CULTURAL HERITAGE MODELS

.


INTRODUCTION
Studies on Underwater Cultural Heritage (UCH) sites induce the use of computerized techniques to handle, preserve and analyse the information produced.For complex archaeological sites, such as deep underwater shipwrecks, there is a special need for systems that support efficient storing and allow for a smart querying of data evolving over time.
This work is centered on the Xlendi shipwreck, named after the place where it was found off the Gozo coast in Malta.The shipwreck was located by the Aurora Trust, an expert in deep-sea inspection systems, during a survey campaigns in 2009.The shipwreck is located near a coastline known for its limestone cliffs that plunge into the sea and whose foundation rests on a continental shelf at an average depth of 100 m below sea level.The shipwreck itself is therefore exceptional; first due to its configuration and its state of preservation which is particularly well-suited for our experimental 3D modelling project.The examination of the first layer of amphorae also reveals a mixed cargo, consisting of items from Western Phoenicia and Tyrrhenian-style containers which are both well-matched with the period situated between the end of the VIII and the first half of the VII centuries BC.The historical interest of this wreck, highlighted by our work, which is the first to be performed on this site, creates a real added-value in terms of innovation and the international reputation of the project.A photogrammetric modelling of Xlendi shipwreck, done in 2014, was presented in a previous work (Drap et al., 2015) in the frameework of the GROPLAN project1 .The University of Malta have already organized photogrammetric campaings over the years and starting archaeological excavation.We have now five years survey to observe the site change over the time.
In 2001, the UNESCO Convention for the Underwater Cultural Heritage established the necessity of making all data publicly available on URL2 .This work is fully build upon this philosophy, proposes a way to share and analyze archaeological data.Indeed the documentation of underwater sites is becoming a growing concern and a large amount of research lab is focusing on this topic (Scaradozzi et al., 2013), (Scaradozzi et al., 2017).In that context, CH sites are tending to allow data exchange and accessibility in a worldwide network scale instead of single isolated database.Note here that relational data base systems tend to be rigid especially when it concerns the schema modifications and extensions.This particular feature does not meet our expectations where the scheme of historical sites need to be often updated as new CH discoveries are made, i.e, new typologies can be often discovered as new dives are made in Xlendi.In our view, the key factor to achieve a semantic exploitation of a shared open CH data consists of constructing a dynamic system that supports schema flexibility, data accessibility and knowledge sharing.Toward the development of CH applications that provide multimedia access to distributed collections of CH resource, we cite recent projects that among others, ADS3 , ARIADNE4 , EUROPEANA5 and STITCH6 .
The main objective of this work is the development of an in-formation system capable of establishing a methodology to acquire, integrate, analyze, generate and share numeric contents and associated knowledge in a standard machine readable format.In this paper, we present a knowledge graph based Web tools, where semantic data are stored in a graph database and schematized by an explicit specification of a conceptualizationontologies (Doerr, 2009) (Guarino et al., 2009).The graph structure meets our expectation of a schema flexibility since data can be separated from the schema model.In this way, the schema modification can be performed without changing the data instances or worrying about foreign keys, nested queries, and complex joins.Hence, new data can be inserted into the existing graph without loss of application functionality.
Ontologies have a meaningful role to play in supporting knowledge sharing expectations, as it provides a shared and common understanding of data and, in some cases, services and processes that exist within a domain (in our case, photogrammetry).Such common understanding facilitates communication between people and information systems (machines) and an enhanced ability to search for information across different knowledge repositories.This conceptualization allows the coupling of concepts within an ontology to information and processes within the organization being represented.In fact, ontologies allow application designers to fully understand the meaning and context of the information being modelled.Such common understanding is supported by the Linked open Data (LOD) structure which, following semantic web best practices (Lóscio et al., 2017), (Auer et al., 2013) and the Linked Data principles7 , will meet our expectations for data accessibility via the use of URIs (W3C Uniform Resource Identifier) to identify resources and allow people (and machines) to look them up on the web.
LOD knowledge graphs can be represented by a set of connected together triples (subject-predicate-object) defined by W3C for RDF8 stored within a triple store, e.g.DBpedia (Auer et al., 2007) knowledge graph bases which has been constructed using structured sources such as Wikipedia infoboxes.Such triples structure expresses information by defining relationships between data objects which allow resource to be identified, disambiguated and meaningfully interlinked.For example, encoding the fact that "Amphorae is an Artifact" in the knowledge graph is done by storing two vertices, one for "Amphorae" and one for "Artifact", with a directed edge originating with Amphorae and pointing to Artifact of type "isA".Moreover, this infrastructure will play a major role in providing a reasoning functionality where, the navigating between individual entities (near one another) in a graph is more obvious than trying to aggregate all attributes related to a given entity from different tables (as in a relational database).For example, the question "Is amphorae an artifact ?" can be answered by simply walking through the graph, starting at "Amphorae" and walking to "Artifact", testing edges and concepts along the way.Hence, data can be easily consumed and combined with other linked data.
In the context of Xlendi shipwreck, we adopted knowledge graph techniques to represent the photogrammetrical information of the site where data are stored within a triple store and schematized by a photogrammetry driven ontology Arpenteur9 .This ontology has been developed in a collaborative work between archaeologists, ontology designers and surveyors (Drap et al., 2015).Arpenteur ontology conceptualizes the whole photogrammetry process and involved CH artifacts in terms of their typologies, photogrammetric measurement and spatial representation, as depicted in (Ben Ellefi et al., 2018b).
The paper is organized as follows: Section 2 presents the surveying process of Xlendi shipwreck by photogrammetry.The storage system architechture of Xlendi semantic datasets is depicted in Section 3. The web tools are presented in Section 4 where: (i) Section 4.1 details the interactive queries performed in back-end of the web-tools with a focus on what ontologies can do and describing how they can be used in support of archaeological research; and (ii) Section 4.2 presents our implementation of 2D/3D web tools where an example use case is detailed.Finally, Section 5 concludes and presents future work plans.

PHOTOGRAMMETRY
Five photogrammetric surveys were done from 2009 to 2018.We had the opportunity to improve the method and try new experiments over the time.The first one, in 2009, was only a recognition photographic campaign done to identify the wreck.We process the 80 images later and use the others survey to scale and orient this one.
The second survey was done in 2014, thanks to the collaborative project GROPLAN, by COMEX with CNRS LIS and CCJ.We used the COMEX submarine Remora 2000 and a trifocal system used to acquire synchronized images on site.This approach allowed us to work safely at this depth and the trifocal system gives us a proper scale without intervention on the site.A huge amount of images were obtained and elaborated with the Photoscan software, which was used for the following campaigns as well.This campaign was also used to develop and test new known object extraction approaches in a site in an attempt to automatically identify and measure known artifacts on the site (amphorae and grinding stone).Nevertheless, several problems were still present.The overall cost of the operation remained high, we did not have a precise reference to the vertical and even if some surface artifacts could be lifted by ROV we could not consider an archaeological excavation.
The successive campaigns, 2016, 2017 and 2018, were organized by the University of Malta, with the participation of University of Dundee, CNRS LIS and CCJ.The idea was to perform the survey with a team of professional divers and archaeologist using rebreather.These three successive campaigns allowed the University of Malta to implement solutions to the problems identified during the first campaign.Scuba divers have drastically reduced the cost of the mission over the flexibility offered by a team of four professional divers has allowed to establish a method of shooting photographs with excellent light management leading to excellent recordings.
Over the years, the survey method has been refined.Timmy Gambin and his team built up procedures and tools to organize excavation and survey by divers at this depth.Cement dead bodies with coded targets were placed around the wreckage to ensure continuity of the reference system over the dive, a stable tripod with a bubble level and coded targets was installed to give a reference to the vertical.
Finally the last year in 2018 a real archaeological excavation was carried out at 110 meters depth with a water suction (with   A rigorous approach allowed us to obtain consistent records documenting sampling over the years (2009 -2018) and excavations completed in 2018 (Gambin et al., 2018).All of these surveys are now available in 2D or 3D on the project website and formalized by an ontology.
This ontology takes into account the manufactured items surveyed, as well as the method used to measure them, in this case, photogrammetry (photogrammetric data as oriented photographs, cameras, 3D points and their projections as well as camera distortion and precision estimator).
The surveyed items are therefore represented from the measure-ment point of view and has access to all the photogrammetric data that contributed to its measurement in space.
A 2D and 3D web interface, accessible on the project web site, allows access to all these data and semantic queries through them and over the web.

XLENDI SEMANTIC DATASETS
The University of Malta has been surveying the Xlendi site since 2009 and the dataset has been enriched by data collected over time, as detailed in Section 2. These successive temporal states of the same site required the development of a management and query system able to handle data according to a spatio-temporal context.In order to implement such query system, we opted for a full materialization strategy to store the different temporal versions of the datasets.Using this strategy, all different versions of the evolving dataset are stored explicitly in separate datasets within the triple store.This storage strategy will avoid any preprocessing cost in order to store the new versions and also ensure that the query processing over versions is usually efficient as all the versions are already materialized in the archive.
As a knowledge base storage system, we used the Apache Jena Fuseki10 component which provides a SPARQL server that implement Apache TDB11 triple store for persistent storage and provides the SPARQL protocols for query, update and REST update over HTTP.
In this triple store system, we have separated the dataset that contains Xlendi artifacts descriptions from the temporal datasets which contains photogrammetrical information corresponding to a given excavation.

Xlendi artifact Dataset
The artifact dataset contains information about the typological dimension of artifacts and the 3D theoretical model description: the set of 3D points describing the model, the rotation matrix and translation to align the multi-temporal models to a common coordinate reference system.The dataset is published as Linked Open Data (LOD) following the semantic web best practices (Lóscio et al., 2017), (Auer et al., 2013) and the the Linked Data principles (coined by Tim Berners-Lee)12 : (1) URIs to identify the things in your data, (2) 'HTTP://' URIs so people (and machines) can look them up on the web, (3) When a URI is looked up, return a description of the thing in the W3C Semantic Web Format (typically RDF, RDF-Schema, OWL), (4) Include links to related things.

Xlendi Temporal Dataset
Successive dives on the Xlendi wreck have resulted in seven temporal datasets corresponding to the surveys dates listed in table 1.These temporal survey datasets describe the photogrammetric process in term of camera settings, interior and exterior orientation parameters, extracted and matched 2D points and their corresponding 3D points.We modeled Xlendi temporal resources by Arpenteur ontology on the following concepts: 'Transformation 3D', 'RotationMatrix', 'RadialDecen-teringDistortion', 'Point 3D', 'Photograph', 'PhotoManager', 'MeasuredPointManager', 'ImagePointManager', 'ImagePoint', 'Item Point3D', 'DigitalCamera' and 'BundleModel'.Similar to the artifact dataset, seven SPARQL user interfaces, allowing to query temporal datasets, are made accessible online via our 2D/3D Web tools.For example, the SPARQL GUI for the 2018-09-21 dataset is available on URL15 .

XLENDI WEB-BASED TOOLS
In this section, we describe our Web tools accessible via http://www.lsis.org/groplan/svg/xlendi/xlendi.html.We set up a set of predefined user-friendly queries, as detailed in Section 4.1, to exploit and analyze Xlendi knowledge database.The Web tools do not require specific knowledge of Semantic Web technologies and SPARQL query language to run the userfriendly queries.On the other hand, users with Semantic Web knowledge can perform advanced SPARQL queries over Xlendi datasets via the web client user interface YASGUI available online in our Web tools (described in Section 3).

Querying Xlendi Semantic Datasets
Xlendi semantic data are processable via SPARQL queries that are performed in the back-end of our Web tools and presented to the user as the following queries.• Select artifacts having the same typology: This user-friendly query corresponds to a SPARQL query over the artifacts dataset retrieving resources having the same typology as the selected one.Listing 1 depicts an example of sparql query going through the knowledge graph structure where the typology can easily retrieved since be the 'hasTypolog-yName' and 'hasName' are directly related data properties to the vertice (concept) 'Amphorae'.
Listing 1: A SPARQL query to retrieve artifacts having similar typology as 'Amphore A02' PREFIX a r p :< h t t p : / / www. a r p e n t e u r .o r g / o n t o l o g y / A r p e n t e u r .owl#> SELECT d i s t i n c t ?s i m i l a r N a m e WHERE { ?amp a a r p : Amphorae ; a r p : hasName " Amphore A02 " ; a r p : hasTypologyName ?t y p o .
? s i m i l a r a a r p : Amphorae ; a r p : hasTypologyName ?t y p o ; a r p : hasName ?s i m i l a r N a m e .
F i l t e r ( ?amphorae != ?amp ) } ¥ • Select similar artifacts having a Hausdorff distance less than a given threshold: A SPARQL query over the artifacts dataset to retrieve the shape (the theorical 3D models) representing each one of the compared models.Listing 2 shows an example of a SPARQL query for retrieving the 3D shapes, i.e. a set of theoretical 3D points retrieved via the directed edge (object property) 'hasMea-suredPointManager' that relates to the concept 'Amphorae' in the ontology.The two shapes are aligned into a common coordinate reference system by retrieving the corresponding orientation matrices and translations.The hausdorff measure is performed to compare the two 3D shapes.
Listing 2: A SPARQL query to retrieve 'Amphore A02' 3D shape PREFIX a r p :< h t t p : / / www. a r p e n t e u r .o r g / o n t o l o g y / A r p e n t e u r .owl#> SELECT d i s t i n c t ?x ?y ?z WHERE { ?amp a a r p : Amphorae ; a r p : hasName " Amphore A02 " ; a r p : h a s M e a s u r e d P o i n t M a n a g e r ?p t s .? p t s a r p : h a s A S e t O f 3 D P o i n t W i t h O b s ?p t .? p t a r p : hasX ?x ; a r p : hasY ?y ; a r p : hasZ ?z .} • similar artifacts having a given dimension property (width, height, length or volume) value difference less than a given threshold: A SPARQL query over Xlendi artifact dataset to crawl the dimension property values and select the similar ones based on the threshold.As can be seen in Listing 3, the height value corresponding to an instance of the concept 'Amphorae' can be retrieved via object property 'hasHeight' which is inherited (inferred) from the concept 'SpatialObject' in the Arpenteur ontology.
Listing 3: A SPARQL query to retrieve artifacts having height distance less than 5 cm from 'Amphore A02' PREFIX a r p :< h t t p : / / www. a r p e n t e u r .o r g / o n t o l o g y / A r p e n t e u r .owl#> SELECT d i s t i n c t ?s i m i l a r N a m e ?d i s t a n c e WHERE { ?amp a ?t y p e ; a r p : hasName " Amphore A02 " ; a r p : h a s H e i g h t ?h1 .
? s i m i l a r a ?t y p e ; a r p : hasName ?s i m i l a r N a m e ; a r p : h a s H e i g h t ?h2 .
BIND (ABS ( ?h1 − ?h2 ) AS ?d i s t a n c e ) .F i l t e r ( ?d i s t a n c e < 0 .0 5 ) .F i l t e r ( ?t y p e IN ( a r p : Amphorae , a r p : G r i n d i n g S t o n e ) ) .F i l t e r ( ?s i m i l a r != ?amp ) .} ¥ • Get the survey images for the source artifact for a given excavation: While artifacts desciption are stored in the artifacts dataset, the photographs information are stored within different temporal datasets corresponding to survey dates.Hence, for a selected source artifact, a federated SPARQL query is performed over: the artifacts dataset to retrieve the bounding box corresponding to the selected source.
the temporal dataset to select the set of 3D points that fits into the bounding box and return the corresponding photographs.
Listing 4 shows an example of a query performed directly over the SPARQL endpoint (HTTP query interface) of the temporal dataset while an embedded SPARQL service will allow queering the Xlendi artifact dataset in the body of the same query.For a given 3D point, the corresponding photograph can be retrieved via the ontology properties path: 'hasImagePointManager/hasASetOfObservation/hasPhotog raph/hasFullFileName.
Listing 4: A federated SPARQL query to retrieve photograph corresponding to 'Amphore A02' PREFIX a r p : < h t t p : / / www. a r p e n t e u r .o r g / o n t o l o g y / A r p e n t e u r .owl#> SELECT d i s t i n c t ?p h o t o g r a p h WHERE { ?pt3D a r p : hasX ?x ; a r p : hasY ?y ; a r p : hasZ ?z ; a r p : h a s I m a g e P o i n t M a n a g e r / a r p : h a s  Most often a 2D representation in plan is better than a 3D representation.In addition, the lack of standard to express 3D models makes portability difficult.Nevertheless the 2D representations must be generated from 3D models to ensure the necessary accuracy.In this work we have opted for the two modes of representation: 2D using the orthophotos themselves generated from the 3D models resulting from photogrammetric surveys.Figure 4 is obtained from an orthophoto produced with the photogrammetric survey obtained in 2014.We used the Raphaöel JavaScript Library 17 to represent the 2D implementation of different artifacts in Xlendi site.
We have chosen not to directly use the orhtophoto as an interface to the end user because, despite the metric qualities of such a document, a key aspect of this work is missing: the human interpretation of the knowledge-based site.
The drawing produced here is by Mrs. Gina De Angelis, architect, who interpreted the orthophoto and produced this drawing by identifying each artifact.The two blue squares are the quadrats used for the excavation in 2018, the colors code the year of removal of the artifacts.
The 3D Web tool is accessible via the URL 18 .We provide a 3D view of the different models corresponding to Xlendi artifacts.The artifact positions map the 3D views of the Xlendi orthophotos corresponding to a given site survey period.The 3D viewer is powered by 3DHop (Potenziani et al., 2015).As described in Section 4.1, an advanced information analysis of the Xlendi site is presented to the user via different user-friendly queries (SPARQL queries are performed in back-end of the Web tool).Moreover, the web tools contain download options for 3D files corresponding to Xlendi artifacts and their typologies.
We draw the reader intention that our main intuition behind our Web tools is to provide a panoramic bird's-eye view for Xlendi shipwreck to allow a maximum exploitation of the available CH information by users (typically archaeologists).Let us take the 17 https://dmitrybaranovskiy.github.io/raphael 18http://www.lsis.org/groplan/hop/xlendiTimeLine/xlendi.html Figure 6: A Sparql query result tab view of the morphological attributes corresponding to Amphorae A02 as the selection in the Figure 4 Figure 7: An image display tab of artifacts in the Xlendi 2D Web tool corresponding to the Amphorae A02 as the selection in the Figure 4 example of amphorae A02, a panoramic view of this artifacts consists of: • a drawing view of the Xlendi orthophoto where the amphorae A02 is colored in red, see Figure 4.
• the set of morphological attributes describing this amphorae can be fetched via the SPARQL tab in our 2D web tool, as shown in Figure 6.
• a 2D image of Amphorae A02 can be depicted via the tab Image in the 2D Web tool, as can be seen in Figure 7.In the 3D Web tool, this image is displayed automatically on artifacts click.
• a 3D model of Amphorae A02 can be displayed in our 2D Web tool via the tab Model 3D, as depicted in Figure 8.The 3D model is also available for downloaded via a userfriendly query.
• a 3D view of the complete Xlendi site highlighting in red the position of the amphorae A02 shape within the shipwreck, as can be seen in Figure 5.
• available photographs taken in the UCH site of the selected amphorae A02 and an orthophoto depicting the complete shipwreck where the amphorae A02 is colored in yellow, see Figure 9.Note here that different surveying photos and orthophotos are available for different survey dates and can be fetched via a user-friendly query in our 2D/3D Web tools.

CONCLUSION
This paper presents an approach for retrieving, via the Web, photogrammetric data as well as non graphical data without any computation but only with querying on several semantic datasets acquired over the time.This is performed following Semantic Web best practices, where Xlendi data is modeled by the Arpenteur ontology and stored in an online published linked open dataset providing an online SPARQL service.The web approach presented here offers request on surveys data made on Xlendi wreck from 2009 to 2018.
This project addresses two types of problems: first the link between representation and the site.We see it in the 2D interface that is proposed, the statement while relying on a precise 3D measurement technique here is a 2D interpretation of the photogrammetric model.The interpretation is performed by a human, a specialist in the field who relies on incomplete metrics and his knowledge of the field.This 2D document, produced by an expert, highlights the geometry of the artifacts studied and represents only that one.In the 3D interface also proposed in this project, both the web interface and the Virtual Reality interface, the 3D model is there as an interface to non-graphical data and also supports the expression of useful model visible parts, among others, to evaluate the ratio measure/deduction.Hence, in this project, we chose a knowledge graph solution in which the semantic data are stored in a database of graphs and represented by a set of aligned ontologies.This approach allows us to turn resolutely towards an approach that allows sharing, exchanging, reusing and analysis of archaeological data in an effort to standardize formats.
Parallel to the website with a 2D/3D interface we are currently working on a virtual reality interface.The operator in total immersion in the 3D model can, on the one hand, follow the evolution of the morphology of the site over the years, and on the other hand visually assess the relevance of the choice of the typology of the artifacts.The combined use of photogrammetry process, 3D measurements, 3D/2D visualization tools and an ontological model provides the ability to analyze and compare data in a way that we have not yet finished exploring.

Figure 1 :
Figure 1: Shooting images: one diver for the camera and two other for the light.Photo: Dave Gration.

Figure 2 :
Figure 2: The bubble tripod and two cement blocks.pump immersed at 20 meters), 2x2 meters quadrat, photogrammetric monitoring, removal of artifact and appearance of lower layers.

Figure 3 :
Figure 3: Excavation on Xlendi by divers with rebreather at 110 meters depth.Photo: Dave Gration.

Figure 4 :
Figure 4: Xlendi Orthophoto drawing in our 2D Web tool (powered by Raphaël 16 ) /3D Web-based ToolsOur Web tools offer the user a 2D and 3D interactive views of Xlendi shipwreck.The 2D and 3D Web tool are depicted respectively in figure4and Figure5.

The
Figure 8: A 3D view display tab (powered by 3DHop) of artifacts in the Xlendi 2D Web tool corresponding to the Amphorae A02 as the selection in the Figure 4.

Figure 9 :
Figure 9: Xlendi orthophoto and photographs corresponding to the selected amphorae A02.

Table 1 :
Table1a statistical report for each Xlendi temporal datasets by survey dates.The goal of this report is to highlight the quality of photogrammetrical information within each temporal datasets in term of the distribution of photographs, 2D/3D points and the total number of triples.
The distribution of photographs, 2D/3D points and total number of triples for each of Xlendi temporal datasets