SURVEYING MEDIEVAL ARCHAEOLOGY : A NEW FORM FOR HARRIS PARADIGM LINKING PHOTOGRAMMETRY AND TEMPORAL RELATIONS

The paper presents some reflexions concerning an interdisciplinary project between Medieval Archaeologists from the University of Florence (Italy) and ICT researchers from CNRS LSIS of Marseille (France), aiming towards a connection between 3D spatial representation and archaeological knowledge. It is well known that Laser Scanner, Photogrammetry and Computer Vision are very attractive tools for archaeologists, although the integration of representation of space and representation of archaeological time has not yet found a methodological standard of reference. We try to develop an integrated system for archaeological 3D survey and all other types of archaeological data and knowledge through integrating observable (material) and non-graphic (interpretive) data. Survey plays a central role, since it is both a metric representation of the archaeological site and, to a wider extent, an interpretation of it (being also a common basis for communication between the 2 teams). More specifically 3D survey is crucial, allowing archaeologists to connect actual spatial assets to the stratigraphic formation processes (i.e. to the archaeological time) and to translate spatial observations into historical interpretation of the site. We propose a common formalism for describing photogrammetrical survey and archaeological knowledge stemming from ontologies: Indeed, ontologies are fully used to model and store 3D data and archaeological knowledge. Xe equip this formalism with a qualitative representation of time. Stratigraphic analyses (both of excavated deposits and of upstanding structures) are closely related to E. C. Harris theory of “Stratigraphic Unit” (“US” from now on). Every US is connected to the others by geometric, topological and, eventually, temporal links, and are recorded by the 3D photogrammetric survey. However, the limitations of the Harris Matrix approach lead to use another representation formalism for stratigraphic relationships, namely Qualitative Constraints Networks (QCN) successfully used in the domain of knowledge representation and reasoning in artificial intelligence for representing temporal relations.


INTRODUCTION
The paper presents a first reflexion based on sixteen years of interdisciplinary cooperation between Medieval Archaeologists from University of Florence (Italy) and ICT researchers from CNRS LSIS of Marseille (France), aiming towards a connection between 3D spatial representation and archaeological knowledge for interpreting the Mediterranean Middle Ages (Drap et al., 2005).In the last decade, we witnessed the significant improvements of photogrammetric technics, starting with SIFT descriptor in 2004 (Lowe, 2004), that allows for automatic matching of thousands of homologous points.This opens the way to more automatization of the photogrammetric process, autocalibration and automatic 3D dense cloud generation.These new photogrammetric tools, which now can easily replace the terrestrial laser scanner still used at the beginning of this new century in archaeology, bring with them the same lack as with laser scanner: no semantic data is linked with this accurate and plethoric geometry.At the same time the stratigraphic analysis applied to upstanding building in the framework of medieval archaeology is from one hand absolutely mandatory and on the other hand seems to have reached a crucial point of its evolution.Italian Medieval Archaeology, in parallel with a close relationship with historical research, contributed to the development of archaeological methodologies at European and world level.A specific and particularly outstanding contribution was provided in the field of * Corresponding author non-destructive urban and territorial analyses.Such was indeed the (successful) attempt to embed the stratigraphic theory in the study of historical buildings, extending the principles of siteformation-process to the architectural-formation-process.Based on the outcome of a pioneer study carried out in '970s by Tiziano Mannoni, and on a further methodological definition carried out in the Eighties , it was finally possible to establish a standard set of techniques for the stratigraphic investigation of medieval (and historic) buildings.An achievement of particular importance since, for the first time, a non-destructive archaeology could build reliable stratigraphic sequences at territorial scale.Since some decades, the Harris Matrix approach (Harris, 1979) has been extensively used for stratigraphic analysis.The translation of the Harris paradigm in upstanding structure (Brogiolo, 1988), i.e. in full 3D context, brings some inconsistencies which become more and more visible with the development of new 3D surveying tools.Many scholars argue on the possibility to increase the Harris Matrix with specific fields suitable for better defining and analysing upstanding buildings stratigraphy (in Italy, where the upstanding building stratigraphy is very used, especially among medieval archaeological researches, see the status question is in Gallina (Gallina, 2012)).In the same time, the recent advance on Knowledge Representation, as the development of Ontologies also in Cultural Heritage, changes the way to manage knowledge and extends the possibilities to connect with other research fields.The presented work aims at linking photogrammetric survey and stratigraphy in order to propose a new representation of temporal relations stemming from temporal qualitative networks thanks to the Allen's approach .This is done by developing a common framework describing knowledge used in photogrammetry as well as in stratigraphy fully based on ontologies.An Ontology describing the photogrammetric process and the measured artefacts (ashlar bloc, observed relevant surface and then Stratigraphic Unit and connected concepts) is aligned on the wellknown ontology used now since a long time in Cultural Heritage: CIDOC CRM (Doerr, 2001).First of all, we try to obtain the following results (like suggested by (Gallina, 2012), p. 80): 1. Maximum readability and immediacy of the drawings and schedules, avoiding, as much as possible, obscure symbolism and hypertrophy of the documentation 2. Simultaneous presence in the same media of all kinds of information (text, drawing, pictures), in order to really simplify the information sharing.It is evident that the Harris Matrix itself is not too much self-evident and insignificant, and that the US cards without photographs and drawings are equally unclear.This rigid separation of the information must be tackled 3. The formalism used to store and represent this complex and heterogeneous knowledge cannot be rigid and too much structured because archaeological data and the knowledge itself are changing over time (Drap et al., 2007).
The significant development done in CIDOC CRM (Niccolucci et al., 2015) (and now with CIDOC-CRMBa, an extention in the field of building archaeology (Ronzino, 2015))is very useful also to explore some theoretical concepts underlying the construction of the Harris Matrix.We have developped an ontology to represent both the photogrammetric process and the measured objects, here ashlar blocs and stratigraphic unit.The objects here are modelled according to the point of view of the measurement process but indeed these artefacts or concepts as stratigraphic unit can be seen also under the point of view of Cultural Heritage or conservation.This is the reason why we have aligned our ontology on CIDOC CRM.
The case study of this work is the Castle of Shawbak in Jordan where the University of Florence is working since almost thirty years and LSIS since sixteen years (Vannini et al., 2009).We are thus working on a huge quantity of data acquired over time according to the evolution of the technology and the team evaluation.We present in this paper both photogrammetric survey and their link with archaeological data through a common formalism based both on pure XML and on ontology.

TIME IN ARCHAEOLOGY
The topic of time in archaeology is most important and very complex, though it has been too often neglected.Regarding the relative chronology, Harris, during the second half of last century, has introduced a very innovative system.In 1979 he published the first textbook devoted to the science of archaeological stratigraphy, where he defined the concepts of units of stratification (US, both deposits and interfaces) and the main important stratigraphical rules.The physical relationships between deposits and interfaces aim the archaeologists to deduce the stratigraphical relationship, thats means, in other words, the relative chronology of the US.To allow archaeologists to organize all the stratigraphical relationships between the different US, Harris designed also a diagram, the Matrix.The Matrix changed the paradigm of the archaeology from the one-dimensional concept embodied in the Wheelerian section drawing to a four-dimensional model that combines the three physical dimensions with that of time, the forth dimension.In this sense, it is like the clock face of twelve hours and the Gregorian calendar of a twelve-month year, which are diagrammatic ways in which time, which does not exist in any material form, can be seen.More than any other science, archaeology is a time-related discipline and the Harris Matrix has given that emphasis a revolutionary force in its ability to translate the physical evidence of stratification into calendar of relative time, unique to each site, but universally comparable through the Matrix diagrams.(Harris, 2003) p. 10.This important and not yet out-dated tool suffers the lack of the possibility to express the concept of duration.In fact a floor, for example, may have been laid before a hearth or oven was constructed on top of it, but the floor itself as an active, used surface, may endure as long as, if not longer than, the hearth.The matrix only shows the temporality of production, not use.
And this has been a major criticism of the matrix that it does not refer to how long a deposit took to form or how long a structure was in use (Lucas, 2001), p. 161-2.This criticism is more yet relevant when we consider not only a single feature, like the example of a floor, but the a more complex situation like the following example: The house where I am writing this paper was built towards the beginning of this century, in the courtyard of an ancient farm whose structure is still visible.From my open window, I see an interweaving of houses and constructions, most of them dating back to the 19th century, sometimes including parts of earlier constructions from the 18th or 17th century.The 20th century here looks so localized, so secondary: it is reduced to details, such as windows, doors or, within houses and flats, furniture.Right now, the present here is made up of a series of past durations that makes the present multi-temporal.
But, despite the lack of the concept of duration, the Harris Matrix is until now an useful tool to analize archaeological stratigraphy.
And since representing space equals, in archaeology, to representing time, after 2009 (Drap et al., 2009) (Pruno and Drap, 2012) our team started to work on the modeling of Archaeological Time, in order to embed this aspect within the system and to eliminate the separation of duration from sequence in the temporality of a site, especially trying to include developing a tool for the automated elaboration of Harris Matrixes as well as the integration of non linear Time Models (Althusser, 1969) (Nicolis and Prigogine, 1977) (Allen, 1983) (Lucas, 2005) into the chronological structure of the Harris Matrix itself.
Chronology is indeed too limited a model for representing the archaeological time as it is for Time itself and its use for interpreting the past may heavily bias our full understanding of any archaeological history.

PHOTOGRAMMETRY AND TIME
Specific photogrammetry tools dedicated to stone by stone measurement have been under development since 2002 to help archaeologists to easily produce photogrammetric surveys.These tools are now integrated in a more complex system which allows automatic production of 2D or 3D representations from archaeological database queries.The graphic 2D documents produced through this process look like the handmade drawings done by archaeologists using ortho-rectified photos.
The 3D GIS, merging photogrammetry and ontologies, is the last step of this chain and aims at the automatic production of 3D models through archaeological database queries: These 3D models are in fact at the same time a graphic image of the database and the actual interface through which the user can edit the dataset.
This approach enables automatic 3D thematic representation and new archaeological analysis, through bidirectional links between 3D representation and archaeological data We propose now to work on the link between 3D measurement and temporal relations.Indeed, archaeologists use a set of rules to determine temporal relationships by observing physical relationships.The table below describes the physical conditions necessary for establishing the temporal relationships.The temporal relations are deduced from the physical relations, and, often these physical relations can be deduced from the relative position of the US between them using the photogrammetric measurement of their components.As the US are built on measured objects it will be possible to compute the veracity of the physical relations on the one hand to deduce the temporal relations if they have not been informed by the archaeologist or to ensure a coherence of Time relations .

Physical relations Stratigraphical relations Gli si appoggia (ad A si appoggia B)
A Anteriore B Coperto da (A coperto da B) A Anteriore B Tagliato da (A tagliato da B) A Anteriore B Riempito da (A riempito da B) A Anteriore B Appoggiato a (A appoggiato a B) Table 1.Links between physical relations and stratigraphy

Ontology
This work is based on a close link between, on the one hand, the software engineering aspects and the operative modeling of the photogrammetry process, the needed computation and artefacts measurable by photogrammetry in the context of this project and, on the other hand, with the ontological representation of the same photogrammetry process and surveyed artefacts.The present implementation is based on a double formalism, JAVA, used for computation, photogrammetric algorithms, 3D visualization of photogrammetric models and Cultural Heritage objects, then OWL for the definition of ontologies describing the concepts involved in this photogrammetric process as well as the surveyed artefacts.The same point of view presides over the elaboration of the JAVA taxonomy, but software engineering constraints involve differences in the two hierarchies of concepts.For each ontology concept a procedural attachment method has been developed with OWLAPI.As each concept present in the ontology has an homologous class in the JAVA tree each individual of the ontology can produce a JAVA instance and can take benefit from its computational capabilities.In the same way, each JAVA class has a counterpart in the ontology and can produce an individual of that ontology.
Thus, reading an XMl file used to serialize a JAVA instance set representing a statement can immediately (upon reading) populate an ontology; similarly reading an OWL file can generate a set of JAVA instance counterparts of the individuals present in the ontology.The link between individuals and instances persists and it can be used dynamically.The huge advantage of this approach is that it is possible to perform logical queries on both the ontology and the JAVA representation.We can thus read an ontology, visualize in 3D the artefacts present in the ontology and graphically visualize the result of SQWRL queries in the JAVA viewer.
Here the photogrammetric survey is expressed as an ABox: An ontology describing the photogrammetric process as well as the measured objects was populated by the measurements of each block and a set of corresponding data (USM owner, etc.).
It is therefore this ontology which contains the metric information as geo-positioning of each block and all the physical and stratigraphic relationships provided by archaeologists.
The reading and the geometrical interpretation of this ontology can lead us to validate or invalidate the physical relations by a topological analysis of the relations between the various components of the USM but also to check the temporal relations consistensy by extracting a graph of Constraints and then using a logical solver.
Moreover, a first possible interpretation of these data should be the generation of a graph of temporal relations where the nodes, instead of having a coded position as in the Harris graphs, should have as the coordinate of the USM center of mass projected in USM main plane.
As illustrated in Figure 3, the graph is superimposed on an unrealistic representation of the stone-to-stone reading of the building.The ontology developed in the framework of this project takes into account the manufactured items surveyed, as well as the method used to measure them; in this case, photogrammetry.The surveyed item is therefore represented from the measurement point of view and has access to all the photogrammetrical data that contributed to its measurement in space.Two ontologies are aligned in this context; one dedicated to photogrammetrical measurement and the geo-localisation of the measured items, whereas the other is dedicated to the measured items, principally the archaeological artefacts, describing their dimensional properties, ratios between main dimensions, and default values.These ontologies are developed closely linked to the Java class data structure that manages the photogrammetric process as well as the measured items.Each concept or relationship in the ontology has a counterpart in Java (the opposite is not necessarily true).Moreover, surveyed items are also archaeological items studied and possibly managed by archaeologists or conservators in a museum.It is therefore important to be able to connect the knowledge acquired when measuring the item with the ontology designed to manage the associated archaeological knowledge.CIDOC CRM is a generic ontology that does not support the items that it represents from a photogrammetric point of view, a simple mapping would not be sufficient and an extension with new concepts and new relationships would be necessary.
This modelling work is based on a previous study that started from the premise that collections of measured items are marred by a lack of precision concerning their measurement, assumptions about their reconstruction, their age, and origin.It was therefore important to ensure the coherence of the measured items and potentially propose a possible revision.This previous work was done in the context of underwater archaeology with similar problems, for more information, see (Cure et al., 2010) The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W3, 2017 3D Virtual Reconstruction and Visualization of Complex Architectures, 1-3 March 2017, Nafplio, Greece This contribution has been peer-reviewed.doi:10.5194/isprs-archives-XLII-2-W3-267-2017(Hue et al., 2011) (Papini and Drap, 2009) (Seinturier, 2007) (Serayet et al., 2011).
The extension of the CIDOC-CRM ontology is structured around the concept E22 Man-Made Object.The root of ItemMesurable developed in this project extends this concept.The mapping operation is done in Java by interpreting a set of data held by the Java classes as a current identification of the object: 3D bounding box, specific dimension.These attributes are then computed in order to express the right CRM properties.
Several methodologies can be chosen regarding mapping two ontologies.For example, Nicola Amico and his team (Amico et al., n.d.) choose to model the survey location with an activity (E7) in CRM.They also developed a formalism for the digital survey tool mapping the digital camera definition with (D7 Digital Machine Event).We see here that the mapping problem is close to an alignment problem which is really problematic in this case.Aligning two ontologies dealing with digital camera definition is not obvious; a simple observation of the lack of interoperability between photogrammetric software shows the wildness of the problem.We are currently working on an alignment/extension process with Sensor ML which is an ontology dedicated to sensors.Although some work have already been achieved (Hiebel et al., 2010), (Xueming et al., 2010), but not enough to clearly hold the close range photogrammetry process, from image measurement to artefact representation.
In addition we use the concepts of time representation present in CIDOC CRM in order to represent the Harris relations and then Allen's formalism about the time intervals.

Positive and negative US
Since representing space equals, in archaeology (and not only), to representing time, after 2009 (Pruno and Drap, 2012) (Drap et al., 2009) we started to work on the modeling of Archaeological time, in order to embed this aspect within the system.
By analyzing and breaking up an archaeological deposit into basic, discrete stratigraphic units, archaeologists are able to create (relative) chronologies of all activities performed (by man or by nature) on a site over time, in order to describe and interprehem.
Following the definition by (Harris, 1979) and further developments by (Harris, 1979) (Mannoni, 1976) and (Brogiolo, 1988), stratigraphic units, or contexts, are those material traces of past events which have been preserved in the archaeological record.They can be positive (i.e. Layers), negative (i. e. Cuts) or neutral (i.e. Interfaces).By analyzing and breaking up an archaeological deposit into basic, discrete stratigraphic units, archaeologists are able to create (relative) chronologies of all activities performed (by man or by nature) on a site over time, in order to describe and interpret it.Stratigraphic units are eventually placed into a Harris Matrix, a conceptual scheme of stratigraphic relationships which is the backbone of historical interpretation of almost every archaeological site excavated after 1980s.
As regards to that, until now, we have been working mostly on the survey and representation of positive USM, although since 2009-10 the research for modeling negative and interface units was also started (see figure 5).
The latter ones are in fact 3D surfaces (not volumes) possessing stratigraphic relationships with other units.They are not formed by ashlars and mortar, since they are only signs of, say, a destruction, like with the traces left by an earthquake or by human (partial to total) destruction of a material feature.Picture 5 shows a negative USM represented by a cloud of 3D points extracted using the perimeters drawn on the photographs by an archaeologist.
Figure 5. Negative USM represented by the set of 3D points extracted using the perimeters designed on the photographs by archaeologist.

Toward duration for representing Stratigraphic Unit
Once solved the problem of surveying and representing Stratigraphic Unit, through ontology and then graphically, we have to face some limits of Harris paradigm.As we saw before, one of the main improvement we are working on is considering US as interval of time and no more as instants.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W3, 2017 3D Virtual Reconstruction and Visualization of Complex Architectures, 1-3 March 2017, Nafplio, Greece This contribution has been peer-reviewed.doi:10.5194/isprs-archives-XLII-2-W3-267-2017Indeed infering all the Allen's relations directely from the physical relations between US is not trivial and currently the only way to do that is using the expertise of archaeologist.
Nevertheless we start with a few deduced relationships from Harris formalism to Allen's one.Considering a Stratigraphic Unit A it should have a start point named Amin and an end point named Amax.The constraint C12 = {precedes} states that v1 precedes v2, the constraint C23 = {same} states that v2 is the same than v3, the constraint C26 = {precedes} states that v2 precedes v6, the constraint C34 = {precedes} states that v3 precedes v4, the constraint C45 = {precedes} states that v4 precedes v5.
The QCN is represented in Figure 8. Intervals algebra However, the stratigraphic relations between U S often involve the notion of duration which cannot be represented with the Harris Matrix approach.In this case intervals are more suitable temporal entities.From stratigraphic relations between US the archaeologists can assign intervals to US and we can consider relations between intervals from the Allen's interval algebra (Allen, 1983).This algebra stems from 13 atomic temporal relations B = {p, m, o, s, d, f, pi, mi, oi, si, di, f i, eq}.
More formally the domain D is defined by the set of intervals of the rationals line   Archaeologists should be equipped with suitable tools allowing them to complete information on the bounds in the best possible way.
Moreover an important issue is the QCN consistency checking problem.Indeed, a QCN may be inconsistent for several reasons.Errors can be made on the interpretation of stratigraphic relations, discontinuities involved by the 3D complexity of studied structures may exist, the survey may be incomplete (for instance lack of photographies, • • •).Efficient tools have to be developed in order to check the consistency and to pinpoint the inconsistencies in each step of the process from the formal temporal representation to the photogrammetric 3D survey.
Concerning the QCN, three main approaches have been proposed for consistency checking in the artificial intelligence community, stemming from discrete CSP (Dechter et al., 1991), from SAT (Condotta et al., 2016), and more recently from Answer Set Programming (ASP) (Brenton et al., 2016).

CONCLUSION AND FUTURE WORK
Beyond the scientific problems, an important obstacle addressed by this work is that a heavy change of paradigm has a lot of implication on the on-going work.Changing paradigm means also changing vocabulary, managing new concepts, changing wellknown tools for unknown and not completely debugged tools.
In a future work we plan to investigate the ASP approach for several reasons.ASP is a unified formalism for both representing and implementing the consistency checking problem expressed by a logic program which is very close to natural language.We already used this formalism when dealing with the revision of geographical information within the framework of the REVIGIS project3 (Benferhat et al., 2010).More precisely we compared the CSP, SAT and ASP approaches.CSP focuses on the direct resolution of inconsistencies by means of propagation mechanisms while SAT and ASP concentrate on the identification of inconsistencies.However an experimental study on a real world application benchmark showed that ASP gave better results than SAT (Benferhat et al., 2005).Moreover, ASP can easily interface with ontologies and Java.

Figure 1 .
Figure 1.High resolution photogrammetric survey used to extract relevant archaeological artefact.

Figure 2 .
Figure 2. Stratigraphic units identified by their ashlar blocs when they are positive.

Figure 3 .
Figure 3. Positive stratigraphic relations with the USM node coordinate comming from photogrammetric measurement.
For example, the Harris relation A is Contemporaneo to B can be translated by A starts B in Allen's relations and the Harris relation A Anteriorita B can be translated by A precedes B or A meets B. using the bound of intervals as expressed in Table 2Harris relations seen from an interval point of view.In order to represent the Stratigraphic Unit duration, using time intervals we use Qualitative Constraints Networks (QCN) successfully used in the domain of knowledge representation and reasoning in artificial intelligence for representing temporal relations.The Allen's intervals algebra has been used for representing and reasoning with archaeological information in the context of archaeological documentation (Accary-Barbier and Calabretto, 2008), dating process (Belussi and Migliorini, 2014), however as far as we know this forrmalism has not been used yet for stratigraphy in archaeology.More formally Definition 1 Let B be a set of basic relations, a QCN N is a tuple (V, C) where V = {v1, • • • , vn} is a set of n variables representing temporal entities, C is a mapping that assigns to each tuple (vi 1 , • • • , vi j ) of variables of V , a relation C(vi 1 , • • • , vi j ) ∈ 2 B .The stratigraphic relations between U S are temporal binary relations and several QCN could be proposed according the choice of the temporal entities and the set of basic relations.Time points algebra Considering temporal entities as time points, archaeologists can provide temporal relations between the centers of mass of the US, deduced from stratigraphic relations between US.More formally, the domain D is defined by the set of rational numbers, (line points) D = Q equipped with the linear order < and the time points algebra stems from 3 atomic relations B = {precedes, f ollows, same}.These relations are defined as follows: precedes = {(x, y) ∈ Q × Q : x < y}, f ollows = {(x, y) ∈ Q × Q : y < x}, same = {(x, y) ∈ Q × Q : x = y}.

Figure 6 .
Figure 6.Time points algebra atomic relations The set of time points algebra relations is denoted by 2 B , where each relation is a disjunction of atomic relations.The set 2 B

Figure 7 .
Figure 7. Example of relations between US .

Figure 8 .
Figure 8. QCN when temporal entities are time points.
Each atomic relation is defined by constraints on the bounds of intervals.For example, the atomic relation s (for starts) is defined by s = {(x, y) ∈ D × D : x − = y − and y + > x + }, the atomic relation m (for meets) is defined by m = {(x, y) ∈ D × D : x + = y − }.

Figure 9 .
Figure 9. Allen's intervals algebra atomic relations The set of Allen's intervals algebra relations is denoted by 2 B , where each relation is a disjunction of basic relations.The set 2 B is equipped with the operations: Union (∪), intersection (∩), composition(•), inverse( − ).The following example illustrates the notion of QCN when the temporal entities are intervals.