The Integration of Semantic Networks in Virtual Exhibitions

: In this article a use case is presented how a semantic network can be used to enrich the existing virtual exhibition “They Shared their Destiny. Women and the Cossacks’ Tragedy in Lienz 1945” about the fate of women during the Cossack tragedy in Lienz. By connecting via CIDOC CRM information about people, events, finds and places the goal was not only to make this information interoperable, but also to integrate the resulting knowledge graph into the exhibition, thus providing a further navigation level and enhancing the visitors’ experience. First, a short introduction to the existing exhibition and the presented project is given. In the second part, the scientific background of CIDOC CRM and its semantically enriched 3D content is outlined. In the third part the implementation and the project as a use case is described with respect to the data modelling and the integration of the semantic network into the 3-dimensional environment as well as the integration of spatial aspects and other internet resources. At the end, there is a summary with an outlook on future planned projects.


INTRODUCTION
After an eventful history, in June 1945 in Lienz about 22,500 Cossacks (including about 3,500 women and children) were handed over to the Soviet Union by the British Administration, contrary to previous agreements. About 4,100 managed to hide in the mountains and forests, others decided to commit suicidesome with their familiesto escape extradition, still others handed over their children to local families beforehand. The events became known as "the tragedy on the river Drau" (Stadler et al. 2005). In occasion of the 75th anniversary of this event, an exhibition was planned to shed light on the fates of the Cossack women, who until now have mostly remained hidden behind the curtains of history. Due to the restrictions during the Covid-19 crisis, the exhibition had to be re-planned as a virtual exhibition on short notice.

Figure 1. Virtual Exhibition
In a nutshell, Blender (Blender Foundation 2019) was used to reconstruct the exhibition room, including the posters on display and the artifacts exhibited in the showcases. From this room, nine scenes from different viewpoints were exported as 360° images. These were assembled into a tour with Marzipano (2021) and enriched with further information on posters and objects about people, places, events etc. Visitors of the virtual exhibition are thus * Corresponding author able to see the room at predefined viewpoints in 360°, to enlarge the posters, to look at the finds and to obtain further information on some aspects through external links. This virtual exhibition can now be visited in the desired language (English, German, Italian, Russian) at https://www.kosaken-lienz1945.com/virtuelle-ausstellung--63364468-de.html ( Figure 1) (for detailed description for technical implementation see Danthine et al. in review). The small follow-up project "Virtual Exhibition in All Dimensions", which is based on the data from this exhibition, has three main goals: a) to collect more information about people, artifacts, places and events, and link them into a semantic network; b) to integrate this network into the 3-dimensional environment and finally c) to increase the amount of information provided, the story-telling possibilities and the navigation options for visitors.

SCIENTIFIC BACKGROUND
When talking about scientific information processing, semantic datasets or generally data from cultural heritage institutions, the FAIR principles and the London Charter are always mentioned as main guiding foundations. Why this is the case will be explained in more detail in the following two sections. However, first the main points of the two guidelines important for this project will be addressed: Wilkinson et al. (2016) recommended in their article that digital assets should be Findable, Accessible, Interoperable and Reusable. The principles are primarily targeting the machine accessibility of data, "i.e., the capacity of computational systems to find, access, interoperate, and reuse data with none or minimal human intervention" (GO FAIR 2021). In the case of the London Charter, it is mainly point 5, the sustainability, and sub-point 5.2. that concerns the project: "5: Strategies should be planned and implemented to ensure the long-term sustainability of cultural heritage-related computer-based visualisation outcomes and documentation, in order to avoid loss of this growing part of human intellectual, social, economic and cultural heritage." "5.2: Digital preservation strategies should aim to preserve the computerbased visualisation data, rather than the medium on which they were originally stored, and also information sufficient to enable their use in the future, for example through migration to different formats or software emulation" (londoncharter.org 2009, p. 10).

Semantic Background
"They [the artifacts in Chinese museums] represent an extremely valuable cultural heritage resource, and yet access and exploitation of the data is constrained due to the distributed and heterogeneous nature of the resource. Therefore [sic] it is highly desirable to find an avenue to integrating these distributed and heterogeneous systems." (Bao et al. 2005, p. 626) A problem faced by all museums and the entire cultural heritage field, as well as research regarding the Cossacks in particular, is their wide (geographically) dispersal: not only the artifacts are scattered accross several countries and institutions, but also the histories of the different persons often extends to several countries or even continents; thus, there is a great variability of sources, perspectives and interpretations in regard to exhibits, persons, places and events. Furthermore, the data and their structure are not only different from institution to institution, but they are for the most part not scientifically studied or evaluated, as well as often buried in archives and collections in the most diverse places and consisting of the most various source genres. Precisely because of these differences between the data of the various organizations the International Council of Museums (ICOM) founded the "International Committee for Documentation/Comité International pour la Documentation" (CIDOC), which has been developing a Concept Reference Model (CRM) (Bekiari et al. 2021)  Their main features are listed below: CRMinf: handle scientific argumentation. CRMsci: model specific scientific observations. CRMgeo: manage spatiotemporal information. CRMdig: model the digital provenance of the metadata. CRMba: encode metadata concerning archaeological buildings.
CRMarchaeo: represent archaeological excavations. This extension was developed together with CRMsci in the course of ARI-ADNE, the Advanced Research Infrastructure for Archaeological Data Networking in Europe, which uses CIDOC CRM as the conceptual background. In this sense, CIDOC CRM has gained acceptance not only in the field of museum studies, but also in the entire cultural heritage sector (see also Moraitou et al. 2019), as highlighted by Bruseker et al. (2015, p. 33): "The CRM responds well to the need to reach the greatest possible audience in a sustainable way. The primary audience to be addressed are cultural heritage professionals considered in the broadest sense: from archaeologists to exhibition designers to building restoration technicians […]. It has wide adoption and recognition as both a conceptual and information sharing tool and increasingly as the basis for the production of information systems implementations." The preparation and mapping of all data of the project according to the CIDOC CRM fulfill the "Interoperability" of the FAIR principles, which demands for example that all (meta)data should "use a formal, accessible, shared, and broadly applicable language for knowledge representation" (GO FAIR, 2021). For the creation of FAIR data in archaeology, see, e.g., Hiebel et al. (2020) or Hiebel et al. (2021). However, the project should go one step further and not only provide the information processed in this way separately but integrate them directly into the exhibition and thus visualize them directly in the virtual museum, making them available to visitors. As shown in the next section, the problems of linking semantically enriched data (mostly structured in accordance with CIDOC CRM) with 3D models and their subsequent scientific use have been discussed in various articles for a while.

Semantically enriched 3D content
Virtual museums do not have to be necessarily presented as 3dimensional spaces (a good overview about various definitions, types and terminologies of or about virtual museums is provided in Schweibenz 2019). Many major museums, such as the British Museum (https://www.britishmuseum.org/collection), now make their collections fully or partially available to the general public through searchable databases. In these cases, too, the linked information is usually prepared according to the CIDOC CRM ontology. However, such virtual collections do not include storytelling and correspond to the "content-centric variants", defined in Geser and Niccolucci (2013). Such versions of virtual object presentations are opposed to the "communication-centric versions" (Geser and Niccolucci 2013). Virtual museums falling into this category can also be collections of individual images, videos or 3D models with their respective information. The linking of such a collection with the Semantic Web has already been described by Ghiselli et al. (2005). While the different variations of virtual museums are of course not mutually exclusive (see, for example, the compilation of virtual ways to visit the British Museum at: https://blog.britishmuseum.org/how-to-explore-the-british-museum-from-home/), they all have one thing in common: they are not limited to their own collection. Instead, they could also include digital objects and information from other institutions (Schweibenz 2019, p. 14). This is especially possible if all edit and provide their data based on a common data model like CIDOC CRM. Another work that aims to make semantic information useful for (virtual) museums is for instance the one presented in Kaczmarek (2008). Here, the author wants to simplify the contextualization of different objects in virtual space and among each other by semantic annotations (e.g. that hanging lamps cannot be placed on tables). However, Kaczmarek does not use CIDOC CRM for this purpose and the target of his research are the museum employees rather than the visitors. However, most of the works regarding the semantic enrichment of 3D models and objects aimed at developing systems for integrative work and annotation of 3D models, and were not developed specifically for virtual museums. Because "the use of digital 3D representation would introduce significant improvements, as it would allow for the inclusion of a comprehensive and holistic perspective, which is inherent to any architecture and sculpted monument" (Apollonio et al. 2018, p. 90), Apollonio's objective was to develop a 3D centered information system for the restoration of Neptune's Fountain in Bologna. Here, all the people involved can note and document all the work undertaken directly on the 3D model or on parts of it (Apollonio et al. 2018). A similar approach concerning the semantic annotation of entire or parts of 3D models can be found in a paper by Bruseker about a case study "of a reconstruction of a kitchen in a Roman domus in Grand" (Bruseker et al. 2015, p. 33) or by Guillem about "entering virtual reconstruction paradata, metadata and data in order to allow the tracing of knowledge provenance in reconstruction […] of the Lioness Lintel (Hephaisteion, Athenes)" (Guillem et al. 2015, p. 383). Soler et al. present the software Agata which "allows specialists to interact in real time with high resolution polygonal models, and to annotate different raster and vectorial information directly onto them" (Soler et al. 2017, p. 49). Yu and Hunter in contrast adapt the X3D (Extensible 3D Graphics) standard as an extension to the Open Annotation (OA) data model "to enable the semantic annotation of points, surface regions and volumetric segments on 3D cultural heritage artefacts." (Yu and Hunter 2014, p. 580). Auer et al. also present with the MayaArch3D a "web-based research platform bringing spatial and non-spatial databases together and providing visualization and analysis tools" (Auer et al. 2014, p. 33) with special attention to the integration of 3D models. "An infrastructure to systematically enrich 3D shapes in a collection by using propagated annotations" by applying CIDOC CRM is described in Rodriguez-Echavarria et al. (2012, p. 41). Serna et al. (2012, p. 33 andSerna et al. 2011) write about an "interactive semantic enrichment tool for 3D CH collections that is fully based on the CIDOC-CRM schema and that fully supports its sophisticated annotation model". Also Kuroczyński and Hauck are using CIDOC CRM for their Cultural Heritage Markup Language (CHML), which "describes CH objects from the 3D reconstruction point of view" (Hauck and Kuroczyński 2015, p. 250). Here, it is used for the "design of a custom webbased virtual research environment based on the WissKI project [...] for archaeology, art and architecture studies with a related interactive virtual museum" (Kuroczyński et al. 2015, p. 54). An overview of these and further works in this field is given by Moraitou et al. (2019) and Apollonio et al. (2018). However, all the projects presented here and elsewhere have one thing in common: " University to a smart museum with the help of semantics and the "small space approach" for the Internet of things: "The aim is to transform a museum to a collaborative work environment where cultural heritage knowledge becomes usable and creatable by visitors and professionals themselves. […] The IoT technology enables cultural objects to interact with people, environments, other objects, and transmitting the related information to users through multimedia facilities" (Korzun et al. 2017, pp. 346-347). Also, the goal of Mata et al. is to give visitors more options both in a physical museum "based on a semantic model of a museum environment that reflects its organization and spatial structure" and by Augmented Reality (Mata et al. 2011, p. 497). Hunter and Gerber (2010) are looking to provide visitors with more interaction by annotating 3D models and to offer museum employees a better system to use the socially tagged objects through the HarvANA system (Harvesting and Aggregating Networked Annotations).

Data modelling
The virtual exhibition and the historic reality are both modelled with CIDOC CRM classes and properties.  (Bekiari et al. 2021), which as mentioned is specifically focused on the cultural (heritage) sector, is an event-centric ontology where Physical Things (E18), Places (E53), Actors (E39) and Conceptual Objects (E28) are related through a Temporal Entity (E2) (Figure 3). The components of the virtual exhibition (posters, photos, 3D representations of showcases with objects or objects by themselves) are modelled as E73 Information objects that relate with P67 refers to (or sub-properties P70 documents, P138 represents, …) to the instances of the historical reality that the virtual exhibition tries to present to the public. One of the best ways to convey a historical reality is to tell a story. The event-centric model of the CIDOC CRM ontology offers a great potential to do exactly this. Events in the lives of specific people who were part of the Cossack tragedy are modelled as E7 Activities and are related in a temporal sequence through the P134 continued property. These activities can be further specified through CRM classes (E9 Move, E65 Creation, E67 Birth, E69 Death) or SKOS (Simple Knowledge Organization System) concepts (like War, Residence, Marriage, Execution, Migration, Deportation, Repatriation, Detention). They took place at specific locations or in the case of an E9 Move started at one location (P27 moved from) and ended (P26 moved to) at another E53 Place (Figure 4). In a GIS (geographic information system), the sequence of activities with the included moves can be visualized in their temporal order, thus bringing personal histories on a map. The tabular information about the different entities was mapped according to the formal definition of the CIDOC CRM with Karma (Center on Knowledge Graphs 2021), a tool from the Semantic Web community. The resulting knowledge graph, which represents the information of the structured data with the concepts of the CIDOC CRM, is implemented in RDF (W3C 2021), a data format able to relate logical statements within a network: "In general, a semantic network is a directed graph consisting of nodes, which represent domain objects, and links, which represent semantic relations between them. In the museum case, nodes represent historical objects (exhibits, associated persons, etc.) and links represent relations between such objects." (Petrina et al. 2017, pp. 677-678) RDF is "an XML based language that deal [sic] with semantic interoperability and that it is [sic] made by three components univocally identified by an URI" (Lo Turco et al. 2019, p. 434). The three components are 1) an object, which is saying something about another thing -2) the subjectin regard to something by using 3) a predicate. The advantage of expressing information in the W3C standard RDF is that it can be machine-processed and enable Linked Data based discovery and access to content" (Geser and Niccolucci 2013, p. 24).

Integration of a semantic network into a 3D environment
As explained above, the project intends to connect its semantic knowledge graph with the posters and objects of the virtual exhibition in order to visualize connections of the people, events and places mentioned in, or connected to them. The same goal is one of the contents of the work of Petrina et al. (2017), but with the inclusion of visitors' annotations. In doing so, she addresses an important point: "When considering relations between cultural objects, then the data become linked, and the number of links can grow in the quadratic law" (Petrina et al. 2017, p. 677).
Since the reconstruction of the virtual environment was done in Blender (for a detailed description of the workflow see Danthine et al., in review), our preferred option was to convert the knowledge graph into a 3D model consisting of different linked entities or nodes. Another possibility would have been to define the corresponding Sparql query for, e.g., individual keywords on the posters, which then redirects the visitor to the representation of the semantic graph ( Figure 5) with GraphDB (Ontotext 2021). The latter would offer the advantage that the semantic network always presents the latest state, since the representation is always rebuilt based on the RDF triple store behind it. However, this option has the major disadvantage that the navigation in the semantic network takes place outside the 3-dimensional environment and the visitor must be redirected according to his "path". In this way, the connections between individual posters and finds within the exhibition cannot be represented on the level intended by the project. The conversion is not simple, since it must be done automatically due to the size of the network. Though, to our knowledge there is almost no software solution in this direction, especially since it should ideally also be possible to control the appearance of entire groups of entities, so that this does not have to be done manually for the entire network either. However, the Knowledgebase Builder (Inforapid 2021) offers a solution. It works in the background with triples. While it is possible to query Wikidata directly as Sparql, queries of other Sparql endpoints are not possible. The data from the Triplestore must be prepared first in an appropriate way, so that everything is correctly displayed in the Knowledgebase Builder, and so that it is possible to control their appearance (e.g. the colors) by assigning the data to different categories. Then again, this software offers the great advantage that both 2-dimensional and 3-dimensional representations of the network are possible and can be generated automatically as well as exported as an image or directly as a 3D model. The 3D model of the graph exported by Knowledgebase Builder as .glb can be imported directly into Blender (.glb is the binary file format of the GL Transmission Format .glTF from Khronos; see https://www.khronos.org/gltf/. Within the "JPEG of 3D" format, it is possible to store information about the 3D model like node hierarchy, cameras, materials, animations and the 3D meshes themselves). In Blender, the nodes can then be moved so that they are located in the desired positions ( Figure 6).

Figure 6. Visual integration of semantic network into the 3D model of the exhibition room
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M- 1-202128th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021 In contrast to other solutions mentioned in the previous section, here not just parts of 3D models are annotated but the whole semantic network is imported. Since the converted semantic networks consist of individual interconnected 3D geometries, this means that everything that is exhibited is also mutually connected. The relation of the information objects of the virtual exhibition to the historical instances enables visitors not only to move freely through the virtual exhibition, read the posters and view the objects, but also to navigate individually in the semantic knowledge graph showing the connections between events, people, places and objects.  Figure 7, due to the density of the visualised information network, is that in Peggetz 1945 she meets most of the women featured in the other posters of the exhibition. Without the network, these encounters would only become clear in retrospect after reading all the exhibition posters attentively. Now, through the network, visitors can follow this one specific thread of Olga Protopopov's story. Furthermore, in the example given here, it is possible to decide at the Peggetz node (or any other entity) whether to follow the story of another woman also involved in the Cossack tragedy in 1945 or continue to "walk along" the path of Olga Protopopov's life all the way to her final home in Australia (Protopopov 2000).

Integration of spatial aspects & (other) internet sources
As clearly shown in this example, one aspect has been completely disregarded so far, that is the spatial one. Which persons are where and when? Which events happen at which place and during which time? This information is stored in the semantic network, whose spatial extent cannot be however visualized. Therefore, the goal of the project is to visualize this information too, by connecting the corresponding nodes of the semantic network with a spatial extent to a WebGIS. There is a wide variety of options available for creating a WebGIS, be it the proprietary ArcGIS online (ESRI 2021) or an open-source solution such as QGIS (QGIS 2021) and Leaflet (Leaflet 2020) or Lizmap (3Liz 2021). In both cases, the connection from the network to the WebGIS goes through links in the nodes or vice versa in the attribute table associated with the GIS geometries.
The plugin "SPARQLing Unicorn" (Thiery and Homburg 2021) offers the possibility to query the semantic network directly within QGIS via Sparql and thus to get the geometry points with the corresponding attributes as a normal layer (Figure 8). A WebGIS offers the more direct solution. However, the hosting in this case lies with the administrators. Exporting the queries from QGIS and importing the layer into ArcGIS online is less direct but offers the advantage of having an existing and maintained platform. For the project presented here, which is still ongoing, both options are currently under evaluation. Analogous to the links inside the respective node, web pages such as Wikipedia articles, information from Wikidata or other content freely available via the internet should be made accessible through the respective entity in the integrated knowledge graph.

Further works
In order to publish the virtual exhibition extended in the described way as an interactive experience, various possibilities and solutions are currently being evaluated. These range from the frameworks 3DHOP (3DHOP 2021) or Aton (Fanini 2021) to game engines like Unity (Unity Technologies 2021) or Godot (Linietsky et al. 2021), whereby the latter is considered more promising due to the better possibilities of making the 3D knowledge graph interactive and accordingly more clearly arranged. An integration in WissKI (German acronym for: Scientific Communication Infrastructur) (Germanisches Nationalmuseum 2021) is intended at a later stage, as the system is now under revision to be applied for the presentation of linked data of the University of Innsbruck. WissKI is a Virtual Reseach Environment (VRE) based on Drupal which provides the intersection between triplestore and content management system (CRM). This would allow to extend the number of additional components of the virtual exhibition without limitations.

CONCLUSION
The goal of the project is to explore the possibilities to add semantic information to the already existing virtual exhibition concerning the women involved in the Cossack tragedy in Lienz 1945. Furthermore, aim of this work is to show how a semantic representation in an event-centric ontology could enable visitors of a virtual exhibition to experience the historical reality beyond the exposed artifacts and information. For this purpose, additional information about people, places, events and objects mentioned or displayed in the exhibition was acquired and mapped on the basis of CIDOC CRM ontology. CIDOC CRM is an eventbased ontology that has been and is being developed for museums and cultural heritage institutions and is widely used. While the literature mostly addresses the annotation of parts of 3D models or the possibility of enriching (virtual) museums with semantic data, this project discusses how to integrate a 3D representation of the knowledge graph into the 3-dimensional museum environment and making it navigable. This should enable the visitor not only to move within the virtual environment and visit the exhibition in a linear way, but also to "move" through the semantic network and thus follow the historical storylines they are interested in.