UNDERSTANDING UNDERWATER PHOTOGRAMMETRY FOR MARITIME ARCHAEOLOGY THROUGH IMMERSIVE VIRTUAL REALITY

: Underwater archaeological discoveries bring new challenges to the field, but such sites are more difficult to reach and, due to natural influences, they tend to deteriorate fast. Photogrammetry is one of the most powerful tools used for archaeological fieldwork. Photogrammetric techniques are used to document the state of the site in digital form for later analysis, without the risk of damaging any of the artefacts or the site itself. To achieve best possible results with the gathered data, divers should come prepared with the knowledge of measurements and photo capture methods. Archaeologists use this technology to record discovered arteacts or even the whole archaeological sites. Data gathering underwater brings several problems and limitations, so specific steps should be taken to get the best possible results, and divers should well be prepared before starting work at an underwater site. Using immersive virtual reality, we have developed an educational software to introduce maritime archaeology students to photogrammetry techniques. To test the feasibility of the software, a user study was performed and evaluated by experts. In the software, the user is tasked to put markers on the site, measure distances between them, and then take photos of the site, from which the 3D mesh is generated offline. Initial results show that the system is useful for understanding the basics of underwater photogrammetry.


INTRODUCTION
Before visualizing an object or a site in a virtual environment, it needs to be documented first.Photogrammetry has been a common method of precise recording for over a decade (Addison 2000).It offers photorealistic visualization of an object or site in its current form.However, objects are often damaged (e.g.eroded due to the natural processes such as temperature or water) or destroyed altogether.Photogrammetric techniques require multi-viewpoint photogrammetry, where data from multi-viewpoint photographs are post-processed to generate three-dimensional (3D) information (Green 2009) of an object or a site.In the past, photogrammetry was for an archaeologist an almost impossible task, unless one had dedicated personnel with expertise.Nowadays any archaeologist with a good camera, a computer, and some training can actually do photogrammetry (Green 2009).Moreover, archaeologists typically develop expertise in a class of archaeological material such as pottery or even ships (Bowens 2009).It occurs much less often to develop skills for working in a particular environment, such as under water.The paper tries to address the above issues by developing gamification techniques to teach the basics of photogrammetry to interested parties (i.e.surveyors and archaeology students).
It has been evident through literature that cultural heritage has been employing different technologies such as serious games and gamification techniques (Anderson et al., 2010) and virtual environments (Sylaiou et al., 2009;Papagiannakis et al., 2018).Gamification can be defined as the application of game elements and digital game design techniques to non-game problems (Deterding et al., 2011).Gamification is therefore the deliberate implementation of features of games in contexts that is not normally associated with games.Companies and Universities incorporate gamification to encourage participants/students to feel a certain way, exhibit a certain behaviour and/or perform a certain action, which may not occur otherwise (Liarokapis et al., 2017).Gamification has been applied to a number of application domains including cultural heritage (Papagiannakis et al., 2018).In terms of virtual reality (VR), gamification elements were recently proposed in a kinesthetic experimental application of sculpturing Cycladic figurines (Koutsabasis and Vosinakis 2018).
Using immersive VR, we have developed an educational software to introduce maritime archaeology students to photogrammetry techniques.To test the feasibility of the software, a user study was performed and evaluated by experts.The main task of the application is to allow users to put markers on the site, measure distances between them, and then take photos of the site.Based on the collected photographs a 3D mesh is generated as an offline process.Initial evaluation results indicate that the system is useful for understanding the basics of underwater photogrammetry, especially if no prior experience exists.
The rest of the paper is structured as follows.Section 2 presents background work in VR and photogrammetry.Section 3 presents implementation details as well as the architecture of the system.Section 4 illustrates the methodology and section 5 our results.Finally, section 6 presents conclusions and future work.

Virtual Reality
Origins of VR dates to the 1960s, when Sutherland presented at MIT the first head-mounted display (HMD) system named "The Sword of Damocles" (Sutherland 1965).The system was capable to draw simple wireframes directly to the viewer's eyes through the display he or she was wearing.The term itselfvirtual reality (VR) -was popularized in the late 80s by Jaron Lanier from VPL Research, Inc. (Steuer 1992) and is described as a technology that allows users to visualize a virtual environment.According to the Reality-Virtuality Continuum (Milgram and Kishino, 1994), VR is separated the most from actual reality and all the information the user would see is artificial.
Even though the user is separated from the actual reality while wearing a VR headset, it is possible to interact in several different forms.To make the feeling of 'being there' more evident, different interaction methods have gaining a lot of attention during the past years.The most obvious type of interaction is through dedicated controllers, gestures or voice commands.In terms of collaborative experiences co-located virtual environment exist such as CAVE systems (Cruz-Neira et al, 1993).In the CAVE system, users don't wear any VR headset and they are able to see each other and interact directly.Alternatively, multiple immersive VR headsets can be used, which is a much cheaper solution (compared to the CAVE systems).The user can be located in the same room or anywhere else on the globe and communicate through the internet (Doležal et al, 2017).
Archaeologists use VR to show their results to the public.A characteristic example presented a CAVE system showing a 3D representation of their archaeological findings from Troy and Egypt (Kirchner and Jablonka, 2001).The system consists of a 120-degree film screen and a handcrafted interface that allows users to interact with the 3D environment.Similar system incorporating the CAVE technology in museums was used to capture the atmosphere of the Dunhuang caves.The feeling of playing as an archaeologist was emphasized by adding a simple interaction in form of a flashlight.The user holds a physical flashlight which illuminates the dark virtual scene (Lutz and Weintke, 2001).
Another interesting way of using VR in archaeology is for the analysis of the 3D excavation data.An example of such usage is a system named Archave, which displays all the components of the excavation with recorded artifacts and features in a CAVE environment (van Dam et al, 2000).One of the more recent project Immersia from 2014 uses immersive CAVE system to provide archaeologists with interaction tools to exploit immersive reconstructions.This was achieved by providing interaction tools to the users to carefully explore the provided environment (Gaugne et al, 2014).
In terms of underwater VR simulations, the 'Amphibian' provided an immersive SCUBA diving experience through a terrestrial simulator (Jain et al., 2016).The system is capable of simulating buoyancy, drag, and temperature changes through various sensors.VR has become widely popular over the past two decades as a medium for exploring archaeological sites (Anderson et al., 2010;Sylaiou et al., 2009;Papagiannakis et al., 2018).In a fewer instances, it has been also used for teaching or training purposes.However, for maritime archaeology VR has not been so popular and only in the past few years there have been some experimental applications.
In 2016, a VR application allowed for the exploration of the underwater cultural heritage by providing geolocated and multiresolution textured 3D models of underwater archaeological sites (Bruno et al., 2016).Users have the potential to receive historical, archaeological and biological information and contents about the submerged exhibits and structure of the archaeological site.In 2017, methods for realistic modelling and VR presentation of underwater archaeological assets was presented (Liarokapis et al, 2017).In particular, the Mazotos shipwreck site was selected because it's importance as well as accessibility issues (the site is located 44 meters underwater).The work allows virtual visitors to (a) get an idea of how the shipwreck looks like in VR and (b) educate themselves based on accurate archaeological information about the amphorae and the shipwreck.
Finally, in another approach, VR was employed to teach the basics of handling an airlift to future marine archaeologists (Kouřil and Liarokapis, 2018).A system for simulating sand in immersive VR was presented, allowing for real-time excavation.The environment is enhanced with caustics and fog, to simulate the lighting conditions underwater.To enhance the immersion, the terrain is populated with procedurally generated vegetation and rocks.It is worth-mentioning that the work presented in this paper has a similar scope, which is to teach future marine archaeologists the basics of photogrammetry.

Photogrammetry
Most of the possibilities offered by contemporary digital tools can be exploited in education both for learning as well as for teaching.That implies as well in the field of photogrammetry and remote sensing.Various tutorials or interactive e-learning environments are available on the Web for more than a decade now (Fluehler et al, 2005).When we refer to the term educational in most cases, we refer mainly to simple photogrammetric tools, designed for educational aspects, or commercial software with limited functionality.
Experience show that standard photogrammetric software may not be the best option for photogrammetry education.Several standard photogrammetric tools such as Z/I SSK, DVP, or EOS Photomodeler and Pictran were available from Technological Educational Institute of Athens.By using those software in the past students were able to follow and perform all stages of a typical photogrammetric project (Kalisperakis et al, 2004).But, as authors state, it is still essential that students come to terms with the basic theory and grasp the fundamental photogrammetric concepts.That's why commercial software are not intended to educate the user/student but to provide a userfriendly environment for the full spectrum of photogrammetric tools for execution, completion and compilation of photogrammetric tasks with minimum clicks.

Kalisperakis et al. having the above in mind, developed in
Matlab an open source educational package called MPT (Matlab Photogrammetric Toolbox) (Kalisperakis et al, 2004), for basic photogrammetric operations.The main concerns of the authors focused on the introductory course of photogrammetry.For that reason, the Toolbox was developed to help students not to learn 'how to do it' but to realize 'what is actually being done'.The development of that tool was intended for use in the class as well as a mean to the students to self-educate themselves.Another application and a rather old one is ARPENTEUR (Architectural PhotogrammEtry Network Tool for EdUcation and Research), a web application for digital photogrammetry mainly dedicated to architecture (Grussenmeyer and Drap 2000).Its aim was to develop a digital photogrammetric workstation on the Web and 3D modeling tools dedicated to architectural survey.ARPENTEUR photogrammetric orientation approach was mainly based on two step orientations (relative and absolute orientations).The modeling process is based on the measure of homologous points visible on any image pairs from the project.
The E-FOTO project's (Coelho and e Silva Brito, 2002) aim was the implementation of an educational, digital photogrammetric workstation.The modules implemented in this project were Image Rectification, Interior-Exterior orientation modules, Image Triangulation/Bundle Adjustment, Stereogram Normalization, DEM Extraction, Ortho Rectification and Digital Restitution modules.The intention of E-FOTO was the development of an e-learning environment that would make possible the adaptation of it to University curricula and the creation of extension courses and symposia.The main idea was to use the interactivity allowed by the Internet, experimenting with tools such as websites, virtual classrooms, chat, e-mail, forums, digital libraries.Moreover, a partial e-learning approach based on the development of an educational software was introduced, sv3DVision (Aguilera and Lahoz, 2006), and the design of a pilot course, Close-Range Photogrammetry and Computer Vision.The main idea was that the students instead of acquiring knowledge, they develop skills that were going to allow them to perform their future profession efficiently, considering their learning and self-evaluation capabilities.
Snavely et al. presented structure-from-motion and image-based rendering algorithms that operate on hundreds of images downloaded as a result of keyword-based image search queries like "Notre Dame" or "Trevi Fountain" (Snavely et al, 2008).The idea behind it was that using search queries in the internet, a large amount of famous places imagery captured from every angle can provide ideal datasets for 3D reconstruction.Following that, the authors proceeded to the development of a novel photo explorer interface for browsing large collections of photographs in which the user can virtually explore the 3D space by moving from one image to another.A tool for selflearning, tests and exercises with real photogrammetric data has been developed by (Luhmann 2016).As is stated, PhoX is able to process almost any kind of imagery, from scanned or digital aerial images to close-range images.The software's main purpose is data analysis, interactive image measurements, simulation and visualisation of results.It is used by the students of Institute for Applied Photogrammetry and Geoinformatics of Jade University of Applied Sciences during their Bachelor studies, as a module with increasing complexity.
A rather recent development, GRAPHOS (inteGRAted PHOtogrammetric Suite) has been developed within an ISPRS scientific initiative (Gonzalez- Aguilera et al, 2018).GRAPHOS allows the processing of aerial and terrestrial sets of images, producing dense metric 3D point clouds.It encompasses a mixture of robust photogrammetric and computer vision algorithms in order to provide automated processing, flexibility in handling any image and type of camera, quality control and assessment, and didactic aspects of algorithms and their performance.GRAPHOS includes various tools for performing analyses and quality checks on the produced dense point clouds like: deformation maps computation, point-based quality analyses by picking points or measuring distances and angles directly on the point cloud, with the use of reference point clouds to provide an accuracy assessment.Nevertheless, no educational software focused on data acquisition process, which is anyway mostly empirical.This study focuses on the data acquisition aspect of photogrammetric workflow, trying to implement VR to train students for this particular task.

SYSTEM IMPLEMENTATION
The project is implemented in the Unity engine mainly for the HTC Vive with the possibility of using some other VR device (i.e.: Oculus Rift and Windows Mixed Reality).Free SteamVR and VRTK assets were used for the core VR functionality.The basis for the environment was overtaken from the dry visit scenario of the Mazotos shipwreck site (Liarokapis et al, 2017).The locomotion there however is managed by teleportation to avoid motion sickness (Kolasinski, 1995).This was not suitable for teaching the photogrammetry properly, so free movement was used instead.The free movement allows to move towards the direction where the user is looking as well as ascending and descending.This type of movement is much closer to the real underwater movement.For the virtual environment, a photogrammetric 3D processing of the undisturbed Mazotos shipwreck site was used.This 3D model was created in 2012, five years after the original photos were captured (Demesticha, et al, 2014).Since, the original photos were captured for creating a simple photo mosaic, without 3D geometric information, the data set was barely sufficient for a 3D model.Nevertheless, holes and missing parts were re-processed, with semi manual procedures, using more recent data sets.The generated high-quality model however consists of over 7 million polygons, which is not eligible for the real-time rendering, especially in combination with VR.
The raw model also contains mistakes or unwanted polygons, such as pieces of the mesh "flying" in the air or the sea floor surrounding the shipwreck.We improved the generated model by cleaning the unwanted polygons.We also reduced the number of polygons to the acceptable amount for real-time game engine, in our case to approximately 500000 polygons.To properly generate the 3D scanned model, measurements of the site are needed to be taken by the diver.In the virtual environment, the user can do that by placing markers on the ground.Then the user must measure distances between all placed markers.This can be done by setting the origin point near one marker and moving to the other marker and setting the destination point.Measurements consist of the origin and destination point coordinates and the distance between those points.All valid measurements are stored into a file.An overview of the system architecture is presented in Figure 2.
Figure 2. Overview of the system The second part of the experiment is taking pictures.Pictures are taken from the virtual camera object on user's interaction.
The virtual camera is visualised by a simplified 3D model (see Figure 3).The taken picture is saved immediately as file, when the user presses the appropriate button.This effect is simply achieved by showing the rendered screen of a camera type object into a texture.While taking the real measurements, a small user error is introduced in the measured data.Automatic calculation of marker distances is thus not an option.On the other hand, accidental or totally off measurements should not be considered.Valid measurements are only those, who start and begin close to marker (see Figure 4).

METHODOLOGY
Fourteen participants took part in the pilot experiment and all had given their written consent.They were all healthy individuals with age ranging between 25-45.Initially, the workflow of the experiment was explained to them.The users were 'materialized' within the VR scenario, next to the Mazotos shipwreck.The VR scenario is composed by the Mazotos shipwreck photogrammetric model, included within a grid frame.Apart from the shipwreck and the grid frame, in the scene the user can see a photographic camera and a table titled "measurement taken".The users had to produce a series of subtasks including: • Position around the site Ground Control Points (GCPs) useful for the photogrammetric exercise; • Take linear measurements between the previously generated GCPs; • Save the taken measurements (at the bottom of the "Measurements taken" table there is a tab called "Save measurements"); • Take photographs following the photogrammetric schemes.
Users had two HTC Vive controllers with specific commands connected with some of the controller's buttons.Figure 5 presents the customisation that was done for the controller's button relative to any action the user can take within the VR environment.
Figure 5. Controls used in VR for the experiment Each controller's button has the ability to trigger one or more actions.The description of each action connected to the relative button is detailed below: Within the VR environment the controllers are visualized as two (left and right) hands.Each hand has an index finger.The markers will be created at the tip of the left hand's index finger.Once created, or any time the User position his/her hand close enough to the GCP marker, the latter highlights in green, showing the GCP number and the relative depth (relative to the water surface positioned at 15 m above the seafloor).Every time one GCP is deleted, the newly created GCP will fill the empty spot in the GCP's sequence.If the User has taken measurements involving the deleted GCP marker, each of these measurements will be deleted from the saved data and therefore lost.
Moreover, to take a measurement between two GCP's markers, the user will have to position the right hand's index finger over the marker, press and hold the top axe of the Touchpad and move from one marker to the other.This action will unroll a "measuring tape ribbon", until the Touchpad is released.Therefore, the User has to position the starting and ending point of the measurement he/she wishes to take.In the event the User is not happy with the taken measurement, a second measurement over the same markers will overwrite the previous one.
Figure 6.Area that testing took place The area of evaluation was limited to only be a small part of the shipwreck as show in Figure 6.Participants had 20 minutes in their disposal to put control points inside the indicated polygon.They were instructed to place no more than 6 GCPs inside the indicated area.They were also expected to measure the distances between the GCPs that will be placed trying to create triangles (trilateration).Then they could capture as many images as they desire as long as they are within the time limit.The images that they captured should only refer to the indicated area.If the user captures the whole area of the shipwreck, the evaluation process will be done only for the indicated area.

EXPERIMENTAL RESULTS
The evaluation was done considering the following parameters.First of all, the area the user's model covers/area of the actual model (it refers to the amphorae concentration inside the indicated polygon).Other parameters that were taken into consideration include: number of GCPs, area GCPs cover/indicated polygon area, number of Images captured and mean cloud to mesh distance.Results of the task itself are illustrated in Figure 7.In terms of results, all users seem to have fully covered the designated AOI, but the number of photos they acquired in doing so, varies significantly.The EU2 have gathered 85 photos, while the mean is 40 and half users have gathered less than 20.This large discrepancy may be explained by the EU2 experience in the specific underwater site, the need to acquire a complete 3D model, including oblique photos to fully replicate the 3D nature of the object and the light absorption, which affects visibility and forces for smaller object to camera distance.

Figure 7. Experimental results
All other users did not have the same understanding of the site, nor the absorption effect was fully imposed, hence even photos from a large distance were adequate for photogrammetric processing.
Data acquired by the users were processed in Photoscan and the resulting 3D model was compared against the reference 3D model in CloudCompare.The 3D model of the VR application was used as reference 3D model.Final results do not fully represent the completeness of the final 3D model, as figures in table XX, represent the projected area covered.Nevertheless, the final RMSE of 3D models from all users are acceptable, with small differences from EU2, representing the expert user in this evaluation.This may be attributed to impeccable photos acquired from the virtual environment.Photos captured did not suffer from absorption nor any other noise effects, rendering them perfect for 3D modelling.

CONCLUSIONS AND FUTURE WORK
This paper presented a novel VR application that can assist maritime archaeology students to learn basic photogrammetry techniques.Our implementation shows that the VR photogrammetry gamification techniques allows the creation of accurate measurements to be made.It should be mentioned that the underwater VR experience is not as in reality.There are several aspects missing in the VR, including anxiety, nitrogen narcosis, and motion restrictions.These aspects cannot be easily replicated in VR, hence it a different experience.Having said so, time constraint is the only difference to UW and close-range photogrammetry simulation, which increases possible educational applications in photogrammetric education, providing invaluable experience in data acquisition techniques.Future improvements include addition of water absorption and gaussian noise in photos, in order to make the results more realistic, both in terms of acquisition and 3D modelling.

Figure 1 .
Figure 1.Orthophoto of Mazotos shipwreck from legacy photos from 2007, processed 2012 for orthophoto creation and 2017 for VR 3D modelling.

Figure 3 .
Figure 3. Virtual representation of a digital cameraThe user takes pictures by a virtual digital camera object.The camera has a preview screen to show, what the camera sees.This effect is simply achieved by showing the rendered screen of a camera type object into a texture.While taking the real measurements, a small user error is introduced in the measured data.Automatic calculation of marker distances is thus not an option.On the other hand, accidental or totally off measurements should not be considered.Valid measurements are only those, who start and begin close to marker (see Figure4).

Figure 4 .
Figure 4. Virtual simplification of a distance measurement