3D UNDERWATER MINE MODELLING IN THE ¡VAMOS! PROJECT

The project Viable Alternative Mine Operating System (¡VAMOS!) develops a novel underwater mining technique for extracting inland mineral deposits in flooded open-cut mines. From a floating launch and recovery vessel a remotely-operated underwater mining vehicle with a roadheader cutting machine is deployed. The cut material is transported to the surface via a flexible riser hose. Since there is no direct intervisibility between the operator and the mining machine, the data of the sensor systems can only be perceived via a computer interface. Therefore, part of the efforts in the project focus on enhancing the situational awareness of the operator by providing a 3D model of the mine combined with representations of the mining equipment and sensor data. We present a method how a positioning and navigation system, perception system and mapping system can be used to create a replica of the physical system and mine environment in Virtual Reality (VR) in order to assist remote control. This approach is beneficial because it allows visualizing different sensor information and data in a consistent interface, and enables showing the complete context of the mining site even if only part of the mine is currently observed by surveying equipment. We demonstrate how the system is used during tele-operation and show results achieved during the field trials of the complete system in Silvermines, Ireland.


INTRODUCTION
The project Viable Alternative Mine Operating System (¡VAMOS!) funded by the European Union's Horizon 2020 research and innovation programme develops a prototype mining system to extract raw materials from inland water-bearing areas. It uses a remotely operated mining vehicle, which is launched from a waterborne carriage. As the mining vehicle cuts the rock face with a roadheader small rock fragments are created which are transported to the surface with a built-in dual-stage pump using a flexible riser hose. The dewatering plant filters sediment from the slurry and then returns the excess water to the mine. ¡VAMOS! looks at flooded open-cut mines which have been considered depleted in the past because with previous mining techniques it was not economically viable anymore to continue operations. Today, with rising prices of certain rare ores it might become interesting again to re-open abandoned mines in order to access deeper seated minerals. However, conventional mining techniques require high treatment and dewatering costs. Moreover, from an environmental perspective it is desirable that the water table of these flooded inland mines is not changed. Therefore, the ¡VAMOS! project aims to develop a new remotely controlled underwater mining machine and associated launch and recovery equipment, which provides a mining technique that is environmentally and economically more viable than the state-of-the-art.
The complete ¡VAMOS! underwater mining system was recently successfully demonstrated and tested during field trials in Silvermines, Ireland in October 2018. The test site, which is an inoperative opencast barite mine, is depicted in Fig. 1. During the trials the mining machine was deployed in depths of up to 57 m, cutting in different rock types under various operating conditions, and pumping the cut material to a deposit onshore as a slurry. The left image in Fig. 2 shows the launch and recovery vessel which is used as a floating platform for deployment of the mining vehicle depicted in the middle image of Fig. 2. The vessel is anchored and positioned using wires and electrical winches. The system is tele-operated via a fiber optic link from a control cabin shown in the right image of Fig. 2. The challenge in tele-operating a large underwater mining vehicle is that there is no direct intervisibility, which makes precise control difficult. The perception data of the mining vehicle can only be communicated via a computer interface. One part of the efforts in the ¡VAMOS! project to enhance situational awareness of the operator is the creation of a 3D model of the mine above-the-water and underwater, which captures the mining site as detailed as possible. It is well known that a map of the environment in addition to the raw sensor data is extremely helpful in supporting remote control and enhances spatial awareness (Nevatia et al., 2008).
An above-the-water model is created from terrestrial laser scanning and camera images. The scans are registered into a consistent model and a high resolution point cloud as well as a lower resolution mesh for faster rendering of the complete scene are created.
The model of the underwater site is acquired using multibeam sonar data captured using an autonomous/remote operated underwater vehicle (AUV/HROV). The models are visualized to the operator in a Virtual Reality (VR) system. Since we know the location of all mining assets using the positioning information from Global Navigation Satellite System (GNSS) and an underwater positioning system based on a short baseline (SBL) and inverted ultra-short baseline (iUSBL) acoustic positioning network, the mining vehicle, launch and recovery vessel and AUV can be rendered accurately in the 3D scene. The models are fully articulated and sensor information, such as the positions of the cutter boom, tracks and perception data, is displayed in real-time. This way a replica of the physical system as well as the environment is built which provides real-time and contextual information in a consistent and easy comprehensible way.
All navigation sensor data is combined and fused to provide real-time, accurate and precise information about the localization and orientation/attitude of all ¡VAMOS! systems. This navigation information and the mine perception data from multibeam, 3D sonar and structured light systems are used to update the 3D model both with real-time data and with off-line survey data. The right image in Fig. 2 shows the operator remotely controlling the underwater mining machine using the VR interface. The VR system is used during planning, e.g., for finding a suitable landing position on the bottom of the pit, as well as during operations. Moreover, live data from some of the environmental sensors is fed into the system to visualize measurements, such as suspended sediments in the water, for real-time monitoring.
In this paper we describe the positioning, navigation and awareness system of the ¡VAMOS! underwater mining system and show results of detailed underwater 3D modelling of a submerged inland mine. We report on the results of the field trials at the Silvermines site in Ireland. Results on the first ¡VAMOS! field trials in Lee Moor, United Kingdom, can be found in (Almeida et al., 2018b).

RELATED WORK
The combination of underwater mapping technology, especially 3D reconstruction techniques using photogrammetry, and visualization in a VR environment has become popular in the field of cultural heritage conservation and underwater archaeology (Drap et al., 2007). Underwater cultural heritage is typically difficult to access and exploitation of underwater archeological sites for a large-scale public audience and tourism is not sustainable. This sparked an interest to create immersive experiences and virtual tours using VR (Haydar et al., 2011). Increasingly, there is an interest not only to show the 3D underwater environment but to augment it with additional educational and archaeological information (Skarlatos et al., 2016. Similarly, VR has been applied to assist teleoperation of mobile robots, remote control of industrial manipulators as well as the piloting of underwater robots (Lin , Kuo, 1997). The difference here is that for a teleoperation scenario live sensor data needs to be fed back into the system to update the virtual environment in order to create a faithful representation of the current environment of the remote system. In combination with simulation of the environment, VR is also used in pilot training of Remotely Operated Vehicles (ROV) for underwater operations (Lin , Kuo, 2015).
As an underlying technology game engines, such as the Unity 3D cross-platform game engine (Unity Technologies, 2019), are increasingly used in research and industry for serious applications, e.g., simulation and teleoperation of robots using VR (Chin et al., 2018).
Bathymetric data gathering is often performed using acoustic sensors, such as single beam or multibeam sonar (Surtees, 2009, Denny et al., 2015. Typically in industrial applications such as underwater mining operations, water turbidity is high and the visibility with optical sensors and cameras is very low. In these applications sonar sensors have the advantage of providing a significantly higher measurement range in water and measurements are also possible in very turbid environments. Sound speed is related to water temperature and salinity, which requires calibration of the speed of sound for accurate measurements.

Perception and Surveying Sensors
The perception sensors installed on the mining vehicle are depicted in the left image of Fig. 3. A Kongsberg M3 multibeam/imaging sonar (1) is installed on a pan-and-tilt unit with rotary encoder feedback, which can be automatically panned/tilted for 3D scanning and mapping. Additionally, a Coda Octopus Echoscope 3D sonar (2) provides real-time scans with high update rate of the work surface. Two custom-built structured light scanners (3) with rotating line lasers and LED flashes are installed on each side of the sensor bar. Moreover, cameras and LED lights (4) are mounted on the pan-and-tilt unit, pointed at the cutter head, and pointing backwards to the backhoe bucket.
As a support for the mining operations, an autonomous/remote operated underwater vehicle (AUV/HROV) was developed in the project, which is depicted in the top image of Fig. 4. For surveying the mine, a 70 deg downwards tilted multibeam sonar or a 3D sonar can be mounted. A detailed description of the AUV/HROV can be found in (Martins et al., 2018). For the above-the-water survey a Riegl VZ-400 terrestrial laser scanner with a custom-built camera mount shown in the bottom image of Fig. 4 is employed.
The sensor data of the mining vehicle and launch and recovery vessel is directly streamed to the control center via an umbilical and optical fiber. The AUV/HROV can stream high-bandwidth sensor data on the surface via a long-range wireless local area network (WLAN) connection to the control center. During dives an acoustic link can be used for control commands. In order to stream multibeam and 3D sonar scans to the control center in real-time during dives, a small support boat with a WLAN access point can be connected with a short umbilical to the AUV/HROV.

Positioning and Navigation
For geo-referencing we estimate the pose of all assets relative to a fixed GNSS Real-time Kinematic (RTK) base station, which is mounted on top of the control cabin. The pose of the launch and recovery vessel is determined using three multi-frequency GNSS antennas mounted on top of the crane tower, see middle image of Fig. 3.  On the water surface the AUV/HROV can directly infer precise positioning information using GNSS. This is achieved with two antennas, such that also the heading can be directly computed from the RTK solution. Additionally, a FOG based INS and a Nortek Doppler Velocity Log (DVL) are used for pose estimation. During dives of the AUV/HROV the acoustic positioning system can be used as additional source for position determination.

Time Synchronization and Calibration
Time synchronization is achieved using the network time protocol (NTP) and pulse-per-second (PPS) signals. All computer systems and sensors are synchronized this way to GNSS time. This allows to also correlate offline measurements to data logs in retrospect based on timestamps.
The sensors are individually calibrated depending on the sensing modality. Calibration of the sensor poses is challenging because many different sensors are employed and it is costly to design calibration fixtures which are visible, for example in optical sensors as well as in sonar sensors. Moreover, considering the large size of the mining vehicle, very large calibration targets would be necessary for accurate calibration, such that they would be visible in multiple sensors. Therefore, sensor mounting positions are estimated from a combination of manufacturing CAD models, laser scans taken of the vehicles after integration, GNSS positions, and tape measure measurements.

Above-the-water and Underwater Mine Mapping
The above-the-water model is created by registering the laser scans using the Iterative Closest Point (ICP) algorithm implemented in 3DTK (Nüchter, 2019) and mapping color information from camera images on the point cloud. The complete point cloud is georeferenced using GNSS position measurements and is sub-sampled to provide a static high resolution point cloud for visualization. Additionally, the point cloud is meshed and imported as a simplified triangular mesh with color since this allows faster rendering of the complete scene.
The bathymetry is initially created from a multibeam survey using the AUV/HROV and later updated with additional surveys and perception data from the mining vehicle. The initial survey is processed offline using a continuous-time SLAM algorithm, which optimizes point cloud consistency globally, i.e. for all the sensor measurements of the complete map. Sensor measurements from the perception sensor systems, such as multi-beam sonar, 3D imaging sonar, and structured light scanners, are fused into a consistent 3D representation.
Mapping algorithms based on a signed distance function (SDF) voxel map are used to integrate measurements taken with varying accuracy and noise properties. A SDF map is a beneficial surface representation because noisy measurements are smoothed over multiple observations. This offline mapping approach is described in (Bleier et al., 2017).
As the mine changes over time due to the mining operations themselves, the internal representation of the mining environment needs to be constantly updated based on new sensor observations. For real-time processing the model is only updated in a small window of 3 -5 m around the initial survey. This way only a small number of cells need to be updated

Virtual Reality based Human-Machine Interface
The raw sensor data, terrain maps, positioning and navigation data, and system information is transferred to the human-machine interface (HMI) via local area network (LAN). The HMI is based on a custom-built VR application build on top of the Unity gaming engine (Unity Technologies, 2019). It provides a 3D VR model of the entire mining operations. This includes models of all vehicles and vessels as well as the riser system and relevant static structures. The model is dynamically adjusted, such that it faithfully replicates the current state of the real operations. As output device, standard monitors are chosen and a computer mouse is used for interacting with the 3D scene in VR. This input scheme was selected because it integrates well with the monitor wall used in the control center for displaying camera streams and data visualizations from other ¡VAMOS! subsystems as well as the joystick and touchscreen based control interfaces for the mining vehicle and vessel. It is also possible to use the VR system with more immersive output devices, such as VR headsets like Oculus Rift or HTC Vive. However, for prolonged operations problems like simulator sickness and increased eye strain can be an issue. dynamically adjusted using measurements from a tide gauge. The bottom images show detail views of the vessel during deployment of the mining vehicle together with the underwater terrain.
The VR system is used for control of the mining vehicle, the launch and recovery vessel and the AUV/HROV. It provides a range of functionalities to support tele-operation as well as assistance during planning of operations. For example, it provides guidance in maneuvering the vessel to lower or pick-up the mining vehicle at the chosen position. It enables path input for autonomous surveys using the AUV/HROV and it provides awareness when maneuvering the mining vehicle on the pit floor and during cutting operations.  . 8 shows results of the underwater bathymetry created using data from multibeam sonar surveys of the AUV/HROV compared to results from a pre-survey created by a surveyor using a single beam echosounder. It can be seen that the updated bathymetry includes more detail than the initial pre-survey of the site.

Fig
Overall the system proved effective in providing the expected functionalities and increased awareness for the operator. Using a VR based HMI turned out to be very useful because it allows free viewpoint rendering of the digital replica. Virtual views for the pilot are created with detailed view of the cutter-head sweep, AUV/HROV bathymetric mapping and work face profiling, see Fig. 7. This way a clear overview of the entire system can be created independently of water turbidity. Additionally, environmental sensor measurements, such as concentration of suspended sediments from acoustic Doppler current profilers (ADCP), are visualized in the 3D environment to provide real-time feedback to the operator.
For the field trials the system is sufficiently accurate to map newly extracted bathymetric surfaces with processing times that are adequate to keep up with mining progress. It was demonstrated that overall positioning accuracy is sufficient for vessel maneuvering and driving of the mining vehicle. The mining vehicle was lowered to the pit floor and detached from the lifting cable. After driving the mining vehicle away it could be placed back into position and successfully be re-attached to the lifting cable. To achieve this, a bullet attached to the lifting cable has to be aligned with an error of less than 0.5 m for latching onto the recovery mechanism mounted on top of the mining vehicle.

CONCLUSIONS
In this paper we showed how a virtual reality 3D replica of the real environment can be applied for assisting tele-operation of an underwater mining vehicle. We use this model for immersive data visualization of the mining operations for planning during development and operations during the testing phase.
A 3D model of the operations is valuable to effectively monitor the events and the mining process below the water surface. Moreover, it enables the use of a smaller and cheaper sensor kit since only the areas where change is expected need to be monitored an updated continuously with surveying equipment while the full context of the mine site is still visualized to the human operator. Furthermore, the ¡VAMOS! positioning, navigation and awareness systems are an enabling technology for future driver-less operation.