LBS AUGMENTED REALITY ASSISTIVE SYSTEM FOR UTILITIES INFRASTRUCTURE MANAGEMENT THROUGH GALILEO AND EGNOS

There is a continuous and increasing demand for solutions, both software and hardware-based, that are able to productively handle underground utilities geospatial data. Innovative approaches that are based on the use of the European GNSS, Galileo and EGNOS, sensor technologies and LBS, are able to monitor, document and manage utility infrastructures’ data with an intuitive 3D augmented visualisation and navigation/positioning technology. A software and hardware-based system called LARA, currently under development through a H2020 co-funded project, aims at meeting that demand. The concept of LARA is to integrate the different innovative components of existing technologies in order to design and develop an integrated navigation/positioning and information system which coordinates GNSS, AR, 3D GIS and geodatabases on a mobile platform for monitoring, documenting and managing utility infrastructures on-site. The LARA system will guide utility field workers to locate the working area by helping them see beneath the ground, rendering the complexity of the 3D models of the underground grid such as water, gas and electricity. The capacity and benefits of LARA are scheduled to be tested in two case studies located in Greece and the United Kingdom with various underground utilities. The paper aspires to present the first results from this initiative. The project leading to this application has received funding from the European GNSS Agency under the European Union’s Horizon 2020 research and innovation programme under grant agreement No 641460.


INTRODUCTION
GIS is an important tool for managing and visualising geospatial data. A current trend in Geographic Information Sciences is the development of integrated geospatial data infrastructures, which unite geographically and semantically disperse national geodatabases. The INSPIRE initiative (INSPIRE, 2016) is the EU directive aiming at the unification and interoperability of national geodata infrastructures throughout Europe. The EU-funded pilot project HUMBOLDT (HUMBOLDT, 2016) is a real test implementation of the standards defined in this directive. Another related EU project, is ORCHESTRA (ORCHESTRA, 2016), which provides a framework for field data management, services for sensor access, visualisation, user-authentication and management, format conversion, coordinate transformation etc.
The modelling of geospatial data in 3D is an active research area in geoinformation science. CityGML (Kolbe et al., 2005) is a specialisation of the Geography Markup Language (GML), the XML grammar for expressing geographical features.The focus of existing research is mainly targeted on above surface city models. VISTA project (Beck et al., 2007) is a prominent exception. This project aims to create integrated 3D underground models in the UK and use automated recognition for some of the modelling tasks.
Mobile GIS extends GIS from the office to the field by incorporating various technologies: mobile and tablet devices, wire- * Corresponding author less communication and positioning systems. Mobile GIS enables on-site capturing, storing, manipulating, analysing and representation of geospatial data. In addition, mobile AR extends traditional 2D or 3D visualisations by overlaying registered visualisations over video footages. There is a clear trend towards mobile devices.
Mobile AR systems are using high-precision Real Time Kinematic (RTK) Global Navigation Satellite Systems (GNSS). Usually, these systems use bulky hardware and limited sensor fusion. The next generation of the mobile AR prototype the integration of a RTK GNSS receiver and performing sensor fusion with Inertial Measurement Units (IMUs) is evident for providing a highqualitative and functional AR experience.
Several publication have been made the last few years that are coupling existing underground utilities data with AR techniques and methods for providing x-ray viewing of underground utilities either for collision avoidance or monitoring. Much of this work has been generated by applications and devices developed in the framework of experimental research projects that try to tackle specific user needs and improve accuracy, usability, and reliability of technology. This paper attempts to describe innovative methods of integrating cutting edge technologies within the realms of augmented reality, geodatabases, GNSS systems and positioning sensors into a mobile system. An exhaustive review on the state of the art of the different technologies employed is attempted and the LARA project integration framework is presented providing useful insights on methods, interoperability requirements and on site validation test beds.

3D Geographic Information System (GIS) & Geodatabase
A geographic information system or geographical information system (GIS) is a system designed to capture, store, manipulate, analyze, manage, and present all types of spatial or geographical data. Generally, the term describes any information system that integrates, stores, edits, analyses, shares, and displays geographic information. GIS applications are tools that allow users to create interactive queries (user-created searches), analyse spatial information, edit data in maps, and present the results of all these operations (Clarke, 1986), (Maliene et al., 2011).
The implementation of a GIS system is often driven by jurisdictional (such as a city), purpose, or application requirements. Generally, a GIS implementation may be custom-designed for an organisation. Hence, a GIS deployment developed for an application, jurisdiction, enterprise, or purpose may not be necessarily interoperable or compatible with a GIS that has been developed for some other application, jurisdiction, enterprise, or purpose. So, there is an active GIS market and as a result a broad range of applications which involve the use of a combination of digital maps and georeferenced data. GIS software can be sorted into different categories. The first main distinction between the different GIS applications is between the open source and commercial proprietary software. In the following list some of the most wellknown open source or commercial solutions are presented: 1. Open source GIS software:

(f) TatukGIS
A geodatabase is highly related to a GIS system. Most of the GIS software solutions support spatial databases. In particular, a spatial database, or geodatabase is a database that is optimised to store and query data that represents objects defined in a geometric space. Most spatial databases allow representing simple geometric objects such as points, lines and polygons. Some spatial databases handle more complex structures such as 3D objects, topological coverages, linear networks etc. Although, the typical databases are designed to manage various numeric and character types of data, additional functionality needs to be added for databases to process spatial data types efficiently. These additional spatial data type called geometry or feature.
In addition to typical SQL queries such as SELECT statements, spatial databases can perform a wide variety of spatial operations. The following operations and many more are specified by the Open Geospatial Consortium standard (OGC): • Spatial measurements: computes line length, polygon area, the distance between geometries, etc.
• Spatial functions: modify existing features to create new ones, for example by providing a buffer around them, intersecting features, etc.
• Spatial predicates: allows true/false queries about spatial relationships between geometries.
• Geometry Constructors: Creates new geometries, usually by specifying the vertices (points or nodes) which define the shape.
• Observer functions: queries which return specific information about a feature such as the location of the center of a circle.
2.2 Augmented reality (AR), Virtual Reality (VR), Real-time pose estimation algorithms Both Virtual Reality (VR) and Augmented Reality (AR) refer to the use of computer-generated pictures (computer graphics) for representing data-generally a rendered 3D scene. In the case of VR, the complete view is synthetic, and there is no direct link to the real environment of the user. For example, the recent advent of VR wearable devices (e.g. Oculus Rift, HTC Vive, Google Cardboard) makes it possible to place the user in a complete virtual world, where all his/her surrounding is replaced by virtual objects. By contrast, in AR, the link with the real environment is not broken. Instead, the user can visualise at the same time the real world and virtual objects unambiguously linked to the real world (Pagani, 2014). The real world is seen either directly in case of see-through AR (using semi-transparent AR glasses), or indirectly by using real-time video acquisition and display. The link between virtual objects and real world is in most cases the proper and seamless positioning of the virtual objects in 3D in the real scene, which implies that the exact position and orientation of the user's eye or of the used video camera -has to be computed in real time to generate AR content. This led to the first definition of augmented reality as "a system that (1) combines virtual and real, (2) is interactive in real time and (3) is registered in 3D" (Azuma et al., 1997). Thus, an AR system generally makes use of two different techniques: 3D rendering as for VR on one side, and real-time pose estimation on the other side.
Use of rendering technologies, such as programmable pipelines and shaders, allows the creation of digital physically-based materials that closely match real life counterparts in the way light interacts and reflects from the virtual objects. Due to the increased performance and parallelism of modern day GPUs, these digital materials can be applied and evaluated in real time. Real world light conditions can also be probed and used as input to such digital materials as to virtually reproduce the real light conditions (Debevec et al., 2012). This are essential to make the AR virtual objects augment the real world without looking out of place, this way breaking the immersion.
Pose estimation refers to the computation of the position and orientation of an object in 3D. This pose consists of a 3D translation and a 3D rotation and can be defined using 6 parameters. In AR, image-based pose estimation has mostly been used in the past, because AR scenarios usually require a camera as a sensor, and because minimising reprojection errors in the image leads to an optimal pose for the task of image augmentation. This type of pose estimation is referred to as inside-out tracking. Pose The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B1, 2016 XXIII ISPRS Congress, 12-19 July 2016, Prague, Czech Republic estimation based on image content has been achieved successfully in the past using specific patterns placed in the scene (referred to as markers) (Pagani et al., 2011) or using pre-learnt natural features in the scene (Zoellner et al., 2008). Recent methods such as simultaneous position and mapping (SPAM) (e.g. PTAM (Klein and Murray, 2007)) attempt to compute the camera pose without knowledge of the underlying scene, but for AR they require an initialisation with a known reference. Recently, additional sensors have been used for computing the pose more robustly. For example, visual-inertial tracking refers to the use of an IMU coupled to the camera in a Bayesian filtering framework to compute a robust pose under non optimal image conditions (Bleser and Stricker, 2008).

Galileo -GPS -GLONASS -IMU sensors
Vast investments were made to improve accuracy, integrity and availability of the global navigation satellite systems (GNSS) and therefore enhance their capabilities in commercial, professional and military applications. There are two main GNSS systems widely used in several applications, the Global Positioning System (GPS) by USA and the GLONASS by Russia. Two other GNSS systems are under development, the Galileo by EU and the BeiDou by China (Madry and Scott, 2015). If the aforementioned systems are used with a single point positioning, an accuracy of a few meters can be achieved because of a number of positioning error sources imposed (Petovello, 2014). Hence, various methods have been developed to eliminate the GNSS positioning error sources. The satellite-based augmentation system (SBAS) system improves the accuracy at the level of a meter. Furthermore, the differential positioning systems can eliminate the positioning error sources hence improve accuracy at the level of a few centimetres (Gerhard et al., 1996). Moreover, other methods such as RTK (L. Kislig, 2011) and Precise Point Positioning (PPP) (Chassagne, 2012), (Lachapelle et al., 2006) are able to acquire positioning accuracy at the level of centimetre.
The LARA device requires such positioning system in order to estimate its position with an accuracy of a few cm. One of the main targets of LARA is to take advantage of the existing GNSS systems (GPS and GLONASS) and the improved GALILEO and EGNOS systems. The positioning receiver will be portable with low power consumption and long autonomy time. An IMU will be integrated in order to estimate the pose of the LARA device as required.

A NEW INTEGRATION APPROACH: LARA PROJECT
3.1 LARA project -state of the art LARA is a H2020 co-funded project which aspires to develop a system, both software and hardware-based, by integrating different innovative components of existing technologies. The aim of LARA is to design and develop an integrated navigation/positioning and information system which coordinates GNSS, AR, 3D GIS, and geodatabases on a mobile platform for monitoring, documenting and managing utility infrastructures on-site. The LARA system will guide utility field workers to locate the working area by helping them see beneath the ground, rendering the complexity of the 3D models of the underground grid such as water, gas and electricity. The project sets four (4) concrete and measurable objectives in developing a highly innovative and adaptive application taking advantage of the Galileo and EGNOS.
1. Design, develop and exploit an entirely practical mobile device for utility field workers.

Make use and integrate mobile AR interfaces.
3. Improve on site interaction and navigation application.
4. Commercialise the final product and the related services as well as address relevant markets. The geodatabase will be supported by the PostgreSQL and Post-GIS. PostgreSQL, often simply Postgres, is an object-relational database management system (ORDBMS) with an emphasis on extensibility and standards-compliance. A very important characteristic of the PostgreSQL that is cross-platform and runs on many operating systems including Linux, FreeBSD, OS X, Solaris, and Microsoft Windows. The PostGIS is an open source software program that adds support for geographic objects to the PostgreSQL object-relational database. PostGIS follows the Simple Features for SQL specification from the Open Geospatial Consortium. As a result, the combination of the PostgreSQL and PostGIS is an excellent solution to support spatial data. Figure  1 presents the main architecture decisions regarding the GIS and geodb module.

AR & VR
In the LARA project, the AR system is implemented on a tablet with a rear camera. This tablet serves as a window onto the world, which means that the user can use the tablet as a scanning device to look at the real scene. In this context, the images acquired by the rear camera are directly and instantaneously displayed in full screen on the tablet, in such a way that the discrepancy and latency are kept minimal.
Because the use case is to visualise GIS data as a realistic 3D model, the system contains a module that converts a GIS (shape) file into a 3D model useful for 3D rendering. In the current version, the module converts pipes described as segments between geolocalised 3D points into cylinders with the correct diameter and a digital material that imitates the real life materials (plastic, metal etc). This conversion is done in an offline stage before the system is used for a particular location. Note that this type of conversion is fundamental for using real world data in an AR application.
For the rendering part of AR, the pipeline has been converted from an onscreen rendering paradigm to an offscreen rendering paradigm. This means that the rendering of the virtual objects as well as the image composition with the real video frame as background is made entirely in memory rather than on the screen as usually done in rendering applications. This proves to be extremely useful for embedding the AR module into a larger application that manages the the Graphical User Interface (GUI) separately. In this way, the augmented images are passed as memory chunks rather than portions of screen, and can be used as wished by the application GUI. The pose estimation part of AR makes use for the first time of Galileo positioning information. Because this pose estimation is an outside-in pose estimation approach, errors on the exact position of the objects might appear in the image. This is compensated by using known points in the real world in order to partially correct the pose given by the Galileo system.

Galileo -GPS -GLONASS -IMU sensors
The positioning receiver will be tailored to the LARA needs and targets. A first prototype of the receiver has been developed. The positioning receiver includes a GNSS module that is able to receive multiple constellations. Namely, the GPS and GLONASS systems as well we the newly developed Galileo system which offers improved accuracy, satellite availability and integrity compared to the aforementioned traditional systems. The module is able to work with the BeiDou system. The satellite availability and integrity is secured due to the large number of satellites but enhancement of the accuracy is needed for the needs of the LARA project. The module is able to work with SBAS (EG-NOS), DGNSS and RTK which means that according to its specifications is possible to work between the accuracy of about a meter, 25cm and 1cm respectively (in the best case). The receiver is equipped with an IMU used for the pose estimation functionality while the pose estimation algorithm and other algorithms are implemented on a powerful low cost microprocessor. Flexibility and autonomy is targeted in order to bring a positioning receiver in the market which can communicate via high level User Interface (UI) installed on most commercialized tablets. The main concept of the receiver is depicted in Figure 2.

Real-time pose estimation algorithms
In the LARA approach pose estimation parameters are initially recovered through the use of GNSS and IMU sensors. GNSS provides the location parameters, while IMU provides the attitude parameters. In the case that the recovered parameters' accuracy is lower than the system specifications, a photogrammetric pose estimation module is triggered. The photogrammetric approach (photogrammetric resection) (Dermanis, 1991), (Patias, 1993) uses at least Figure 2: Concept of the LARA receiver 3 control points to compute the pose estimation parameters. In cases when there is lack of control points (less than 3, or only one) pose estimation parameters can be computed using vanishing points, a control point and a vertical or horizontal distance originating from the control point.

The integrated LARA solution
Working methodology: The LARA system has been designed taking into account user requirements obtained from studies with final users and technical requirements, during the first months of the project resulting on a compilation of requirements and constraints that the system must accomplish.
The particular characteristics of the LARA system suggest following the continuous development methodology to be able to assure a final integrated solution with optimal results. To satisfy this methodology, three iterative sprints are created through the 24 months of the project, with their respective prototypes delivered. In these prototypes' functionalities and complexity will be included gradually, to check, evaluate and validate the prototypes' lab trials and pilot tests will be performed with the help of the end-users participating in the project.
LARA system consists of a set of modules mutually dependant due the required information exchange to achieve its full functionality. This information exchange between modules requires a beforehand planning, taking into account the different programming languages of the modules, mainly Java and C++. In addition, is the necessity to orchestrate the information exchange which includes the decision about the format in which the information will be sent among others.
Logical architecture: LARA system contains a set of depending modules, due to the real time behaviour of the system; interdependence between different modules is very high. This implies that communication between different modules is one the most important issues of the integration process of the system. LARA system modules are the following: Interfacing and data flows: The link between the modules implies that the information must flow between them, assuring the communication. Communication is made by means of a set of interfaces that defines how different modules are interconnected, the module information requirements and output. Figure 4 depicts how the information flows between the modules and their mutual interactions. The PR with the SF modules have common purpose as they receive and analyse signals from the position receiver sensor and determinate the quality of the signal. This signal and the quality with different flags are sent to the UI through the UI-PRSF interface. At the same time module 3DG needs this position data to obtain, show and work with the different data layers of the underground infrastructure of the working area. All this information exchange is done through the UI-3DG interface. The AR module needs a shaped view of the entire underground infrastructure, to render the appropriate 3D model; this shaped view is transmitted by the 3DG-AR interface. Finally, the AR module renders the 3D model and sends it to be displayed using the UI-AR interface.
Physical architecture: The LARA solution, as described in the previous paragraph, consists of a set of modules. Modules PR and SF compose the receiver sub-system. This sub-system needs specific hardware that is not included in common tablets. As a result, this module is implemented on a separated device. The remaining modules are included in the sub-system which can be included to any market available tablet device. Communication between the sub-systems is made by a USB connection and Bluetooth. Figure 5 depicts the physical architecture of the LARA platform. Particularly, Figure 5 shows how the PR and the SF are joined to create the positioning sub-system which is directly connected to the tablet using USB / Bluetooth.

Implementation:
To implement the LARA solution, a Java project with Java 8 is created. This project acts as the system backbone, as it receives the information from all modules and sends it where it is needed. The interfaces are created using Java 8 and Java FX. This Java project includes, as a module, the GIS module, which uses GeoTools technology to provide its functionalities. It also makes use of PostGIS to provide spatial objects for the Post-greSQL database.
The communication between the positioning sub-system and the Java project is JNI (JNI, 2016) with a C console which provides functionality to interact with the receiver sub-system, once request for the position is sent, the positioning sub-system stores this value on a database on the tablet, from where it is taken by the orchestrator.
The AR module is written in C++ as it needs high performance to render the 3D model and calculate all mathematical operations needed. To interact with it, the orchestrator (UI) will make use again of JNI (JNI, 2016) through a console which will return the augmented image to the UI module.
System synchronisation: UI acts as an orchestrator of the system, due to the "real time" behaviour of the system. Many problems were raised while integrated the whole work as a single solution.
The main problem is that we need not only the position of the user but also the pose. This information is taken from the receiver sub-system, and must be taken at the same time a camera image is taken to be augmented. However, the different system refresh rates make the synchronisation task rather complicated. To solve these problems we have taken the following approach: 1. Select UI as an orchestrator that will interact and decide where and when to send the information.
2. Use the same rate in PR and AR, as the most restrictive is the AR module the rate is fixed at 30 information items per second, the PR module will send repeated information if no new information is available.
3. Create a process that will ensure that the image taken has an appropriate position and pose ( Figure 6) The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B1, 2016 XXIII ISPRS Congress, 12-19 July 2016, Prague, Czech Republic The testing and validation of the LARA solution has the following objectives: • To demonstrate and evaluate the developed and implemented techniques, and methodologies of the LARA prototype.
• To collect feedback for its improvement. Currently the developed prototype have been tested under laboratory conditions. The testing and validation procedure, under real world conditions, is scheduled for the second semester of 2016. The testing and validation will take place in two different locations: in the municipality of Kozani, Greece, and the municipality of Birmingham, UK. The field trials will focus on system functioning and performance under real world conditions. Figure  7 and 8 present the information related to the pilots areas.

SUMMARY AND FUTURE WORK
We are developing a software and hardware-based system, namely LARA. This system is able to effectively handle underground utilities geospatial data. LARA is incorporating different novel modules in order to develop an integrated navigation/positioning and information system. This system is coordinating different Figure 8: The municipality of Birmingham pilot area technologies, such as GNSS, AR, 3D GIS and geodatabases on a mobile platform for monitoring, documenting and managing utility infrastructures on-site. In practise, the LARA system is designed to guide the utility field workers in locating the working area by helping them see beneath the ground and rendering the complexity of the 3D models of the underground grid such as water, gas and electricity.
At this stage, we have already finished with the system design.
During the design phase we collected information and the real users' needs not only from the end-users participating in the project, but also from others outside LARA consortium.
The V0.5 system prototype is ready and presented during the mid-term review, which was successfully passed the evaluation. We scheduled two more prototypes; V1.0 to be used during the 2 pilots foreseen within project lifetime and V2.0 at the end of the project.
The capabilities and benefits of LARA system are planned to be verified in two real case studies located in Greece and the United Kingdom. Both case studies will take place with various underground utilities. The final stage of the project concerns the definition of the business models to be used for LARA system commercialisation.

ACKNOWLEDGEMENTS
The LARA project leading to this application has received funding from the European GNSS Agency under the European Unions Horizon 2020 research and innovation programme under grant agreement No 641460. LARA is co-funded for the period 2015-2017.