OUTDOOR AR-APPLICATION FOR THE DIGITAL MAP TABLE

In many fields today, it is necessary that a team has to do operational planning for a precise geographical location. Examples for this are staff work, the preparation of surveillance tasks at major events or state visits and sensor deployment planning for military and civil reconnaissance. For these purposes, Fraunhofer IOSB is developing the Digital Map Table (DigLT). When making important decisions, it is often helpful or even necessary to assess a situation on site. An augmented reality (AR) solution could be useful for this assessment. For the visualization of markers at specific geographical coordinates in augmented reality, a smartphone has to be aware of its position relative to the world. It is using the sensor data of the camera and inertial measurement unit (IMU) for AR while determining its absolute location and direction with the Global Navigation Satellite System (GNSS) and its magnetic compass. To validate the positional accuracy of AR markers, we investigated the current state of the art and existing solutions. A prototype application has been developed and connected to the DigLT. With this application, it is possible to place markers at geographical coordinates that will show up at the correct location in augmented reality at anyplace in the world. Additionally, a function was implemented that lets the user select a point from the environment in augmented reality, whose geographical coordinates are sent to the DigLT. The accuracy and practicality of the placement of markers were examined using geodetic reference points. As a result, we can conclude that it is possible to mark larger objects like a car or a house, but the accuracy mainly depends on the internal compass, which causes a rotational error that increases with the distance to the target.


INTRODUCTION
Due to the ever-increasing performance of smartphones, many people carry more calculating performance in their pockets than a datacenter 35 years ago. The most powerful supercomputer in the year 1985, the Cray-2, had the equivalent performance of the iPhone 4 released in the year 2010 (Mick, 2015). Not only the performance, but the quality of the sensors does increase as well. With each generation, we get higher resolution cameras, and the accuracy of the GNNS gets more precise with new GNSS satellite constellations and better antennas (European GNSS Supervisory Authority, 2018). The combination of all these technological advances allows the interactive merging of the real world with the virtual world. AR is already used in many parts of our every-day lives, as virtual measuring tape, to virtually try on new sunglasses, or as entertainment in countless AR games. Navigation on a smartphone using the GNSS is nowadays a standard feature that replaced many dedicated navigation systems. A very successful app that combines AR with a geo-position is Pokémon Go! which lets you hunt virtual monsters in the real world. Another example is vGIS, an application that displays underground pipes to minimize accident hazards on construction sites. Google Maps received an AR mode in 2019 that visualizes junctions and points of interest in pedestrian navigation. We wanted to evaluate how we can integrate AR into a software system for situation visualization like the DigLT and what the possible problems are. * Corresponding author 2. TECHNOLOGY

DigLT
The Fraunhofer IOSB develops the digital map table (DigLT) 1 . The software consists of a server (DigLT Core ) that provides all services and geo-data and a web-based application (DigLT Web ) that allows multiple users to work at the same situation at the same time. A role concept makes it possible to create, visualize and analyze complex situations as a team. As a web-application, the software supports a multitude of different devices, from a smartphone and tablet to laptop and desktop PCs up to large table displays and video walls. The system is easy to adapt to various tasks. The DigLT has a Virtual Reality (VR) version, where multiple users can meet at a virtual table for a briefing. The DigLT supports multiple layers of geodata like maps or points of interest, and users can place markers or plan routes. An AR extension could improve the visualization on a mobile device if the marker is not only shown on the map but also visible in the real world using AR. Figure 1 shows the user interface of the DigLT. On the left, the default desktop view, on the right on a mobile device in portrait mode is shown.

Global Navigation Satellite System
The Global Navigation Satellite System (GNSS) consists of all globally available navigation satellites. It includes the US Navstar GPS, the Russian GLONASS, the European Galileo and China's BeiDou satellites. Already over 30% of all GNSS receivers can receive signals of all available satellite systems  (European GNSS Supervisory Authority, 2018, p. 20). The GNSS positioning system ist based on distance measurement by comparing the transmission and reception times of satellite signals. A position on the globe can be acquired by using the signals of four satellites. The distance measurement is based on the assumption that the satellite signals move at the speed of light, meaning that the clocks in the transmitter and receiver should be synchronized as precisely as possible. Because of that, even a deviation of a millisecond causes an error of almost 300 kilometers. In contrast to the satellites, smartphones do not have high-precision atomic clocks. An essential part of the positioning is the most accurate recording of the reception time of the signals at the receiver. In smartphones, the C/A-Code, Civilian Access Code is used for this purpose. Because of the signal length, the best-case scenario in the positioning of current smartphones is about 3m.
Apart from errors due to timing inaccuracies, there are other types of errors that can affect GNSS precision. One source of error in the positioning is the multipath error. This error occurs in environments in which the direct view of the satellites is obscured. Susceptible places for multipath errors are places like cities, forests, valleys, and canyons. The error is caused by the reflection of the satellite signals, which then cover a longer distance or arrive twice and thus the result is of the positioning is altered. There are possibilities to distinguish a reflected from a direct signal. GNSS signals usually are right-handed circularly polarized. The direction of this polarization changes after a reflection. This effect allows signals, which have experienced an odd number of reflections to be excluded due to the direction of their polarization (Sickle, 2020). Besides multipath, there are other sources of error in positioning. The ionosphere is a layer of ionized plasma of electrons, which is located between 50 and 1000km above the earth's surface. It creates a delay in the satellite signals. How strong this effect occurs depends mostly on the angle of the satellite to the planet. The lower the satellite and the more distance the signal has to travel in the ionosphere, the greater is the delay. Furthermore, the density of the electrons fluctuates due to the changing solar radiation, and other influences, which in turn changes the delay. Receivers that can receive several frequencies of the same satellite can correct this error partially. This can be achieved with a dual frequency GNSS receiver. While this functionality is standard for professional receivers, more and more smartphones have such a receiver. In addition to the ionosphere, the troposphere also delays the satellite signals, but this delay is not frequencydependent. A way to counter this is the Satellite Based Augmentation System (SBAS). The principle is, that ground stations are known reference points that send correction data to geostationary satellites, which in turn send them to the receiv-ers of the users. The support of positioning by correction data is often referred to as differential GPS (DGPS).

AR
The term augmented reality refers to the extension of reality with virtual content. According to the definition of Azuma (Azuma, 1997), an augmented reality system must have three characteristics: 1. the combination of the real world and virtual world 2. real-time interactivity 3. registration of the environment in three dimensions To enable AR on a device, it needs at least one screen that supports displaying the real environment and the virtual contents placed on top of it, a camera that captures the environment and a processor that must be powerful enough to handle the necessary calculations in real-time. Due to the strong growth in the last few years, the increased performance of processors and cameras in smartphones has made a significant contribution to the direct access to an AR capable device. Google's ARCore, a development environment for AR applications, requires at least Android version 7.0. According to a statistic from statista.com in May 2019, 57.9% of all Android smartphones AR capable. For detecting the position of the device in three-dimensional space, ARCore uses the image data of the camera and the acceleration data of the IMU 2 . From this data, an algorithm generates an approximation of the position and orientation of the device for a virtual camera. The more precisely the cameras are superimposed, the more credibly and more immersive is the AR experience. Thus the virtual object seems to be fixed to the real environment. How exact the recognition is done depends strongly on the illumination of the object and nature of the environment. The higher the contrast of the surfaces and objects in the field of vision of the camera, the better the recognition calculation works. ARCore offers a function that displays all identified feature points and surfaces of the environment. Figure 2 shows in purple the detected floor area and the detected feature points as white dots. The more structured a surface is, the better and the more feature points ARCore can detect.

Magnetic Compass
The magnetic compass in smartphones consists of three Hall sensors, which measure the magnetic field strength in the environment. A Hall sensor measures the magnetic field strength in only one direction. Therefore you need three sensors, which are arranged at a 90°angle to each other, to measurement in any orientation of the smartphone. Every smartphone contains several permanent magnets in built-in speakers and other small parts. The measurement results of the Hall sensors are therefore always the sum of the magnetic field of the internal permanent magnets and the external magnetic field of the earth (ST Microelectronics, 2010). Since the position of the permanent magnets relative to the sensors does not change, they produce a constant offset, which is compensated by a calibration of the sensors. The calibration is performed by rotating the smartphone around each of its axes. Since the strength of the magnetic field on earth varies constantly and depends on location, in practice, the calibration must be repeated before each use of the compass. After calibration, the compasses in smartphones should have an accuracy that is completely sufficient for the usual scenarios. calibration is absolutely essential, otherwise, the values are extremely unreliable. The application was created with the Unity Engine. For detecting the position of the smartphone in the room, we used AR-Core developed by Google and integrated into the Unity SDK. A similar technology, ARKit 3 , exists for Apple devices. Furthermore, we used two assets from the Unity Assetstore: The In-App Web Browser 4 asset and the GPS + AR Location 5 asset. The browsers asset serves as a communication interface to the DigLT. It includes a simple web browser that allows the operation of the DigLT within the application without changing the window. The GPS + AR Location Asset contains the necessary logic for placing objects at geographic coordinates. We developed a special plugin with Android Studio to directly access the geographical coordinates of the smartphone. Figure 3 shows the structure of the mobile app.

Android Plugin
Unity already contains a function to access the geographic coordinates of the device. The output of this function is provided through the Float format. Since Unity uses floats with C# 32bit precision, the output is limited to seven digits. With six decimal places, a latitude has an inaccuracy of 11cm, with five it is 1.1m. In Germany, we could always have an inaccuracy when using the internal Unity function of 1.1m in latitude and about 8cm in longitude. We developed the android plugin to completely exclude any loss of accuracy due to the data type and to access the coordinates in double format. The plugin also allows access to all received, unprocessed GNSS signals. It is the basic building block for the development of an improved positioning algorithm, which could use different correction data from the Internet.

Communication with the DigLT
The DigLT Web is accessed via In-App Web Browser. It is a typical browser that is capable of running javascript and allows bidirectional communication between Unity and the webpage it displays. By integrating the already existing Web-Application, the android app has all the functionality and can extend it using AR. This makes it possible to not only display data that is received from the webpage, but also to add new markers by selecting them in AR.

AR + GPS Location Asset
To place an object at geographic coordinates, they have to be converted into coordinates that can be understood by Unity. To achieve this, a vector that is aligned to the north-south axis is calculated from the current geographic coordinates of the smartphone to the geographic coordinates where the object is placed. The Unity coordinate system in which the objects are placed is also aligned to the north-south axis using the compass. One meter in the real world corresponds to one meter in Unity. The Unity-Coordinates for an object to be placed are therefore the vector, added to the current Unit-Coordinates of the smartphone. Through this method, the compass is causing an error in positioning with increased distance to the placed object. The placed object rotates with the angle change of the compass around the position of the smartphone. In the application, this is noticeable as repositioning of the placed Objects, which is very perceptible during fast movements of the smartphone. The AR + GPS Location Asset includes various smoothing methods to make the display appear as reliable as possible. Without the smoothing methods, all objects appear to be jumping with the GNSS signal arriving every second. The objects jump a few centimeters to several meters. Also, the objects rotate around the user's position due to the shaking of the compass, which returns a new value for each frame. At a distance of 15 meters from the placed object, a jump of 15°already produces a deviation of four meters. Combined with the repositioning every second due to the incoming GNSS signal, this yields an extremely unstable impression of the application. In case an object is outside the field of view of the camera, a function has been implemented, which indicates the object position by an arrow at the edge of the screen.
The AR + GPS Location Asset was used in this work exclusively in "Ground Relative Mode". In this mode, the altitude is ignored and the marker is placed in an adjustable number of meters placed over the detected floor area. If no floor has been identified under the object yet, the object is placed on a set number of meters above the nearest detected floor area. If you are standing on a mountain slope and place the marker far away, it will, as shown in Figure 4a, hover in the air, because ground surfaces are only recognized in a few meters around the user. If you move closer to the marker, as shown in Figure 4b, it slides down to the now correctly detected ground surface. After returning to the starting point, as shown in Figure 4c, the marker remains on its lower detected ground surface.

Purpose of the evaluation
The goal of the evaluation is to find out how accurately and precisely a marker can be placed. For this purpose, we measured the deviation of the placed marker from geodetic reference points. The measurements also serve the purpose of getting a feeling for the practical suitability of the application.

Test Setup
We selected three locations for the measurements. Measuring site 1 has a publicly accessible GNSS test field with several ground markings whose coordinates are known to the millimeter. In a radius of about 100 meters are no trees or buildings, making this measurement location the least susceptible to multipath errors. Measuring site 2 has a geocache in a metal box whose position is known to the decimeter. This point is in the city center and is therefore prone to multipath errors. The behavior of the application in the forests should also be tested on measuring site 3. Since there is no precisely determined reference point in a wooded environment in the immediate vicinity, the coordinates of a prominent crossroads in a nearby forest area were determined using Google Maps. Due to the inaccuracy of the reference point, the measurement results from measurement location 3 cannot be used to make any statements about absolute accuracy. However, statements about reproducibility and precision are possible. We measured from three distances of 5, 10, and 15 meters to illustrate the influence of the compass. The distance from the measurement setup to the reference point was determined with a tape measure. Figure 5 shows the different measuring points. The lines drawn represent the two directions from which the measurements were taken. The measurements were performed with a Samsung S10e and Xiaomi Mi8. The Xiaomi Mi8 was used in addition to the Samsung because of its Dual Frequency GNSS. As shown in Figure 6, both smartphones were fixed on a camera tripod and not moved during the measurements. During the first measurements, we noticed that the screws in the self-made holder influence the compasses of the smartphones. In addition, the smartphones influenced each other through their magnetic components. The holder was therefore revised several times, glued joints replaced the screws and we placed the smartphones further apart. Before each measurement, the compasses of both smartphones where calibrated. Attention was paid that the movement was performed as uniformly and equally often as possible on both devices. The reference angle was taken from a standard analog magnetic compass, which was aligned with the smartphone holder. The displayed angle of this compass was entered into the application to have a precise value as a reference. The app placed two capsules into the scene: A green one at the position calculated with the corrected compass and a grey one using the internal compass of the smartphone. A virtual reference is placed on the real reference point in augmented reality to measure the distance between the reference point and the capsules places in geo-coordinates. Since the ground surfaces are often registered a few centimeters above or below the real ground, the virtual reference was placed from above as best as the localities allowed. The height above ground was not included in our measurements, but still has an impact on the visual impression. The measurement was started manually on both smartphones, by pressing the start buttons simultaneously. A measurement routine recorded all relevant values over two minutes and 30hz. This includes: 1. Deviation of the marker from the reference in north-south direction with the integrated compass in meters 2. Deviation of the marker from the reference in east-west direction with the integrated compass in meters 3. Deviation of the marker from the reference in north-south direction with the corrected compass in meters 4. Deviation of the marker from the reference in east-west direction with the corrected compass in meters 5. Compass direction of the internal compass in degrees 6. Compass direction of the reference measurement (this value remains constant throughout each measurement) 7. Longitude and latitude 8. X, Y, Z coordinates of the reference placed in augmented reality in the global coordinate system from Unity 9. X, Y, Z coordinates of the smartphone in Unity's global coordinate system 10. Timestamp in milliseconds 11. The estimated accuracy of the GNSS measurement from the Android system Furthermore, the app took a screenshot every 10 seconds.  Samsung S10e Figure 8 shows the distribution of the measuring points around the reference, which is located in the origin. The green dots, which represent the deviation without the influence of the compass, show a relatively small drift of about half a meter along both axes. The distribution can be explained by the drift in positioning by the GNSS. For the black dots, which represent the uncorrected positions of the marker, we see a circular deviation. This behavior can be explained by the compass, which makes the capsule rotate around the position of the smartphone. With the black dots, the drift by GNSS is much more difficult to detect, because they are additionally more scattered due to the jitter of the compass. A drift of the green marking was only noticed during the measurements at the second and third locations.

Results
Here, the green mark was repositioned nearly every second. Figure 9 shows a drift of about 12m along the north-south axis and about 7m along the east-west axis. This strong drift could be explained by the multipath error, which is potentially amplified by the surrounding trees at location 3 and the houses at site Figure 9. Site: 3, Direction: 2, Distance: 5m, Device: Samsung S10e XiaomiMi 8 Samsung S10e standard deviation, corrected marker, north-south axis 2.23m 2.7m standard deviation, corrected marker, north-south axis 2.27m 2.59m Table 1. Standard deviation of all measurements 2. The corrected marking on the Xiaomi differs slightly less overall than on the Samsung, as shown in Table 1. This result fits the expectation that the Xiaomi with its Dual-Frequency GNSS has somewhat higher accuracy and is less prone to the multipath error. The standard deviations of the distance to the reference point from the measurements with the integrated compass increase with increasing distance, while for the standard deviations of the measurements with the corrected compass, no such clear trend can be seen. This behavior is due to the jittering of the compass and the fact that the capsule is moved further with constant angular error but increasing distance. Figure 10 shows this behavior quite clearly. The green dots, which represent the deviation without the influence of the compass, are so closely grouped that they can hardly be identified as single measurements. The black points, which represent the deviation with the influence of the compass, are clearly divided into three groups, which represent the single measurements in the different distances. Due to the high angular error in these measurements, the circular paths on which the black dots move can be seen very clearly.
XiaomiMi 8 Samsung S10e standard deviation 43.9°46.1°m ean value 3.3°6°T able 2. Compass standard deviation and mean value The compass of both smartphones, and thus the entirety of the measurement data, has a standard deviation of more than 40°, as shown in Table 2. In addition, the mean values deviate from zero and are shifted clockwise by a few degrees on both smartphones.

CONCLUSION
The compass has a considerable influence on the accuracy of placement. Because of this, the precision error increases with larger distances. This is the case with both tested smartphones and there is no clear advantage of any of the devices. According to the findings of this work, the dual-frequency GNSS of Xiaomi makes a small positive difference in accuracy and precision. However, it is unlikely that this slight advantage enables usage scenarios, which are not possible with a single-frequency receiver. The augmented reality tracking of ARCore is very reliable and stable with rare exceptions. The GNSS of both smartphones delivered an expected accuracy, which is roughly in line with the common perception that current smartphones can be detected to an accuracy of about three to four meters under the right conditions. The performance was very dependent on the environment. The measurements showed that the combined inaccuracies of compass and GNSS leave an unpredictable overall impression during positioning. Under optimal conditions, it is possible to identify a car, a side of the road, or a small house by a set marker. Under bad conditions like in cities, or even if the compass points 40° in the wrong direction despite correctly performed calibration, it may not even be possible to identify an entire block of houses. For future work, we would like to develop new methods to increase the precision and stability of the integrated compass. A dynamic compass could be implemented that is using the GNSS position of the smartphone during movement could allow the application to derive a more accurate direction.