INTEGRITY - A TOPIC FOR PHOTOGRAMMETRY ?

: Photogrammetric methods and sensors like LIDAR, RADAR and cameras are becoming more and more important for new applications like highly automatic driving, since they enable capturing relative information of the ego vehicle w.r.t its environment. Integrity measure the trust that we can put in the navigation information of a system. The concept of integrity was ﬁrst developed for civil aviation and is linked to reliability concepts well known in geodesy and photogrammetry. Currently, the navigation community is discussing how to guarantee integrity for car navigation and multi-sensor systems. In this paper, we will give a short review on integrity concepts and on the current discussion of how to apply it to car navigation. We will discuss which role photogrammetry could play to solve the open issues in the integrity deﬁnition and monitoring for multi-sensor systems.


INTRODUCTION
The quality and trust that be can put in a vehicle navigation solution and derived parameters get more and more interest especially in the context of highly automated or autonomous driving. Thus, it is mandatory to develop a theoretical framework how to measure this trust or the corresponding uncertainty. Best practice examples are available from civil aviation (DO-229D, 2006, ICAO Doc 9849, 2005. There, a set of four quality indicators are defined namely: accuracy, integrity, continuity and availability. They are directly linked to the idea of a navigation service fulfilling correctly and safely the different navigational operations. In this context, integrity measures the ability of the navigation system to timely warn the user when predefined positional error thresholds so-called alert limits are transgressed (Hofmann-Wellenhof et al., 2003).
In the past 30 years, different algorithms have been investigated for monitoring integrity of GPS-based aircraft navigation, (DO-229D, 2006, ICAO Doc 9849, 2005. Integrity risk evaluation involves both assessing the fault detection and exclusion capability and quantifying the impact of undetected faults on position estimation, which is similar to the concepts of external and internal reliability well established in geodesy and photogrammetry, (Teunissen, 1990, Salzmann, 1991, Förstner, Wrobel, 2016. One established solution in GPS based positioning is Receiver Autonomous Integrity Monitoring (RAIM) cf. (Parkinson, Axelrad, 1988, Brown, Sturza, 1990, Brown, 1996, Blanch et al., 2012, Hegarty et al., 2015 which tries to exclude faulty system states autonomously and independently, i.e. only using redundant GPS observations. Global navigation satellite systems (GNSS) deliver absolute position and velocity as well as time information (P, V, T). In urban areas, the GNSS navigation performance is restricted due to signal obstructions and multipath. An overview on GNSS related studies for integrity in urban areas is given in (Zhu et al., 2018) and the cited references therein. Consequently, multisensor systems including GNSS, inertial measurement units (IMU), LIDAR and cameras are used to enhance the navigation performance. In addition and sometimes more important, * Corresponding author tasks such as environmental perception, relative location w.r.t landmarks or semantic interpretation of a scene should be performed. By integrating map or 3D environmental information, another partially independent source of information can be used to increase the navigation performance and contribute to integrity monitoring, (Toledo-Moreo et al., 2009, Quddus et al., 2006, but generally the map itself has no dedicated integrity information. Although well established procedures for integrity monitoring exist for GPS-based aircraft navigation, for sensors and fusion algorithms used in automotive navigation these concepts are still lacking.
Consequently, this paper will summarize some ideas on the current situation and will discuss directions for future developments as well as the potential role that photogrammetric research could play in order to contribute to integrity monitoring for multi-sensor systems.
The remainder of the paper is structured as follows. Sec.II introduces the concept of integrity as it is defined in aviation and discusses the issues when just simple transferring this concept to further applications. Next, multi-sensor systems for environmental perception and navigation are introduced with special focus on LIDAR and images. Current research on integrity concepts for multi-sensor systems are presented and potential research in the photogrammetric domain is highlighted.

CONCEPTS AND DEFINITION OF INTEGRITY
In this section, we will give the definitions of integrity and explain the underlying concept. A lot of effort has been put to develop the definitions and to adopt them to the state of the art navigation devices, especially in the context of GPS positioning.

Definitions
Integrity can be considered as one out of four quality measures, namely accuracy, integrity, continuity and availability that characterize the navigation performance during a specific operation, e.g. a landing approach, (DO-229D, 2006, ICAO Doc 9849, 2005. Accuracy measures the positional deviation between the unknown true, i.e. actual position and the one computed by the navigation system. Depending on the type of operation, specific predefined thresholds between these two quantities (alert limits, AL) should not be transgressed.
Integrity measures the capability of a navigation system to warn the uses in time when predefined quality criteria (namely the alert limit for the positional error) are transgressed, (Hegarty et al., 2017). Subsequently, the integrity risk IR is given by the probability that during the operation an error occurs that leads to an position error transgressing the specified threshold and that the user is not warned in time.
where P (M D) indicates the probability of missed detection of a failure, P (U F ) the probability of the occurrence of an unscheduled failure, P (M I |DF ) the probability of an incorrect failure isolation, and P (DF ) the probability of detected failure.
Integrity is violated either when an error occurred but it cannot be detected or but it cannot be correctly isolated. Making parallels to testing theory applied for outlier detection in least-squares estimation (Mikhail, Ackermann, 1982, Teunissen, 1990, Koch, 1999, Förstner, Wrobel, 2016, the first term (detection) refers to the overall model test while the second part (isolation) to a local observation test, which tries to identify an error in an individual observation. Subsequently, the integrity risk can be interpreted as a type I error, i.e. the rejection of a true null hypothesis, also referred to as false positive.
We should emphasize that only a small integrity risk is accepted in the range of 10 −6 to 10 −8 . This implies that very rarely events must be tackled like occurring 1/year.
Continuity defines the capability of a navigation systems to complete an operation (e.g. landing approach, docking of a ship, parking of a car, etc.) without sending any alarm. The system should provide the predefined accuracy and integrity during the period of operation. Subsequently, the continuity risk is defined as the probability of an alarm occurring during operation which would lead to an interruption of the service.
(2) where P (F A) denotes the probability of false alarm, P (U F ) the probability of an unscheduled failure, P (N I |DF ) the probability of no isolation of the fault, and P (DF ) the probability of a detected failure.
Cases of wrong alert can occur if a false alert is flagged, i.e. an error is detected while no error occurred or if an error occurred that can be detected but not isolated. The continuity risk represents a type II error, i.e. non-rejection of a false null hypothesis, which is also referred to as false negative. Thus, a compromise must be found between continuity and integrity since they are linked via the probability of misdetection and probability of false alarm.
Availability describes the percentage of time, during that the specified functionality, i.e. accuracy, integrity and continuity of navigation system is fulfilled.
Two aspects are worth noting in this definition of integrity: 1. The time component, i.e. we need real time algorithms and the user should be warned in a predefined time-span of a few seconds, so that he or she can change to an other navigation devise or start an action to bring the system back to save operation.
2. Specifications are needed. Navigation is interpreted as a sequences of well-defined elementary operations. We should be able to describe a nominal operation including allowed or accepted error thresholds. This implies that the navigation task can be subdivided into individual elementary tasks that can be well specified. For car navigation this could be a formalization of turning left or turning right as it is known from driving schools. This includes a representation of traffic regulations.

Integrity Monitoring
In order to monitor integrity, two further quantities are introduced: position errors PE and protection levels PL. Different computational approaches exist to assess the values, like: where A denotes the design matrix, i,.e. the matrix of partial derivatives of the functional relationship w.r.t. the parameter, P the observation weight matrix, δl the vector of errors in the observed minus computed values that is transferred by the so called slope matrix S onto the parameters yielding the vector of position errors δx. The position error describes the deviation between the true or actual position and that one computed by the navigation system. In real applications, this quantity is not accessible, since a ground truth is not available but only the computed position, (Betaille et al., 2007). Thus, for real time application as navigation we need to define the trust that be could put into our computed positioning solution.
In photogrammetry, a similar dilemma is present, e.g., when evaluating the performance of an image analysis software in real time applications, (Förstner, 1987). In addition, not only the geometrical correctness but also the semantic correct allocation of an object to a measurement must be ensured. Ground truth can only be obtained in post processing and from extensive labeling.
In aviation and in the context of RAIM, specific strategies are applied to overbound the unknown observation errors and to compute a protection level P L for the estimated position. It is worth noting that the need for a sound description of the contributing observation and system errors is common to all integrity monitoring concepts, which is similar to the approach of (JCGM 101, 2008). In general a quadratic summation of different error contributions to the GNSS observation, like e.g., from tropospheric or ionospheric propagation effects are accounted. A normal distribution is assumed for each individual effect and the error terms are given as variances σ 2 i . Finally, the effects are combined using variance-covariance propagation: where k is a confidence level, and Σ is assumed as diagonal matrix containing the resulting variances of the observations after variance-covariance propagation of the individual contributing effects.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition) Using a singular value decomposition of the design matrix A = U Λ V T , the link between the typically used, covariance-based DOP values as quality indicators and the protection levels can be established by and P L * = k ||δx|| ≤ ||S|| ||δl|| = λmax ||δl||.

Stanford Diagram
When testing navigation systems, the so called Stanford integrity diagram is often used to display the performance of the system, (Walter et al., 2003, Tossaint et al., 2007. In this case, a ground truth is given by the test set-up and thus both the position error and the protection levels can be evaluated. Four different cases are indicated in the Stanford diagram: • P L > AL: the navigation system is said being unavailable. The first decision is made based on the comparison between the specified alert limit AL and the computed protection level P L that indicates the confidence we put in our solution.
• In the opposite case, a second classification based on the position error P E, i.e. the deviation between the true position and the computed position is made.
-P E < AL and P E < P L: nominal operation -P E < AL and P E > P L: misleading information MI -P E > AL : hazardously misleading information HMI, which means an integrity risk.
Taking the example of an alert limit of AL = 10 m, the following figures depicts a simulated time series (44500 epochs) of absolute value of the vertical position error and the computed protection levels assuming a normal distributed observation error.
This time series is classified following the above-mentioned categories and represented in terms of the Stanford diagram. Each pixel represents the number of epochs or data points of the time series that are belonging to the respective category.

Current situation
Attempts are currently being made to transfer these established concepts to mobile platforms, especially for highly automated driving. However, we should keep in mind that the idea of a direct transfer may be only of partial success, since significant obstacles and currently unsolved challenges for a direct transfer of concepts from aviation are: 2. The much more difficult signal propagation conditions for GNSS in an urban environment compared to a flight due to signal obstructions leading to a weakened navigation geometry (e.g. expressed by larger PDOP values) and a smaller redundancy (e.g. a smaller number of visible satellites), cf. Fig. 3 and Fig. 4. 3. The required accuracies in the field of highly automated driving can only be achieved with carrier phase-based positioning. For this, integrity concepts for the positioning method Precise Point Positioning (PPP) and Network RTK must be developed first, which includes the uncertainty and trust that we can put in the entire data flow from the (worldwide) distributed GNSS reference stations via the communication channels to the rover positioning. First approaches are discussed in (Volckaert et al., 2019, Kreikenbohm et al., 2019. However the large complexity    (Zhu et al., 2018) and .
More generally, if we refer to the above introduced definitions, we have to verify how much information we have at hand concerning: 1. the time requirement: Even if there are some assessments available how long it take for a back-seat driver to (re-)take the control of the vehicle, the time to alert has to be specified. We do not further comment on this issue but will restrict ourselves to the positioning part.
2. the identification of the sequence of the guidance tasks: Is the navigation or the way how to get from A to B determined with a sequence of well defined maneuvers? If a path planning is used, the answer could be yes.
3. the specification of normal operation for each elementary navigational task: To our understanding the required information is currently not yet available.
Thus, an interesting alternative is to push the concepts of reliability and sensitivity well-established in geodesy and photogrammetry towards dynamic sensor networks, which could be seen as a representation of traffic nodes.

Role of photogrammetry and image analysis for integrity of multi-sensor systems
The latest approaches reported at conferences significantly expand the consideration of sensors in the integrity calculation. Initial considerations of integrity using cameras are made in, e.g. (Zhu et al., 2019a, Zhu et al., 2019b. For a better understanding of feature matching errors in navigation, (Yang et al., 2018) analyzed manually the occurrence of such errors and found that they are linked to the numbers of features extracted, the lightening conditions, the occurrence of repeated patterns (similarity). Basic concepts for statistical testing for images and computer vision are explained in (Förstner, 1994). Uncertain geometry and geometric relations are presented in (Förstner et al., 2000, Perwass et al., 2006. Application of outlier detection are reported in (Taglioretti et al., 2015).
Approaches to include classification and assignment uncertainty in LIDAR point clouds can be found in  or (Hassani et al., 2018). Different types of errors occurring in laser scanner data for a generation of a reference map based on pole shaped landmarks were discussed in (Hofmann, Brenner, 2009). They state that the false positives for pole extraction can be subdivides into three groups: (i) Varying point density in the data set which is a function of changes in speed, distance to the object, and vehicle attitude, (ii) Laser propagation effects, like signal reflections or penetration through windows. (iii) Incompletely detected poles.
The identification of observations that do not fit the model (e.g. finding outliers and incorrect correspondence) is much more challenging with LIDAR and camera observations, since the enumeration of the possible cases yields exponential increase of computation time in the number of observations. For this reason, heuristic techniques are typically used today, for example the Iterative Closest Point (ICP (Besl, McKay, 1992, Rusinkiewicz, Levoy, 2001) algorithm, or the Random Sampling Consensus (RANSAC, (Fischler, Bolles, 1981)) principle. Both procedures do not offer guarantees, since they may find only often local minima. RANSAC works globally, but as a randomized algorithm it cannot guarantee the optimality of the result.
Collaborative navigation is one interesting option to improve the navigation performance of each node and to increase the redundancy by additional sensors and the exchange between platforms. Collaborative position can be either understood as a sensor fusion on a single platform. (Kealy et al., 2015 or sharing data from multiple platforms (Fox et al., 1999, Bonnifait, 2017. (de Ponte Müller, 2017) investigated the properties of different sensors and their potentials for bridging other sensors and making synergies in collaboration. , Garcia Fernandez, Schön, 2019 developed a simulation framework for collaborative positioning based on an extended Kalman Filter framework. They underline the improvements when adding additional observations from aiding vehicles, hence extending the idea of classical geodetic networks. in addition extroperceptive sensors, like cameras and LIDAR, play an important role to improve the overall navigation performance. The studies showed also that not all positioning directions benefit in a similar way when investigating urban scenarios with almost parallel facades

Open Questions
What's about the quality and integrity of the other state parameters, like velocity, attitude and time, which are also available in navigation systems? If a prediction and time evolution of integrity becomes important, these parameters must be considered: i.e. starting with a given protection level and position error at one epoch we are able to predict their future values based on considerations of spatio-temporal similarity of error sources or based on collaboration between different vehicles or platforms. Assigning integrity to photogrammetric observations, this would also include attitude or platform orientation as an important link from the relative nature of these observations to a global frame.
Is a purely stochastic error bounding and error propagation reasonable or adequate? In Eq. 5 we presented the current strategy of bounding observation uncertainty by a sum of variances of the elementary effects. Remaining systematic errors are also treated stochastically. A more natural approach could be using interval mathematics that just bound the remaining errors (Jaulin et al., 2001), for GNSS (Schön, 2016, Schön, Kutterer, 2005, Drevelle, Bonnifai, 2009, Dbouk, Schön, 2020, for images (Rohou et al., 2017, Kenmogne et al., 2018, for car navigation (Wörner et al., 2016). A state propagation via Kalman filtering is described in (Xiong et al., 2013, Chen et al., 1997. A final decision about the adequate modeling strategy is complicated. To the author's understanding, it depends on the state of maturity of the sensor systems and corresponding analysis software used. If most of the error sources are well understood and modeled and corrected, then it is probably fair to assume that the remaining parts are random errors. Then, still the correct error distribution is needed to be assigned. Some error sources may not yield white noise stochastic processes for the observations but rather result in flicker or random walk properties as well a fractional noises related to atmospheric propagation.
How to the strengthen best the navigation geometry?
The key for monitoring and guaranteeing integrity is the strength of the navigation geometry, the redundancy in observations and the quality of the sensor data. Images and LIDAR cannot derive absolute positioning by its own, but they can significantly increase the geometric strength of the navigation problem especially in urban environments, cf. the discussion in (Garcia Fernandez, Schön, 2019). Are there sensors that can directly compensate the shortcomings (like RADAR, images and LIDAR concerning the condition of operations, illumination, operation during rain)? How prone are these sensors to outliers? Are outliers due to a basic physical limitation or is this behavior linked to the prize of the sensor and the maturity of the used technology? The benefits of investigation integrity are to make a clear inventory of all possible (also rarely) occurring effects; so to be forced to better understand the sensors and error models, to assess spurious effects and to describe an upper bound (not necessarily) in a purely stochastic sense.

CONCLUSIONS
In this paper, we motivated the notion of integrity which is currently intensively discussed in the navigation community, especially in the context of highly automated or autonomous driving. The transfer of the concepts of integrity as defined in aviation towards multi-sensor road applications or the completely new adequate definition are open issues. Geodesy and photogrammetry with the well-established concept of reliability at hand can significantly contribute to this discussion.
Observables like images or ranges from LIDAR or RADAR are important complementary observations w.r.t. GNSS for enabling an integrity monitoring for urban navigation with multisensor platforms. The processes needed to come up with an integrity concept include thoroughly checking the implicated sensors and their error budgets but not only on average performance but with special focus on maximum possible deviations, critical conditions and rarely occurring events at the 10 −9 level. These studies in themselves will be beneficial, since valuable knowledge will be generated.