HIERARCHICAL AERIAL TRIANGULATION OF OBLIQUE IMAGE DATA

: Oblique imaging sensor systems are becoming a standard for photogrammetric applications such as city modeling, with increasing need for frequently updated geospatial data in urban areas. The ground processing software has to consider and support this development, providing a highly performant workflow for projects that can consist of tens or hundreds of thousands of images in different view angles: nadir (vertical), left, right, backward, forward. This is a challenge also for the oblique aerial triangulation (AT), both the large number of images to process and the high amount of overlap in-between them, where more than 50 image measurements for a given ground point are common. At the same time, the large overlap even in a single view generally provides stable sub-blocks for each viewing direction and, therefore, allows for separating the oblique AT by view and performing a hierarchical approach: The nadir block is triangulated first, updating the exterior orientation parameters of all nadir images and providing adjusted ground points. This nadir reference can be used to tie the oblique views’ sub-blocks in subsequent AT runs to eventually provide adjusted exterior orientations for all images, with comparable accuracy to an integrated AT – but providing a significant performance gain. The hierarchical AT is a three-step approach: 1. adjustment of the nadir imagery, 2. adjustment of each oblique views independently by cardinal direction (East, West, North, South) while tying it to the nadir reference, and 3. computation of combined statistics and report. The paper details our AT approach and evaluates its parameterization for different oblique data sets, comparing results and performance to a fully integrated AT.


INTRODUCTION
The increasing need for accurate and frequently updated geospatial data in urban areas has resulted in the development of oblique sensor systems with increasing resolution and performance. Examples are the Leica RCD30 Oblique as well as the hybrid imaging and LiDAR sensor systems: Leica CityMapper and CityMapper-2. All of which feature five image sensors in nadir and oblique viewing directions: left, right, backward and forward; oblique mounting angles are generally 45º off-nadir. Their main application is city mapping, which requires capturing large image overlaps especially in densely built-up areas, typically ≥ 80 % forward and ≥ 60 % cross-strip (cp. Gerke et al., 2016).

Motivation
Post processing software, in our case Leica HxMap, must keep up with the development and provide a highly performant workflow for projects of many images with significant overlap. This includes an oblique aerial triangulation (AT), which has been a general field of recent research (Wiedemann & Moré, 2012;Rupnik et al., 2015;Gerke et al., 2016;Toschi et al., 2018). The AT performance can be generally improved in various ways, which include the numerical solution of a given bundle adjustment normal equation system (by matrix re-ordering, parallelized factorization or iterative solution) as well as sub-dividing a large project into sub-blocks to reduce memory requirements and runtime and further allow for distribution of AT runs on a cluster. _______________________________ * Corresponding author Our new hierarchical AT is a further optimization approach that separates oblique data sets by their viewing direction and carries out individual triangulation runs that only contain images for each of the individual views: nadir, East, West, North, South in case of the above-mentioned oblique sensor systems. The nadir view's sub-block is comparable to a classic photogrammetric block (however, with more overlap and, therefore, more stable); it is adjusted first and then acts as reference, to which oblique views are tied afterwards. Rigid registration is achieved by using a grid of most reliable, nadir-based ground points as connection points. As a result, adjusted exterior orientations will be provided for all images in the project, nadir and oblique. Compared to a single, integrated AT of the entire imagery, a total of five subsequent runs are carried out, reducing the number of observations and parameters and, accordingly, the size of the normal equation system to approximately 20 % of the full size (with some overhead for connecting the views), which obviously provides a significant benefit for performancewith a potential impact on geometric accuracy that is investigated and quantified within this paper.

Considerations and Discussion of Related Work
Classic nadir imagery is captured and processed based on an (nadir) AOI that is divided into flight sessions, strips, takes and images in topological order, both for the image take capture (exterior orientation) as well as ground location (products). This situation is substantially different for oblique imagery, especially if the four oblique footprints are significantly offset from the take's corresponding nadir one (cp. Figure 1). Figure 1. Example of Leica CityMapper-2S nadir and oblique ground coverage from one image take. Image data was captured over Munich, Germany, in 2021 (see also chapter 4).
Depending on forward overlap and side-lap, the same area on ground might be captured by nadir and oblique views offset by some 5 takes (along strip) or strips (across strip). This is illustrated in Figure 2. For oblique projects, the AOI is required to be covered by imagery from all viewing directions, e.g., to texture facades in a derived 3D city model. Therefore, areas on the edges, i.e., outside full five-view coverage, are usually not required for product generationwhich in turn means compromising the quality in these locations has practically no effect on final products. Figure 2. Illustration of an oblique sensor's block layout, with a central (usually large) five-view coverage and partial coverage near the block edges. If flown in East-West direction, left/right or backward/forward sensors images are looking North/South or East/West, respectively.
Regarding an oblique AT, we carried out an (unpublished) investigation on limiting the processing to the nadir-looking images anda stable sensor system providedpropagate the resulting corrections of the exterior orientation parameters of each image take to the corresponding oblique images. This has been described and successfully used earlier by Wiedemann & Moré (2012); however, with left and right oblique sensors' fields of view overlapping the corresponding nadir. This could save the oblique views' AT runs altogether, but in our case the approach does not fulfill accuracy requirements. The main reason for that is very likely the extrapolation error from a given nadir image extent to oblique images at a significant distance in our cases (see Figure 1). There needs to be some means of AT for such an oblique image configuration.
A hierarchical approach as presented here for the geometric AT is already provided in Leica HxMap for the relative radiometric adjustment (or "radiometric AT") to provide a seamless nadir mosaic first and then fit oblique views to that radiometric reference. See Gehrke and Beshah (2016) for a general description of the radiometric normalization approach.

HIERARCHICAL TRIANUGLATION APPROACH
The general purpose of an AT, whether integrated or hierarchical, is the update of exterior orientation parameters for all images in the block. Furthermore, it needs to provide meaningful statistics to judge the results' quality. Therefore, our hierarchical AT is implemented as three-step approach: 1. Adjustment of the nadir sub-block. 2. Adjustment of each oblique sub-block, tying it to the nadir reference. 3. Computation of combined statistics.
The hierarchical AT requires a common set of tie points, usually collected by automatic point matching (APM) on the full block.

Reference AT: Nadir View
The typical city modelling project with ≥ 80 % forward overlap provides 4-fold coverage of each ground point in a single flight strip (and view), along with ≥ 60 % side-lap up to 3 x 4 = 12 image measurements or rays per ground pointa very stable nadir block that can be triangulated as such. Results are (final) exterior orientation parameters for all nadir images as well as adjusted coordinates for all tie points. Ground control points provide a geodetic datum, which derived products are required to adhere to after the AT. In general, this goal is achieved by parameterizing a datum transformation between the images' GPS/IMU-based orientations, if present. Such a model must then be used in the nadir reference AT run, so it provides final adjusted orientations and tie points in the desired datum.

Oblique Views' ATs
The subsequent AT of any of the oblique views needs to tie the images to the nadir reference to eventually provide adjusted exterior orientations for all images in the project. Oblique subblocks in that regard are defined by the cardinal view (or world view: East, West, North, South) rather than by the oblique sensor installation, which would correspond to different cardinal views when capturing data in different flight directions. Each cardinal view provides continuous ground coverage and, therefore, a subblock that can be expected to be similarly stable as the nadir. The main challenge of the hierarchical AT is tying these oblique sub-blocks to the nadir at an accuracy level that is close to a fully integrated AT. We address this by introducing connection points, which consist of ground control (if available) and, to provide a rigid, local connection throughout, a significant number of tie points, with both types of points in the same datum after the nadir AT adjustment. Suitable tie points can be determined by their external reliability (Hinsken & Boehrer, 2010), which correlates with the number of image measurements: the more rays, the more reliable the point. Two-ray points should be avoided by any means because they are badly constrained. A meaningful threshold for the number of rays has to be found, essentially with the following types of options, considering the nadir overlap described before:

•
Points ≥ 3 rays, which is the minimum requirement.

•
Points ≥ 5 rays that would be in neighboring strips (at 80 % forward overlap), which results in favorable intersection angles, especially for vertical accuracy. • Points ≥ 10 rays, a set of "most reliable" points (with 80 % / 60 % overlaps).

•
Points evenly distributed across the block, with the locally largest number of rays.
The latter criteria can be achieved by dividing the block into a grid of cells, picking the point with most rays in each cellbased on the idea of Cell-Based Analysis (CBA). This concept has been originally introduced by Hinsken & Boehrer (2010) to provide localized quality criteria for an AT block; see example in Figure  3. Some of which criteria are derived from the most reliable tie points per cell, so these points can in turn be collected from such CBA.
Both connection points and control points are introduced as ground observations into the oblique AT. The above connection point options have been tested and compared with regard to accuracy and performance, which discussed in the next chapter.

Combined AT Results and Statistics
The main AT result is the adjusted orientation for each image, which is updated from each of the individual runs. Each of these runs provides statistical parameters that can and should be considered for detailed analysis. However, the analysis as well as quality control (QC) reporting in a production environment require overall statistics. An AT run is judged by several key parameters that include: σ0 a posteriori in comparison to σ0 a priori; number of iterations; percentage of blunders; residuals of the observationsimage measurements, control and check points, exterior orientation -; datum shift and boresight misalignment; and standard deviations of the computed parameters. Based on the results of the individual AT runs per view, error propagation and computation of control, check and tie points for the combined CBA set-up is required.

HIERARCHICAL TRIANUGLATION EVALUATION
For detailed analysis of the hierarchical AT in comparison to the integrated one, we use a Leica CityMapper data set captured at a nadir GSD of 0.123 m over Heilbronn, Germany, by a total of 3,050 images that cover a test field with 52 surveyed ground points, almost all of which measured in all five sensor views. Nadir overlap is 80 % forward and 60 % across strip. All ground points are used as check points in our evaluation, however, with an average datum shift removed after AT. Internal accuracy is judged by the standard deviation of the ground point computation, external accuracy results from the RMS errors of the coordinates. Further verification has been carried out on a number of different data sets, especially including one with less overlap to verify feasibility of the hierarchical AT in general. The example shown below is a Leica CityMapper block in Stillwater, Oklahoma, with 70 % forward overlap, which is significantly larger than Heilbronn, featuring 33,510 images in total at 0.050 m nadir GSD. There were no surveyed ground points available, so we performed a relative comparison of a large number of tie points between hierarchical and integrated AT, carried out for both Heilbronn and Stillwater. The verification includes different connection point selection criteria to eventually identify the best solution, predominantly in terms of accuracy but also considering the performance impact.

Internal Accuracy
For the Heilbronn block, the internal accuracy for different configurations of the hierarchical AT is shown in Table 1 in comparison to the integrated AT, which presumably provides the best possible overall accuracy. Results are separated by using all measurements from all views as well as rays from only single views, corresponding to the individual sub-block runs of the hierarchical AT. As expected, and also in line with previous research (Gerke et al., 2016), the best accuracy is achieved from the integrated AT:

AT / Rays
Using the large number of rays from all view angles, we achieve ~ 0.1 GSD horizontally and < 0.2 GSD vertically. However, the hierarchical AT comes very close, regardless of the underlying connection point selection method. For the nadir view, it stands out that the horizontal accuracy is superior to the vertical (0.1 GSD vs. 0.5 GSD), which is typical for nadir blocks, depending on sensor and flight configuration, i.e., baseline vs. height ratio. The accuracy retrieved from individual oblique views is generally larger (~ 1.0 GSD) due to the increased oblique object distances at otherwise identical baselines. Corresponding to the 45º view direction, horizontal and vertical numbers are similar. As expected, the absolute accuracy figures as shown in Table 2 are generally larger than the internal accuracy, ~ 0.3 GSD horizontally and ~ 0.2 GSD vertically, with similar, view-dependent behaviour. The hierarchical AT indicates to provide slightly better accuracy in the horizontal component than the integrated AT. In terms of the connection point selection, there is little difference between the approaches, with the exception of using only "most reliable" points, for which results are worstwhereas especially the CBA-based approach is superior. The case of running only the nadir AT sub-block and propagating its orientation corrections to oblique images is exemplarily added to Table 2 along with Direct Georeferencing without any AT. It is obvious that both cannot keep up with the AT results, regardless whether carried out hierarchically or fully integrated.

Relative Comparison
Different AT runs for the Heilbronn data set generally result in similar statistics, first of all σ0 a posteriori = 0.48 px for the integrated one, while varying between 0.42 px and 0.48 px between the individual runs of the hierarchical (for comparison: σ0 a priori = 0.5 px in all cases). The number of iterations within an AT run behaves similarly as well: 5 iterations for the integrated AT and 5-6 for each hierarchical run. A relative comparison on ground is provided by the differences of adjusted point coordinates, which can include or even solely be based on tie points and, thus, be very dense. Table 3 shows the results for the Heilbronn block, with RMS values derived from tie points in the central part with full five-view overlap, about 1,100 points ≥ 3 rays and 700 points ≥ 10 rays (exact point numbers differ slightly due to, e.g., blunder detection in different AT set-ups). In comparison to the GSD of 0.050 m, the differences are a larger than in Heilbronn (0.4-0.6 GSD here), but again at or below the level of standard deviations (RMS: σXY = 0.027 m and σZ = 0.031 m for points ≥ 3 rays). Hierarchical AT is feasible here, delivering similar results.

Connection Point Analysis
While the geometric accuracy discussed so far is clearly the driver for selecting suitable connection points that tie nadir and oblique sub-blocks, the actual number of points is of interest as well, because using fewer points means better performance (see below). The number of connection points in comparison to the theoretical maximum of all tie points that have at least 2 nadir rays is compared in Tables 5 and 6. The smallest selection of meaningful connection points is achieved by the CBA approach. Looking at ray-based selection, there are even more "most reliable" points ≥ 10 rays in Heilbronn than retrieved through the CBA while the resulting accuracy is worse. Selecting points ≥ 5 or even ≥ 3 rays results in significantly more points with no gain in overall accuracy.

HAT
In conclusion, the CBA-based point selection provides the best possible accuracy for a hierarchical AT along with the minimal amount of inherently evenly distributed connection points. The fact that Stillwater does not have any points with ≥ 10 nadir rays due to its smaller image overlap underlines the benefit of the CBA's locally best point as opposed to a fixed number of.

Performance
The main driver for the new hierarchical AT was performance improvement over a large integrated AT. Performance numbers for Heilbronn and Stillwater blocks are listed in Tables 7 and 8 It is very obvious that the hierarchical AT provides superior performance compared to the integrated AT. The initial nadir subblock run requires about 3 % of the runtime of the full-size AT, then we observe increased run-time for the oblique sub-blocks, largely depending on the number of connection pointswith the biggest increase for Stillwater's East or West views that use most connection points (cp. Tables 6 and 8). The dependency on the number of connection points supports the superiority of the CBAbased point selection also in terms of performance.

USAGE IN PRODUCTION
The hierarchical AT approach as described is implemented in Leica HxMap ground processing software and regularly used in for the oblique workflow. In this context, we discuss two typical production blocks captured by a Leica CityMapper-2S sensor system over Munich, Germany, in September 2021 and processed by Hexagon Geosystems. Ground resolution is very high: GSD = 0.035 m (Table 9).

Munich Central Munich South
Flight  Table 9. Munich production blocks' statistics.

AT Runs and Results
For the purpose of this investigation, the blocks have been processed in the now-default hierarchical approach also using the classic integrated AT. Control points have been introduced as such, with the datum transformation modelled as 3D shift. In a nutshell, the AT runs for both blocks achieved very similar results and statistics, e.g. σ0 a posteriori: 0.29 px vs. 0.27 px in Munich Central and 0.28 px vs. 0.27 px in Munich South (with σ0 a priori = 0.28 px / 0.27 px). Datum shifts are compared in Table 10, with very close results in both AT approacheseven though the datum shift is determined from all data of all views in the integrated compared to nadir-only in the hierarchical AT.

Internal Accuracy: CBA
The localized, internal accuracy of an AT is provided by the CBA as illustrated in Figure 3, derived from the hierarchical AT, and Figure 4, a difference plot compared to integrated AT. The CBA generally provides ground-based statistics such as ray intersection standard deviation: horizontally, vertical, and 3D from all or selected views.  According to the CBA derived from the hierarchical AT result, the internal accuracy when using all available view angles can be expected < 1 GSD in most of the central part of the data set, which is taken further into product generation; an exception is seen in the very West in a forested area (Kreuzlinger Forst, see Figure 5). Quality degrades outside the nadir coverage for the "hanging" oblique images (compare Figure 2). The integrated AT provides similar or slightly better internal accuracy, with the main differences limited to the border areas.

External Accuracy
Residuals of control and, most important for the evaluation of external accuracy, check points are shown in Tables 11 and 12. Here the hierarchical AT falls back compared the integrated AT, however only the horizontal component. Considering the small GSD of 0.035 m, it should be noted that the standard deviation of the ground point survey (modeled as 0.020 m in the AT) generally impacts the order of residuals.

Performance
Performance numbers for the Munich production blocks are listed in It should be noted the oblique sub-block runs are geometrically tied to nadir reference and have to be processed afterwards, but they are independent from each other and could theoretically be run in parallel. This would reduce the AT time to about 1 hour.

SUMMARY AND OUTLOOK
We outlined our new approach for a hierarchical AT of oblique images and showed its potential in terms of performance and accuracy; the latter is generally comparable to an integrated AT. A CBA-based approach for selecting reliable connection points to rigidly tie sub-blocks from different oblique views to the nadir reference has been discussed and analyzed. Based on that, exterior orientations retrieved from hierarchical AT can be used for product generation the same way, without compromising quality compared to integrated ATwith the benefit of a significant performance improvement. The hierarchical AT is implemented in Leica HxMap and used as the default work-flow for large oblique data sets in production.
Results have been presented and discussed.
Looking into the future, we expect a further increasing demand, resolution and frequency in oblique urban mapping, along with increasing requirements on processing performance, including the oblique AT. In that regard, it should be pointed out that the hierarchical processing can be carried out in addition to other AT optimization. A speed improvement of the hierarchical AT itself can be achieved by running the individual and independent oblique ATs in parallel on a cluster (after their common nadir view reference AT), which would reduce overall clock-time by another 60-70 %.