IMPLEMENTING FUNCTIONAL MODULARITY FOR PROCESSING OF GENERAL PHOTOGRAMMETRIC DATA WITH THE DAMPED BUNDLE ADJUSTMENT TOOLBOX (DBAT)

: The Damped Bundle Adjustment Toolbox (DBAT) is a free, open-source, toolbox for bundle adjustment. The purpose of DBAT is to provide an independent, open-source toolkit for statistically rigorous bundle adjustment computations. The capabilities include bundle adjustment, network analysis, point ﬁltering, forward intersection, spatial intersection, plotting functions, and computations of quality indicators such as posterior covariance estimates and parameter correlations. DBAT is written in the high-level Matlab language and includes several processing example ﬁles. The input formats have so far been restricted to PhotoModeler export ﬁles and Photoscan (Metashape) native ﬁles. Fine-tuning of the processing has so far required knowledge of the Matlab language. This paper describes the development of a scripting language based on the XML (eXtensible Markup Language) language that allow the user a ﬁne-grained control over what operations are applied to the input data, while keeping the needed programming skills at a minimum. Furthermore, the scripting language allows a wide range of input formats. Additionally, the XML format allows simple extension of the script ﬁle format both in terms of adding new operations, ﬁle formats, or adding parameters to existing operations. Overall, the script ﬁles will in principle allow DBAT to process any kind of photogrammetric input and should extend the usability of DBAT as a scientiﬁc and teaching tool for photogrammetric computations.


INTRODUCTION 1.1 Background
Bundle adjustment is a crucial part of photogrammetry and computer vision.Several commercial photogrammetric software include a dedicated module for the computation of bundle adjustment, e.g., Trimble Inpho and ERDAS (Lumban-Gaol et al., 2018).In other software, the bundle adjustment might be hidden in a more black-box manner.Open source solutions to the bundle adjustment problem also exist in the form of toolboxes and/or dedicated functions.Examples include the Apero module from the Apero-MicMac photogrammetric suite (Rupnik et al., 2017) or the DGAP developed by the University of Stuttgart (Cramer, 2006).From the computer vision domain, other open source options also exist, for example Sparse Bundle Adjustment (SBA) (Lourakis, Argyros, 2009), multi-core bundle adjustment (MCBA) (Wu et al., 2011), or the Bundler software (Snavely et al., 2008).Most of these solutions are written in a low-level language such as C or C++.
The Damped Bundle Adjustment Toolbox in Matlab (DBAT) is written in the high-level Matlab language (Börlin, Grussenmeyer, 2013a).One goal is to provide open-source, easy-to-use implementations of core photogrammetric computation, including computation of posterior statistical quality estimates.
Several demo examples and data sets are included in DBAT.However, unless only minor modifications of the examples are needed to suit the user needs, programming skills have been * Corresponding author required to use DBAT on the user's own data.Furthermore, the user has so far been restricted to either Photomodeler or Metashape (Photoscan) formats as input.

Aim
The aim of this paper is to present a modular, script-based, processing of photogrammetric data.The goal is to reduce the level of needed programming skills and allow for a wider range of input data sources.

Background
As the name suggests, the Damped Bundled Adjustment Toolbox started out as a toolbox to study different damping strategies known in non-linear optimisation (Börlin, Grussenmeyer, 2013a, Börlin, Grussenmeyer, 2013b).The source code has been available on GitHub since February 2014 1 .
A recent development was to modularise the bundle projection model to allow users to develop novel projection models for, e.g., non-standard lenses or underwater applications (Börlin et al., 2018, Menna et al., 2018, Börlin et al., 2019).

Current capabilities
The current DBAT release 2 includes the following capabilities:

Bundle adjustment
The bundle adjustment proper, with or without self-calibration.Fixed and weighted prior observations are supported, e.g., control points and camera stations, as are check points.The parameters to be estimated by the bundle are selectable at the parameter level, e.g., for individual camera parameters.Furthermore, the parameters can be block-invariant (the same for a whole block), image-variant (individual for each image), or anything in between.
Posterior covariance estimates are computed from the bundle result, including correlations and significance levels, point and image quality statistics.

Photogrammetric processing
Besides bundle adjustment, DBAT supports several other core photogrammetric computations, e.g., spatial resection, forward intersection, and absolute orientation.

Software compatibility DBAT can input data from
Photomodeler-style text export files and point tables.Furthermore, DBAT can read and post-process native Metashape (Photoscan) .pszfiles.

2.2.4
Network analysis and quality DBAT includes functions to filter points and analyse camera networks to detect network problems.The filter allows removal of points with low point count and/or intersection angles.The analysis functions can detect both structural rank problems, e.g., caused by missing observations or gaps in the network, and numerical problems, e.g., caused by weak networks.

2.2.5
Output and report generating DBAT can present the results graphically in 2D and 3D and in text files.The available 2D plots present the raw data (Figure 1) or image and point quality statistics, such as image coverage, ray count, ray intersection angles, posterior standard deviations, etc., (figures 2-3).Other 2D plots illustrate the evolution of the bundle process (Figure 4).The 3D plots either show the final camera network or the evolution of the camera network and OP parameters during the bundle process (Figure 5).
The text output is mainly a PhotoModeler-style text summary report file that contain the processing input and setup, any detected problems, the estimated values and posterior covariances.Any detected high correlations among the estimated parameters are also listed.
2 DBAT v0.8.5.1, released January 13, 2019.Listing 1: The metadata section is intended for project information but is extendable to contain any structured information.

SCRIPT-BASED PROCESSING
Script-based processing uses a single DBAT function rundbatscript to load and execute a script.The script is written in the XML language (extensible markup language).Advantages of the XML file format include that they are human-readable, structured, and can easily be extended.Furthermore, since they follow the XML standard, other software may be used for searching or organising the files.
The DBAT XML file contain four major sections: metadata, input, processing, and output.

Metadata
The metadata section of the XML file is intended for project information, software and hardware that were used to obtain the measurements, the image source, etc.The only DBAT processing of the metadata is to write it in the report file.For an example of a metadata section, see Listing 1.

Input
The input section of a DBAT XML file contain the data and data sources needed for the processing.The main subsections include camera information, control and check information, image information, image measurements and other prior observations.

Camera information
The camera information is specified in XML format and can either be present in the main XML file or in a separate file.The basic camera information include the camera name, the camera unit, the sensor and image sizes, and the the nominal focal length.The camera unit specifies the unit used for the internal parameters, typically mm or pixels.For a calibrated camera, the estimated values of the interior orientation parameters are also specified.The aspect parameter can either be computed from the sensor and image sizes or specified directly, in which case the sensor width is computed to match the aspect.The projection model specify what interpretation -Photogrammetry or Computer Visionof the (Brown, 1971) lens distortion model is used (Börlin et al., 2019), together with the number of radial and tangential coefficients (nK and nP, respectively).Calibrated distortion coefficients are listed directly (K and P, respectively).See Listing 2 for some examples.Listing 2: Camera information can be specified either in a separate XML file (top example) or directly in the main XML file (middle example).If the camera is calibrated, the estimated interior parameters are also specified (bottom example).

Control and check information
The control and check information is expected to be listed in "commaseparated" data files, although it is possible to specify other separator characters.A format string allows for flexibility in the format used in the data file and is specified as an argument to the file directive (see Listing 3).The typical information to include is the point id, a label, the point coordinates and the coordinate uncertainties.The uncertainties can be specified per coordinate (sx, sy, sz) or jointly (sxy, sxyz).
The control and check point sections can contain filter directives.This enables a single, master, file to be used as a joint control and check point source.Listing 5: The image points are read from a comma-separated text data file.In the format string, im and id refers to the image id and point id, respectively.The uncertainties can be present in the data files (top example) or specified via attributes to the file directive.Uncertainty values may be overridden by a combination of the ignored format string and uncertainty attributes (bottom example).

Image information
The images information block uses a similar file directive and format string as the control information block (see Listing 4).The typical information is the image id and the image path, although only the image id is necessary for the processing.The image path name is only necessary if the images are to be visualised.

Image measurements
The image measurements are specified in the image pts subsection and follows the same pattern as the previous (see Listing 5).The typical information in the format string is the image id, the point id, the image coordinates, and the measurement uncertainty.Multiple files may be specified as sources by multiple file directives.The uncertainty values in the text files may be overridden by a combination of the ignored format string and file attributes sx, sy, or sxy.

Prior observations
Prior EO observations can be specified via the prior eo directive (see Listing 6).Furthermore, if pre-computed initial EO or OP values are to be used by the bundle, the data sources are specified via the initial eo and initial op directives, respectively.
1. Apply a sanity check on the input.
(a) Optionally, filter points on low ray count.

Set up initial values for the bundle.
(a) Optionally, apply secondary sanity checks and/or filtering given the initial values.
3. Specify what parameters the bundle should estimate.

Sanity checks
The sanity checks can be used to catch blunders in the data files, poor initial values, and network problems.The supported sanity check operations are check ray count Check that no point has too few rays.Can be used to catch, e.g., missing points in the data files.
check ray angles Check that no point has a too small ray intersection angle.Requires initial or estimated values for the camera and object point positions.Can be used to understand why a bundle fails.
check projection Check whether the object points are projected outside the images.If used on initial values and a wide tolerance, the operation can catch grossly incorrect initial values.If used on the final values and a strict tolerance, the operation can detect object points that have converged to an incorrect position behind a camera.
check structural rank Check whether the structural rank of the Jacobian (design matrix) is too low.Can be used to, e.g., detect holes in the network.
check numerical rank Check whether the numerical rank of the Jacobian (design matrix) is too low.Can be used as a debugging tool to understand why a bundle operation failed.

Filtering
The filter points operation can be applied to remove points that have too few rays and/or a too small intersection angle.The latter operation requires that the EO and OP positions have been estimated.

Set initial values
The initial values can be set by specifying constant or pre-loaded values or as a result of performing a photogrammetric computation.As a special case, if the datum is not set by the control points, the set datum operation can set the datum.Currently, only the dependent relative orientation is supported.

Execute the bundle
The bundle adjustment operation executes the bundle adjustment.This is typically the last operation specified.

A full operations example
Listing 7 shows the full operational flow for a camera calibration script.Since the camera calibration situation is assumed to be reasonably controlled, only a single sanity test is included to catch major blunders, such as to include the wrong files.For less controlled settings, more sanity checks and/or filtering might be necessary.Listing 7: The full operation sequence for a camera calibration.After an initial ray count sanity check, the camera constant is initialised to the nominal focal length.The other parameters are set to default values (principal point at the image center, unit aspect, zero skew and zero lens distortion).The initial EO parameters are computed by spatial resection, followed by forward intersection to set the initial object point coordinates.All OP, EO, and IO parameters except skew are estimated by the bundle.

Output
The output section contain information about what kind of results are of interest for the user.This includes different kinds of plots and files.The plotting can either show some quality parameters graphically, e.g., image coverage, or show how the parameter estimates have evolved during the bundle iterations, see section 2.2.5.Output files include report files that summarise the processing, DBAT XML camera files to store calibrated cameras, or comma-separated table files that can be imported into other software.

Data integrity
During processing of all input text data files, each non-blank, non-comment line in the data file is expected to match the format string.To reduce the probability of accidentally processing bad data, any mismatch between the number of expected and actual fields triggers an error.

File structure
The input and output sections may have an attribute base dir to specify a common base directory for the files.Non-absolute path names within the same section are assumed to be relative to the specified base directory.This simplifies grouping of data files while retaining flexibility for special cases.

Language extensions
The combination of the XML language and the Matlab language makes it easy to extend the DBAT scripting language.An extension with a new operation requires adding two things: A Matlab function that performs the operation and adding the name of the operation to the list of known operations of the

Figure 1 :
Figure 1: An input image with measured coordinates.

Figure 3 :
Figure 2: Image and external orientation statistics.Point count

Figure 5 :
Figure 4: Parameter evolution during the bundle iterations.
The images subsection refers to a separate text file.
The set initial values directive allows the user to set individual or groups of parameters to specific values or to use pre-loaded values.An alternative is to use the spatial resection and/or forward intersection operations.Spatial resection requires initial values of the IO parameters and measurements of control points.Forward intersection requires initial values of the IO and EO parameters.3.3.4Specifyparameters to estimateThe parameters to estimate are set by the set bundle estimate params directive.It allows the user to specify exactly parameters should be estimated by the bundle and what parameters should be treated as fixed.A standard bundle would estimate the EO and OP parameters only.A self-calibration project would also estimate some or all IO parameters.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W17, 2019 6th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 2-3 December 2019, Strasbourg, France script parser.Any parameters to the function can be specified as XML attributes to the operation.Adding a parameter to an existing function would only require the latter.Similarly, extending the input capabilities may simply require adding a keyword to the format string and support for the keyword in the corresponding Matlab function.The modularity provided by the scripting allows a simple setup of the DBAT processing with little to no programming knowledge.At the same time, the language provides flexibility of what parameters to estimate or what operations to perform and in what order.Furthermore, the ability to read text files with a general format enables input from many sources.The combination of the XML and Matlab languages makes it relatively easy to add new function or file format to the DBAT scripting language.These qualities suggest that DBAT XML scripting should be useful both for users that want to independently verify a black-box computation by another software, or teach students the inner workings of the bundle adjustment process.The modularity concept is particularly interesting for teaching purposes, as it enables a step-by-step learning of the photogrammetric workflow in lieu of a one-click black-box solution.The implementation of the XML file also presents a simple interface with which the workflow may be explained systematically.APPENDIXThis appendix presents a full example script for camera calibration.The example includes the main XML file and the associated text files.The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W17, 2019 6th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 2-3 December 2019, Strasbourg, France