AN ITERATIVE APPROACH FOR SELF-CALIBRATING BUNDLE ADJUSTMENT

Obtaining accurate image interior and exterior orientations is the key to improve 3D measurement accuracy besides reliable and accurate image matching. A majority of cameras used for those tasks are non-metric cameras. Non-metric cameras commonly suffer various distortions. Generally, there are two ways to remove these distortions: 1) conducting prior camera calibration in a controlled environment; 2) applying self-calibrating bundle adjustment in the application environment. Both approaches have their advantages and disadvantages but one thing is common that there is no universal calibration model available so far which can remove all sorts of distortions on images and systemic errors of image orientations. Instead of developing additional calibration models for camera calibration and self-calibrating adjustment, this paper presents a novel approach which applies self-calibrating bundle adjustment in an iterative fashion: after performing a conventional self-calibrating bundle adjustment, the image coordinates of tie points are recalculated using the newly obtained self-calibration model coefficients, and the self-calibrating bundle adjustment is applied again in the hope that the remaining distortions and systematic errors will be reduced further within next a few iterations. Using a “virtual image” concept this iterative approach does not require to resample images or/and re-measure tie points during iterations, only costs a few additional iterations computational resource. Several trails under various application environments are conducted using this proposed iterative approach and the results indicate that not only the distortions can be reduced further but also image orientations become much stable after a few iterations.


INTRODUCTION
With the rapid development on digital imaging technology and imaging platforms, using stereo photogrammetry technique to conduct various 3D measurement and reconstruction becomes a popular choice and routine tasks for some applications. For example, using non-metric cameras mounted on UAV platforms to capture images is now very common for 3D measurement and reconstruction. Obtaining accurate image interior and exterior orientations is the key to improve 3D measurement accuracy besides accurate and reliable image matching (Alsadik, 2016). A majority of cameras used for those tasks are non-metric cameras. Non-metric cameras commonly suffer planimetric distortions, namely radial distortion, decentering distortion and the orthogonality and affinity (linear) distortion and so on. Generally, there are two ways to remove these distortions: 1) conducting prior camera calibration in a controlled environment (Feng, 2002;Zhang 2000;Tsai, 1986;Zhang, 2002); 2) applying selfcalibrating bundle adjustment in the application environment (Grodecki, Dial, 2003;Fraser et al., 2002;Jacobsen, 2003;Borlin et al., 2019). Both approaches have their advantages and disadvantages. For example, self-calibrating bundle adjustment does not require the cameras to be pre-calibrated therefore the calibration tasks can be performed in the application environments which is very attractive. Prior camera calibration can make the application process easier (bundle adjustment without self-calibration) but there are risks that the calibrated coefficients may be out-of-date or have been changed due to various reasons, and some experimental results show that the systematic errors still exist even if the camera distortions have been pre-calibrated.
Most applications/software (photogrammetry and computer vision packages) prefer to the self-calibrating bundle adjustment approach probably because the difficulty or not practical to perform prior camera calibration (for example, Pix4Dmapper, PhotoScan, Smart3D, VisualSFM). However, there is no universal calibration model available so far which can remove all sorts of distortions on images and systemic errors of image orientations. Various models have been developed in the past. Choosing suitable self-calibration models for particular applications and environments is a complicated analysis and process (Tecklenburg et al., 2014;Jia et al., 2015;Zhao et al., 2017;Snow et al., 1993;Fraser, 1997;Brown, 1971;Ebner, Fritsch, 1986;Gruen, 1996;Gruen, Beyer, 2001).
Instead of developing additional calibration models for camera calibration and self-calibrating adjustment, this paper presents a novel approach which applies self-calibrating bundle adjustment in an iterative fashion: after conducting a conventional selfcalibrating bundle adjustment, the image coordinates of tie points are re-calculated using the newly obtained self-calibration model coefficients, and the self-calibrating bundle adjustment is applied again in the hope that the remaining distortions and systematic errors will be reduced further within next a few iterations. Using a "virtual image" concept (the image only contains geometric extents without any visual information), this iterative approach does not require to resample images or/and remeasure tie points during iterations, only costs a few additional iterations computational resource. The details of the proposed approach are presented and several trials and results are described in the next sections.

ITERATIVE APPROACH
An iterative approach for camera calibration was developed by the author and others recently (Wang et al., 2019;Cui et al., 2018). This approach has been successfully applied to prior camera calibration situations -performing camera calibration in well control environments with sufficient control points. The basic idea was developed further and extended here to provide an iterative solution for self-calibrating bundle adjustment to suit more broad applications.
The main drawback of the conventional self-calibrating bundle adjustment (CSCBA) is that only a single self-calibration model is employed during the whole exercise. The proposed iterative approach for self-calibrating bundle adjustment assumes that there are still some considerable errors left after CSCBA and those errors can be corrected further using the same or a different self-calibration model. The new approach treats CSCBA as its units and each CSCBA unit can choose the same or a different self-calibration model. The iterative approach is illustrated in Figure 1. The current iteration's results (self-calibration model coefficients) are used to calculate the new image coordinates of tie points for the next iteration, therefore CSCBA can be applied again and again until the criteria of convergences are met. A CSCBA unit and "Virtual image coordinates" computation complete one iteration cycle. The differences between two iterations are the image coordinates of tie points. Normally after each iteration, the images need to be resampled and the tie points need to be re-measured/matched. However, these labor extensive tasks can be eliminated or avoided through the introduction of a "virtual image" concept. Instead of actually resampling the input image into a new image using the coefficients of the self-calibration model, only the extents of the new image (so called a virtual image) are calculated and only the coordinates of tie points on the virtual image are calculated. Using this idea the new image coordinates of tie points can be calculated very easily and quickly, and most importantly, very accurately, therefore makes the iteration approach very efficient.
Assuming after a few iterations, the criteria of convergences are met, multiple sets of coefficients of self-calibration models together with image orientation parameters have been generated. A final resampling process may be required to generate distortion-free images. This is achieved through applying the above virtual image concept to calculate the image coordinates of each pixel on the next virtual image extents until the final virtual image extents are reached. Because the final image colors/intensities are only to be resampled once therefore the iterative approach can maximally preserve the original image qualities if the convolution kernel is chosen carefully. Self-calibrating Bundle Adjustment and Model means the selected self-calibration model). The left diagram is Iterative Approach Configuration One: the different self-calibration models can be applied alternatively among iterations; the right diagram shows Iterative Approach Configuration Two: the different self-calibration models can be applied independently in their own iterations.
Various configurations of how CSCBA units fit into the iteration scheme can be designed based on how many self-calibration models to be used and how complicate they are. Figure 1 shows two kinds of configurations which could be commonly adapted. In Configuration One different self-calibration models are chosen for sequential CSCBA units, while in Configuration Two Model A is iterated first until stable, then Model B is iterated until the finial satisfied results are derived. For demonstration purpose, Australis model (Snow et al., 1993, Fraser, 1997 is used as the planimetric distortions self-calibration model throughout this paper. Australis model can be briefly described using Equation 1. { ∆ = ̅ 1 2 + ̅ 2 4 + ̅ 3 6 + 1 ( 2 + 2 ̅ 2 ) + 2 2 ̅̅̅ + 1 ̅ + 2 ̅ ∆ = ̅ 1 2 + ̅ 2 4 + ̅ 3 6 + 1 ( 2 + 2 ̅ 2 ) + 2 2 ̅̅̅ (1) where k1, k2, k3 represent the radial distortions, p1, p2 represent the decentering distortions and b1, b2 represent the orthogonality and affinity (linear) distortions, xo, yo are the image coordinates of the principal point, x, y are the image coordinates of any pixel, r is the radial distance (from the principal point).
For non-metric cameras, the focal length and the principal point are usually unknown. Those three coefficients are estimated using an interior calibration model.
Besides self-calibration model coefficients, the six image exterior orientation parameters can be estimated simultaneously.
If the tie points' spatial coordinates are also going to be estimated, the unknowns can be sorted into two groups and solved in an alternative fashion to avoid huge matrix handling during the computation process.
There are some flexible options in the iterative approach worth to be mentioned: 1. Self-calibration model can be the same or different in each iteration cycle (Iterative Approach Configuration One in Figure 1); 2. Different self-calibration models or part of the coefficients can be applied separately and multiple iterative solutions can be integrated into one solution (Iterative Approach Configuration Two in Figure 1). This is particularly worthwhile for some situations where the selfcalibration model coefficients could be highly correlated. For example, it was our experience that interior orientation parameters are highly correlated to some Australis model coefficients if all of them are put into one model. In the following experiments, the interior orientation model is applied independently first and Australis model is applied afterwards; 3. The iterative approach suits both controlled network or free network (only free networks were tested in the following experiments); 4. Outlier detection and robust estimate techniques could be implemented during iterations (for simplicity assumed no outliers in the following experiments).

RESULTS
In order to demonstrate the effectiveness of the proposed iterative self-calibrating bundle adjustment method, three trials were conducted. The camera information and image parameters are listed in Table 1. Trail A is a close-range photogrammetric application using a Canon EOS 5Dr full-frame DSLR camera; Trial B is an underwater close-range photogrammetric application using a GOPRO HEREO 5 BLACK camera; and Trial C is an aerial photogrammetric application using a SONY RX1R full-frame DSLR camera mounted on a UAV platform.
Because all three cameras are non-metric cameras, besides the planimetric distortions need to be removed, the interior orientation (the focal length, the principal point) also need to be estimated. From author's experiences, it's better to separate the interior orientation estimate from the planimetric distortions estimate, therefore, the Iterative Approach Configuration Two (see Figure 1) is employed in all three trials: the interior orientation model (this model includes three parameters: the corrections of focal length, the image coordinates of the principal point, df, dxo, dyo) was applied in the first iterative solution, then the planimetric distortions model (Australis model, coefficients are k1, k2, k3, p1, p2, b1, b2) was applied afterwards.

Trail A
Trail A use the camera of Canon EOS 5Dr, the camera information and image parameters are listed in Table 1. A consecutive relative orientation was initially performed to link all images together and using relative orientation results as the initial values for the bundle adjustment. The iterative approach firstly applies to obtain the interior parameters. The corrections of interior orientation parameters (df, dxo, dyo) and the root of mean squares (σ) in each iteration are shown in Table 2. It can be seen that after 5 iterations, the root of mean squares σ reaches 0.11 pixel, comparing to 49.10 pixels if only one iteration is applied (conventional method). The lines in Figure 2 clearly show that the corrections of interior orientation parameters trend to be very small and stable after 5 iterations.
The iterative approach then employed Australis model to estimate the planimetric distortions coefficients (k1, k2, k3, p1, p2, b1, b2) together with final image exterior orientation parameters. The planimetric distortions coefficients and the root of mean squares (σ) are shown in Table 3 and Figure 3. The lines in Figure  2 clearly show that the planimetric distortions coefficients trend to be very small and stable after 6 iterations. The results of this example successfully demonstrated that using a combination of two self-calibration models in the proposed iterative approach can adequately remove image distortions and recover the focal length and the principal.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-97-2020 | © Authors 2020. CC BY 4.0 License.    Although those 120 tie points in Trial A have accurate spatial coordinates pre-measured using a total station (with accuracy about 2mm), the above exercise deliberately avoided to use control information during the process, only performed the free network adjustment. In order to check the absolute spatial accuracies in each iteration, those 120 tie points' spatial coordinates (free networks) were transformed into the absolute coordinate system (controlled network) and the statistical comparisons are listed in Table 4. The RMS of coordinate spatial discrepancies (dXYZ) and mean values of those tie points clearly show the improvement after each iteration.

Trial B
In order to illustrate the possible broad applications of the proposed iterative approach, GOPRO HERO 5 BLACK camera was chosen for an underwater close-range stereo measurement application in Trial B. The camera information and image parameters are listed in Table 1. The procedure is exactly the same as Trial A (except no control points to be compared). The corrections of interior orientation parameters are shown in Table  5, and changes of those parameters are shown Figure 4. Table 6 and Figure 5 show the results of planimetric distortions coefficients (Australis model). In both iterative processes, the agreed solutions were reached after 4 iterations. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)  Even in underwater environment, the results clearly demonstrated iterative approach's effectiveness.

Trial C
The third experiment is a kind of common aerial photogrammetric application. In Trial C, 6 aerial consecutive stereo images were extracted from a survey, which were captured using a UAV. The camera information and image parameters are listed in Table 1. The procedure is exactly the same as Trial B. The corrections of interior orientation parameters are shown in Table 7, and changes of those parameters are shown Figure 6. Table 8 shows the results of planimetric distortions coefficients (Australis model). It is worth to mention that among three cameras used, the camera used in Trial C seems less distorted and this evidence was demonstrated through the results listed in Table 8 where the convergence criteria was met just after 2 iterations.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)

CONCLUSION
Instead of developing additional models for camera calibration and self-calibrating adjustment, or evaluating and making complicate decisions on choosing suitable models for certain applications, this paper proposes a novel iterative approach for self-calibrating bundle adjustment. The proposed iterative approach treats the conventional self-calibrating bundle adjustment as its units and each unit can choose its own selfcalibration model to form some flexible configurations for various requirements. The results from several experiments demonstrated that the proposed iterative approach can effectively and efficiently improve the accuracies of conventional self-calibrating bundle adjustment and stabilize the results within a few iterations. Although the idea and implementation of the proposed iterative approach is relatively simple and easy, it could avoid the complication of choosing self-calibration models and improve the triangulation accuracy. It is believed that better and stable accuracies of bundle adjustment could be obtained if the proposed iterative approach were adapted by existing self-calibrating bundle adjustment algorithms. The further development includes testing a large amount of images, detecting/removing outliers and automating the process further.