BUNDLE ADJUSTMENT WITH POLYNOMIAL POINT-TO-CAMERA DISTANCE DEPENDENT CORRECTIONS FOR UNDERWATER PHOTOGRAMMETRY

Uncontrolled refraction of optical rays in underwater photogrammetry is known to reduce its accuracy potential. Several strategies have been proposed aiming at restoring the accuracy to levels comparable with photogrammetry applied in air. These methods are mainly based on rigours modelling of the refraction phenomenon or empirical iterative refraction corrections. The authors of this contribution have proposed two mitigation strategies of image residuals systematic patterns in the image plane: (i) empirical weighting of image observations as function of their radial position; (ii) iterative look-up table corrections computed in a squared grid. Here, a novel approach is developed. It explicitly takes into account the object point-to-camera distance dependent error introduced by refraction in multimedia photogrammetry. A polynomial correction function is iteratively computed to correct the image residuals clustered in radial slices in the image plane as function of the point-to-camera distance. The effectiveness of the proposed method is demonstrated by simulations that allow to: (i) separate the geometric error under investigation from other effects not easily modellable and (ii) have reliable reference data against which to assess the accuracy of the result.


INTRODUCTION
Any measurement process is inevitably affected by errors. Observation residuals may be caused by random and gross errors, as well as by systematic errors which derive from an incorrect or incomplete functional model (Vlcek, 1969). The potentially attainable accuracy thus depends on the proper choice of the mathematical model and subsequent system calibration, which leads to a compensation of systematic errors. In photogrammetry, the standard systematic errors that go beyond the collinearity conditions are well understood and the Brown/Beyer (Gruen & Beyer, 2001) model of self-calibration has proved to compensate well those errors for digital cameras. However, it is well known that this model has strong limitations in multi-media applications, where optical rays travel across media with different refraction coefficients. The deviation from the standard pinhole camera model is caused by the bending of the light rays as the medium changes. By the way, this effect also exists in air and aerial triangulation, but it is far less significant and thus usually ignored in practice. Although this phenomenon is known, the simple pinhole camera model with Brown/Beyer distortion formulation is still the preferred approach in practice in underwater photogrammetry (Shortis, 2015). The induced residual systematic errors may appear as systematic residual patterns in image observations (Fig. 1) and produce deformations of the photogrammetric model in object space. This work builds on the studies presented in (Menna et al., 2018) and , further extending the proposed residual pattern mitigation approach. The article is structured as follows. The problem of camera calibration and systematic residual patterns in image observations is briefly presented. An overview of the investigation previously conducted (Menna et al., 2018, Menna Figure 1. Image residuals for a DSLR camera equipped with a dome port after standard self-calibration. The image format is 6016×4016 pixels

ON CAMERA CALIBRATION AND IMAGE RESIDUALS SYSTEMATIC PATTERNS
Camera calibration is essential for applications where accuracy matters, to reduce systematic errors introduced by the measurement and modelling processes. The Brown/Beyer (Gruen & Beyer, 2001) model of self-calibration is currently considered the standard, satisfactorily compensating for systematic errors in most typical cases in arial as well as close-range photogrammetry.

Figure 2.
Distance-dependent distortion. First row: points on the same straight line but at different distances from the camera entrance pupil are projected at the same radial position on the image plane. Second row: points on the same straight line but at different distance from the camera entrance pupil are projected at different radial position on the image plane.
When this standard model proves insufficient, other approaches have been proposed in the literature, such as collocation (Rampal, 1976) or finite elements (Lichti & Chapman, 1997;Reznicek & Luhmann, 2019;Tecklenburg et al., 2001). However, these approaches are computationally expensive and have not experienced a strong spread. Legendre algebraic polynomials, as generalization of the polynomial models proposed by Ebner (1976) and Grün (1978), and Fourier series were investigated to integrate the self-calibration model especially for analogue and large format digital cameras (Tang, 2013). Abnormal systematic effects in image residuals were recently highlighted in a DSLR camera equipped with inexpensive zoom kit lens (Detchev & Lichti, 2020). The cause of the observed patterns was identified in (most likely) manufacturing imperfections of inexpensive aspherical lens components. The study showed that plotting the residuals as function of image coordinates a parabolic effect is highlighted, which can be modelled by a set of second order polynomials. Systematic residuals in the image plane may not only be dependent on the position of the image observations within the sensor, being also a function of the point position in the object space (distance from the camera's perspective centre). This is the case, for example, with images acquired by a moving camera equipped with a rolling shutter sensor (Vautherin et al., 2016;Geyer et al., 2005). A point-to-camera distance dependent distortion also occurs in multimedia photogrammetry (Kotowski, 1988). In the latter examples, the image residuals after bundle adjustment do not necessarily show apparent systematic patterns. Rather, the residual function is dependent on the distance between the object point and the camera's perspective centre (Maas, 1992;Li et al., 1997;Telem andFilin, 2010, Mulsow, 2010).
It is known that a rigorous calibration approach in multimedia photogrammetry would require the correct mathematical formalisation of optical effects due to refraction. While the theoretical mathematical foundation is well established, in reality there are many factors that cannot be satisfactorily modelled and can lead to uncompensated systematic errors. For this reason, the most common practice in underwater photogrammetry is to use the standard photogrammetric model and calibrate the camera system under prevailing working conditions (Shortis, 2015). This way, the refraction effects are absorbed into the estimated orientation parameters (implicit modelling). However, in adhering to this practice, one must always bear in mind that the camera network is a critical issue. Only a highly redundant and geometrically strong camera network including nadir and oblique images would mitigate residual systematic errors. A less robust image configuration could result in a considerable increase in deformations in object space .

Distance-dependent distortion in underwater photogrammetry
Figure 2 exemplifies the distortion introduced by the presence of water in the case of a planar water/air separation interface, e.g. a camera with a flat port. Because of the presence of water and the separation surface, a point P in water is always projected on the image plane in p'' along the blue (refracted) optical rays. This position p'' is different from the position p' where the point would be projected following the red (non-refracted) optical ray if there were no water. The refracted and non-refracted image positions differ for the radial component Dr. In the case where the camera's projection centre O (entrance pupil) lies on the planar water/air interface, the introduced radial The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) component Dr is the same for any object point lying on the blue optical ray, regardless of its distance from the centre O. Under these conditions, the introduced radial effect can be effectively compensated by the Brown/Beyer (Gruen & Beyer, 2001) model. When the entrance pupil does not lie on the water/air separation surface, the radial component depends on the distance Z0 between the perspective centre O and the planar interface, the angle of incidence e and the distance Z of the point P from the perspective centre O (Kotowski, 1988;Luhmann et al., 2013). This dependency can be easily visualised in Figure 2. A point P1 in water is projected on the image plane in p'' along the blue (refracted) optical rays. Moving along the optical ray, we find the points P2 and P3 at different distances from the camera's perspective centre O. All these points would be projected in the same image position p'' if the distance Z0 were zero (perspective centre O on the water/air interface). On the contrary, for each point a different radial component Dr is introduced. This effect can only be partially modelled by increasing the principal distance and adjusted radial distortion coefficients. When the water/air separating surface is spherical (dome port) and the centre of perspective (lens entrance pupil) coincides with the centre of curvature of the surface (dome centre), the effect of refraction becomes practically negligible She et al., 2019;Menna et al., 2016;Kotowsky, 1988). However, as the optical centre moves away from the centre of curvature, the projections of the object points on the image plane undergo increasingly significant radial displacements, whose magnitudes increase as this misalignment increases. This radial component is also a function of the distance between the object point and the camera centre, just as in the case of a flat surface. Further details can be found in .

Dealing with refraction: rigours modelling, iterative refraction correction and mitigation of image residuals systematic patterns
Different mathematical models have been proposed to rigorously tackle the refraction effect in multimedia environments. Mass (1992,1995) explicitly models the introduced additional distortion effects. Kotowski (1988), Li et al. (1997), Telem & Filin (2010) and She et al. (2019) trace the optical path through the various interfaces. Jordt-Sedlazeck & Koch (2012) introduce the non-single view point interpretation. While it is universally recognised that it would be preferable not to neglect the effect of refraction by properly modelling the phenomenon, it is difficult to quantify the improvement achieved over the implicit modelling approach in real cases (Kahmen et al., 2020). This is due to factors, such as water turbidity, back scatter induced by suspended particles, optical aberrations introduced by the water/air interface, which are not explicitly modelled in the proposed models. Alternative methods aim at implicitly absorbing all the adverse effects based on iterative corrections to generate refraction-free images (Skarlatos & Agrafiotis, P., 2018) or depth maps , also supported by machine learning approaches for aerial image-based bathymetry mapping (Agrafiotis et al., 2020). In Menna et al. (2018), we investigated the combined effect of image quality degradation underwater and geometric errors caused by refraction in real cases and introduced a stochastic approach, consisting of radial weighting of image observations. In , we focused on the geometric components of the systematic errors induced by the unmodelled refraction effects through simulations. We proposed mitigation measures to reduce the systematic patterns in image observations and confirmed the importance of camera network geometry in reducing the deformation in the object space caused by the incomplete functional model. For flat ports, we demonstrated that the use of a stochastic approach, consisting of radial weighting of image observations, improves the accuracy in object space up to 50%. Also, we proposed mitigation measures using iterative look-up table corrections, based on the "Masson d'Autume method" (d'Autume, 1972), that can reduce the evident systematic residual patterns in the case of dome ports. Despite the improvements, the study confirmed that the distance dependency ( Figure 2) of the residuals still remains uncorrected.
Moving from this considerations, we have extended the study of the mitigation of systematic effects in image space, this time explicitly considering their dependence on the distance between object point and camera.

METHODOLOGY: ADDRESSING THE DISTANCE-DEPENDENT DISTORTION
The proposed correction method starts with a self-calibrating bundle adjustment solution using classic Brown/Beyer formulation. An example of the initial image residual plot as function of radial position on the sensor and point-to-camera distance is sown in Figure 3a. Image observations and corresponding residuals are then collected into radial slices of step size typically equal to 64 pixels (Figure 4a). Different slice dimensions can be adopted depending on the pattern behaviour (spatial frequency) and image observation density over the sensor. For each radial slice, a polynomial correction function is fitted on the image observation residuals as function of the pointto-camera distance (Figure 4a). For each observation, the correction is computed based on the fitted polynomial function.  Figure 4. Radial component of image residuals versus relative didtance from camera for a cell slice of 64 pixels at radial distances respectively of 3232 pixels before (a) and after (b) the correction. In red the estimated polynomial correction function.
At the next iteration, the corrected image observations are input to a new self-calibrating bundle adjustment with Brown/Beyer model. The process is iterated until the convergence criterium is reached (difference in the sigma naught ratio). Figure 3b shows the final image residual plot as function of radial position on the sensor and point-to-camera distance after correction. Figure 4b demonstrates how the applied correction reduces the image residuals and eliminates the point-to-camera distance trend in the same radial slice analysed in Figure 4a. The corrections applied at each observation are stored in a lookup table that might be used to resample the original images, as proposed in Song et al. (2019) and Agrafiotis et al. (2020). The correction method is implemented by the authors in MATLAB in the form of a plugin for DBAT software (Börlin & Grussenmeyer, 2013).

Simulated dataset and processing
The present investigation employs the same simulated dataset presented in . The choice of testing our methodology on simulated rather than real-case scenarios is driven by a double necessity: we only intend to address the geometric component of the residuals introduced by refraction, avoiding other sources of uncertainty that are not easily discernible. In addition, in simulations reference data is available for accuracy checking of the proposed method. The employed simulations are derived from a real case dataset (Table 1), collected on a coral reef within the Moorea Island Digital Ecosystem Avatar (IDEA) project (https://mooreaidea.ethz.ch/). The images were acquired by SCUBA divers with a Nikon D750 24Mpx full frame DSLR camera (pixel size ≈ 6 µm) mounting a 24 mm prime lens in a NiMAR pressure housing. The surveyed reef plot measures about 20x10m 2 , featuring a rough and hilly surface, at an average depth of about 12 m and with a depth variation of about 1.6 m. Results of the photogrammetric processing aiming at monitoring across time the reef evolution are discussed in Nocerino et al. (2019Nocerino et al. ( , 2020 From the full camera network, we extract a subset comprising a single round-trip nadir strip (130 images). This weak camera geometry allows to better emphasize the residual errors which we aim to compensate with the proposed method. From the photogrammetric processed data, a subsampled 3D point cloud (1point/10cm) is extracted and used as input to run the simulations. Image observations are generated using a refraction index of 1.34 and neglecting the water/air interface (port) according to the formula derived in . Lens distortions are fixed to zero and white noise is not added for the purpose of analyzing the sole effect of the refraction induced systematic errors. Four simulated datasets are created: A. FP0, entrance pupil on the planar water/air interface (flat port); B. FP30, entrance pupil at 30 mm from the flat port; C. FP100, entrance pupil at 100 mm from the flat port; D. DP+30, misaligned dome port with dome center 30 mm ahead the entrance pupil .
The chosen offset values of 30 mm and 100 mm both for flat and dome ports are quite large. But, although still possible, for example in the case of using wrong extension tubes, they are here used to stress the related systematic errors and proposed corrections. Minimal constraints bundle adjustments with self-calibration are performed in DBAT on the simulated image observations. As the systematic errors on the simulated dataset are expected to have only a radial component in image space, only radial distortion additional parameters are solved within the self-calibration solutions. Accuracy assessment is performed computing, through a similarity transformation, the difference (error) between the coordinates of triangulated image observations (sparse point cloud of tie points) and the original 3D point cloud used to generate the simulated image observations. 12.000 ± 3.74e-06 12 ± 3.09e-07 k1

RESULTS AND DISCUSSION
[mm -2 ] 9.90683e-05 ± 1.53e-09e-09 9.8891e-05 ± 1.26e-10 k2 [mm -4 ] -3.05246e-08 ± 6.88e-12 -3.047e-08 ± 5.68e-13 k3 [mm -6 ] 1.39758e-11 ± 1e-14 1.3764e-11 ± 8.29e-16 max distortion pixels -206.4 -205.9 RMSE residuals pixels 0.009 0.001 Table 2. Results of the bundle adjustment with self-calibration for the simulated dataset before and after the polynomial distancedepend correction correction showed the highest standard deviations and residual errors and where the image residuals RMSE has been reduced by approximatively the 93% post correction. Figure 5 reports the result of the accuracy assessment in Z (the weakest measurement direction) for the tie points estimated for all the simulated datasets. As with the internal assessment of bundle adjustment (Table 2), there is a clear improvement in the statistics, which in the most critical case reaches around 86% reduction in error in Z. Compared to the results obtained in , we note that the method adopted here is effective in reducing image residuals and errors in the object space with both misaligned dome port and planar surfaces (flat port) at different distances from the entrance pupil. The reason lies in the fact that the proposed new correction method explicitly addresses the pointto-camera-distance depend effect due to the refraction of optical rays.

CONCLUSIONS
In this work, we have addressed in detail the problem of the error introduced by refraction in multimedia photogrammetry as a function of point-to-camera distance. We showed how this error becomes more prominent as the distance between the camera perspective centre and the water/air interface increases. Systematic patterns of image observations residuals are a warning bell indicating that the employed calibration model is unable to fully model the physical phenomenon. However, a correction of residuals in the image plane alone does not necessarily completely eliminate the source of the systematic errors. Here, we have demonstrated with simulated data that only by explicitly considering the dependence of image residuals on the point-to-camera distance is it possible to significantly reduce image residuals and thus improve the accuracy potential. We must emphasize here that the systematic corrections applied are very smallin the subpixel domain. The simulations were instrumental in validating our proposed method, allowing us to separate the geometric error due to refraction from other effects not easily modellable (degradation of image quality). The next step will be to apply the proposed correction method to real cases. This will allow us to further improve our knowledge of the phenomenon of refraction in multimedia photogrammetry and better quantify the effect of the geometric component in the residual systematic errors on the attainable accuracy.