EVALUATION OF INTERIOR ORIENTATION MODELLING FOR CAMERAS WITH ASPHERIC LENSES AND IMAGE PRE-PROCESSING WITH SPECIAL EMPHASIS TO SFM RECONSTRUCTION

For optical 3D measurements in close-range and UAV applications, the modelling of interior orientation is of superior importance in order to subsequently allow for high precision and accuracy in geometric 3D reconstruction. Nowadays, modern camera systems are often used for optical 3D measurements due to UAV payloads and economic purposes. They are constructed of aspheric and spherical lens combinations and include image pre-processing like low-pass filtering or internal distortion corrections that may lead to effects in image space not being considered with the standard interior orientation models. With a variety of structure-from-motion (SfM) data sets, four typical systematic patterns of residuals could be observed. These investigations focus on the evaluation of interior orientation modelling with respect to minimising systematics given in image space after bundle adjustment. The influences are evaluated with respect to interior and exterior orientation parameter changes and their correlations as well as the impact in object space. With the variety of data sets, camera/lens/platform configurations and pre-processing influences, these investigations indicate a number of different behaviours. Some specific advices in the usage of extended interior orientation models, like Fourier series, could be derived for a selection of the data sets. Significant reductions of image space systematics are achieved. Even though increasing standard deviations and correlations for the interior orientation parameters are a consequence, improvements in object space precision and image space reliability could be reached.


INTRODUCTION
For optical 3D measurements in close-range and UAV applications, the modelling of the interior orientation is of superior importance in order to subsequently allow for high precision and accuracy in geometric 3D reconstruction and modelling. In general, a standard 10-parameter model based on Brown (1966) and Brown (1971), summarised in Luhmann et al. (2019), is introduced to the mathematical implementation. This standard model is based on the thin prism theory for spherical lens constructions, as they can be found in most high-quality lenses and especially in the past decades. With high-quality cameras and lenses used in combination with close-range image bundles of signalised objects, as they are mostly found in optical 3D metrology, this model allows for an adequate calculation of the image ray deflection with respect to the resulting accuracy levels. For optical 3D measurements, focussing on 3D surface reconstruction with structure-from-motion (SfM) techniques, it can be found that not necessarily high-quality cameras and lenses are used. It is rather a question of applicable payload in UAV photogrammetry and economic purposes. Besides, some modern lenses are constructed of aspheric and spherical lens combinations, not even manufactured of glass and being more instable. Furthermore, camera systems include image preprocessing like low-pass filtering or internal distortion corrections. This might lead to effects not being considered with the standard model for interior orientation.
Additional correction parameters are well known from aerial photogrammetry. Tang (2013) and Tang et al. (2012) refer to a * Corresponding author more flexible and effective self-calibration model that allows for non-correlated parameter estimation, such as Fourier parameters. For aerial photogrammetry, significant improvements were obtained. Tang et al. (2012) conclude, that a 3 rd order Fourier series might be used to model significant radial distortion, as it is often found in close-range photogrammetry. However, a 3 rd order Fourier series with 96 unknowns may lead to overparametrisation. It is recommended to use the Fourier series at least in combination with the Brown radial distortion model in order to get reliable correction results. They doubt the necessity to apply the additional correction terms to close-range applications due to much smaller scale ratios. Besides, Tang (2013) refers to correlations between the focal length and the radial distortion parameters, when additionally applying the Fourier terms to the interior orientation model. Tournadre et al. (2015) discuss the necessity of additional parameters in interior orientation modelling for critical configurations, mostly known in UAV photogrammetry. Best results, for typical aerial flight arrangements, can be obtained in MicMac with its F15P7 camera model, which refers to a higher polynomial in radial distortion (K1 to K15) related to a principal point of symmetry and an additional 7 th order Fourier model estimated in three steps. Griffiths & Burningham (2019) summarise that for stable results highly redundant image networks with strong convergent geometry are necessary. Using single-scale nadir images, as given in linear photogrammetry, the standard model is not suitable. Nevertheless, for comparison purposes they used a DJI Phantom 4 Pro+ single-scale data set with the standard model applied by Agisoft Metashape and MicMac with respect to a labcalibration. Likewise, Peppa et al. (2019) compare results given by data sets of a DJI Phantom 4 Pro and a DJI Phantom 4 RTK system. Maximum distortions of 46µm and 42µm, respectively, are found. They refer to image pre-processing purposes given for the P4RTK and the necessity of further analyses with respect to the internal distortion corrections and correlations. Theoretical analyses by Hastedt & Luhmann (2015) already show a significant influence of the image arrangement, including tilted images and cross flight scenes, to the interior orientation and the subsequent object space accuracy. Detchev & Lichti (2020) introduced second order polynomials to model observed systematics for Canon cameras with zoom lenses. Menna et al. (2020) presented an approach, applied to a simulation process, to minimise residual systematic patterns observed in underwaterphotogrammetry with a hemispherical port. After an initial selfcalibration bundle adjustment, the observations are corrected with respect to their median residual error, quantified within defined cells, representing the image space and pattern type.
These investigations are focussing on the evaluation of interior orientation modelling for data sets that underlie the described technical conditions (aspheric lenses, distortion corrections, filter corrections) with respect to remaining systematics given in image space after bundle adjustment. Four typical systematic residual patterns ( Figure 1) can be identified within a charge of different data sets representing a variety of typical SfM applications. The impact of different interior orientation models is evaluated in image and object space. Changes in interior and exterior orientation parameters are investigated with respect to the bundle adjustments results.

Agisoft 1.6.4 (special edition)
For data processing, Agisoft Metashape 1.6.4 in a special edition is used, which includes, besides the Brown standard model of interior orientation, additional Fourier correction terms based on Tang (2013, 2.3.2) and an extended radial-symmetric lens distortion model based on Brown (1971, 2.3.1).

IAPG-bundle adjustment based on Ceres-Solver
In order to evaluate additional correction terms for interior orientation modelling, an own bundle adjustment implementation at IAPG was used (to be published in late 2021). This solution is based on the Ceres Solver technology (http://ceres-solver.org/). Currently, the bundle adjustment is restricted to data sets with one camera, using distances for scaling (no control point solution) and is used for a general evaluation of interior orientation model influences on data sets (without explicit scaling) within these investigations. Besides the standard Brown interior orientation model, an extended radial-symmetric lens distortion model (2.3.1) and a "local" lens distortion model (2.3.3) are implemented.

Models of interior orientation
All data sets are based on the standard 10-parameter Brown model, referred to as model B, including radial-symmetric lens distortion, decentring distortion, affinity and shear. The following models are added as further correction terms.

Radial-symmetric lens distortion model
The standard model of radial-symmetric lens distortion based on Brown (1971) is described as follows (Luhmann et al. 2019): The linear component is eliminated setting K0 = 0. Including all terms of the Seidel series, referred to as model B+O, the extended radial-symmetric lens distortion model results as follows: ∆ ′ = 0 + 1 2 + 1 3 + 2 4 + 2 5 + 3 6 + 3 7 (2) 2.3.2 Fourier series model Tang (2013) gives the general form of the self-calibration model of the bivariate Fourier series with equations (3) and (4). The Fourier series model is applied in addition to a radial-symmetric lens distortion model.  Detchev & Lichti (2020) proposed the application of a "local" lens distortion model given by Lichti et al. (2015) for the reduction of remaining residual systematics using optics with at least one aspheric lens component. The local distortion model, referred to as model B+L, is added to the standard distortion corrections by using a second-degree polynomial:

EVALUATION RESULTS
Figure 1 exemplarily shows four typical systematic patterns of image residuals. These patterns (pattern types I-IV) result from a variety of evaluated data sets (Table 1, Figure 2) from UAV and close-range applications that underlie the mentioned image processing procedures and different technical conditions like the usage of at least one aspherical component.
Even though the residuals are in most cases lower than one pixel and within most precision requirements, the patterns indicate unknown sources and influences on the bundle approach and subsequent 3D processing.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) Table 1. Overview of evaluation data sets with different conditions in image processing and application type; all cameras are operated with global shutter, image configurations are given in Figure 2.
Therefore, the aim is to better understand the given systematics and to evaluate different interior orientation models to minimise these effects. Influences on the interior and exterior orientation parameters, their reliability and the impact in object space are considered. All results are denoted as relative changes to the model B reference.

Figure 1.
Typical residual patterns with systematics for data sets processed with B in Agisoft Metashape (pattern types with respect to data sets and their specifications are given in Table 1). Exterior orientation parameters: Influences on selected data sets are evaluated -Object space: Evaluation in object space is given by the residuals of control and check points

Standard Brown model and Fourier series (B+Fx)
All data sets are evaluated by applying all possible B+Fx interior orientation models in Agisoft Metashape. While no significant influence could be observed for data sets of pattern types I and II, significant reductions of image space systematics could be gained for data sets of pattern types III and IV.
For data set 4+5, which is a combined processed data set of UAV and terrestrial images, best reduction in image space systematics could be reached with a 3 rd order Fourier. This reduction is most significant for the Canon data ( Figure 3, Figure 5). Consequently, the correlations of the interior orientation parameters increase up to 100%. Besides high correlations of the Fourier terms to the principal distance and among each other, a significant increase of the correlations of the standard Brown parameters to the principal distance can be observed. The correlation from K1 and K2 to the principal distance increases with every higher order of Fourier polynomials, maintain the same order of magnitude for the following intermediate term (e.g. B+F16 and B+F32). In addition, a significant change of the principal distance value is given for B+F48 with -8px and for B+F96 with -20px with an increase in its standard deviation of about factor 10. Figure 6 includes graphs of the correlation effects for the DJI FC220 camera. They follow the same manner as it is given for the Canon camera. Nevertheless, the DJI FC220 shows an increase in principal point correlations with highest order Fourier terms. Likewise, the value change of the principal distance is of high impact with a change of +80px for B+F96.    The combined processing of data set 4+5 is based on a set of 18 control points, which are all used for datum definition. The standard processing B results in an RMSXYZ of 0.021m, the B+F96 bundle results in an RMSXYZ of 0.015m. Therefore, the results indicate an increase in object space precision by simultaneously reducing the image space systematics applying B+F96 for the data set processing 4+5.
In order to evaluate the influences of the interior orientation model on the different cameras and acquisition types that are given for data set 4 and 5, the Canon data was processed separately. While the correlations stay equal, the principal distance significantly changes with respect to the standard processing up to -55px for B+F48 and -35px for B+F96.
Hence, the significant changes in principal distance of -20px and -55px resp. need a closer look on the resulting parameters in exterior orientation. Figure 7 exemplarily visualises the difference vectors from standard processing camera positions to B+F96 camera positions. In general, the absolute changes in camera positions increase with each Fourier term up to 110mm for B+F96, which equals ~factor 5 of object space precision. The image scale varies from 1:100 to 1:7200, given with the maximum imaging distance of the UAV flight of 34m and a median imaging distance of 1m for the terrestrial images. The mean ground sample distance for the data set 4+5 results to 8mm. The variation of the principal distance of up to -20px for the Canon camera and +82px for the DJI camera lead to the deviation vectors of the camera positions. The vectors show that the remaining influences are equally compensated in XYZ directions within the bundle adjustment. Due to the mixed UAV and terrestrial close-range data set processing, the absolute value changes with respect to the principal distance and their impact cannot be estimated. Furthermore, the impact on the resulting 3D model has to be investigated. For data set 7, image space systematics are reduced with B+F48 and higher order polynomials. It has to be considered that the residuals are generally larger than for all other data sets. Figure 8 shows the reduction of the image residual systematics, which is of higher success as for the Canon.
The correlations in Figure 9 indicate nearly the same effects. Most conspicuous are the correlations between the principal point coordinates and the principal distance. They decrease to 0%, as is generally expected in close-range photogrammetry. With respect to these effects, a Fourier model with at least 48 parameters should be applied. However, the principal distances significantly change for B+F48 and B+F72 with respect to the standard processing B by +100px while this effect is reduced to +40px for the B+F96 processing. Therefore, the Fourier model B+F96 indicates best results and reliability.  The iPhoneX data set is processed using 3 ground control points and 4 check points. The object space precision increases from RMSXYZ = 0.023m to RMSXYZ = 0.021m for the control points and from RMSXYZ = 0.062m to RMSXYZ = 0.049m for the check points from B to B+F96.
While the application of the Fourier model for pattern types I and II do not influence the image space systematics, another particularity can be identified for data set 2 and 3. Both cameras are of identical design, although they are not physically the same. Having a closer look at the correlations of interior orientation parameters in Figure 10, different graph characters result. For both cameras, these graphs are almost identical. The correlation between the radial-symmetric distortion parameters to the principal distance is extremely low. Additionally, correlations of the principal point y-component to the principal distance decrease from 100% to 40% with higher order Fourier terms. In addition, the correlations for the decentring distortion P1 and P2 to the principal point components are in general expected being high. They increase to their expected values by applying higher order Fourier terms.
In order to assess the influence of the Fourier model on a standard UAV scene and with respect to the given rather unexpected correlations behaviour, the changes in camera positions are plotted with respect to the standard processing in Figure 11.
As expected for a UAV scene, the changes in camera positions are evenly compensated over the whole data set with an absolute maximum of 42.5mm. The results remain nearly constant for all orders of Fourier series. The vectors are aligned towards the centre line of image acquisition and indicate a trend in negative Z-direction. With a ground sample distance of 14mm and a value change for the principal distance of approximately +3px, this leads to 42mm in object space. A larger principal distance and lower Z-values for the camera positions of same magnitude indicate a consistent bundle adjustment result.  In standard UAV applications, a combined processing of images underlying different pre-processing steps does not lead to different results in image and object space compared to their individual adjustments (e.g. data set 1).
For the pattern type IV, which is represented by a Sony RX1RM2 data set, a reduction of the image systematics is reached for one coordinate component by applying the Fourier model ( Figure  12). Unfortunately, no object space reference is available for this data set. However, the finding of reducing the exceptional systematics in image space should not be neglected.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition)

Extended Brown model (B+O)
All data sets are evaluated by applying the B+O interior orientation model in IAPG-bundle adjustment and Agisoft Metashape. The implementation of this interior orientation model focusses on the evaluation of pattern types I and particularly pattern type III.
For pattern type I a very slight reduction in the image centre can be achieved with the extended Brown model. For pattern type III the effect is of higher impact for the DJI (4), Canon (5) and iPhoneX (7) data, as visualised in Figure 13 - Figure 15. Nevertheless, the general pattern still remains. The data is processed separately.   With the achieved improvements, the extended Brown model was introduced into the Agisoft Metashape special edition. A significant increase in object space precision cannot be observed with B+O. For the data set 4+5 the object space precision statistically decreases by 2mm which is beneath the control point accuracy. For the iPhoneX data the object space precision remains unchanged.
Applying the extended Brown model, the standard deviation for the principal distances increase by factor 10 compared to the standard model B, likewise the B+F96 results. However, the standard deviations of the principal point and other distortion parameters keep within the same order as they are resulting with the standard Brown model B.

Extended Brown model and Fourier model (B+O+Fx)
As expected, applying the extended Brown model and a F96 model leads to over-parametrisation. Some single data sets can be processed with some effort in data organisation and processing steps. Nevertheless, for the Canon (5) and Sony (6) data the residuals remain the same as for the B+F96.

Standard Brown model and "local" distortion model
Since the previously described investigations are not able to model several specific systematic patterns, the approach published by Detchev & Lichti (2020) is evaluated. The described "local" lens distortion model is able to compensate for specific systematics in image space, appearing as deflection. This behaviour was suggested by the authors that were using a Canon SLR camera with a zoom-lens with at least on aspheric optical component.
Therefore, the residual components vs. coordinate components for the data sets of this investigation are evaluated in order to identify deflection in their residual distribution that may lead to an improvement applying the "local" lens distortion model. Some effects are only obtained for the iPhoneX data set. Figure 16 shows the plots of residual components vs. coordinate components with special asymmetric distribution for vx/x and vy/y. Hence, the application of the local distortion model is not able to model the given effects. The residual distribution remains the same for B and B+L processing. This is also true for the data sets 2, 3, 4 and 6 while all data sets show linear, horizontally arranged residual distributions. However, the influence of the applied "local" lens distortion model on the Canon data (data set 5) is of interest. Even though the residual distribution does not indicate an improvement in image space systematics applying the B+L model, the data set was analysed with respect to complete data analyses and the fact of technical similarities of camera/lens combination. Figure 17 shows the resulting residual components plotted against the coordinate components for the B+L processing. Solely for the Canon data set, the additional "local" corrections seem to lead to an over-parametrisation, resulting in deflected residual distributions besides a decrease in image space precision.

CONCLUSIONS
With the analysis of a variety of different, but typical SfM data sets, systematics in image residuals are observed after bundle adjustment. Four systematic pattern types are identified. These can be linked to the data sets camera/lens configuration, image processing steps or bundle configuration. Different interior orientation models are applied and analysed for better understanding of the processes respectively reductions of systematics in image space during bundle adjustment, with special emphasis to SfM reconstruction.
The analysed data sets already represent a variety of used cameras, lenses, platforms and imaging configurations as well as different pre-processing influences. Therefore, a variety of successful and unsuccessful analyses are given in order to allow for an advice on the usability and reliability of the chosen interior orientation models and subsequent influences on the bundle adjustment.
In general, the applicability of different interior orientation models and their impact are strongly dependent on the data sets themselves. In the following, the analyses are summarised with respect to the data sets and the applied interior orientation models. An overview is given with Table 2.

Conclusions with respect to data sets
For data set 1 (pattern type I and II), even though this includes different types of imaging data with and without image preprocessing, no reduction of the systematics could be investigated by the different evaluations. Separated processing of the single data sets do not lead to any significant deviations with respect to the combined processing.
Data set 2 (pattern type I), underlying image distortion correction steps, and data set 3 (pattern type II) without pre-corrections, do not indicate improvements in image space, applying any of the described interior orientation models. Nevertheless, the data sets show similar effects in interior orientation correlations.
Obviously, high correlations for the principal point and principal distance are present, applying the standard model B, being reduced to expected values with additional Fourier correction terms. Both cameras are of the same construction design but not physically equal. Both data sets are arranged in the same manner in two different flying heights, using the same ground control points. As the observed correlations are again almost equal, the resulting image space systematics are probably due to some identical physical components of the imaging hardware.
Most effects are observed for data sets 4 and 5 (pattern type III). The application of the Fourier series B+F96 significantly reduces the image space systematics by simultaneously increasing the correlations of interior orientation parameters to the principal distance and their standard deviations. In addition, the Fourier terms lead to a significant change of the principal distance value, subsequently causing absolute changes of exterior orientation translation values of up to 11cm. With B+F96, the object space precision could be improved.
Applying the extended Brown model to the data sets 4 and 5, bundled separately, leads to a significant reduction of the image space systematics by only influencing the principal distance value and standard deviation. To allow for a reduction of the systematics for pattern type III without high impact on other parameters of interior and exterior orientation, the B+O model should hence be preferred.
Data set 4+5 and 7 tend to over-parametrisation and singularities within the bundle adjustment using different interior orientation models.
Data set 6 (pattern type IV) could only be evaluated with respect to the minimisation of image space effects, as no object space information for subsequent object space evaluation is available. The pattern could significantly be reduced using the Fourier series B+F96.
Data set 7 (pattern type III) behaves as data set 5 in terms of interior orientation parameters and correlations, as well as object space precision. The residuals are the highest among all investigated data sets. Best effects in the reduction of the systematics are given with B+F96, considering best conditions for the correlations of the interior orientation parameters.

Conclusions with respect to applied interior orientation models
Some data sets tend to over-parametrisation and singularities within the bundle adjustment using different interior orientation models. This is particularly the case for higher order polynomials or combinations of different models, as expected.
Applying the Fourier series in addition to a standard Brown model (B+Fx), an increase in correlations to the principal distance can be observed. In general, the standard deviations of interior orientation parameters increase as well, except for data sets 2 and 3. The additional correction terms lead to large changes of the principal distance values and correlations with each main order polynomial while staying with its following intermediate Fourier term. This has a high impact on the exterior orientation parameters, causing value changes of the translation coordinates.
An extended Brown model (B+O) allows for a reduction of residual systematics for pattern type III without influencing too many other interior orientation parameters and correlations. Causing less impact on the principal distance and only on this, the model is favourable to model this type of data sets. While the improvements in object space with the B+Fx model are low, the model B+O seems to provide a suitable and reliable solution. Table 2. Overview on conclusions with respect to the data sets

FURTHER INVESTIGATIONS
Some specific advices in the usage of extended interior orientation models, like Fourier series, could be derived for a selection of the data sets respectively resulting pattern types of image space systematics. The results indicate very different behaviours of data sets and their parameters with respect to the analysed effects. Their impact on object space precision could partially be evaluated. Further investigations are focussing on the evaluation of the subsequent 3D modelling processes. The results are still open to their impact on 3D models, in particular focussing on the presented value changes in exterior orientation parameters for UAV and/or close-range arrangements.