CALIBRATING A LENS WITH A “LOCAL” DISTORTION MODEL

This paper is about camera calibration where an abnormal systematic effect was discovered. The effect was first encountered in a multi-camera system used for close range 3D photogrammetric reconstruction. The objectives for this research were two-fold. The first objective is to identify the source of the systematic error, and the second objective is to model the error as rigorously as possible. The first objective was met after acquiring several calibration data sets where the camera bodies, the lenses, and the image formats were varied. It was concluded that the source of error is the lens system. The second objective was also met. The so called “local” lens distortion was modelled using second order polynomials as the plots of the residuals vs. the image coordinates resembled parabolic shapes. Overall, the final room mean square error for the residuals after applying radial and “local” lens distortion was reduced from 1/2 to 1/6 of a pixel or a 200% relative estimated error improvement.


INTRODUCTION
Camera calibration is an essential quality assurance procedure in the field of photogrammetry especially when it comes to precision applications. Calibrating cameras became even more important when inexpensive off-the-shelf digital units started to be used commonly in close range applications. Publications by Fraser (1997), Shortis et al. (1998), Clarke and Fryer (1998), Habib and Morgan (2003), Mills et al. (2003), Chandler et al. (2005), Remondino and Fraser (2006), and others explored the use of low-cost amateur cameras and their calibration for metric applications. By and large, they applied a self-calibrating bundle adjustment (Ackermann, 1981;Granshaw, 1980;Kenefick et al., 1972) where the unknown parameters to solve for, in addition to the object space point coordinates and the exterior orientation of each image, are the principal distance, the principal point coordinates, and a set of additional parameters: radial lens distortion, decentring lens distortion (Brown, 1966(Brown, , 1971, and affinity / in-plane distortion coefficients. These papers also emphasize on the importance of imaging / network geometry for achieving reliable results. This paper explores a peculiar type of distortion found in Canon DSLR cameras sold in a bundle with inexpensive zoom kit lenses. The distortion was discovered when eight Canon DSR cameras were calibrated in a system. The system was used for infrastructure deformation monitoring and, more specifically, measuring deflections in concrete beam specimen. The cameras were rigidly mounted to a steel frame above the specimen being monitored. All cameras had previously been calibrated in a lab with the principal distance set to 35 mm with no issues. This time around, the lenses were set to 22-28 mm, and the cameras were calibrated in-situ with a portable test field, i.e., the cameras remained mounted to the steel frame, and the test field was moved in front of them. After the radial lens distortion was removed, a systematic effect with a different signature for each camera could was noticed. Several different types of residual plots were generated for the purpose of detailed analysis: quiver, radial distance, radial orientation, histograms, and image coordinates. For example, see Figure 1 for a sample residual quiver plot. Since the residual signatures did not resemble those of decentring lens or in-plane distortions, several hypotheses were brainstormed. This paper has two research objectives. The first objective is to find out the source of the peculiar residual signature. For that purpose several data sets were collected and a number of bundle adjustments were run. The methodology and results / discussion sections on finding the source of the residual signature contain the details. The second objective is to model the source of the error. Two methods are proposed for that purpose and again a few bundle adjustments were run for validation. Thus, a set of methodology and results / discussion sections for modelling the source of the error are also included.

METHODOLOGY: FINDING THE SOURCE OF THE RESIDUAL SIGNATURE
Note that the cameras had been previously tested to have stable interior orientation parameters over time. So the following hypotheses were brainstormed in order to explain any other factor(s) which may have caused this abnormal residual signature: 1) system instability, i.e., the relative orientation between the multiple cameras in the system may have been changing; 2) instability in the object space, i.e., the portable test field may have been deforming; 3) issues with sensors, i.e., out-of-plane distortions may exist for each sensor; 4) artefacts from the image processing chain, i.e., the errors may have been introduced in the conversion from raw to .jpeg file formats; and 5) issues with lenses, i.e. some lens distortion other than radial, decentring or in-plane may exist for each lens .
In order to narrow down the potential causes for the residual signature, calibration data sets were acquired with the following: two different camera bodies (with CMOS APS-C solid state sensors); two different lenses (an EF 35 mm f/2.8 prime and an EF-S 18-55 mm f/3.5-5.6 zoom kit lens set to 24 mm); a stationary test field (see Figure 2); and images in both raw and .jpeg file formats. Overall eight data sets were collected: each camera body with each of the lenses separately with both raw and .jpeg images. Each calibration data set consisted of 30 images taken from varying positions and orientations in order to obtain strong network geometry. Note that this time around, it was the test field that remained stationary the entire time, while the cameras were moved.

RESULTS AND DISCUSSION: FINDING THE SOURCE OF THE RESIDUAL SIGNATURE
The eight different calibration data sets were run in a selfcalibrating bundle adjustment with the 1 and 2 radial lens distortion coefficients as additional parameters. Note that the results between the raw and .jpeg image adjustments did not differ significantly. The results, i.e., the image space root mean square error (RMSE), for the four calibration adjustments are listed in Table 2. Note that both the prime lens solutions performed at an RMSE level of better than 1/10 of a pixel, while both the zoom kit lens solutions performed at 1/2 of a pixel. In addition, residual quiver plots for the adjustments are shown in Figure 3, Figure 4, Figure 5, and Figure 6. Note that the residuals for the 35 mm prime lens solutions in Figure 3 and Figure 4 exhibit no distinct pattern in terms of both magnitude and direction. The magnitude of the maximum residuals is 1/3-1/2 of a pixel. Conversely, the residuals for the 24 mm zoom kit lens solutions in Figure 5 and Figure 6 were first unacceptably large in terms of magnitude. The maximum residuals had a magnitude of 2.5-3.5 pixels. Secondly, the pattern of the residuals was not random, and most importantly the signature was the same for both camera bodies. Given the results after processing all the acquired data sets, through the process of elimination, a conclusion can reached that the source of the peculiar residual pattern must be the lenses, not the system, the camera bodies, the test field or the .jpeg image format.

METHODOLOGY: MODELLING THE ERRORS
Since the magnitude of the residuals after removing the radial lens distortions for the 24 mm zoom kit lens solutions was unacceptably large, the next step is to model the remaining systematic effects. If the camera body-lens combination at 35 mm can perform at an image space RMSE level of better than 1/10 of a pixel, then adding more additional parameters to the self-calibrating bundle adjustment for the 24 mm solutions may improve the 1/2 of a pixel RMSE.
With regards to the EF-S 18-55 mm f/3.5-5.6 series of zoom kit lenses, photographers online had reported a "complex" (DPReview.com, 2008) or "wavy" (Rockwell, 2011(Rockwell, , 2006 distortion, especially pronounced at the wide angle and difficult to remove in commercial image processing packages. No photogrammetric / scientific literature was found on the subject so far. After researching the manufacturer specifications of this lens series, it was discovered that the lens contains eleven elements in nine groups, and one of the lens elements is aspherical (Canon Inc., 2011). Aspherical lens components are usually used to keep the lens system less heavy and minimise any dulling effects in terms of radiometric image quality. Thus, plausible reasons for the distortion could be imperfections in the production or the assembly of the lens elements. For example, the use of an inexpensive aspherical lens element could be the distortion culprit.
In order to model distortion, residual plots for the radial distance, orientation, and image coordinates were generated (see Figure 7 and Figure 8). Figure 7 clearly shows a wide band of residuals (~ 1-3 pixels). Also, the band of residuals in Figure 8 is not flat, but rather it follows a systematic pattern. Given the parabolic shape of the systematic pattern, a variation of the "local" lens distortion model in Lichti et al. (2015) is proposed to be applied (see equations (1) and (2)). Essentially, the systematic errors are empirically compensated by second-degree polynomials with coefficients 1 to 4 . = 1 ̅ 2 + 3 ̅ 2 (1) with ( , ) being the measured image point coordinates and ( , ) being the principal point coordinates.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)

RESULTS AND DISCUSSION: MODELLING THE ERRORS
In order to test and validate the proposed "local" lens distortion model self-calibrating bundle adjustments with the following additional parameters were run on the Canon T3 (1100D) with 24 mm zoom kit lens:  Radial lens distortion coefficients ( 1 and 2 only)  Radial and decentring lens distortion coefficients ( 1 , 2 , 1 and 2 )  Radial and "local" lens distortion coefficients ( 1 , 2 , 1 , 2 , 3 and 4 ) The image space RMSEs for the three adjustments are listed in Table 3. The RMSE for the self-calibrating bundle adjustment with 1 , 2 , 1 , 2 , 3 , and 4 as additional parameters was 1/6 of a pixel. Note that all the "local" lens distortion coefficients were significant. Also, the residual plots no longer exhibited any major systematic effects (see Figure 9 and Figure 10). Moreover, with the exception of some outliers, the residuals were now bounded by ±2-3 μm on all the plots (again, see Figure 9 and Figure 10). Finally, there were no major correlations between the interior orientation and the exterior orientation parameters.  Table 3. RMSE of the residuals for the different calibration adjustments assessing the necessary additional parameters Since the "local" lens distortion model is not readily available in most self-calibrating bundle adjustment software packages, this experiment meant to also check if applying decentring lens distortion would at least partially compensate the newly discovered "local" distortion. Decentring lens distortion did in fact seem to partially compensate the "local" distortion. For example, the RMSE for the residuals was 1/5 of a pixel. However, upon closer scrutiny, the residual plots still exhibited some systematic effects, and adding 1 and 2 caused major correlation issues with and . Overall, using "local" lens distortion coefficients over the decentring lens distortion coefficients is warranted. The effect of the "local" distortion coefficients would most likely be even greater when outliers in the data are detected and eliminated. radial distance or orientation after removing the radial and "local" lens distortions for the 24 mm lens calibration of the Canon T3 (1100D) Figure 10. Plots of residuals vs. image coordinates after removing the radial and "local" lens distortion for the 24 mm lens calibration of the Canon T3 (1100D)

CONCLUSIONS
This paper introduced the readers to a newly discovered "local" lens distortion present in the Canon EF-S 18-55 mm f/3.5-5.6 zoom kit lenses. The lens distortion is especially pronounced at the wider angle (18 -~ 28 mm). The presence of the lens distortion most probably is due to manufacturing imperfections in the aspherical component of the lens system. In the residual vs. image coordinate plots it manifests itself as a parabolic effect best modelled by a set of second order polynomials. While this local distortion may be partially compensated via the use of well-established lens distortion parameters (e.g., decentring lens distortion coefficients), it is advised that the use of the "local" lens distortion coefficients is more rigorous.
Future work for this research project would involve testing more zoom kit lenses and other prime lenses, using a 3D test field, removing outlying observations, and evaluating the effect of including the "local" lens distortion parameters on the object space reconstruction.