A RIGOROUS AND FLEXIBLE CALIBRATION METHOD FOR DIGITAL AIRBORNE CAMERA SYSTEMS

This paper presents a novel family of rigorous and flexible mathematical self-calibration additional parameters (APs) for airborne camera calibration. It is pointed out, that photogrammetric self-calibration can – to a very large extent – be considered as a function approximation problem in mathematics. Based on the mathematical approximation theory, we suggest that Fourier series (trigonometric polynomials) be the optimal mathematical basis functions for camera self-calibration. The whole family of so-called Fourier Self-calibration APs is developed, whose solid theoretical foundations are Laplace Equation and Fourier Theorem. The Fourier APs are mathematically rigorous, orthogonal, flexible, generic and efficient for calibrating the image distortion of frameformat airborne cameras. We show the theoretical and practical advantages of Fourier APs over the popular polynomial APs and physical APs. The good performance of Fourier APs is illustrated in the many practical tests on camera system calibration, including the DMC, DMCII, UltracamX, UltracamXp, DigiCAM cameras.


INTRODUCTION
Camera calibration is an essential subject in photogrammetry and computer vision.Self-calibration by using additional parameters (APs) has been widely accepted and substantially utilized as an efficient calibration technique in the photogrammetric society since 1970s.Many studies were investigated on self-calibration APs in the last decades.Traditionally, two types of self-calibration APs were developed for analogue single-head camera calibration: physical and mathematical.The development of physical APs was mainly attributed to D. C. Brown (Brown, 1971) for close-range camera calibration and these APs were later extended by attaching additional polynomials for aerial application (Brown, 1976).El-Hakim & Faig (1977) proposed the mathematical APs by using spherical harmonics.Ebner (1976) and Grün (1978) built the second and fourth order algebraic polynomial APs, respectively.These early works showed the remarkable significance of selfcalibration APs in camera calibration and orientation, while the concerns have been raised as well on overparameterization and high correlations (Kilpelä, 1981;Clarke & Fryer, 1998).The popular polynomial APs were often criticized as "have no foundations based on observable physical phenomena" (Clarke & Fryer, 1998).These APs, though being widely used for many years and even in digital era, however, might be inadequate to fit the distinctive features of digital airborne cameras, such as push-broom, multihead, virtual images composition, various image formats (Honkavaara et al., 2006;Cramer, 2009).A considerable progress was made for the digital camera calibration.Many tests were carried on to re-test the traditional APs.Cramer (2009) and Jacobsen et al. (2010) reported comprehensive empirical tests, in which lots of different APs were employed to compensate the image distortion.However, many of the APs are purely the combinations of the traditional APs while lack of solid physical or mathematical foundations.Moreover, the successful incorporation of navigation system demands calibrating the overall system rather than camera lens distortion only.The systematic effect caused by direct georeferencing, such as shift/drift and misalignment, must be calibrated for the photogrammetric applications (Honkavaara, 2004;Cramer et al., 2010).One major challenge of whole system calibration is that each calibration effect must be decoupled from others.Serious considerations are desired on high correlations.All the above motivate our present work on airborne camera self-calibration.One intrinsic deficiency of the polynomial APs is revealed.Theoretically, algebraic polynomials are not proper for self-calibration purpose.After examining many mathematical bases, the Fourier series (trigonometric polynomials) are favored as the optimal mathematical basis functions for building self-calibration APs.Then, a whole family of so-called Fourier APs is developed.The Fourier APs are mathematically rigorous, flexible, generic and efficient for calibrating the image distortion of frame-format airborne cameras.The advantages of Fourier APs are demonstrated over the polynomial APs.The performance of Fourier APs is evaluated in several empirical tests.The rest of the paper is organized as follows.The mathematical principle of self-calibration APs is briefly reviewed and the Fourier APs are constructed in Section 2. Empirical test results are demonstrated in Section 3. The discussions are made in Section 4 to compare Fourier APs with other counterparts.This work is concluded in the end.

FOURIER SELF-CALIBRATION APS
The collinearity equations, which are the mathematical fundamental of photogrammetry, read as in Eq. ( 1).
(1) where and denote the distortion, and indicates the random error.The denotations of other parameters can be seen in textbooks such as Kraus (2007).The distortion terms are twovariable functions whose form is unknown.They have to be approximated by some models, i.e., self-calibration APs.Thus, the key work is to develop a proper parametric model which can accurately represent the actual distortion.Do note that there are several kinds of basis functions available in mathematics, whose combination can well approximate any unknown function.Thus, the distortion, on which even little is known, can be modeled by using the linear combination of specific basis functions.The unknown coefficients of the linear combinations can be computed in the adjustment process.Or in a loose sense, the coefficients can be fixed by the noisy image measurements during the least-squares adjustment.It quite resembles the problem of least squares fitting to the irregular spaced data in mathematics (for the mathematical materials on the fitting problem, the readers are referred to the textbooks such as Rao (2001)).Therefore, photogrammetric selfcalibration canto a very large extentbe considered as a function approximation or, more precisely, a curve fitting problem in mathematics.Then, we need to find a proper group of basis functions for the self-calibration purpose.Various mathematical basis functions can be taken into account.We start with algebraic polynomial since of the historical prevalence of the polynomial APs.

Polynomial APs
Based on the standard 60% forward overlapping level and the 3×3 and 5×5 grid pattern of image point distribution, Ebner (1976) and Grün (1978) proposed the polynomial APs of second and fourth order, respectively.Although they constructed their polynomial APs in a quite different way from mathematical function approximation, we showed that the mathematical principle of both sets of APs is the well-defined Weierstrass Theorem (Tang et al., 2012a).This theorem indicates, that any function can be approximated with arbitrary accuracy by a polynomial of sufficiently high degree.We also showed how to derive both sets of APs from the rigorous principle of function approximation.It can also be proved that the grid pattern is not prerequisite for applying both APs.The main effect of the irregular image point distribution is degrading correlations but not decaying calibration.This is the exact reason why the Ebner and Grün APs still work quite well in the cases where the ideal regular grid pattern is unsatisfied.In fact, we have developed in our previous work a new family of so-called Legendre Self-calibration APs, which is established on the basis of the orthogonal univariate Legendre polynomials (Tang et al., 2012a and2012b).The mathematical relations can be derived between Legendre APs and the two historical polynomial APs.From both theoretical analyses and practical tests, Legendre APs must be preferred to the two conventional ones.In fact, the Legendre APs can be considered as the superior generalization of the Ebner and Grün models in many senses.However, there are one intrinsic deficiency of all polynomial APs, including Ebner, Grün and Legendre APs.That is, all polynomial APs need to eliminate four high correlated parameters (NOT six, since two of them, i.e. two constant terms, are purely the principle point shifts).The elimination imposes four constraints on the polynomial APs.These four constraints, which are caused by high correlations, violate the mathematical principle of polynomial APs.Because according to the approximation theory and the Weierstrass Theorem, all unknown parameters in should be fully independent of those in .Therefore, algebraic polynomials are not the optimal mathematical bases for building self-calibration APs.The discussions on this theoretical deficiency of the polynomial APs were also made in Tang et al. (2012b).
Our view was shared in Ziemann (1986) that algebraic polynomials "are undesirable from a mathematical point of view because of the high correlation between the different terms".Another inconvenience of the polynomial APs is that they usually require large number of unknown APs for calibration, compared with other counterparts (the APs in Brown (1976), for example).

Fourier self-calibration APs
Besides the algebraic polynomials, other basis functions are the potential alternatives for developing mathematical APs.We briefly review the mathematical basis functions for function approximation applications.Three groups of quite useful basis functions can be derived from the renowned Laplace Equation, depending on the defined coordinate system: Fourier series (trigonometric polynomials) for the Cartesian coordinates, Bessel functions for the cylindrical coordinates and spherical harmonics for the spherical coordinates.Wavelet functions and rational functions are also useful for specific implementations.
For more on mathematical approximation theory, the readers are referred to textbooks such as Oliver et al. ( 2010).
For the self-calibration purpose, wavelet functions are not appropriate since of its non-analytical form, which precludes them from being incorporated into the collinearity equations.The form of rational functions is rather inconvenient for building APs and can burden the adjustment.For the three basis functions derived from Laplace Equation, Fourier series should be favored since of the Cartesian coordinates in the rectangular image format.Actually, the APs using spherical harmonics (El-Hakim & Faig, 1977) encounter the similar difficulty of high correlations as algebraic polynomial APs.We develop below the self-calibration APs on the basis of Fourier series.Bivariate Fourier series are given as follows. (2) where .The Fourier Theorem indicates that any two-variable functions, defined in , can be approximated with arbitrary accuracy by the combinations of bivariate Fourier series of sufficiently high degree.Let and denote the width and length of the image format, respectively.and indicate the metric coordinate of the image measurement, .Denote Then, the linear combination of the bivariate Fourier series is used to approximate the distortion terms and , respectively.The general form of self-calibration APs is given in Eq. ( 3).
(3) where , , and are unknown coefficients to be computed in adjustment.and are the maximum degrees chosen by the users (Generally speaking, more complicated the distortion is, larger and it requires).The number of unknown APs is .
As this family of self-calibration APs (with respect to the chosen and ) is established based on Fourier series, they are named as Fourier self-calibration APs.Do note that in Eq. ( 3) the unknown parameters in is fully independent of those in .Particularly, the Fourier APs of (16 unknowns) are given with lexicographic order in Eq. ( 4). (4) The Fourier APs of with 48 unknowns are depicted in Eq. ( 5).
(5) Theoretical comparisons can be made between Fourier APs and algebraic polynomial APs.The Weierstrass Theorem and Fourier Theorem are the main mathematical foundations behind the polynomial APs and the Fourier APs, respectively.It is guaranteed in theory that both APs of proper degree can well represent the unknown distortion function and effectively compensate the image distortion.However, the Fourier APs own a remarkable theoretical advantage over the polynomial APs.In contrast to the polynomial APs which need to remove four parameters due to high correlations, all Fourier terms are independent of IO parameters and EO.There is no need to eliminate any parameter of Fourier APs and there is no constraint imposed between the unknown parameters in and those in of Eq. (3).Therefore, Fourier APs rigorously obey their theoretical principle (Fourier Theorem) while polynomial APs do not.From a mathematical viewpoint, the Fourier APs are more rigorous and more proper for camera self-calibration than the polynomial APs.

Overall system calibration
As mentioned previously, the systematic effects caused by direct georeferencing must be compensated.The effects of important interests include the misalignment between camera and navigation instruments and the shift/drift effect in direct georeferencing (if present).For the overall system calibration, one of the most challenging works is to minimize the coupling effect of different correction parameters.The decoupling is of vital importance, in the sense that each systematic error must be independently and appropriately calibrated and the calibration results are block-invariant.For this purpose, we suggest the joint application of the Fourier APs (for calibrating the image distortion, as the conventional APs do) with the other correction parameters, i.e., the three interior orientation (IO) parameters used for correcting the principle point offsets and the focal length deformation, and GPS/IMU shift/drift and misalignment correction parameters.The low correlation must be warranted among these calibration parameters and between them and exterior orientation (EO).As will be seen in Section 4, the correlations between Fourier APs and EO, and between Fourier APs and other correction parameters, are fairly small.The low correlation is another advantage of Fourier APs.

PRACTICAL TESTS
The Fourier self-calibration APs are tested by using the data from the recent DGPF (German Society for Photogrammetry, Remote Sensing and Geoinformation) project, which was performed under the umbrella of DGPF and carried out in the test field Vaihingen/Enz nearby Stuttgart, Germany.This successful project aims at an independent and comprehensive evaluation on the performance of digital airborne cameras, as well as offering a standard empirical dataset for the next years.The readers are referred to Cramer (2010) and DGPF website (2010) for the project details.Four flights' data of the frame cameras are adopted: DMC (GSD 20cm, ground sample distance), DMC (GSD 8cm), UltracamX (GSD 20cm) and UltracamX (GSD 8cm).Each camera was flown at two heights.For each flight, we are interested in two most often contexts: the in-situ calibration one and the operational project one.The former context is with high side overlapping (≈60%) and dense GCPs and the later with low side overlapping (≈20%) and few GCPs.

In-situ calibration context
The system calibration strategy in Section 2.3 is adopted for all the blocks.Particularly, IMU misalignment, horizontal GPS shift (factually insignificant in the tests), IO parameters and Fourier APs of are employed.Higher degree's Fourier APs seem unnecessary in these tests since they deliver insignificant accuracy refinement.The derived external accuracy, indicated by "self calibrating", would be compared to the theoretical accuracy and the "without APs" one, for which the same correction parameters except Fourier APs are used.The derived external accuracy is demonstrated in Fig. 1.By comparing "Self calibrating" with "Without APs", the refinement by applying Fourier APs is significant in all tests, up to 10 cm in the DMC (GSD 20cm) block.Moreover, all the "self calibrating" accuracy reaches very close to the theoretical one.It means that the optimal accuracy has been achieved in all the tests.All the "self calibrating" accuracy reaches to around 1/5 GSD in the horizontal directions and 2/5 GSD in the vertical directions in the four blocks.It is also interesting to notice that although the DMC and UltracamX cameras are differently manufactured, very similar external accuracy can be obtained by using Fourier APs in the blocks of similar configuration, i.e., similar GSD, similar forward and side overlapping levels and similar GCPs distribution.This fact, independent of the used cameras, coincides well with our photogrammetric accuracy expectation.Now look at the estimation of the precision of image measurements.The posterior std.dev.estimation is 1.58, 1.44, 0.88 and 0.79 for DMC (GSD 20cm and 8cm) and UltracamX (GSD 20cm and 8cm) blocks, respectively.These values are around 0.12 pixel, which are 12 and 7.2 for DMC and UltracamX cameras, respectively.They match very well the expected precision of the automatic tie point transfer techniques, which are 0.1-0.2pixel for aerial images.GCPs and p60%-q20% ('without APs' indicates without using Fourier APs only)

Operational project context
There are 4 GCPs and 20% side-overlapping level in each block, much weaker than the in-situ calirbation context.The IMU misalignment, IO parameters and the Fourier APs with are employed in the adjustment.This derived external accuracy is analogously denoted as "self calibrating" one.Due to 4 GCPs available only, the GPS/IMU observations have to be weighted carefully to achieve best accuracy.We also evaluate the quality of the in-situ calibration in last subsection.The calibration results of IO parameters and image distortion in Section 3.1 are utilized as known and fixed values in the adjustment of the corresponding "reduced" operational blocks, i.e., the cameras are assumed being calibrated and need no further self-calibration.The derived external accuracy is named as "after calibration".We compare "after calibration" with "self calibrating", "without APs" and theoretical ones.The adjustment accuracy results are illustrated in Fig. 2. On the one hand, the self-calibrating Fourier APs help improve the external accuracy.It is noticed that for the blocks of weak geometry, the accuracy in vertical direction is generally worse than 1/2 GSD.By comparing the closeness between "selfcalibrating" accuracy and the optimal "theoretical accuracy", it is observed that self-calibration gains less significant refinement in the operational blocks than in the in-situ block (Section 3.1).On the other hand, the "after calibration" yields more remarkable refinement in all four operational blocks, particularly in the vertical accuracy.Further, the "after calibration" accuracy is very close to the optimal theoretical one in every block.Therefore, these tests not only recognize the sufficient accuracy obtained by Fourier APs in the operational projects, but also confirm their great efficiency in the in-situ calibration.Besides, from the considerable distinction between "self-calibration" and "after calibration" accuracy in Fig. 2, it demonstrates the necessity of in-situ calibration in the presence of the significant image distortion.
It is worth mentioning that the Fourier APs have also been assessed by the flight data of other airborne cameras in other test fields, like medium-format DigiCAM, large-format UltracamXp and single-head DMC II cameras.The similarly good performance of Fourier APs is confirmed while the details are not published here.

External accuracy
We compare the external accuracy obtained by different APs in two blocks, the in-situ and operational blocks of 'DMC (GSD 20cm)'.The results are illustrated in Fig. 3.It is obvious that "Ebner (12)" and "Fraser (10)" derive rather worse accuracy than the others.Yet, the reasons for their poor performance are quite different.The poor performance of "Ebner (12)" is because the second order algebraic polynomials are insufficient to approximate the image distortion.Polynomials of higher degree are required for better approximation, according to the Weierstrass Theorem.The considerable improvement is clearly achieved by using higher-order polynomial APs, e.g., the fourth order Grün models and the fifth order Legendre APs.On the other hand, "Fraser (10)" fails mainly due to that the classical distortion model of close-range camera is in general inappropriate for airborne camera calibration.The distortion in aerial images is not dominated by the radial distortion as in close-range camera.The good performance of extended "Brown (21)" model is mainly since this model includes several high order polynomial terms, which although were supposed to compensate film deformation and unflatness (Brown, 1976).Do note that Fourier APs (16 unknowns) achieve similar accuracy with Legendre polynomial APs (66 unknowns), but use much fewer unknowns.Certainly, fewer APs imply better efficiency.Fewer APs also help improve the stability of block geometry against overparameterization, which is always a main concern for applying the self-calibration APs.Mathematically, fewer terms indicate, that Fourier series, instead of algebraic polynomials, are the more proper basis functions to approximate the unknown distortion.This is a significant practical advantage of Fourier APs over polynomial APs.Thirdly, the correlations with IMU misalignment are similar for the Fourier APs and the Legendre APs.Although the correlations of Fourier APs (92%) seem slightly worse than those of Legendre APs (100%) in '< 0.1', the difference is negligible since the 'max' correlation of Fourier APs is 0.20.The 'Intra-corr' shows the orthogonality of the Legendre APs and the Fourier APs.This orthogonality, though having no significant impact on calibration and accuracy, helps the stability of adjustment process.Thus, the Legendre APs and the Fourier APs perform similarly best in correlation analyses.They minimize the coupling effect in calibration.Therefore, they can derive more reliable calibration results than the others.It is also expected that they will produce similar calibration result, as will be demonstrated in next subsection.

IO parameters and IMU misalignment calibration
We consider the calibration results of IO parameters and IMU misalignment.

CONCLUSIONS
We present a novel family of self-calibration Fourier APs for calibrating the image distortion of frame-format digital airborne cameras.We point out that photogrammetric self-calibration can to a very large extentbe considered as a function approximation problem in mathematics.Fourier APs have solid mathematical foundations: Laplace Equation and Fourier Theorem.It is guaranteed, that the Fourier APs are rigorous, orthogonal, generic, flexible and effective for airborne camera calibration.
The high performance of the Fourier APs was identified in several empirical tests.These tests show that Fourier APs can fully calibrate the image distortion of the current airborne cameras, including the DMC, DMC II, UltracamX, UltracamXp and DigiCAM cameras.Optimal accuracy can be achieved by using Fourier APs of proper degree.In principle, the Fourier APs can be used for calibrating frame cameras of large-, medium-and small-format CCDs, mounted in single-and multihead systems.
We also make comparisons between the Fourier APs and other APs.In contrast to the physical APs, Fourier APs performs much better in low correlations, which are of vital importance in multi-sensor calibration.Fourier APs possess theoretical and practical advantages over the popular polynomial APs.It is shown that Fourier APs are theoretically preferable, since the Fourier series are the more appropriate (or optimal) mathematical basis functions to develop self-calibration APs, rather than the algebraic polynomials.Moreover, Fourier APs show an important practical advantage, that in general Fourier APs need fewer parameters for effective calibration.In fact, we believe that the Fourier APs of first order or second order are sufficient for most airborne camera calibration tasks.This advantage helps to reduce the overparameterization effect and improve the block stability.

Fig. 3
Fig. 3 Accuracy comparison of different APs in two blocks: the in-situ block (above) and the operational block (below)4.2Correlation analysesCorrelation analyses deserve more importance in integrated sensor orientation than in the conventional aerial triangulation.Heterogeneous observations from multi-sensors may contain various systematic errors.Each systematic error must be calibrated independently from the others.The coupling effect must be minimized and low correlations between different correction parameters must be guaranteed.It is illustrated in Table.1 the correlation analyses of different APs in the DMC (GSD 20cm) in-situ block.The correlations are examined between APs and EO, IO parameters, IMU misalignment correction parameters and the intra-correlations among APs themselves (denoted by 'Intra-corr' in Table.1).'< 0.1' denotes the percentage of correlations smaller than 0.1 and 'max' denotes the maximum correlations.As 'Ebner (12)' and 'Fraser (10)' deliver rather poor accuracy, their performance in correlations is not demonstrated.
It is quite interesting to check whether different APs can deliver coincident results.The 'Brown (21)', 'Legendre (66)' and 'Fourier (16)' are compared.We examine two in-situ calibration blocks: DMC (GSD 20cm) and UltracamX (GSD 20cm).It is illustrated in Fig. 4 and Fig. 5 the calibration results of IO parameters and IMU misalignment, respectively.It is found in Fig. 4 that in most cases, the Legendre APs and the Fourier APs derive quite coincident results in IO parameters, while the Brown models get diverse results, particularly the focal length calibration in the DMC (GSD 20cm) block.From Fig. 5, three APs yield similar results on IMU misalignment calibration, while the results of the Legendre APs and the Fourier APs are closer.It is reasonable and expectable, that the Legendre APs and the Fourier APs obtain coincident calibration results, since both calibrate effectively the image distortion and have very low correlations with IO parameters and IMU misalignment.The high correlations between 'Brown (21)' and IO parameters are probably the reason causing the deviation in IO calibration results.With the decreased correlations with IMU misalignment parameters, the 'Brown (21)' models obtain more consistent results with the other two APs.

Fig. 4
Fig. 4 Calibration results of IO parameters derived by three different APs in the blocks of DMC (left) and UltracamX (right) cameras (unit: ) International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012 XXII ISPRS Congress, 25 August -01 September 2012, Melbourne, Australia

Table 1 .
CorrelationSome interesting observations can be obtained from Table.1.Firstly, all APs hold very low correlations with EO.Secondly, Legendre APs and Fourier APs deliver much lower correlations with IO and IMU misalignment than the Brown and Grün counterparts.