CRITICAL ASSESSMENT OF CORRECTION METHODS FOR FISHEYE LENS DISTORTION

A fisheye lens is widely used to create a wide panoramic or hemispherical image. It is an ultra wide-angle lens that produces strong visual distortion. The distortion modeling and estimation of the fisheye lens are the crucial step for fisheye lens calibration and image rectification in computer vision and close-range photography. There are two kinds of distortion: radial and tangential distortion. Radial distortion is large for fisheye imaging and critical for the subsequent image processing. Although many researchers have developed calibration algorithms of radial distortion of fisheye lens, quantitative evaluation of the correction performance has remained a challenge. This is the first paper that intuitively and objectively evaluates the performance of five different calibration algorithms. Upto-date research on fisheye lens calibration is comprehensively reviewed to identify the research need. To differentiate their performance in terms of precision and ease-using, five methods are then tested using a diverse set of actual images of the checkerboard that are taken at Wuhan University, China under varying lighting conditions, shadows, and shooting angles. The method of rational function model, which was generally used for wide-angle lens correction, outperforms the other methods. However, the one parameter division model is easy for practical use without compromising too much the precision. The reason is that it depends on the linear structure in the image and requires no preceding calibration. It is a tradeoff between correction precision and ease-using. By critically assessing the strengths and limitations of the existing algorithms, the paper provides valuable insight and guideline for future practice and algorithm development that are important for fisheye lens calibration. It is promising for the optimal design of lens correction models that are suitable for the millions of portable imaging devices.


INTRODUCTION
With the development of computer technology, non-metric camera has been more widely used in the field of photogrammetry and computer vision.Fisheye lens has a wide field of view up to 180 degree.Because of its remarkable range of view, it is widely applied in the field of military, surveillance, street maps, panorama stitching and so on.However, due to the special imaging process of fisheye lens, the distortion is a main obstacle for practical use.The traditional pinhole model is no longer able to meet the need of fisheye lens correction.We need to consider more parameters and introduce different distortion models for different lens.
Distortion has been researched for many years, and it was first introduced by Conrady in 1919.Following his work, Brown (Duane, 1971) proposed the radial distortion, tangential distortion and thin prism distortion model which has been widely used for image distortion.On the basis of previous work, some different distortion models are proposed.Zhang (Zhang, 2000) proposed a flexible technique for estimation of radial distortion, which only required the camera to observe a checkerboard from a few different orientations, and the method is widely applied because of its easy implement.Fitzgibbon (Fitzgibbon, 2001) showed how to use the division model to solve the problem of nonlinear lens distortion, and Miguel Alemán-Flores (Alemán-Flores et al., 2014) then extended the division model.In this model, they combined hough space with the division model, and a single fisheye lens image can be corrected without any attached condition.Devernay and Faugeras (Devernay and Faugeras, 2001) proposed the fieldof view(FOV) model and perform edge extraction and polygonal approximation.Claus (Claus and Fitzgibbon, 2005) proposed the rational function model and built a general distortion model for large field-of-view lens.Carlos (Ricolfe-Viala and Sánchez-Salmerón, 2010b) introduced a robust metric calibration.In their paper, they proposed using cross-ratio constraint which is independent of perspective projection as a template for calibration, so that we can get two sets of point: the distorted points and the corrected points.Zhu (Zhu et al., 2011)proposed a lifting strategy based on an elliptical model for the correction of fisheye image.Lee (Lee et al., 2011) introduced another method for wide-angle distortion correction with hough transform and gradient estimation.There are also other methods,such as virtual grid (Arfaoui and Thibault, 2013), parabolic perspective projection (Zhang, 2012), vanishing point (Hughes et al., 2010) and so on.
Although different camera distortion models were proposed, there is no simple and intuitive measure to evaluate the correction quality of each model.After the projective transformation of the imaging process, a line in the world should be correctly recorded as a line in the image.But the line in the image is distorted to be curve because of the lens distortion.The purpose of lens correction is to rectify the curve in the image into a straight line, and to reflect the real geometry in the world.Therefore, for the correction results of different distortion models, the degree of correcting the distorted curved lines into the undistorted straight lines is the intuitive measure of the correction performance.A line in the Euclidean space corresponds to a clustering point in the Hough space.By analyzing the statistical consistence of co-linear feature points at different directions in the Hough space, we can conclude how many points lie in a straight line and how close they are to the straight line.The count of each bin in the Hough space then represents the degree of correction.The larger the count is, the more feature points are corrected for the line associated with that bin.
The paper is divided as follows.First, brief descriptions of five different distortion models are given.Second, lens calibration via cross-ratio constraint is briefly described.Third, we propose a Hough-based measure for evaluating distortion model.Finally, experimental results and conclusions are presented.

FIVE DISTORTION MODELS
The distortion model is a mapping from the distorted image to the corrected image, which is mathematically formulated according to the internal geometry of camera lens.The traditional pinhole model is a ideal model without taking lens distortion into account.The 3-D space points can be transformed to image coordinates with only the intrinsic and external parameters.However, the distortion is inevitable especially for wide-angle lens.Therefore there are various models proposed to solve the distortion and to refine the transformation.Some model are metric methods and other not.In the paper, we compare five metric distortion models according to their popularity and relevance in the state of the art.

Rational and Tangential Model(RT Model)
Radial and tangential distortion model is a conventional model which is widely accepted.According to the model, the formula of the model takes into account the radial, tangential and prism distortion.Given one point (u d , v d ) in the image, its corresponding corrected point (u, v) can be given such that where u and v represent the undistorted image coordinates, namely the ideal position in the image; u d and v d are the corresponding observed points with distortion; δu(u, v) and δv(u, v) are the distortion in u and v direction respectively, which is the sum of three types of distortion: the radial, tangential, and prism distortion.It can be written by the following equation: where r d = ∆u 2 d + ∆v 2 d represents the distance from the image point to the distortion center, which can be defined as (u0, v0), so The radial distortion is modeled by the first part of the formula.It can be given by where δur(u, v) and δvr(u, v) represent the displacement of point position in the direction of u and v respectively.The coefficients k1,k2,k3,... represent the degree of the polynomial function, and usually the first and the second radial symmetric distortion parameter k1,k2 are dominant and others negligible.
The tangential distortion is also known as decentering distortion, which arises from the decentering of lens.This is modeled can be recorded as where p1, p2 represent the coefficients of tangential distortion.
Another kind of distortion is called the prism distortion which comes from the tilt of lens when the lens are not perpendicular to the optical axis, and it is modeled by s1, s2.Usually, the radial distortion is in a dominant position, and the prism distortion is relatively insignificant and can be negligible (Weng et al., 1992).
2.2 Logarithmic Fish-Eye Model(FET Model) Basu and Licardie (Basu and Licardie, 1995) proposed a alternative model fitting for fisheye lens, which based on the logarithmic fisheye transformation.Given a point (u, v) representing the distortion-free cartesian coordinates in the image, we can denote (r, θ) as the corresponding polar coordinates, so r = √ u 2 + v 2 , θ = arctan(v/u).Then the corresponding distorted polar coordinates (r d , θ * ) can be given by where r d is the distorted radius, s represents the scale factor and λ controls the amount of distortion over the whole image.Then the corresponding distorted cartesian coordinates The inverse mapping is given by

Polynomial Distortion Model(PFET Model)
The model is one of the most frequently used distortion model, and it is similar to the FET model except that r d = G(r), where r is the radial distance for a undistorted image, r d is the distorted one.It can be given by the following equation: where k represents the degree of polynomial function.What is the difference between the polynomial distortion model and the FET model is that G(r) is a polynomial in r.

Division Model
Fitzgibbon (Fitzgibbon, 2001) introduced the division model for the simultaneous estimation of multiple view geometry and lens distortion.The division model is written as follows: or can be recorded as In this model, the formula is represented at the form of division function, so its Taylor expansion can be infinite.Therefore, the remarkable advantage of this model is that we can correct severe distortion using fewer terms than the polynomial model.Thus, it is more suitable for wide-angle lens.Additionally, the inversion of one parameter division model is simple and can find the roots of a second degree polynomial, so for most camera lens, a single parameter division model is adequate.

Rational Function Model(RF Model)
To adapt to the imaging process of perspective and non-perspective imaging system, Grossberg (Grossberg and Nayar, 2001) presented a general image model which used a set of virtual sensing elements to express the mapping from the incoming scene rays to the physical imaging sensors.It considers the camera as a "black box" and ignores the concrete process that the light goes through the camera internal optical sensors.Following this idea, Claus and Fitzgibbon (Claus and Fitzgibbon, 2005) extended image plane to 3-D scene, and built a general model for highly distorted lens, which called the rational function model.Given a distorted image point (u d , v d ) and corresponding distortion-free point (u, v), the mapping can be formulated by the quadratic given by Equation 11.This model can be written as a linear combination of the distortion parameters, in 3 × 6 matrix A, and a six-vector x, of monomials in u d and v d .Define x as the follows: So the rational function can be written by where d is a vector in camera coordinates representing the ray direction along which pixel (u d , v d ) samples.So we can have the perspective projection to obtain the undistorted image coordinates.i.e., where a T 1...3 represent the rows of the matrix A.

Figure 1: Principle of cross-ratio invariablity
According to the perspective projection in Euclidean space, straight lines have to be straight after perspective projection.More details can be found in (Devernay and Faugeras, 2001).In geometry, cross ratio remains invariable when they are under perspective projection, as is demonstrated in Figure 1.
Given four points A, B, C, D which lies in a straight line, the cross radio of four co-linear points can be written as where the points A and B are two datum points, and the points C and D are the reference points.When the four point are under perspective projection, there are four corresponding points A , B , C , D in the space.The cross ratio of the new four points are formulated as follows According to projective geometry, if four points are co-linear, the corresponding points also lie on a straight line.Furthermore, the cross ratio is a projective invariant, so that we can conclude that cross ratio of two sets of points is equal, the equation is defined as In our experiment, we use a checkerboard as a calibration plane, and it is projected on the screen.The corner points are detected on the image, and because of the collinearity of points, the cross ratio should be invariable in the image.Based on the work of Carlos and Antonio-Jose (Ricolfe-Viala and Sánchez-Salmerón, 2010b) (Ricolfe-Viala and Sanchez-Salmeron, 2010a) who proposed the metric point correction via the cross-ratio constraint, we detected m × n corner points from the image of the chessboard, where n is the number of the straight lines in the chessboard pattern and m is the number of points in each line.So, q k,l is a point k of the straight line l, l = 1 . . .n, k = 1 . . .m.To find the coordinate of the distortion-free point corresponding to each distorted point q k,l in the image, the difference of cross-ratio between the four sets of points detected in the image and the distortion-free ones must be minimized.On the other hand, a point in the straight lines have to fit the linear equation.Therefore, the equation including the two constraints can be given such that ||CR(q k , q k+1,l , q k+2,l , q k+3,l ) where a l , b l , c l are the set of parameters that define the straight line, which represent the linear constraint in the image; the second part of the equation represents the cross-ratio constraint, and CR(p1, p2, p3, p4) is computed previously which is equal to all four sets of points in the designed planar template.
The equation begins with the coordinates of the detected corner points in the image, and ends with the corresponding corrected ones when the minimization is done.So, with the detected points in the image and the corresponding distortion-free points via the cross-ratio and linear constraint, we can utilize the set of point pairs to get the coefficients of the five distortion models previously proposed in the paper.

HOUGH-BASED MEASURE FOR EVALUATING DISTORTION MODEL
As is mentioned before, there are many fisheye lens calibration methods.Which one is better needs to be determined.Traditional evaluation measure uses error function to represent calibration precision, but different models have different error functions.It is difficult to measure them under the unified framework.Besides, the error functions is not intuitive.So, a simple but effective evaluation measure is needed.Due to the fact that a line on the distorted image must be corrected into a straight line on the undistorted image, so whether lines of the undistorted image are still straight or not is a good measure of the correction performance.We analyse these lines in the Hough space, in which one line of the undistorted image is represented by one point (ρ , θ).Based on the corner points of the straight lines of the chessboard image, the Hough statistics is made for different (ρ , θ) combinations.A large value of the statistical result represents that more points coincide with the same line of (ρ , θ).The line (ρ , θ) of maximal statistical count in the Hough space is found to locate the corner points that are nearly aligned with the most significant line of (ρ , θ) in the image space.By analyzing the statistical consistence of co-linear feature points at different directions in the Hough space, as shown in Figure 3, we can conclude how many points lie in a straight line and how close they are to the straight line.The count of each bin in the Hough space then represents the degree of correction.The larger the count is, the more feature points are corrected for the line associated with that bin.
Evaluation measure consists of two aspects, precision and easeusing.For precision, we analyse the overall and local performance of lens correction.In the overall analysis, all corner points are checked whether they align with each other; in the local analysis, only lines of a certain direction is checked to evaluate the correction performance at that direction.Calculating the ratio of the aligned points after correction and the total points at a direction can evaluate the degree of correction of different models.High ratio is considered to have a better correction effect.For ease-using, little human interaction without compromising the correction performance is welcomed.

EXPERIMENTAL RESULTS
In this section, five fisheye lens calibration models have been tested with a diverse set of data which is captured by the Canon 5DI-II camera and the fisheye lens of a fixed focal length of 8mm.
A chessboard pattern is projected to a large screen by a projector, and taken at Wuhan University, China under varying lighting conditions, shadows, and shooting angles, as the Figure 4 shows.The pattern is made of 10×12 grids, and 99 corner points.The distortion coefficients are obtained by the cross-ratio constraint, which are used to correct the distorted image.The corner points on the red line of Figure 5 are mapped into the Hough space, where the ρ step is set to 14 pixel and the θ step is set to 1 degree.For each θ, one corresponding ρ is calculated to get a (ρ , θ) bin in the Hough space.
Based on the statistical count of Hough space, the red line of Figure 5 are located at the bin of (ρ , θ) as follows: (178,41) in RT model with the count number of 5, (179,38)  The distance threshold that defines whether a corner point is aligned with the line is set to 6 pixels.The lines of different directions in Figure 3 are chosen to evaluate the correction performance of different models along different directions.The result of corrected corner points at different directions is demonstrated in Figure 6, Figure 7 and Figure 8.
For traditional RT model, totally 72 corner points are got on all lines in direction 3 (horizontal direction) and totally 80 corner points on all lines in direction 4 (vertical direction), as shown in Figure 3.The total points is the least whether seen from horizontal line or vertical line.This result is consistent with the overall performance of the corresponding model.

CONCLUSION AND RECOMMENDATIONS
Fisheye lens is widely used in such fields as robot navigation, target detection and so on.Fisheye lens calibration is the first step of image processing.This paper focuses on the evaluation of the calibration performance.Based on the observation that lines become straight after the correction of the distorted image, the Hough-based measure is proposed in this paper to evaluate the calibration performance of different distortion models.
After lots of experimental tests, the proposed evaluation measure works well.For the five calibration models, it is concluded that 1) RT model works the worst; 2) the FET model, PFET model and division model have the similar performance and work better than RT model; 3) RF model works the best in fisheye lens correction in terms of precision.It is consistent with Carlos Ricolfe-Viala's (Ricolfe-Viala and Sánchez-Salmerón, 2010b) result and proves that the proposed evaluation measure is simple and effective.As far as ease-using are concerned, the division model with one parameter can get distortion parameter automatically and only needs one image with lines.It is very convenient and fast.So, in the case of non-restricted calibration, the division model is of practical use.

Figure 3 :
Figure 3: Four directions for evaluating lens correction (⊕ represents distortion center)

Figure 4 :
Figure 4: Distorted image captured by fisheye lens with fixed focal length

Table 3 :
Performance at the direction of 3 and 4 Due to the quantization and rounding error, some of the corrected points may fall outside of the range or be approximated.It results in the black points in the corrected image, for which no distorted points are corresponded exactly.But it has no effect on this paper because we aim at addressing the parameters of different models.With the model parameters, instead of finding the undistorted point for each distorted point, we can inversely find the distorted point for each undistorted point by the numerical iteration or interpolation.
Next is the FET model, we get 81 points on horizontal lines and 78 points on vertical lines, its number is larger than the RT model and less than the PFET model, whose number of points is 85 on horizontal lines and 77 on vertical lines.The RF model has better result than the first three models, the number of points reach 92 on horizontal lines and 76 on vertical lines.The last one is the division model, whose number is 83 on horizontal lines and 76 on vertical lines.Table2 and 3show the local correction performance along different directions and distances to the distortion center.We can see the ratio of RF model is the largest, while the FET, PFET, division model are almost the same and the RT model is the least.
Table1is the overall performance along different directions of each correction model.It can be seen that the FET, PFET and division model have the similar number of points, RT model has the least points and RF model has the maximum points.In order to further analyse which model is better locally, two lines which are far and near to the distortion center, as shown in Figure