A ROBUST METHOD FOR FUNDAMENTAL MATRIX ESTIMATION WITH RADIAL DISTORTION

Fundamental Matrix Estimation is of vital importance in many vision applications and is a core part of 3D reconstruction pipeline. Radial distortion makes the problem to be numerically challenging. We propose a novel robust method for radial fundamental matrix estimation. Firstly, two-sided radial fundamental matrix is deduced to describe epipolar geometry relationship between two distorted images. Secondly, we use singular value decomposition to solve the final nonlinear minimization solutions and to get the outliers removed by multiplying a weighted matrix to the coefficient matrix. In every iterative step, the criterion which is the distance between feature point and corresponding epipolar line is used to determine the inliers and the weighted matrix is update according to it. The iterative process has a fast convergence rate, and the estimation result of radial fundamental matrix remains stable even at the condition of many outliers. Experimental results prove that the proposed method is of high accuracy and robust for estimating the radial fundamental matrix. The estimation result of radial fundamental matrix could be served as the initialization for structure from motion. * Corresponding author


INTRODUCTION
Fundamental matrix describes the epipolar geometry relationship between two images in the same scene.It is independent of scene structure, and only depends on the camera internal parameters and motion parameters.Fundamental matrix estimation is a basic and key issue in computer vision.It plays an important role in many vision applications such as SLAM, motion segmentation, structure from motion, image stitching and dense stereo matching.Moreover, it's one of the core parts of 3D reconstruction pipeline.
Given its vital importance, many methods were proposed in the past decades.(Longust H, 1984) first proposed to apply epipolar geometry constraints to scene reconstruction.The five point relative pose solver with known camera internal parameters (Stewé nius H, 2006) and the six point relative pose solver with unknown focal length (Stewenius H, 2005), the well-known 7-point and normalized 8-point algorithm (Armangué X, 2003) were the frequently used linear estimation algorithm.These methods had high computational efficiency regardless of false matches and outliers, but poor accuracy and stability if taking these factors into account.Robust methods, such as RANSAC, LMedS, MLESAC, MAPSAC, could deal with false matches and outliers based on multiple sampling estimation parameter model, which resulted in heavy computational burden and low efficiency.
However, most of the above-mentioned method are based on the assumption that the camera follows pin-hole model, while with the popularity of unmanned aerial vehicle camera, cellphone cameras and GoPro Hero, the importance of radial distortion models increases, especially in 3D reconstruction and SLAM.A non-minimal method based on 15 correspondences for fundamental matrix estimation with radial distortion was first proposed in (Barreto J, 2005) A number of minimal problems for fundamental matrix estimation with radial distortion have been studied in (Kukelova Z, 2007a(Kukelova Z, , 2007b)), where practical solutions were given in some cases.Fast and robust algorithms for two minimal problems for simultaneous computation of fundamental matrix and two different radial distortion were given from 12 point correspondences based on a generalized eigenvalue formulation (Byrod M, 2008).A numerically stable and efficient solution for the calibrateduncalibrated image registration problem with radial distortion was presented in (José H, 2012).Based on the plumb-line assumption, constraints on the radial distortion center from epipolar geometry were derived (Henrique B., 2013).More recently, a fast and stable polynomial solver based on Gröbner basis method was derived (Jiang F, 2014), which enables simultaneous auto-calibration of focal length and radial distortion.In the meantime, the detail of using numerical Gröbner basis computations techniques was given.A more efficient and stable solution using 10 image correspondences was proposed by using the Sturm sequences method, which can be used in real-time applications (Kukelova Z, 2015).Moreover, a new formulation in which distortion center can be absorbed into the radial fundamental matrix was presented by (Brito J, 2013).These solutions make great progress in numerical stability and efficiency.However, the fast, accurate and robust solutions for the fundamental matrix with radial distortion estimation need to be further studied.
In this paper, we propose a new robust method for estimating the fundamental matrix with radial distortion based on 14 image correspondences.Unlike traditional robust estimation method, our method integrate the outlier removal procedure into the normalized 14-point algorithm, so as to estimate radial fundamental matrix robustly and fast.The main content of the paper is as follows: In section 2, the formulation of radial fundamental matrix is introduced and in section 3, the robust method for radial fundamental matrix estimation is discussed in detail.Finally, in section4, experiments on synthetic data and real images are implemented to prove the accuracy and robust of the proposed method.

Problem Formulation
Assuming that

F
In this problem, we adopt one-parameter division model given with homogeneous coordinates as the following formula: Where  = equality up to a scalar multiple  = the radial distortion parameter di m = distorted image points The distortion center is assumed to be known at first.Combined with the equation ( 1) and (2), using different distortion parameter 1  and 2  in each image, we can get the following equation: After transformation, which can be rewritten as Where = the extended corresponding points.
Using Kronecker products, equation ( 4) can be written as: Where T 11 44 ( , , ) From each correspondence, we obtain a different row vector When there are n correspondence points ， we can get n coefficient row vectors.Stacking these vectors together, we can obtain a 16 n  coefficient matrix A and the following linear equation: So, f must lie in the null space of coefficient matrix A , similar to the eight point for the fundamental matrix estimation.

Radial Fundamental Matrix Estimation
In the ideal case, the effective dimension of the null-space of coefficient matrix equals to 2. However, account to the influence of noise and outliers, the effective dimension is always 0，which means no singular value 0. To deal with noise and outliers, we propose a new strategy to compute the nullspace of A .Introducing a weighted matrix W and multiplying it with the coefficient matrix, solving the linear equation ( 7) can be transformed into a nonlinear minimization problem:  In the same way as the strategy 7-point algorithm adopting for the improving of classical fundamental estimation, we use at least 14 correspondence for the above nonlinear minimization problem, where we can fix one of the coefficients and the true f is a combination of 1 f and 2 f . 1 2 (1 ) We perform the inverse vectorization of f and F can be reassembled as: In order to obtain  , the rank 2 constrain of F should be used and a cubic determinant constraint 1 2 3 det( ) 0 F F F  could be obtained, where the only unknown variable is .Furthermore, rank two of the fundamental matrix must be enforced via singular value decomposition.
And the element i w in weighted matrix W can be updated using the following function： max max 1 max( , ) is the lowest 15% quantiles of the geometric error: max  = the maximum geometric error threshold By introducing this threshold, the probability that the correct inliers is classified as outliers is effectively reduced, and the iterative convergence rate is accelerated.In the following experiment, we set max 1.6 The above algorithm is iteratively computed until the null space of f converges, then we can get the final estimation result of radial fundamental matrix F .Generally, the iterations converge quickly.Simple algebraic manipulation of Eq. (3) for distortion parameters computation leads to: In order to improve the estimation accuracy and eliminate the influence of coordinate transformation, the corresponding point coordinates must be normalized before the iterative computation.
Assuming the image coordinate of a feature point is ( , ) ii xy ， the image coordinates of feature points after normalization is ( , ) ii xy: between feature points and feature center    = the center of all feature points T = the normalized transformation matrix: After normalization, an effective standard coordinate system can be selected to eliminate the influence of coordinate change and the data would be scale invariance and coordinate origin invariance, thus effectively improving the accuracy of estimation results.

Experiments
In this section, we demonstrate the accuracy and robustness of proposed method by experiments on both synthetic data and real images.

Synthetic data:
In the experiment with synthetic data, we generate two images with the size of 640*480, where the equivalent focal length is 800 pixels, the distortion center is at the center of the images and the distortion parameters are [-0.234;-0.145] respectively, regarded as true  .The relative pose of the synthetic images are known and 3D reference points are randomly generated in the cubic space .The 2D correspondence points can be generated according to pinhole projective model, then distorted the points on the image of both cameras according to distortion model.The experiment data includes two groups: in the first group, we add a set of outliers disturbed by the false matches to the 2D correspondence points, whose percentage are 10%, 20% and 30%.We compare our method with the method proposed by (Barreto J, 2005) adding RANSAC loop at the beginning.Fig1 Distortion Parameter Error varying with respect to Gaussian Noise In the second group, we test distortion parameter error changing with respect to varying noise level.A set of different types of Gaussian noise are added to the image pairs to determine their effects on the estimation of the fundamental matrix and distortion parameters, whose means is 0 and variances varying from 0.5 to 2.5.
The result in Fig. 1 shows that distortion parameter error increases with the increase of noise level, while the stability is on the decline.But when the noise standard deviation is less than 2.5 pixel, the proposed method could get high accuracy estimation of radial distortion parameter.After all, compared with methods proposed by (Barreto J, 2005), our method turns out to be more robust to the noise and outliers.When there are severe noise or outliers, the errors of our method are smaller.In other words, the proposed method could cope with outliers effectively，and is better than RANSAC based method.

Real image data:
In the real image experiment, we test our method with image dataset downloaded from the internet.The image sequences is shown in Fig2.

Figure 2 Image sequences
For each image, sift features are extracted by SIFTGPU and then the nearest neighbour method is used to match the feature points for each pair of images.Given these preliminary matches, we can perform the computation of the radial distortion matrix, which can serve as initialization for the following iterative calculation.After iterative calculation, we select 30 correspondence inliers from the initial matching points according to the epipolar distance criterion.As is seen in  As the best of our knowledge, bundler is the state of art method for 3D reconstruction, but not suitable to the distorted images.We adjust it with our proposed method to estimate the relative pose of the initial two images, and the result is shown in Fig 5.The sparse reconstruction result shows that the reconstructed scene is clear, the outlier is very few, and the camera pose is reasonable.All of these reveal the effectiveness of our method in structure from motion, especially the existence of radial distortion.
The proposed method can also be applied to relative pose estimation for satellite imagery.The proposed method are tested by using World-View III imagery, Sequestered Park (MVS3D Mapping, 2016).30 correspondence inliers are shown in Figure 6, while the corresponding epipolar lines are plotted in Figure 7.As is illustrated, feature points (red diamond) are precisely located on the epipolar lines (green line), which means that the relative pose of the two cameras are estimated accurately.

CONCLUSIONS
This paper presented a new robust method to estimate the fundamental matrix with radial distortion and perform outliers removing simultaneously.By multiplying a weighted matrix, which is updated by calculating the distances between the matching points and the corresponding epipolar lines, to the coefficient matrix, the proposed method makes the SVD based solutions more robust, and retain its original characteristics such as fast, global convergence and numerical stability.The experiment results show the effectiveness and robustness of our method.With our solvers, the estimation result of radial fundamental matrix could be served as the initialization for structure from motion.
on the two images , II  taken by different cameras in the same scene.According to epipolar geometry theory, the 33  fundamental matrix has the following relationship with the two points in undistorted case: Fig 3, where the green circle marks feature points, the green line shows the correspondence relationship of the two feature points。The corresponding epipolar lines are plotted in Fig 4, in which feature points (red diamond) are precisely located on the epipolar lines (green line), which means the epipolar geometry relation is estimated exactly.

Figure 5
Figure 5 Sparse reconstruction result

Figure 6
Figure 6 Partial correspondence inliers of satellite imagery

Table 1 .
Comparison of estimation results of the two methods under different percentage of outliersThe mean and standard deviation of distance from point to epipolar line is listed in Table1, from which we can see that under the condition of high level of outliers and false matches in the test dataset, the proposed method can eliminate the potential outliers to a certain extent, thus reducing its influence on the estimation results of the radial fundamental matrix.