CALIBRATION OF PANORAMIC CAMERAS WITH CODED TARGETS AND A 3 D CALIBRATION FIELD

The aim of this paper is to present results achieved with a 3D terrestrial calibration field, designed for calibrating digital cameras and omnidirectional sensors. This terrestrial calibration field is composed of 139 ARUCO coded targets. Some experiments were performed using a Nikon D3100 digital camera with 8mm Samyang Bower fisheye lens. The camera was calibrated in this terrestrial test field using a conventional bundle adjustment with the Collinearity and mathematical models specially designed for fisheye lenses. The CMC software (Calibration with Multiple Cameras), developed in-house, was used for the calibration trials. This software was modified to use fisheye models to which the Conrady-Brown distortion equations were added. The target identification and image measurements of its four corners were performed automatically with a public software. Several experiments were performed with 16 images and the results were presented and compared. Besides the calibration of fish-eye cameras, the field was designed for calibration of a catadrioptic system and brief informations on the calibration of this unit will be provided in the paper.


INTRODUCTION
Conventional perspective cameras, also known as pinhole cameras in the Computer Vision community, have a limited field of view that can be augmented by reducing camera focal length.However, there is a limit for this reduction, due to vignetting and other undesired effects.
To cope with this limitation of perspective cameras, several approaches have been developed to produce omnidirectional or panoramic images, which can have 360° field of view.Several techniques are used to achieve larger fields of view: the use of specially designed optics (fish eye lenses); the combination of several perspective cameras; the use of pure reflexive systems and the combination of refractive and reflexive optics (catadioptric systems) (Yasushi, 1999;Ray, 2002;Sturm et al. 2010).
Several mathematical models have been developed in both the photogrammetric and the computer vision communities for these non-perspective systems.Some of these models are rigorous and based in the physical properties of the imaging systems (Abraham and Förstner, 2005;Van den Heuvel et al., 2006;Schneider et al., 2009) and others are generic or empirical (Barreto and Araujo, 2002;Scaramuzza et al., 2006;Mei and Rives, 2007;Puig et al., 2011).The application of rigorous or generic models is related to the accuracy to be achieved and the application.The determination of the model parameters are achieved in the process of calibration and this fundamental step normally is based in a test field with signalized targets which coordinates can be precisely determined, depending on the calibration technique used.
The accurate measurement of these targets in several images is a key step in the calibration procedure.This procedure can be done interactively on the screen, but this is time consuming and prone of errors.To cope with this problem, several specially designed targets have been used, which can be automatically located and recognized in the digital images (Fraser, 1998).These targets can be set in a flat surface or even in a plotted sheet, a strategy that is used by some commercial software, like Photomodeler (Photomodeler, 2013).These 2D calibration fields are much easier to establish but have some limitations in the case of calibration of omindirecitonal systems.Also, the existing correlations in the parameters to be estimated with bundle adjustment can be reduced when using 3D calibration fields (Fraser, 2013).
The aim of this paper is to present results achieved with a 3D terrestrial calibration field, designed for calibrating digital cameras and omnidirectional sensors.This terrestrial calibration field is composed of 139 ARUCO coded targets.Some experiments were performed using a Nikon D3100 digital camera with 8mm Samyang Bower fisheye lens.Besides the calibration of fish-eye cameras, the field was designed for calibration of a catadrioptic system; brief informations on the calibration of this unit will be provided in the paper.

CALIBRATION OF FISHEYE CAMERAS
The Collinearity mathematical model combined with lens distortion models is generally used in the photogrammetric camera calibration process.However, fisheye lenses are designed following different projections models such as: Stereographic, Equi-distant, Orthogonal and Equi-solid-angle.In general, the fisheye lenses follow the Equi-distant and Equisolid-angle projections (Abraham and Förstner, 2005;Schneider et al., 2009).These rigorous mathematical models can be used in combination with radial symmetric, decentering and affinity distortion models.
In this paper, a low cost fisheye lens (Samyang) is being used with a Nikon D 3100 digital camera.This lens was designed following the Stereo-graphic projection model (Charles, 2009), which is less common than other mentioned models (Ray, 2002).
Camera calibration can be done by an indirect estimation process, in which the inner orientation parameters (IOPs) are recovered.The IOPs of digital cameras are the focal length, the principal point coordinates and the lens systematic errors coefficients (lens distortions: symmetric radial and decentering; and affinity).The calibration, in general, is accomplished using the collinearity equations with additional parameters, as presented in Eq. 1. (Mikhail et al., 2001).
where f is the focal length; (X C , Y C , Z C ) are the point 3D coordinates in the photogrammetric reference system (Eq.2); (x, y) are the image point coordinates in the photogrammetric reference system; (x', y') are the image point coordinates in a reference system parallel to the photogrammetric system with origin in the image centre; (x 0 , y 0 ) are the coordinates of the principal point (PP) and; Δx and Δy are equations describing the systematic errors (Eq.3).
, in which r ij are rotation matrix elements relating object and image reference systems; (X, Y, Z) are the coordinates of a point in the object reference system; and (X CP , Y CP , Z CP ) are the coordinates of the perspective centre (PC) in the object reference system.

By Ax
in which K 1 , K 2 , K 3 are the symmetric radial distortion coefficients; P 1 e P 2 are the decentering distortion coefficients; A and B are the affinity parameters.The radial symetric and decetering distortions formulation were developed by Brown (1971).
The collinearity equations are generally used in the calibration process, but the image acquisition with fisheye lens camera does not follow this condition.In the perspective projection incident and emerging rays are equal (α = β), except by the small deviations caused by lens distortion.With fisheye lens, the rays are refracted toward the optical axis as shown in Fig. 1.
Figure 1 -Geometry of the image aqusition with a fisheye lens camera (adapted from Abraham and Förstner (2005)).
The fisheye lenses are generally designed following the Stereographic, Equi-distant, Orthogonal or Equi-solid-angle projection models.Table 1 presents the mathematical models based on these projections (Abraham and Förstner, 2005;Schneider et al., 2009).Charles (2009) presents a discussion on the technical features of Samyang fisheye lens camera, which was designed following the stereographic concept (Fig. 2).

Projections Equations
Table 1 -Mathematical models for the calibration of fisheye lens camera (Adapted from Ray (2002); Abraham and Förstner (2005)).The analysis of Table 1 shows a sign difference in the focal length between the mathematical models.For the perspective and equi-distant projections the sign is negative, and for the other models the sign is positive.Fig. 3 shows the photogrammetric reference system which justifies this difference.

3D CALIBRATION FIELD WITH CODED TARGETS
In order to facilitate the automation of the calibration process a special 3D terrestrial calibration field with coded targets was build.This 3D field is composed of 139 coded targets, using the ARUCO codification (Muñoz-Salinas, 2012; Garrido-Jurado et al, 2014).These targets and the software for automatic identification and extraction were originally developed for augmented reality.These targets have two main parts: an external crown, which is a rectangle and 5x5 internal squares in five rows and five columns.From the five columns only the second and the fourth one are used to store data.The other three columns are used for integrity check and error detection.Considering that combination, this target structure can encode 10 bits of information, or 1024 values (Figure 4).The first step in the algorithm is edge detection with an adaptive threshold technique.Then, the edge pixels are connected defining contours; short edges are eliminated.With these polygonal approximations only the concave contours with four corners (like rectangles) are stored.The corners are then ordered in counterclockwise direction.Close rectangles are eliminated because some edge detectors create both the internal and outer borders of the crown.The next steps are aimed at the identification of the target.Firstly, the detected rectangle is rectified with a transformation, to reduce the perspective effect.A threshold with OTSU method is applied and this segmented area is divided into 6x6 cells.The internal 5x5 cells hold the code and the rest are part of the external crown.The crown is checked before to ensure that the segmented rectangle is a target and the 5x5 cells are analysed to check if a valid code is achieved.A rotation can be applied to get a valid code.Then, for the valid targets, the corners coordinates are refined by intersection of the adjusted lines.
A public existing software (Muñoz-Salinas, 2012; Garrido-Jurado et al, 2014) was adapted to perform automatically the location, identification and accurate measurement of the four corners of the external crown of targets of the calibration field (Silva, 2012).The original ARUCO software was developed in C++ based on OPENCV free library.
With the adapted software, most of the existing coded targets are automatically located, recognized and the coordinates of the corners of the bounding rectangle are extracted with subpixel precision.Some missing corners can then be interactively measured to provide enough points with suitable geometry for the camera calibration.To improve the image quality the shadows were also segmented and enhanced using a specially designed algorithm (Freitas and Tommaselli, 2013).
A software was implemented in Java for the generation of pictures of the chosen targets.Those pictures were plotted in vinyl sheets and taped over ACPs (Aluminium Composite Plates) resulting in targets with dimensions of 35x35 cm.This size was selected to guarantee the suitable recording of its internal blocks with code even in low resolution and blurred images.The targets were distributed over the field area in row and columns with a separation of approximately 1.4 m.The target number is generated from the approximated position in this 3D matrix.Besides the target code, the software was adapted to store the four corners with an extra label, starting from the top left corner (label 0) in clockwise direction.
The 3D coordinates of the targets corners on the calibration field (Figure 5) were estimated using geodetic and photogrammetric methods.Initially, four reference points (Figure 5.c) were surveyed during eight hours with a double frequency GNSS (Global Navigation Satellite System) receiver.The distances between these points were measured with a Total Station and the discrepancies among the electronically measured and those computed from the 3D coordinates were around 1 mm.Forty-three ( 43

EXPERIMENTS AND RESULTS
A Nikon D 3100 camera with Samyang-Bower 8 mm fisheye lens (see Table 2) was calibrated using the Collinearity, Stereographic, Equi-distant, Orthogonal and Equi-solid-angle mathematical models.These mathematical models were implemented in an in-house developed software package called TMS (Triangulation with Multiple Sensors) (Ruy et al., 2009;Marcato Junior and Tommaselli, 2013) A set of 16 images was acquired (see Figure 6) for the calibration process as follows.Twelve horizontal images were collected at three camera stations.At each station, four convergent images were acquired with changes in positions and rotations.In addition, four vertical images were also collected in the same position but at different heights.This procedure was used to minimize the linear dependency between the interior and exterior orientation parameters.
Self-calibrating bundle adjustment was used to compute parameters with a set of seven minimum absolute constraints.
The 3D coordinates of one target in the object space, the X, Y and Z coordinates of a second one and the Z of a third one were introduced as absolute constraints.The X coordinate of the second point was measured with a precision calliper with an accuracy of 0.1mm.The coordinates of the remaining GCP were introduced in the bundle adjustment as weighted constraints with a standard deviation of 5 cm.A further set of six distances between signalized targets (Figure 5.a) were measured with a precision calliper with an accuracy of 0.1 mm, and these distances were used to check the results of the calibration process.After bundle adjustment the distances between two targets can be computed from the estimated 3D coordinates and compared to the distances directly measured.
Experiments were accomplished with different sets of IOPs.The analysis of Table 3 shows that the standard deviation of unit weight 0  (a posteriori) estimated with the collinearity model is larger when compared to the other models, because the image coordinates residuals are larger.The standard deviation of unit weight 0  for the Stereo-graphic, Equi-distant, Equi- solid-angle and Orthogonal when considering all IOPS are around ½ pixel, the same value of the estimated precision of the corner measurement process.The results for these four models can be considered similar.
Table 3 also shows that the best result (smaller 0  ) is achieved when all the IOPs (f, x 0 , y 0 , K 1 , K 2 , K 3 , P 1 , P 2 , A, B) are considered.However, when analysing the estimated standard deviation of IOPs it was verified that some of them were not significant, mainly P 1 and A. For this reason in Table 4, only the most significant parameters are presented with the estimated standard deviation.With these sets of IOPs an assessment was performed based on some directly measured distances.The estimated standard deviation of the IOPs with Stereographic, Equi-distant, Equi-solid-angle and Orthogonal models are smaller when compared to the collinearity model, according to Table 4.It is also verified that the focal length standard deviation is smaller than 0.4 pixels for all the models.It is important to mention that the estimated parameter K 1 for the collinearity model absorbs the effect of the rays' refraction toward the optical axis as shown in Fig. 1 but this modelling is not enough to recover the inner bundle geometry, in comparison with the other models assessed.
It is also verified that x 0 presents different signs for the models, which is caused by the differences between the photogrammetric reference systems depicted in Fig. 3.The correlations between the IOPs and EOPs were significantly reduced with this 3D calibration field.For example, the correlation between the focal length and the Z coordinate was around 0.7.
Table 5 presents the average, standard deviation and RMSE (Root Mean Squared Error) of the discrepancies in 6 check distances, in the object space, for all the experiments.It can be seen that the errors in the check distances were higher in the experiment with collinearity model, which confirms the previous conclusions.The values for the other models are similar, and it can be concluded that they provide similar results with this level of accuracy in the measurement of image coordinates.

ONGOING RESEARCH
Another system being developed by the research group is a mobile catadioptric omnidirectional vision system composed of a camera and a conic mirror with direct georeferencing system (double frequency GNSS receiver and an IMU).The calibration of this system with a rigorous model in under investigation and some experiments are being conducted with the 3D calibration field.Some previous experiments showed that the same technique can be applied to the automatic measurement of ARUCO targets in the omnidirectional images.Mathematical models relating object and image spaces are under development.The mathematical models use the collinearity equations and equations that represent the conic mirror reflection.

CONCLUSION
The aim of this paper is to present results achieved with a 3D terrestrial calibration field, designed for calibrating digital cameras and omnidirectional sensors.Some experiments were performed using a Nikon D3100 digital camera with Samyang-Bower fisheye lens were also presented with different models for calibration.Experiments were conducted with images from a 3D field calibration.
It was verified that the collinearity mathematical model, which is based on perspective projection, presented the less accurate result, which was expected because fisheye lenses are not based on perspective projection, as it was already discussed in the literature.Stereo-graphic, Equi-distant, Orthogonal and Equisolid-angle projections presented similar results in the studied cases, although Samyang fisheye lens was built based on Stereo-graphic projection.
In conclusion, Nikon D3100 camera with Samyang 8 mm fisheye lens, after rigorous calibration with bundle adjustment, can be used in close range Photogrammetry applications.
In future work, techniques will be developed to fully automate the measurement of image points and the calibration process.Experiments will also be performed to compare rigorous and generalized mathematical models to calibrate fisheye lens camera.
(a) The Samyang-Bower 8mm fisheye lens camera and; (b) the Nikon D3100 digital camera.
Figure 4 -(a) Codification scheme of an ARUCO target and (b) and example of a target.
) images of the calibration field were acquired by a Hasselblad H3D (50 Megapixels) camera with a 35 mm lens, with a GSD of 3 mm.The coordinates of the other points were estimated with on the job calibration, of approximately 3 mm for the photogrammetric points.The coordinates of each one of four corners of these 139 targets (Figure5) can be used either as constraints or as initial values, depending on the strategy used in the calibration trial.Some external distances between targets were also measured with a precision calliper to provide both a control distance and some distances for check.(a) A view of the 3D calibration field with some known distances; (b) the full set of 3D points defined by the targets corners (c) a bird eye view of the calibration field with four GCP in yellow (d) image with shadows enhanced.

Figure 6 -
Figure 6 -Images acquired with Nikkon D 3100 camera with Samyang 8 mm fisheye lens over the 3D calibration field with coded targets.
Figure 7 presents the omnidirectional system and an image of the 3D calibration field collected by this unit.(a) Catadioptric omnidirectional vision system and (b) an image from the 3D calibration field.

Table 2 .
. Technical data of the Nikon D3100 digital camera and Bower fisheye lens.

Table 4 -
Estimated IOPs and standard deviations.