COMPARISON OF TARGET-AND MUTUAL INFORMATON BASED CALIBRATION OF TERRESTRIAL LASER SCANNER AND DIGITAL CAMERA FOR DEFORMATION MONITORING

In the current state-of-the-art, geodetic deformation analysis of natural and artificial objects (e.g. dams, bridges,...) is an ongoing research in both static and kinematic mode and has received considerable interest by researchers and geodetic engineers. In this work, due to increasing the accuracy of geodetic deformation analysis, a terrestrial laser scanner (TLS; here the Zoller+Fröhlich IMAGER 5006) and a high resolution digital camera (Nikon D750) are integrated to complementarily benefit from each other. In order to optimally combine the acquired data of the hybrid sensor system, a highly accurate estimation of the extrinsic calibration parameters between TLS and digital camera is a vital preliminary step. Thus, the calibration of the aforementioned hybrid sensor system can be separated into three single calibrations: calibration of the camera, calibration of the TLS and extrinsic calibration between TLS and digital camera. In this research, we focus on highly accurate estimating extrinsic parameters between fused sensors and targetand targetless (mutual information) based methods are applied. In target-based calibration, different types of observations (image coordinates, TLS measurements and laser tracker measurements for validation) are utilized and variance component estimation is applied to optimally assign adequate weights to the observations. Space resection bundle adjustment based on the collinearity equations is solved using Gauss-Markov and Gauss-Helmert model. Statistical tests are performed to discard outliers and large residuals in the adjustment procedure. At the end, the two aforementioned approaches are compared and advantages and disadvantages of them are investigated and numerical results are presented and discussed.


INTRODUCTION
In the current state-of-the-art, geodetic deformation analysis of natural and artificial objects (e.g.dams, bridges, towers, railroads, landslides,...) is an ongoing research in both static and kinematic mode.In this research, due to increasing the accuracy of geodetic deformation analysis, TLS and a high resolution digital camera are integrated to complementarily benefit from each other.On the one hand, TLS can provide high resolution 3D data in the sub-millimetre range in combination with reflectivity values.Consequently, a reflectance image can be generated using central perspective representation to project the 3D point clouds to a virtual image plane.On the other hand, digital camera can acquire rich and high quality colour images.In the integrated sensor system, high resolution cameras are advantageous due to having high angular accuracy of sub-pixel accuracy image measurements which would improve the lateral accuracy of laser scanners (Schneider & Maas, 2007).In addition, this integration focuses at filling gaps in TLS data to compensate modeling errors and to reconstruct more details in higher resolution (Moussa et al., 2012).
The main purpose of this research is to high accurately estimate extrinsic parameters between TLS and digital camera to compensate the deficiency of the TLS measurements for deformation monitoring of the objects, e.g. in case of large incidence angle, by using high resolution camera.Furthermore, digital images would assist us to detect deformation analysis in both direction of laser beam and perpendicular to laser beam.Moreover, this integration leads to increasing redundancy in the adjustment procedure.
In order to ideally relate digital camera coordinate frame to the TLS coordinate frame, the digital camera is mounted on top of the TLS using clamping system (figure 1, right).To avoid any vibration of digital camera and blurring of images, Nikon wireless mobile utility application is setup on the cell phone and by the usage of Wi-Fi connection, photographs are captured indirectly.The calibration of the aforementioned hybrid sensor system can be separated into three single calibrations: calibration of the camera, calibration of the TLS and extrinsic calibration between TLS and digital camera.The interior orientation of the camera and internal error sources of the TLS can be determined in the laboratory to reach high accurate calibration values.However, external error sources especially atmospheric and object related errors could not be considered in the laboratory and need on-site calibration to be removed as well.Unnikrishnan and Hebert (2005) described the algorithm to estimate the extrinsic calibration parameters of a camera with respect to a laser rangefinder using checkerboard calibration targets.In this method, a plane fits to the manually selected targets on the checkerboard pattern and aligned it with the detected pattern of the image using optimization algorithm.Moussa et al. (2012) proposed an automatic procedure to combine TLS and digital camera based upon free registration using ASIFT and RANSAC algorithm to match reflectance image and RGB image.Absolute camera orientations are obtained on the basis of space resection method.Lichti et al. (2010) presented the self-calibration of the range camera with respect to the rangefinder in a free-network bundle adjustment using signalized targets.Variance component estimation is applied to optimally re-weight observations iteratively.Pandey et al. (2012) proposed the automatic targetless extrinsic calibration of a Velodyne 3D laser scanner and Ladybug3 omnidirectional camera on the basis of the mutual information (MI) algorithm to estimate extrinsic calibration parameters by maximizing the mutual information between the reflectivity values of the laser scanner and intensity values of the camera image.Taylor and Nieto (2012) proposed a method to compute extrinsic and intrinsic calibration parameters between camera and LIDAR scanner.This approach utilizes normalised mutual information to compare images with the laser scans projections.
Particle swarm optimization algorithm is applied to optimally determine the parameters.
In this research, we focus on highly accurate estimation of the extrinsic parameters between TLS and digital camera which is necessary and preliminary step for deformation monitoring.Two different strategies of target-and MI based are applied.In target-based approach, focal length of the camera, exterior orientation parameters between TLS and camera (position and orientation; 6 DOF), exterior orientation parameters between TLS and laser tracker (scale, position and orientation; 7 DOF) and target coordinates are estimated with high accuracy.Laser tracker (LT (figure 1, left)) measurements are carried out with superior accuracy and independently.In addition, its measurements are considered as a reference coordinate frame.
In MI-based approach, extrinsic calibration parameters between TLS and digital camera are obtained with adaptation and modification of the Pandey's work to our research purpose by considering horizontal angle measurement of the TLS as additional parameter into the transformation matrix.

DATA ACQUISITION, INTERFACING AND PRE-PROCESSING
In the target-based approach, data acquisition step comprises image measurements, TLS measurement and LT measurements.
In the MI-based approach, it consists of the RGB image from the digital camera and generated reflectance image from the TLS reflectivity values.
Images are captured with Nikon D-750 24.3 megapixel digital camera and centre of each target is computed based upon detection of the four centriods of the circles within each target and performing averaging.For instance, an exemplary target with detected centriods is illustrated in figure 2. Afterwards, extracted image measurements are rectified based on well known Brown's equations to eliminate the effects of radial and decentring distortions.TLS point targets accurately acquired using "Fit target" mode of the Z+F LaserControl software.
The horizontal angle measurement of the TLS (Az) is defined as a 3x3 rotation matrix to rotate TLS coordinate frame around its Z-axis to the digital camera coordinate frame (Al-Manasir and Fraser ( 2006)).It is written for each captured image and can be considered as additional observation in the adjustment procedure.As can be seen in figure 1 (right), TLS coordinate frame, digital camera coordinate frame and Az are depicted.LT measurements are utilized as an additional observation and they are obtained by pointing to the mounted corner cubes which are located at the centre of each target.
Reflectance images can be generated based on the scanning matrix and central perspective representation.In the first approach, each 3D data is assigned to one pixel based on the scan resolution.It is quite simple and fast.However, as a drawback, straight lines appear as curved lines (Meierhold et al., 2010).In the second approach, TLS data is projected to a virtual image plane on the basis of the collinearity equation (Moussa et al., 2012).
Figure 3. Definition of coordinate systems (Meierhold et al., 2010) As can be seen in figure 3, the maximum and minimum horizontal angle of the TLS is determined to reduce the size of the entire scan of TLS data to project into the image space.In the figure 4, rectified image and generated reflectance image from TLS data are depicted.

Target-based calibration
In the target based approach, focal length of the camera, extrinsic parameters between TLS and camera, extrinsic parameters between TLS and LT and target coordinates are estimated as unknown parameters.In order to obtain exterior orientation parameters between TLS and digital camera, space resection bundle adjustment is employed based upon collinearity equations to determine the condition that a perspective centre, a point in the image space and its corresponding coordinate in the object space are on a straight line (equations 1 & 2).Since aforementioned equations are nonlinear with respect to the parameters, it needs to be linearized to compute parameters iteratively.Therefore, initial starting values are estimated using direct linear transform (DLT) in combination of RANSAC algorithm to robustly estimate the parameters.In addition, the mathematical model to estimate the exterior orientation parameters between TLS and LT is solved based on similarity transformation (equation 3).Furthermore, additional constraint is defined to compute target point coordinates in the object space (equation 4).Thus, least square solutions are computed by means of the Gauss-Markov model (GM) and Gauss-Helmert model (GH).Therefore, four target functions (equations 1-4) are determined as follow: (1) where =   .  .  (6) In equations (1-6), (, ) are the target coordinates in the image space, (, , ) are the target coordinates in the object space, (  ,   ,   ) are TLS point coordinates of the targets, (  ,   ,   ) are the translations between TLS and digital camera, (, , ) are the rotation angles between TLS and digital camera, (Az) is horizontal angle measurement of the TLS, (X C ′ , Y C ′ , Z C ′ ) are the translations between TLS and LT, (  ,   ,   ) are LT point coordinates of the targets, ( ′ ,  ′ ,  ′ ) are the rotation angles between TLS and LT and () is the scale factor between TLS and LT.GM model is a set up linear or non linear relation between observations and unknown parameters.It is merely determined by observations to estimate unknown parameters.In this type of least square adjustment, square sum of residuals are minimized for one type of observation (image measurements).GH model is more complete and sophisticated model compared to GM model and comprising all the unknown parameters and observations that can be updated as unknowns iteratively.

Gauss-Markov model:
In this research, GM model is solved based on equations 1 & 2. In equation 7, v is a vector of residuals, A is a matrix of the coefficients of the unknowns which is so called design matrix, Δ is the reduced vector of parameters (unknown extrinsic parameters) and  0 is the reduced vector of observations.In equation 8,   0 and   0 are the target functions of the equations 1 and 2 which are substituted for the initial values.(8) 3.1.2Gauss-Helmert model: GH model is the combination of the observations and unknowns in the target functions and it is denoted as: Where  ̂ is estimated observation vector,  ̂  estimated unknown vector, A matrix is derivative with respect to the unknown parameters, B matrix is derivative with respect to the observations and w is the vector of misclosures.Thereafter, unknown parameters are computed as follows: Where  ̂ is the estimated reduced unknown vector.Moreover, vector of residuals are computed by: Where   is the cofactor matrix of the observations.In addition, a-posteriori variance factor ( ̂02 ) is calculated as follows (Niemeier, 2002): Where  is the number of constraints and  is the number of unknown parameters.

Variance Component Estimation and Statistical
Test: Variance component estimation is applied to assign optimal weights to the observations in the adjustment procedure iteratively.The statistical test is performed to investigate the adjustment results.Additionally, the uncertainty of the measurements and unknown parameters is computed.In this research,  2 and  test with 95% confidence level are applied to detect outliers.

Mutual Information-based Calibration
Mutual information (MI) is used to detect statistical dependencies or a measure of coupling between signals (Pompe et al., 1998).MI is defined on the basis of Shannon's definition of entropy (equation 13) and is interpreted based upon the amount of information and event that occurs, the uncertainty about the result of an event, and the dispersion of the probabilities when the event occurs (Alempijevic et al., 2006).
Where   is the probability mass function of random variable i.

In this work, MI is determined on the basis of the entropy of reflectance image (H(A)), entropy of RGB image (H(B)) and H(A,B) as a joint entropy (equation 14) and generally it means the amount of information that A contains about B. MI is maximized by maximizing the terms H(A) and H(B) and minimizing the H(A,B)
. In equations (15-17), a and b are the real numbers of the events that random observations A and B of a probabilistic experiment are mapped onto them (Pandey, 2014).MI based approach is an automatic procedure that is usually applied in outdoor calibration without any need of mounted targets in the field.In this approach, extrinsic calibration parameters are estimated by maximizing the mutual information between reflectance image of TLS and RGB image of the digital camera and correlation coefficients are computed.Then, different scan measurements from different horizontal angle measurements of TLS are considered in a single optimization framework and the parameters of interest are computed by means of the gradient ascent algorithm (Pandey et al., 2012).
The main goal of the author is to apply MI based approach for in situ calibration to eliminate systematic errors (e.g.clamping system) and consequently avoiding target based calibration in the field.In order to adapt pandey' algorithm to our work, Az included as additional parameter in the transformation matrix to re-project 3D point clouds to the 2D image correctly.Thus, equation 5 is utilized to perform this projection.
As a drawback of this method, it cannot be applied directly to range sensors without associated reflectivity information.Furthermore, it needs quite good initialization values of 6 DOF.Moreover, in case of speed up of the algorithm, it is significantly slower than target based approach.In addition, in order to obtain better results, reflectivity values of the TLS need to be calibrated in addition to image enhancement and filtering of the RGB image (e.g.brightness and contrast).

EXPERIMENTS AND RESULTS
In this research, two different approaches of target-and MIbased are investigated in two case studies.In the first case study, two different adjustment models (GM and GH models) are solved.GM model is just implemented as preliminary test to achieve primary results very fast.But, due to more complete and accurate results of the GH model compared to the GM model, merely its results from target-based approach are presented.In the second case study, MI based approach from Pandey's work is adapted to this research and its results are presented and discussed.
For the experiments, a calibration room is measured with Z+F IMAGER 5006 in the super high resolution mode with horizontal and vertical angle resolution of 0.0018 °.It has the maximum field of view of 360 °× 310 ° in horizontal and vertical respectively and its accuracy is 0.007 ° rms.Thereafter, 84 images are captured with Nikon D750 digital camera to fully cover our calibration room.Targets are measured in the both image and object space, respectively.The number of measured targets in object space is 25 and number of measured targets in the image space is 395.LT is utilized for validation and check the accuracy of our calibration results with the super high accuracy.

Case Study I:
In this case study, least square solution is solved on the basis of GH model.Employed sensors are TLS, LT and digital camera.Observations are target coordinates in the image space, TLS coordinates of the targets, LT coordinates of the targets and horizontal angle measurements of TLS.Extrinsic parameters between TLS and digital camera (   Variance component estimation leads to obtain accurate standard deviations of the observation.As we can see in table 3, standard deviation of the image measurements is in sub-pixel range since the resolution of the captured images is 0.006 mm.In addition, standard deviation of the TLS measurements is close to half millimeter since we were close to the targets (less than 6 meter) in our laboratory and experiencing less systematic errors.Concerning the Az of the TLS is a bit worse than its nominal value in the user manual that is 0.007 ° since it is written down with 0.001 decimal degree from the display screen of the TLS.Moreover, as we expected, standard deviation of the LT measurements is close to 0.1 mm.Furthermore, A-posteriori variance factor of unit weight ( ̂02 ) is computed for entire measurements and that is equal to 0.8975.As can be seen in figure 7, residuals for all type of the observations (image measurements, TLS measurements, LT measurements and horizontal angle measurements of the TLS) are illustrated.Furthermore, some of the LT measurements residuals are too large which we will investigate them in the future.In the second case study, MI based approach as an alternative approach is applied to compare it with high accurate target based approach.In addition, author is investigating about applicability of MI based approach for in field calibration.In this work, we examine the MI based approach merely for one image.Extrinsic parameters between TLS and digital camera are indicated in  In the figure 9, absolute deviations of the re-projected estimated TLS data and estimated image targets in pixel unit is indicated.

Observations
As can be seen, estimated extrinsic parameters in MI based approach is not as accurate as target based approach and it leads to increasing the deviations.

∆𝑋 𝐶 (m)
∆  (m) ∆  (m) ∆ (Deg.)Δ (Deg.)Δ (Deg.)0.0002 -0.0012 -0.0007 -0.0009 -0.0035 5.0e-05 Table 5. Deviations of 6 DOF in the two aforementioned approaches In the table 5, deviations of extrinsic calibration parameters (6 DOF) between two approaches are indicated.These deviations can be related to the sensors noises or uncertainties of the measurements and unknown parameters.Furthermore, it can be due to remaining outliers and also considering just one image in the MI-based approach.

CONCLUSION
The main objective of this research is to obtain extrinsic parameters between fused sensors (TLS, digital camera and LT).Two different methodology of target based and MI based are applied.As a result of the first methodology, GH model is more accurate comparing to the GM model since we have this possibility to use different types of the observations with different weights in the non linear relations to the parameters.In addition, variance component estimation assists us to automatically assign adequate weights to the observations iteratively and consequently arising high accurate adjustment results.Moreover, statistical tests are beneficial due to rejecting outliers and large residuals which are above the pre-determined test value.MI based approach is come up with the lower accuracy results compared to the target based approach and it did not fully satisfy our purpose for in situ calibration and needs more efforts and investigations.
In the future work, MI based approach for numerous scans and images should be investigated.In addition, high accurate extrinsic parameters from target based approach are utilized in deformation monitoring and analysis to exploit the possibility of images and TLS data simultaneously.

Figure 1 .
Figure 1.The employed sensors.Laser tracker (left), A D750 digital camera and Z+F Imager 5006 TLS and their corresponding coordinate systems (right)

Figure 2 .
Figure 2. Calibration room (left) depiction of detected centroid of circles using sub-pixel target mode of the PhotoModeler software (right)

Figure 5 .
Figure 5. Absolute deviations of the re-projected estimated TLS data and estimated image targets in pixel unit Figure 5 is illustrated to visualize the accuracy of the implemented space resection bundle adjustment algorithm.It

Figure 6 .
Figure 6.Absolute deviations of the estimated image targets and image targets measurements in pixel unit In the figure 6, the absolute deviations of the estimated image targets and image targets measurements in pixel unit is depicted.X-axis corresponds to image numbers and Y-axis corresponds to absolute deviations in pixel unit.For each image, all the targets with their deviations in x and y directions are considered in one column.For instance, in the first column of the figure 6, image one contains three targets which with consideration of their deviations in x and y directions respectively, six colourful blocks are shown.In addition, the maximum magnitude belongs to image 11 since it contains six targets.

Figure 7 .
Figure 7. Residuals of the observations In figure 8, re-projection of the downsampled point clouds into the rectified image by the usage of calibration parameters is illustrated.

Figure 8 .
Figure 8. Re-projection of downsampled point clouds into the rectified image using estimated extrinsic parameters of the calibration

Figure 9 .
Figure 9. Absolute deviations of the re-projected estimated TLS data and estimated image targets in pixel unit -MI based approach

Table 2 .
table 1), extrinsic parameters between TLS and LT (table 2), focal length and target point coordinates in object space are the unknown parameters.Extrinsic parameters between TLS and LT (7 DOF) using GH model

Table 3 .
Standard deviations of the observations