CALIBRATION OF LOW COST DIGITAL CAMERA USING DATA FROM SIMULTANEOUS LIDAR AND PHOTOGRAMMETRIC SURVEYS

Digital photogrammetric products from the integration of imagery and lidar datasets are a reality nowadays. When the imagery and lidar surveys are performed together and the camera is connected to the lidar system, a direct georeferencing can be applied to compute the exterior orientation parameters of the images. Direct georeferencing of the images requires accurate interior orientation parameters to perform photogrammetric application. Camera calibration is a procedure applied to compute the interior orientation parameters (IOPs). Calibration researches have established that to obtain accurate IOPs, the calibration must be performed with same or equal condition that the photogrammetric survey is done. This paper shows the methodology and experiments results from in situ self-calibration using a simultaneous images block and lidar dataset. The calibration results are analyzed and discussed. To perform this research a test field was fixed in an urban area. A set of signalized points was implanted on the test field to use as the check points or control points. The photogrammetric images and lidar dataset of the test field were taken simultaneously. Four strips of flight were used to obtain a cross layout. The strips were taken with opposite directions of flight (W-E, E-W, N-S and S-N). The Kodak DSC Pro SLR/c digital camera was connected to the lidar system. The coordinates of the exposition station were computed from the lidar trajectory. Different layouts of vertical control points were used in the calibration experiments. The experiments use vertical coordinates from precise differential GPS survey or computed by an interpolation procedure using the lidar dataset. The positions of the exposition stations are used as control points in the calibration procedure to eliminate the linear dependency of the group of interior and exterior orientation parameters. This linear dependency happens, in the calibration procedure, when the vertical images and flat test field are used. The mathematic correlation of the interior and exterior orientation parameters are analyzed and discussed. The accuracies of the calibration experiments are, as well, analyzed and discussed.


INTRODUCTION
Nowadays, low cost digital cameras have been applied in many photogrammetric applications.Frequently, these cameras do not have information about their internal geometric characteristics, commonly known as the interior orientation parameters (IOPs).Without this information, the systematic errors in the image measurements cannot be modeled, and therefore the derived metric information in the object space is degraded in terms of accuracy.So, to qualify these cameras for the photogrammetric application, a calibration procedure must be applied to compute the interior orientation parameters.The methodologies to determine the internal characteristics of a camera, which were defined by its Interior Orientation Parameters (IOP), are well known today.Many studies have been performed by several authors (e. g.Brown, 1971;Merchant, 1974;Fraser, 1997;Clarke and Fryer, 1998;Habib and Morgan, 2005;Habib et al., 2006;Remondino and Fraser, 2006;Cramer, 2004;Honkavaara, 2004;Wackrow, 2007;Tommaselli and Telles, 2006;Mitishita et al., 2009;Habib et al., 2010 andMitishita et al., 2010).
Digital cameras are frequently connected to GPS/INS sensors integration to be able to perform the direct georeferencing of images.For this application, high accuracies for the Interior Orientation Parameters are required due to the impossibility of the compensation of IOP inaccuracies via the accommodation of the images' exterior orientation parameters (EOP).There is a high correlation among the elements of the IOP and EOP, which can be seen as an advantage to perform the traditional bundle adjustment aerotriangulation (AT); however the same effect can be considered as a disadvantage for self-calibration via bundle adjustment, especially when the IOP are computed to perform airborne photogrammetric mapping.Traditional self-calibration procedures supported by test field and convergent images do not attain the required IOP accuracies due to the environmental variation among calibration site and job location; then in situ self-calibration methodology has been recommended to solve this problem but difficulties arise to get the prerequisites to perform the calibration in same job mission or using the same images block that is used to perform the mapping project.The basic used prerequisites are a set of convergent images or a survey area with great relief undulation but they are conditions not common for conventional airborne photogrammetric mapping.The use of the coordinates of the camera position stations, computed by GPS/INS sensors integration, as additional observations can be the path to minimize the mathematical correlations among the elements of the IOP and EOP in a process of in situ airborne self-calibration.However, other questions arise when this methodology is used, for example: are the coordinates of the camera position stations accurate enough to perform the calibration?Can the inaccuracies of the coordinates of the camera position stations invalidate the IOP estimation?Does the methodology improve the final mapping accuracies?Inside this subject, this paper shows the methodology and obtained results of the study that was performed to investigate the feasibility to calibrate a low cost digital camera using in situ self-calibration supported by lidar dataset.From the results of the performed experiments, this study tries to answer the questions listed above.This study uses lidar and imagery datasets from a specific airborne photogrammetric system that was development by the scientific cooperation among LACTEC -laser scanner mapping company, and Geodetic Sciences Graduated Program -Federal University of Paraná -Brazil.
The following two sections contain overviews about the airborne photogrammetric system used in this study and the methodologies applied to perform the study of the in situ selfcalibration supported by lidar dataset.Finally, in the last sections, the obtained results from the performed experiments are shown and discussed, as well as the conclusions and recommendations for future work.

AIRBORNE PHOTOGRAMMETRIC SYSTEM
The Kodak DCS Pro Single Lens Reflex (SLR) digital camera, mounted with 35 mm EF f/1.4LCanon lens was connected with Optech Airbone Laser Scanner ALTM 2050 to perform simultaneously airborne lidar and imagery surveys.The CMOS sensor has 14 million effective pixels.The type is 2/3" with the size: diagonal equal 43 mm; width equal 36 mm and height equal 24 mm.The pixel size is 0.008 mm.The images used on this research have 4500 x 3000 pixels.The camera was installed on the same lidar platform, allowing the use of the angular positions of the laser sensor as initial values to compute the images orientation parameters.An acrylic box was specially built to fixate the camera in the lidar platform.
The camera is physically connected with the lidar system via RS232 serial cable to register, along the GNSS-IMU trajectory, the instants that the images are taken.Using post-processing techniques, the GNSS-IMU trajectory is calculated; inside this trajectory the positions and orientation of the sensor mirror are determined for the instants the images were taken.The lever arm was determined via topographic survey.The coordinates are shown in table 1.Using equations 1, the 3D coordinates of the camera station position can be computed.More details about the camera-and-lidar connection can be found in Martins, 2010.ΔX (m) σ (m) ΔY (m) σ (m) ΔZ (m) σ (m) -0.035 0.005 0.244 0.005 -0.055 0.005 Table 1.Lever arm coordinates and their standard deviations.
(1) Attitude matrix from the orientation of the sensor mirror at the moment of the image was taken; = Coordinates of the camera station; = Coordinates of the sensor mirror at the instant the image was taken; = Coordinates of the lever arm;

Test Field
A permanent test field was established to perform the calibration studies.The test area, approximately 4 km 2 in size, lies within the suburban area of the city of Ponta Grossa (State of Paraná, Brazil).Forty-two circles with a 60 cm diameter were painted on the streets within the test area.White paint was chosen to yield a better contrast against the black background of the asphalt.The value of 60 cm was adopted for the flying height of 1000 m, which results in an image space a square of 3 x 3 pixels, approximately.The three-dimensional coordinates of the 42 points were acquired by precise differential GPS survey.The values of root mean square error (rmse) from the X, Y and Z coordinates were 0.0151, 0.0147 and 0.0286 m, respectively.These values were approximated to 0.03 m and used as precisions for the 3D coordinates for the experiments in this work.The area has moderate relief variation.The maximum variation in the Z coordinates of the control points is 57 meters approximately.

Photogrammetric and Lidar Surveys
A cross block layout with four flying lines were planned to get the photogrammetric images from the test field area.Thirty-one images, acquired in four strips, have nearly 60% forward overlap.Two strips with around 50% lateral overlap were taken in opposite directions (approximately west-to-east and east-towest) and other two with the similar lateral overlap (perpendicular flying lines) were taken approximately north-tosouth and south-to-north; the resulting layout is shown in Figure 1.The flying heights of the four strips were about 1000 m.Only thirty-three signalized points were visible in the images.Nine targets were not located due to tree and building occlusions.
The lidar data-set was captured simultaneously with photogrammetric images, using an OPTECH ALTM 2050 laser scanner.The average flying height was 1000 m, resulting for separate strip a mean point density of 3 points/m 2 (nearly 0.5m point spacing).Due to different strips overlaps, the point density varied from 3 to 12 points/m 2 .
The intensity data were recorded.According to the sensor and flight specifications, 0.5 m horizontal and 0.15 m vertical accuracies are expected with the lidar data.The necessary data (lidar trajectories -GPS/IMU technologies) to compute the 3D coordinates of the camera stations positions were also capture simultaneously with the images.

Viability Studies
Experiments of the in situ self-calibration supported by vertical control points and 3D coordinates of the camera position stations are conducted to evaluate the feasibility to compute the Interior Orientation Parameters (IOP) off the Kodak DCS Pro Single Lens Reflex (SLR) digital camera that was previously connected to the airborne photogrammetric system.The 3D coordinates of camera position stations are computed by the lidar trajectories (GPS/IMU sensors integration).Considering the 3D coordinates have inaccuracies caused by bias in lever arm offsets, time integration and GPS observations, the IOP from calibration procedure can model these inaccuracies since there are elements of IOP highly correlated with elements of EOP.These correlations do not cause problems for this research since mounting parameters were not considered here and it is desirable that the IOPs are capable to minimize all displacements of the colinearity of camera's perspective centre, the object point and the corresponding image point to achieve the best accuracies of 3D coordinates of object points computed by photogrammetric procedures.Two groups of in situ self-calibration are performed.The photogrammetric blocks have tie points (natural targets), signalized control points and signalized check points.In the first, the experiments are conducted with the set of signalized control points and their 3D coordinates, computed by GPS survey as well as the 3D coordinates of camera position stations.The main objective of these experiments is to compute standard results of the in situ self-calibration methodology, using the best available layout of control points and flying lines (cross block layout).The obtained results from the calibration procedure are analyzed and discussed.Finally, standard results of accuracies and precisions are calculated to be compared with the similar results from the in situ self-calibration using vertical control points.In the second group, the experiments analyze the viability of the use vertical control points and 3D coordinates of the camera position stations to perform in situ self-calibration.The studies are conducted with cross block layout and conventional block.They use the same set of signalized control points but using only vertical coordinates.The obtained results from the calibration procedure are compared with the calibration results that were computed before using 3D coordinates.The final results point out the feasibility of proposed in situ self-calibration procedure using vertical information from lidar dataset.

RESULTS AND DISCUSSIONS
The first and second experiments of the in situ selfcalibration experiments used a cross block layout and a set of tridimensional control points (X, Y and Z coordinates).As mentioned previously, the main objective of these experiments is to compute the IOP values supported by the best configuration of strips and control points available in this study.The IOP values computed here are considered as standard values for the evaluation of IOP estimated in other in situ self-calibration procedures conducted in this research.
To perform these experiments a set of tie points, control points and check points are defined in the cross block layout; all image measurements were perform by manual operations.The first experiment uses 13 3D control points, 20 check points and 154 tie points.The control points are placed at beginning and end of the strips and only one in the center of the block.The Figure 1 shows the configuration of the images and points in the cross block layout.Thirty-one images and their exposition stations 3D coordinates (Perspective Center) are used to perform the in situ selfcalibration.The second experiment uses the same block but the all check points are changed to 3D control points.To perform the experiments the following precisions are considered for the measurements: 0.004 millimeters (half of pixel) for x and y image coordinates; three centimeters for X, Y and Z coordinates of control points and ten centimeters for X, Y and Z coordinates for the camera position stations.
Table 2 shows the values of Interior Orientation Parameters that were considered significant in their estimations or when the parameter standard deviation from the variancecovariance matrix is at least ten times smaller than its value.Although the first experiment uses less 3D control points, the estimated IOP values are almost equal to those that were estimated in the second experiment.
The precision analysis of the self-calibration experiments by the values of the root mean square errors of the measurements residuals and standard deviation of the significant IOPs, reported in Table 2 and Table 3, reveals acceptable precision for the measurements and estimated parameters in both experiments.Maximum values of coefficient of correlation (close to ±0.8) in both experiments were found among Omega and yo, when the strips flying directions were East to West and West to East.When the strips flying directions were North to South and South to North, the same values of coefficient of correlation occurred but the correlation is among Phi and yo.Table 4 shows the mean absolute values of coefficients of correlation among EOP and IOP from the first and second experiments.
The accuracy of the 3D object point determination in the first self-calibration experiment was performed by statistical analysis of the horizontal and vertical discrepancies from 20 check points available in the block.The planimetric positions of these check points in the block are shown in Figure 1.
Assuming that the RMSE of the X, Y and Z discrepancies reflect the accuracy of the procedure, the obtained values from the analysis, which are shown in Table 5, reveal remarkable accuracies in 3D object point determination.The values of root mean square errors in X, Y and Z coordinates are close to the precision of the check points coordinates.
It is impossible to obtain the accuracies of 3D object points derived from the second self-calibration experiment using the same procedure that was adopted for the first self-calibration experiment because the second experiment does not have check points.However, considering the small differences among common elements computed in both experiments (shown in Tables 6 and 7), the slight differences of the IOPs, the same residuals from the bundle adjustments and similar correlations among EOP and IOP (shown in Tables 2, 3 and  4), it can be concluded that the first and second experiments have slight variability.Also, the 3D coordinates of point object derived from the second self-calibration experiment have similar accuracies to those that were acquired in first self-calibration experiment.
The third experiment aimed to analyze the performance of the in situ self-calibration procedure without 3D control points.It was conducted with the same cross block layout, same configuration of tie points and same measurements precisions that were used in the first and second experiments.Only one state was modified; all 3D control points and check points are changed to vertical control points.The experiment yielded results very close to those that were achieved in the first and second experiments, as can be seen in the values of IOP (Table 2), in the results of residuals analysis (Table 3) and in the values of correlation coefficients among EOP and IOP (Table 4).Even though the mean absolute correlation coefficients among yo and omega or yo and phi are lower than other values in Table 4, it is clear that the correlation behavior among yo and phi or among yo and omega did not change.Based only on these results it is not difficult to conclude that although the procedure uses only vertical control points, the obtained results are similar to other that used 3D control points.However, if accuracy results are considered such as those included in Tables 5 and 6 the conclusion about the procedure is not the same.To perform the accuracy study here, the obtained results from the second experiment were considered as standard because it employed the best available configuration of control points and block layout.Based on this condition, the 3D coordinates of the tie points and the angles of images orientation computed in the third experiment were compared; the main results of this study were included in Table 6 and 7.The root mean square errors computed from the differences of the horizontal coordinates reveal a small systematic error (approximately 1 pixel on the ground) that it was not modeled by IOP or EOP even though the cross block layout was used.A closer look in the results shown in Table 7 reveals that it is possible to assume that probably the systematic error was compensated by the angles of images orientation when the 3D control points were used (first and second experiments).
The fourth experiment had also the main purpose to evaluate the performance of the in situ self-calibration procedure using vertical control points.However, in this experiment the used block layout was a type of conventional aero surveying block that does not have crossed strips.This type of block layout aims to evaluate the feasibility to use lidar vertical control points with 3D coordinates of perspective center to perform in situ self-calibration in lidar and imagery surveying simultaneously.Thus, the experiment was conducted with a sub-block from the cross block layout, showed in Figure 1.It contains two strips that were flown in opposite directions (approximately west-to-east and east-to-west), 96 tie points, 20 vertical control points and 16 3D coordinates of perspective centers.The same measurements precisions used in the previous experiments were adopted here.The Interior Orientation Parameters computed in this experiment were similar to those computed in the experiments using cross block layout.Comparing the four sets of IOP in Table 2, the IOP values from the fourth experiment have the greatest variation between them; however the value of sixteen micros was the maximum variation, occurring in the principal distance.The results from residuals analysis, shown in Table 3, are almost the same the ones that were calculated in the previous experiments.The maximum correlation among EOP and IOP parameters occurred only among omega and yo (close to ±0.7) because the block layout does not have crossed strips; the values of mean absolute correlation coefficients among EOP and IOP are shown in Table 4. Using the same procedure that was applied to perform the accuracy study of the experiment to derive 3D coordinates of the object points, the obtained values of 3D coordinate of the tie point and the angles of images orientation are compared to the values that were computed in the second experiment.The results from this study, shown in Tables 6, reveal that the planimetric accuracy is two times worse than the value from the third experiment (approximately 2 pixels on the ground).The reduced horizontal accuracy in this experiment confirms a small systematic error that was not modeled by IOP or EOP when a 3D control point was not used.In Table 7, the values of the root mean square errors of the differences of the angles of images orientations are approximately two times bigger than the differences that were obtained from the third experiment.Assuming that the angles of images rotations compensate the small systematic errors when the 3D control points were used (second experiment), in the third experiment, due to the use of crossed strips, the lack of non compensation is smaller than that occurred in the forth experiment.However, even with this geometric improvement, the third experiment was unable to compensate correctly the probably small systematic error in the airborne photogrammetric system.7. Main results of the analysis of angles of images orientation performed in the in situ self-calibration experiments

CONCLUSION AND RECOMMENDATIONS FOR FUTURE WORK
This paper has investigated the performance of the in situ self-calibration of a digital SLR camera, connected to a laser scanner system to perform photogrammetric procedures for mapping.The empirical study was conducted to evaluate the feasibility to use lidar vertical control points with 3D coordinates of perspective center to perform in situ selfcalibration in simultaneously lidar and imagery surveys.To perform the experiments, a test field with 42 signalized control points was established in an urban area.Laser scanner and photogrammetric surveys were performed simultaneously.Four strips flown in opposite directions established a cross photogrammetric block layout.3D coordinates of perspective centers were computed by GPS and INS sensors by the camera and lidar system connection.Different experiments of in situ self-calibration were performed using 3D or vertical control points with variation of cross or conventional block layouts.Considering the obtained results from the performed experiments, the following conclusions were drawn: -For the airborne photogrammetric system used in this research, in situ self-calibration can be conducted in the lidar and imagery surveying simultaneously, using 3D

Table 2 .
The interior orientation parameters (IOP) estimated in experiments of the in situ self-calibration experiments

Table 3 .
Main results of the residuals analysis performed in the experiments of the in situ sef-calibration

Mean Absolute Values of the Correlation Coefficients Among Orientation Parameters and IOP
Kappa .0483.3404.0355.0127.0133Kappa .0344.4102.0645.0124.0109Table 4. Main results of correlation coefficients among orientation parameters and IOP from experiments of the in situ selfcalibration

Table 5 .
Main results of discrepancy analysis performed in the first experiment of the in situ self-calibration

Table 6 .
Main results of the analysis of differences of tie points coordinates performed in the in situ self-calibration experiments

DIFFERENCES ANALYSIS -ANGLES OF IMAGES ORIENTATIONS Difference Mean Values of the Differences (seconds) Root Mean Square Error of the Differences (seconds) µ (ω) µ (φ) µ (χ) Rmse (ω) Rmse (φ) Rmse (χ)
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012 XXII ISPRS Congress, 25 August -01 September 2012, Melbourne, Australia coordinates of perspective centers with tridimensional or vertical control points; -When tridimensional control points are used with a cross block layout, the self-calibration procedure achieves the best accuracy for 3D point object determination (lower of one pixel on the ground).Using the same block layout with vertical control points, the accuracy for 3D point object determination was reduced to approximately one pixel on the ground; -When vertical control points are used with a conventional block layout (two strips flown in opposite directions), the accuracy for 3D point object determination was reduced to approximately two pixels on the ground.-Considering the expected horizontal accuracy of laser scanner dataset the valor equal to 0.5 meters, the obtained accuracy for 3D point object determination equal approximately to two pixels on the ground (0.44 meters) is an acceptable value.Future works will focus on performing more experiments of the in situ self-calibration using vertical control points computed via lidar dataset.The experiments will be conducted to investigate the viability to perform in situ selfcalibration procedure based only on lidar dataset without test field and signalized control points.