SUB-CAMERA CALIBRATION OF A PENTA-CAMERA

Penta cameras consisting of a nadir and four inclined cameras are becoming more and more popular, having the advantage of imaging also facades in built up areas from four directions. Such system cameras require a boresight calibration of the geometric relation of the cameras to each other, but also a calibration of the sub-cameras. Based on data sets of the ISPRS/EuroSDR benchmark for multi platform photogrammetry the inner orientation of the used IGI Penta DigiCAM has been analyzed. The required image coordinates of the blocks Dortmund and Zeche Zollern have been determined by Pix4Dmapper and have been independently adjusted and analyzed by program system BLUH. With 4.1 million image points in 314 images respectively 3.9 million image points in 248 images a dense matching was provided by Pix4Dmapper. With up to 19 respectively 29 images per object point the images are well connected, nevertheless the high number of images per object point are concentrated to the block centres while the inclined images outside the block centre are satisfying but not very strongly connected. This leads to very high values for the Student test (T-test) of the finally used additional parameters or in other words, additional parameters are highly significant. The estimated radial symmetric distortion of the nadir sub-camera corresponds to the laboratory calibration of IGI, but there are still radial symmetric distortions also for the inclined cameras with a size exceeding 5μm even if mentioned as negligible based on the laboratory calibration. Radial and tangential effects of the image corners are limited but still available. Remarkable angular affine systematic image errors can be seen especially in the block Zeche Zollern. Such deformations are unusual for digital matrix cameras, but it can be caused by the correlation between inner and exterior orientation if only parallel flight lines are used. With exception of the angular affinity the systematic image errors for corresponding cameras of both blocks have the same trend, but as usual for block adjustments with self calibration, they still show significant differences. Based on the very high number of image points the remaining image residuals can be safely determined by overlaying and averaging the image residuals corresponding to their image coordinates. The size of the systematic image errors, not covered by the used additional parameters, is in the range of a square mean of 0.1 pixels corresponding to 0.6μm. They are not the same for both blocks, but show some similarities for corresponding cameras. In general the bundle block adjustment with a satisfying set of additional parameters, checked by remaining systematic errors, is required for use of the whole geometric potential of the penta camera. Especially for object points on facades, often only in two images and taken with a limited base length, the correct handling of systematic image errors is important. At least in the analyzed data sets the self calibration of sub-cameras by bundle block adjustment suffers by the correlation of the inner to the exterior calibration due to missing crossing flight directions. As usual, the systematic image errors differ from block to block even without the influence of the correlation to the exterior orientation.


INTRODUCTION
Multiple lens and multiple camera arrangements for aerial purposes are in use more as 100 years (Manual of Photogrammetry 1952, Jacobsen 2008).They became a revival with digital cameras supported by direct sensor orientation (Remondino, Gerke 2015).Especially for urban mapping penta cameras are in use since years as e.g.Pictometry camera, Track'Air MIDAS, UltraCam Osprey, Leica RCD30 oblique, IGI penta camera and several individual mid-format and small camera combinations.Due to the difficult and time consuming orientation of multi head cameras in most cases a direct sensor orientation is preferred, using pre-calibration of the subcameras together with a calibration of the camera system and a boresight calibration in relation to the combination of GNSS and giros.
The pre-calibration of such multi-head cameras is required for operational use of these systems due to too time consuming inner and relative self calibration.In most cases the accuracy requirement is limited to presentation scale of the generated product.Several solutions have been published as e.g.Jacobsen 2008, Madani 2012, Rupnik et al. 2014.With progress in Unmanned Aerial Systems (UAS) several self arranged multihead systems came in use and requiring a system calibration e.g.Niemeyer et al. 2013, Li et al. 2013, Detchev et al. 2014.The geometry of the sub-cameras of the IGI penta camera with one nadir and four inclined cameras with 45° nadir viewing direction (figure 1) and the relation of the sub-cameras to each other has been investigated.Especially the exact camera geometry and geometric stability is delineated here.The precise image geometry was determined by bundle block adjustment with self calibration by program system BLUH (Jacobsen 2007, Jacobsen et al. 2010).By self calibration the details of inner orientation can be determined, while the boresight misalignment can be extracted from exterior orientation.As input for the bundle block adjustment image coordinates taken by Pix4Dmapper have been used.Pix4Dmapper is operational software for the determination of tie points, even if it requires intensive operator support, but this is typical for tie point generation of penta cameras with 45° nadir angle for the view direction of the side looking cameras.The bundle block adjustment with BLUH has not been handled with the same projection centre for all sub-cameras from one imaging instant.This is weakening the block adjustment, but it allows a boresight calibration without any pre-condition as for example caused by not exactly simultaneous imaging.

DATA SET
An image flight with the penta camera was made over the ISPRS benchmark test fields Dortmund and Zeche Zollern (http://www2.isprs.org/commissions/comm1/icwg15b/benchmark_main.html).For the nadir camera block Dortmund has 60% endlap and sidelap, while this is 80% for Zeche Zollern (figure 2).In Zeche Zollern the flight lines have been flown twice in opposite direction.The image flight Dortmund was made 905m above ground, while it was 860m for Zeche Zollern.Corresponding to this the ground sampling distance (GSD) for the nadir camera is 10.4cm respectively 9.9cm and for the inclined cameras in block Dortmund 7.5cm x 8.5cm up to 14cm x 29cm for horizontal objects, for facades, perpendicular to view direction, in the far range 14cm x 16cm.For Zeche Zollern the GSD is 5% smaller.The frequency distribution of number of images per object point is shown in figures 8 and 9.Even if object points just measured in one image are excluded from the adjustment and the influence of object points, located just in two images, to the block tie is limited, the image tie leads to a satisfying image connection, only in the periphery the blocks are a little week.The distribution of the real image tie can be seen in figure 10.Program BLUH is using a standard set of 12 additional parameters for standard self calibration (Jacobsen 2007).In addition digital cameras often have systematic effects at image corners, mainly due to not satisfying flatness of the CCDmatrix.Depending on the fixing of the CCD-arrays in the cameras, the flatness may depend on the temperature.This effect can be determined with the special parameters 81 up to 88 (Jacobsen et al. 2010).Also the principal point (x and y) as well as the focal length can be included as unknown.The use of the focal length as unknown was not successful; also the principal point did not lead to realistic results.This is not surprising in an object area with only limited undulation in height and same view direction for all images of a sub-camera.In general the additional parameters are checked for justification, not justified parameters are automatically eliminated by the program.So the number of originally chosen additional parameters is reduced to the required set.Approximately 50% up to 70% of the additional parameters have been significant.The accuracy of the bundle block adjustments is typical for such type of images and the used tie point generation.The sigma0 exceeds the pixel size of 6µm, this is not unusual for such camera systems with matching of images taken with quite different view direction, being often limited to the pixel address, not reaching sub-pixel accuracy.The standard deviation of the horizontal coordinates X and Y based on all ground control points (GCP) is in the range of the GSD.For the height the accuracy is not as good due to not always good intersections.The ground control points are measured 2 up to 11 images in case of block Dortmund, for Zeche Zollern up to 22 images per control point are available.With only 9 not optimal distributed GCP (figure 2) the block Zeche Zollern is not very stable in the height.

IMAGE GEOMETRY
The image geometry can be determined by self calibration with additional parameters.By analysis of image coordinate residuals also remaining systematic image errors can be identified.For this all image residuals for one camera are overlaid corresponding to the image positions.By averaging all overlaid residuals in a chosen image matrix good information about remaining systematic image errors is achieved.Only residuals of object points measured determined at least with 4 images have been used.An averaging in 25 x 25 sub-areas was selected.Due to the high number of image points in the average any vector represents 275 residuals for block Dortmund and for Zeche Zollern even more.The determined systematic image errors are not the same for both blocks, flown with the same camera.Due to the parallel flight lines the systematic image errors are correlated to discrepancies at GCP, so only the same trend of systematic image errors can be expected (figure 12).In addition aerial cameras are changing the geometry from image flight to image flight, so finally only by self calibration optimal results can be achieved.In most cases the radial symmetric distortion dominates the systematic image errors.Also for the investigated data set the radial symmetric parameters are very important even if the used image coordinates are pre-corrected by information included in the calibration certificate.Again, the radial symmetric distortion has the same tendency for the subcameras of both project areas, but it is not the same (figure 13).
The remaining systematic image errors (figure 14) are even more different in both blocks as the systematic image errors.
Figure 12: Systematic image errors for the 5 sub-cameras, left: Dortmund, right: Zeche Zollern ( ! different vector scale ! ) Sub-camera 147 (backward looking camera) shows some problems in both data sets -the systematic image errors are quite large (take care about different vector scale in figure 12) and also the remaining systematic image errors are large -the vector scale for 147 in figure 14 is 10µm instead of 3 µm for the other sub-cameras.
o SX SY RMSZ N Block adjust.
8.2µm 0.10m 0.19m 0.47m 9 At check points -0.05m 0.06m 0.78m 15 Table 3: Systematic image errors from block Dortmund as precorrection for block Zeche Zollern (no self calibration) Nevertheless a pre-correction of the image coordinates by systematic image errors determined in block Dortmund leads to a small improvement of the block adjustment Zeche Zollern, reverse this is not the case, which may be explained by instability of block Zeche Zollern.With exception of sub-camera 147 the systematic image errors are not so large and may only lead to small improvement by pre-calibration of other data sets taken by this penta-camera.,so the sigma0 of the block adjustment is reduced from 9.5µm by self calibration to 8.7µm for block Dortmund and from 8.6µm to 7.2µm for block Zeche Zollern.Nevertheless even such small influence to sigma0 may lead to block deformation if the block is not supported by direct sensor orientation and the number of GCP is limited, but the handled blocks with 60 respectively 54 nadir images are not very large.

BORESIGHT MISALIGNMENT
As mentioned, the sub-cameras have not been fixed to the same projection centres and also no direct sensor information was used to allow an independent calibration.The discrepancies of the projection centre positions against the nadir camera (tables 3 and 4) are larger as possible for boresight.In flight direction (Dlong) differences may be explained by discrepancies in synchronization of the subcameras, but this is not the case across flight direction and in height.In addition there is no correlation between results for block Dortmund to block Zeche Zollern.The projection centre offsets are clearly larger for the results based on adjustment with self calibration, demonstrating the influence of the self calibration to the exterior orientation.Nevertheless the variations of the projection centres for the individual camera poses are clearly smaller for the block adjustment with self calibration (table 5).This is not shown for Zeche Zollern because of limited number of projection centres where all subimages have been used.

CONCLUSION AND FUTURE STEPS
Additional analysis shall be made by joining the sub-camera images together with a single projection centre.In addition the correlation of the additional parameters with exterior orientation will be improved by changing the formulas of the additional parameters without changing the influence in image space.This will improve the determination of the angular relation of the sub-cameras to each other.

Flight
Flight lines, projection centres and ground control and check pointsThe strong overlay of all images is shown by the foot print plots, colour coded for the used sub-cameras (figures 3 and 4).It also shows that only in the block centre the images are strongly overlapped.Crossing flight lines, having some advantages for self calibration, are not available.The possible image connections are shown in figures 6 and 7 colour coded corresponding to the number of overlapping images.By theory it would be possible to have the same image point in 25 respectively 41 images, but in reality it is only 19 respectively 29 and this only for one object point in both blocks.

Figure 3 :
Figure 3: foot prints of all images Dortmund

Table 1 .
Root mean square discrepancies of block adjustment without self calibration

Table 2 .
Root mean square discrepancies of block adjustment with self calibration by additional parameters The improvement of the block adjustment by self calibration is obvious.As usual, especially the height accuracy depends upon the self calibration which can reduce block deformation.

Table 4 .
Discrepancies of averaged projection centre -related to nadir camera, block Zeche Zollern

Table 5 .
Root mean square variation of projection centre position in relation to nadir camera, block Dortmund