A STUDY ON THE VARIATIONS OF INNER ORIENTATION PARAMETERS OF A HYPERSPECTRAL FRAME CAMERA

New low-cost hyperspectral frame sensors have created a new perspective for remote sensing applications. In this work, we investigate some issues related to the geometric calibration of a hyperspectral frame camera based of FPI (Fabry-Pérot Interferometer), the Rikola camera. The approach proposed in paper is to study the changes in internal optical path caused by the FPI and by the splitting prism. The aim is to model the changes in the IOPs with an analytical function and also to estimate the misalignments between sensors. Several experiments were performed. The changes in position of a specific point were analasyzed to confirm that the bundle of rays is deviated. A self-calibrating bundle adjustment was performed and the Interior Orientation Parameters (IOP) of each band were estimated. The IOPs were analysed and it was concluded that a single set of symmetrical radial distortion parameters can be used for all band. Also, the estimated parameters for each image band were analysed as a function of the air gap of the FPI interferometer. It was noticed some correlation between the focal length and the air gap, and an air-gap dependent model was estimated. Thus, instead of considering an IOP set for each band or for each sensor, a single set of distortion parameters and another set of parameters that is “air-gap dependent”, was assessed. Another important issue was the determination of the misalignment angles between the two sensors, which can explain some differences in the recovered camera trajectory when performing the bundle adjustment.


INTRODUCTION
The recent development of light-weight and relatively low-cost hyperspectral frame sensors has created a new perspective for remote sensing applications.One example of this class of new sensors is the Fabry-Pérot interferometer (FPI) hyperspectral frame camera, which was developed by the Technical Research Center of Finland (VTT) and has been used in various environmental remote sensing applications, including forestry and agriculture (Honkavaara et al., 2013).This camera acquires bands in a time sequential frame format mode, providing rigid geometry, which is an advantage when using non-stable platforms such as the Remotely Piloted Aerial Systems (RPAS).The frame geometry makes feasible the simultaneous determination of the exterior orientation parameters (EOPs) of all images by bundle adjustment, relaxing the need for a high grade inertial navigation system, which is mandatory in cameras with pushbroom geometry.
Nevertheless, having direct orientation data (positions and attitude), constraints can be applied with proper weight in the bundle adjustment.Additionally, images can be used to generate complete object models containing both the 3D geometry and the spectral features of each point.The camera model used in this study captures images in the spectral range from 500 to 900 nm, which makes it suitable for agriculture and forest applications.
However, due to its internal geometry there are several issues to be addressed for the proper definition and estimation of the Interior Orientation Parameters (IOP) of this type of camera.In this sense, this work intends to analyse in details the systematic effects introduced by this type of camera, since the traditional mathematical models of photogrammetry may not be suitable to this peculiar system.

MATERIALS AND METHODS
Several light hyperspectral cameras are available nowadays, such as the Cubert S185-FireflEYE SE (Cubert, 2018) and which have important features that make them suitable sensors to be used with RPAS.The original concept of the Rikola camera, produced by Senop (Senop, 2018), was developed by the VTT, (Saari et al., 2009).Fig. 1.a depicts the camera and accessories (GPS receiver and radiometer), Fig. 1.b is a diagram of its internal components and Table 1 presents its main technical data.
The main component of this camera, which allows the acquisition of images at different wavelengths, is the Fabry-Pérot Interferometer (FPI).This interferometer is composed of two parallel plates, each one with mirrored surfaces facing each other and controlled by piezoelectric actuators (Saari et al., 2009).The electromagnetic radiation incident between the plates undergoes various refractions and reflections and the constructive interferences that occur within the plates allow certain wavelengths to be transmitted and others reflected.The transmitted wavelengths are a function of the separation between the plates (air-gap).Therefore, the incident radiation on this type of camera passes initially through the optical assembly  (Tommaselli et al., 2016).
In this camera model, each sensor acquires a specific spectral range in a time dependent sequence: Sensor II has been configured to acquire the visible bands (506-636 nm), while sensor I captures visible and NIR (VNIR) bands (650-820 nm) (Table 2).The internal change of the air gap of the FPI and the splitting prism can cause problems when estimating the IOPs, because it would be unsuitable to define a single set of parameters for this camera.Estimating a set of IOPs for each image band is troublesome because the configuration can change depending on the application.Also, the practical use of such set of IOPs in bundle adjustment software would be complex.On the other side, a single set of IOPs for both sensors can also be unsuitable due to the sensors misalignments and changes in the IOPs due to the internal changes in the rays paths.(Clarke and Fryer, 1998).The camera IOPs are usually the focal length, principal point coordinates and coefficients of lens distortions (Brown, 1966;Brown, 1971).The mathematical model used in the estimation process are the collinearity equations, which relates the ground and image coordinates of a point, the EOPs and the IOPs (Mikhail et al., 2001).The IOPs, EOPs and the ground coordinates of tie points are estimated simultaneously using the least squares method to solve a system of linearized collinearity equations.The image coordinates of the measured points and additional observations, such as the ground coordinates of some points, and coordinates of the camera perspective centre, determined by GNSS.
Experiments performed by Oliveira et al. (2016), using Rikola camera model 2014, showed that variations between the IOPs of different bands exists, both between bands of the same sensor, and between different sensor bands, which may represent a practical drawback when using this type of camera.They concluded that variations in the IOPs occur for each band, probably because the FPI causes slight changes in the optical path.However the major change in the IOPs occurs for each sensor, basically because they are not perfectly aligned.The authors also concluded that for practical applications two sets of IOPs, one for each sensor is an acceptable option.
There are several approaches to deal with the multiple IOPs of this type of camera.The first one, as addressed by Oliveira et al. ( 2016) is to estimate the IOPs for each single band.Another approach, as proposed by Tommaselli et al. (2018) is to estimate a set of 2D transformations with the camera static, and then using this transformations to coregister all bands.Honkavaara et al., (2017) developed an approach for accurate band registration utilizing a few FPI-camera reference bands and an accurate object model.In this case, the available interior orientation parameters (for sensor, band etc.) were used and the EOPs were solved via spatial resection utilizing control points based on the sparse high-quality structure from motion point cloud generated during the block adjustment of the reference bands or camera.In this case, the potential uncertainties of the interior orientations of the individual bands were compensated for the most part due to correlations between the EOPs and IOPs and sub-pixel accuracy were obtained for the orientations.
The approach proposed in paper is to study the changes in internal optical path caused by the FPI and by the splitting prism.The aim is to model the changes in the IOPs with an analytical function and also to estimate the misalignments between sensors.Table 2. Band numbers, air gaps of the FPI, full width of half maximum (FWHM) and wavelengths of the cubes.

Hypercubes acquisition and feature extraction
The experiments were performed with the camera in static mode.Twenty image cubes of a calibration plate with ARUCO coded targets were captured (Garrido-Jurado et al., 2014;Silva et al., 2014) with several positions and orientations, suitable to camera calibration.
The Rikola camera (model 2015) was configured to acquire 25 bands with bandwidths as showed in Table 2. Sensor 2 collects images from bands 1 to 10 (visible range, from 506 nm to 636 nm), while sensor 1 acquires images of bands 11 to 25 (visible and near infrared, from 650 nm to 820 nm).To produce such spectral bands different air gaps were set in the FPI as presented in Table 2.After image acquisition, the ARUCO targets were automatically located in all image bands of all cubes and their corners were automatically measured with subpixel precision (Garrido-Jurado et al., 2014;Silva et al., 2014;Tommaselli et al., 2016).

Tracking points coordinates
The first analysis was performed selecting a cube and tracking the coordinates of specific points over several bands collected with the camera in static mode, aiming at to check if their coordinates vary in a systematic way.Five targets were analysed: four in the image corners and one in the image centre.
Under the assumption that the camera has a perfect geometry, the position of a point over several bands should be exactly the same, except by variations due to noise or brightness and contrast variations.Significant positions changes would occur if the sensors are not aligned or because the changes in the optical paths deviates them from the original directions.The first effect can be explained by a change in the positions of edges and corners caused by exposures variations.Although the integration time is fixed, the FWHM for the bands are different and, thus, exposures will be also different.Moreover, the type of the target material can also affect the spectral response.The process for corners extraction of the ARUCO targets is done by adjusting straight lines to the edge pixels and then computing the lines intersection.This edge detection step is affect by the internal thresholds.An experiment was performed to determine the effects of corners displacement caused by illumination and exposure variations.The distances in the object space between points of the same target (corners) were computed from their coordinates estimated by self-calibrating bundle adjustment considering different IOPs for each band.Also, the distances of neighbouring points between different targets were calculated.
Considering that the real distances are known it is feasible to compute scale factors relating the estimated and the real distances.Scale factors were calculated both in the horizontal direction of the plate, and vertical for four bands: three of the sensor II (most critical case) and one for the sensor I.The results showed systematic variations in the ratio between distances, especially in the vertical direction.To avoid this problem, a solution is to compute the centroid of the four corners of each target, thus creating a single virtual image point instead of four corners.
The use of the corners centroids in the calibration process will reduce the number of points available but the result with this method produced image coordinates with smaller discrepancies for the two sensors.For both sensors, the maximum discrepancy in the image coordinates of different bands were close to 0.5 pixel, indicating that the use of the centroids may produce better estimates in the calibration.However, even removing this problem related to the corners detection it can be seen that the discrepancies on the image coordinates of a same point in different bands are higher than the measurement error (0.5 pixel).This was already expected, because of the misalignment between the two sensors, the optical path deflections caused by the FPI and the internal prism.Although the discrepancies in image coordinates of corresponding points over several bands were apparently a more detailed analysis showed some systematic effects.The image coordinates (rows and columns) of the centroids of these five targets were plotted using the air gap of each band as the reference, for the two sensors individually (Figures 3 and 4).Analysing the generated graphs it was possible to notice that there is a systematic change of the image coordinates as a function of the image gap, although smaller than 1 pixel.This means that different sets of IOPs would be necessary to correctly reconstruct the incoming bundle of rays.

Geometric camera calibration
A set of 20 cubes, collected with the camera static and with suitable rotations around the calibration plate, were processed with self-calibrating bundle adjustment.All targets appearing in all image bands were detected and measured and, from the extracted corners, centroids coordinates were computed.The calibration process was performed with an in-house developed software (CMC -Calibration with Multiple Cameras).Each image band was approached as a single camera and, thus, 25 sets of IOPs were estimated.
Table 3 presents the estimated IOPs and respective standard deviations for one band of each sensor and the estimated variance of unit weight (a posteriori sigma) for an a priori value of 1.The results for the other bands presented similar standard deviations, although the values of the estimated parameters vary as will be showed.It can be seen from the estimated standard deviations that the calibration was successful, with subpixel precision.

Exterior Orientation and sensors misalignment
The first analyses were done with the Exterior Orientation Parameters estimated in the self-calibrating bundle adjustment.
Since the acquisition of the cubes was done with the camera rigorously static, it should be expected that the EOPs were the same for the 25 bands of that cube, except for random variations caused by the measurement noise.Variation should be credited to sensors misalignment or to correlations with the IOPs, since the variations in the IOPs should absorb the image changes caused by the FPI.The EOPs of all bands were plotted and are presented in Figure 5.The first obvious conclusion from Figure 5 is that the EOPs are varying, while it be the same.variations are partially random but some systematic behaviour can be identified.For instance, the values of  rotation and Y o coordinates have a clear jump from sensor II (band 10) to sensor I (band 11).Since  and Y 0 are correlated in the bundle adjustment one hypothesis is a common cause and the most probable one is the sensors misalignment.Estimated values of rotation  and X 0 coordinates decrease as a linear function.Values of  rotation also jump in the transition from sensor II to sensor I and there is a different value in band 1 that can be explained by the image quality of this band.Band 1 is the one with lowest image quality because it is collected with very small air gap and small band width, as can be seen in Table 2. Values of parameter Z 0 are varying around a fixed value except for bands 23, 24 and 25.
The conclusion is that a single set of EOPs could be used but with a sensor misalignment correction.
Having the sets of estimated EOPs, the Relative Orientation Parameters (ROP) can be computed with Equation 1.These parameters were estimated for pairs of images of the same cube but from different sensors, for instance, band 1 (sensor I) and 11 (sensor II).Ten pairs were used in the calculations of the ROP.
The computed values of the angles are presented in Table 4, along with the minimum, maximum and average values.
is the rotation matrix for the sensor I and is the rotation matrix for the sensor II.To assess the significance of the obtained values, the relative orientation matrix was used to transform the coordinates of Sensor II to Sensor I using the data presented in Table 4.The discrepancies among the coordinates before the transformation presented average values of -6.8 pixels for columns and 5.4 pixels for rows.Considering that there are 10 values for the relative orientation angles and that these values are quite small, three transformations have been made: The first using the average value of the rotations, the second using the minimum values and the third using the maximum values.For the first test, the result of the transformation reduced the discrepancies to 2.3 pixels in columns and 3.2 pixels in rows.For the second test the reduction was 4.7 pixels for columns, and 4.5 for lines.The third test, however, obtained the best results, raising the average discrepancy in columns of 0.99 pixels and 0.91 pixels in lines.This approach is not rigorous, since the Euler angles are dependent of the sequence of rotations, but for this assessment they were taken as independent, to enable the use of 3 values for each one.The conclusion is that the misalignment is significant and the estimation of reliable values for these angles should be included in the sensor calibration.A further test was performed by computing the relative rotation matrix with differential angles.The results were much similar to the previous one, and the differential form can be used to estimate de ROP.

Effects of air-gap in the estimated IOPs
The results obtained by Oliveira et al. (2016) showed that there are variations of the IOPs for the different bands captured by the Rikola camera, even within the same sensor, which is probably caused by the variations in the air-gap of the FPI.An open issue is the behaviour of the IOPs for the different bands.Some experiments were performed to evaluate whether a systematic variation of the IOPs exists, and what mathematical model could model the variation of the focal length and other parameters with respect to the air-gap.To this end, the air-gap values were first analysed.It can be seen from the data presented in Table 2 that the air gaps of the first bands of each sensor (bands 1 and 11) are very different from the others.The variation of the air-gap of the others bands is linear.These two bands were thus removed from the set of data to be analysed.
Figure 6 depicts the different focal lengths as a function of the air gaps of the FPI used to produce each image band.It can be seen that there is a linear trend which can be modelled by a straight line, and, thus a "gap-dependent" model can be expressed by Equation 2.
Where GAP is the FPI air-gap, which is set to produce a specific wavelength, according to Table 2; a and b are the slopeintercept parameters of a straight line computed by linear regression from the focal length estimated by camera calibration.The estimated values of these parameters are presented in Table 4.The analysis of Table 4, especially the Coefficient Determination R 2 , shows that 54% of the variation in focal lengths of Sensor I can be explained by the variation in the air gap.For sensor II, however, this value is quite low and, thus, variation in the focal length cannot be associated to variation in the airgap.Removing band 3 from the set of data for Sensor I, produced a R 2 value of 58.2 %.This analysis should be done by considering that variations in IOPs are correlated with variations in the EOPs.Thus, in a future work camera calibration will be done by imposing the same EOPs to all bands, when acquiring hypercubes with the camera static.As it can be seen in Fig, 7, the position of the principal point presents variations when plotted against the air-gap but the resulting straight line presents practically no slope, which may indicate that average values for each sensor may be sufficient as a common value.
Variations in the estimated coefficients of the symmetrical radial distortion, mainly between the two sensors of the camera were also reported by Oliveira et al. ( 2016) when calibrating each band as an independent camera.However, it is known that the effect of symmetrical radial distortion is dependent on the camera focal length.The following analysis compares the effects of the symmetrical radial distortion, determined using the coefficients estimated for each band, and the same effects using a reference band.Finally, the effects in the other bands are determined by changing the focal length.To this end, a reference band and a point in the image corner were selected to perform this analysis.
Further analysis were performed to evaluate whether a single set of symmetrical radial distortion parameters could be used for this camera, considering the hypothesis that the variations of these systematic effects in the different bands are caused by the estimated focal lengths for each band in the calibration process.
The effects of the symmetrical radial distortion, determined using the estimated parameters for each band (Figure .8(a)), were compared, and these same effects using a reference band were determined by means of the focal length variation only (Figure 8(b)).Figure 8(c) depicts the differences between the obtained values and it can be seen that the differences are always smaller than 1 pixel.The obtained results were consistent with the hypothesis, because the differences in the distortion effects were close to the image measurement error.This result is important because it indicates that a single set of distortion coefficients are sufficient to correct the effects of symmetrical radial distortion for this camera, which is expected, since there is only one set of lenses for the camera.
The decentering distortion coefficients also vary for the different bands.The effects of this distortion were analyzed for a point at the image corner and are depicted in Figure 9.The maximum effect of this distortion was 1/5 of the pixel size for the two components (x and y -columns and rows), except for the bands in the transition between sensors (10-11), where corrections of 1/3 of the pixel were computed.In their experiments, Oliveira et al ( 2016) concluded that decentering distortion was significant but the experiments performed in this work showed that the effects of these distortions are under the measurement error and, thus, the set of parameters can be recomputed by self-calibrating bundle adjustment without decentering distortion.The differences can be explained the camera models were different also because the centroid of the corners were used in this work, which reduced the effects of brightness and contrast variations.To assess the impact of using the IOPs estimated by this gapdependent model a further experiment with a bundle adjustment was performed.Firstly, the data from two image bands from the sensors were selected (Band 20, Sensor I; Band 5, Sensor II).The focal length and principal point coordinates were estimated based on the parameters of the gap-dependent model.The symmetric radial distortion parameters were obtained by calculating the effect of this distortion for different radius values, and then adjusting a polynomial to model this effect (symmetrical radial distortion equation).The EOPs were estimated based on the average values obtained in the selfcalibrating bundle adjustment of each band separately.Both the IOPs and the EOPs were inserted as weighted constraints in the bundle adjustment to assess the coherence of the estimated IOPs with the gap-dependent model.The maximum residual in both experiments were less than 0.68 pixels.The estimated standard deviation of the unit weight was 0.063 for the experiment with band 20, sensor I and 0.18 for the experiment with band 5, sensor II.These results show that there is a good fit of the IOPs estimated with the gap-dependent model to the existing data.More research is needed to assess this modelling and it will be done in a future work because changes in the calibration software is required to assume a single set o EOPs and a "gapdependent" set of IOPs.

CONCLUSION AND FUTURE PROSPECTS
Considering the obtained results, it can be concluded that a single set of symmetrical radial distortion parameters can be used for all bands, and, thus, it is possible to perform the calibration of this camera by estimating only a set of symmetrical radial distortion parameters.Also, the estimated parameters for each image band were analysed as a function of the air gap of the FPI interferometer, and they could be used in the bundle adjustment.A future development of an air-gap dependent model in the calibration model is recommended, where the focal length can be stated as a linear function of the interferometer air-gap.
Thus, instead of considering an IOP set for each band or for each sensor, a single set of distortion parameters and another set of parameters that is "air-gap dependent", can be used.Another important issue was the determination of the misalignment angles between the two sensors, which can explain some differences in the recovered camera trajectory when performing the bundle adjustment.

Figure 2 .
Figure 2. (a) Calibration set up, with the camera static and a targets plate; (b) example of one acquired image.

From
the results achieved by tracking 5 points, changes in these points positions from band to band were detected, mainly between image bands collected by different sensors.Since the focal length for each sensor is fixed, the differences in the image point coordinates from different sensors can be explained by several combined factors: (1) the differences of brightness and contrast of the different bands; (2) the misalignment between the two sensors; (3) the deflections in the optical path caused by the changes in the air gap of the FPI;

Figure 3 (
Figure 3 (sensor I) and Figure 4 (sensor II) depicts the image rows and columns of 5 points (upper left corner -ULC, bottom left corner -BLC; upper right corner -URC; bottom right corner -BRC and image centre -IC) as a function of the FPI gap (Columns are represented in blue and rows in red).

Figure 3 .
Figure 3.Effect of air-gap variation in the image coordinates of targets for sensor I.

Figure 4 .
Figure 4. Effect of air-gap variation in the image coordinates of targets for sensor II.

Figure 5 .
Figure 5. EOPs estimated for the image bands of a single cube.

Figure 6 .
Figure 6.Estimated values for the focal lengths for both sensors and linear adjusted values.SENSOR I SENSOR II Par.value  Par.value  a (mm/nm) 8.4 E-05 2 E-05 2.9 E-05 2 E-05 b (mm) 8.6633 1 E-02 8.6460 1 E-02 R 2 (%) 54.4 23.3 Table 4. Slope-intercept parameters and respective standard deviations of two straight lines modelling focal length variations for different bands of the two sensors and the R 2 values.

Figure 7 :
Figure 7: Estimated values for the coordinates of the principal point for both sensors.
Figure 8.(a) Effect of symmetrical radial distortion with the coefficients of each band for a point in the image corner; (b) Effects of symmetrical radial distortion with the coefficients of a reference band and changing the focal length; (c) differences from the two effects.

Figure 9 .
Figure 9.Effect of decentering distortion for a point in the image corner.
Residuals in the image coordinates of two bands when using the IOPs estimated by the gap-dependent model: (a) Band 20, sensor I and; (b) Band 5, sensor II.

Table 1 .
Technical data of the Rikola camera.
Estimation of IOPs is done by camera calibration usually by Bundle adjustment with indirect observations

Table 4 .
Relative orientation between sensors, computed from the estimated EOPs.