TIE POINT GENERATION IN HYPERSPECTRAL CUBES FOR ORIENTATION WITH POLYNOMIAL MODELS

This paper presents a technique for tie point generation in hyperspectral images collected by a camera with time-sequential principle for band acquisition (i.e., non-synchronized bands). In mobile applications, each band is acquired at a different time, which generates different camera positions and attitude angles. Due to the large number of bands, a bundle adjustment with polynomial models can be applied to sample bands and then, EOP of other bands are interpolated. The determination of homologue points in all sample bands is required to ensure geometric robustness. A procedure was developed to extract tie points from a reference band which were then transferred to the other sample bands. The technique uses a Helmert geometric transformation combined with majority voting to estimate point transfer functions, followed by area-based matching. Experiments with image orientation were conducted to apply and assess the technique. The tests showed an increase in height discrepancy when tie points are not located in all sample bands highlighting the relevance of the proposed filtering. The accuracy of the technique achieved less than 1 GSD in planimetry and 2 GSD in altimetry using the tie points with the maximum number of rays. Thus, the polynomial approach enables interpolation of other bands according to the parameters of the polynomial function. * Corresponding author


INTRODUCTION
Lightweight hyperspectral cameras, embedded in unmanned aerial vehicles (UAV), have enabled many applications due to the acquisition of data at high spectral, spatial, and temporal resolution with lower costs when compared to other types of traditional technologies. For example, tree species, plant health or some types of diseases can be detected in forestry or agricultural applications.
Several studies and applications have been developed with hyperspectral cameras aboard UAVs. For example, Aasen et al.(2015) implemented a method for obtaining 3D hyperspectral data from snapshot cameras for vegetation monitoring. Oliveira et al. (2019) developed a technique to generate hyperspectral digital surface models for complex cover studies such as forests. Honkavaara et al. (2017) proposed a band registration technique based on a hyperspectral tunable filter for forest applications. Moriya et al. (2017) performed a study on sugarcane virus detection from hyperspectral images. Jakob et al. (2017) used hyperspectral imaging for mineral exploration. Thus, hyperspectral sensors in UAV platforms have many types of application that can produce good results.
In this context, a type of camera with wide application possibilities is the Rikola hyperspectral camera (Senop Ltd., 2017), which is assembled with two sensors in frame format and allows the image orientation by photogrammetric techniques. The images can be collected with suitable overlap and enable 3D reconstruction, in which radiometric and geometric analysis can be made for more detailed studies on the objects. This camera uses a (Fabry-Perot interferometer) FPI-based technology that acquires unsynchronized spectral bands, that is, it uses a time-sequential mode to produce the spectral bands. Thus, when collecting images in mobile applications, each band is acquired at a different time, which generates different camera position and attitude for each band of the hypercube (as depicted in Figures 1a,b). Due to the large number of images generated by the acquisition of hypercubes, instead of computing a set of six exterior orientation parameters (EOPs) for each image band, an alternative solution is to use bundle adjustment with polynomial models. From some sample bands, polynomial parameters describing the camera trajectory are estimated and then the EOPs of the other spectral bands within the hypercubes can be interpolated (Berveglieri et al., 2017(Berveglieri et al., , 2019a . For this task, tie points are essential to ensure the robustness and accuracy of the polynomial technique like in conventional bundle adjustment (BA).
However, the major problem is the radiometric variation of the bands, which complicates the determination of the same point in multiple bands. Then, a procedure was performed to ensure automatic extraction of tie points in all sample bands, considering the maximum number of rays.

Material
The FPI camera used in this study ( Figure 2)  The images captured by the camera are hyperspectral bands generated within the 500-900 nm spectrum. A radiance sensor and a GPS (navigation accuracy) receiver are connected to the camera. The internal mechanisms of the FPI camera use a variable air gap between two parallel reflective surfaces. Multiple light ray reflections at various air gaps produce different wavelengths. Each band is collected with the FPI tuned to a specific interval. The hyperspectral cube is formed by the time-sequential band acquisition. As the FPI camera uses tuneable filters, the number of bands is set by an operator according to the type of application.
In a conventional photogrammetric aerial survey, the platform movement follows an approximately linear trajectory, which results in band displacements in the flight direction. Such effects cause different positions and attitude angles for each band. The misalignment between the two camera sensors and vibrations during image acquisition also contribute to the band displacement. However, this displacement in each hypercube can be modelled by polynomials as function of time, as presented by Berveglieri et al. (2019). To solve this problem, corresponding points are required to perform procedures of image orientation or co-registration.

Method
The determination of homologue points in sample bands is required for the estimation of the polynomial parameters in the modified bundle adjustment. An important requirement is that corresponding tie points are determined in all sample bands, including hypercubes in forward and side directions (adjacent strips). If tie points are only extracted within the same cube, the quality of the result will be affected due to the short baseline. Therefore, tie points must appear in several hypercubes with suitable geometry. In addition, it is well known that suitable geometric distribution of points in images improve the image orientation procedure in bundle adjustment.
The technique for tie point generation is based on a previous work  and can be described in four main steps as detailed below: 1. SIFT keypoints (Lowe, 2004): distinct points are extracted from four sample spectral bands in each hypercube. In this case, we have adopted the first, last, and two intermediate bands of the image sequence.
2. Estimation of point transfer functions between spectral bands: the first band is used as a reference and then the image matching is performed with the SIFT keypoints to produce pairs of matches. This procedure is done using the reference band with the 1 st intermediate band; the reference band with the 2 nd intermediate band; and the reference band with the last band. In each image pair, the most reliable match pairs are selected using a 2D Helmert geometric transformation combined with a majority voting technique to define a point transfer function between spectral bands. The 2D Helmert geometric transformation (Mikhail et al., 2001) is composed of four parameters (scale λ, rotation θ and translations Tx and Ty). As scale and rotation can be approximately pre-defined by the flight plan, these parameters can be considered fixed. Thus, only the two shifts parameters (Tx and Ty) need to be calculated, as presented in Equation 1, in which (x1, y1) are coordinates of the keypoint in the image 1 and (x2, y2) are coordinates of the corresponding keypoint in the image 2.
The results of each pair Tx and Ty are computed in a normalized accumulator matrix, which represents a space discretized in pixels, as depicted in indicate which matches should be selected. The extracted matches are then used to estimate the four parameters of the Helmert transformation by least-squares method. In this adjustment, if any match still has a residual above a predefined threshold, then the match pair is removed. Finally, a point transfer function between spectral bands is obtained. This approach works well within a hypercube because the parallax effect is very small. 3. Extraction of points in sub-regions in the reference band: the reference band is divided into sub-regions to extract well-distributed points in the image, which will be tie points, if they are matched in all sample bands.
4. Point transfer: from the reference band, the distinct points (from step 3) are transferred to the corresponding band by the function estimated in step 2. In the match band, a window is opened, and the tie point is matched by areabased correlation with sub-pixel refinement by least squares. This step ensures that the distinct points of the reference band are located in all sample bands, including the adjacent hypercubes.
Thus, a set of tie points is generated. Such points appearing in all sample bands ensure a good geometry for estimating the EOPs of the sample bands, which follow the mathematical model defined by the polynomial technique. Consequently, the other intermediate bands can be interpolated by the polynomials resulting from the bundle adjustment, as performed by Berveglieri et al. (2019).

Image acquisition
Experiments with image orientation were performed to apply and assess the automatic tie point generation. Firstly, the FPI camera was set to collect 25 bands per cube. A camera selfcalibrating procedure (Kenefick et al., 1972) was performed in a terrestrial calibration field to estimate the inner orientation parameters (IOPs): focal length, principal point, and the parameters related to lens distortions. More details about the FPI camera calibration are given by Oliveira et al. (2016) and Tommaselli et al. (2019). Another important requirement is the misalignment correction between the sensors. After estimating the IOPs, the boresight angles were determined using a hypercube in the calibration field, as described by Tommaselli et al. (2013).
An aerial survey with UAV was performed with the FPI camera (Figure 4), in which an image block of 21 hypercubes distributed in three flight strips was used for the tests. This block was acquired at a flight height of 80 m with flying speed of 4 m/s, forward overlap of 60% and side overlap of 60%, generating images with 5-cm GSD ( Figure 5). In this case, the integration time was set with 5 ms, which represents an elapsed time of 0.779 s to acquire an entire hypercube. In the object space, the displacement between the first and last band corresponds to 3.6 m (depicted in Figure 1).

Processing
The image orientation procedure was conducted using bundle adjustment with polynomial models, in which four distributed bands were determined as samples to apply the tie point generation technique and estimate the polynomial functions.
Previously, the tie points were produced by the proposed technique using local SIFT, majority voting and transfer function followed by area-based matching with sub-pixel refinement, where only correlation coefficients higher than 70% were accepted for matches. Within the procedure of selecting reliable matches to generate the transfer functions, the goal was to find at least two matches to estimate the parameters of the transfer functions. In addition, the proposed technique was compared with the technique named M-estimator sample consensus (MSAC) (Torr and Zisserman, 2000), which is based on random sample consensus (RANSAC) and is used in match filtering procedures. MSAC is an efficient technique for filtering mismatches when there are many inliers, since there is an underlying mathematical model in which random matches should be fitted to test the solution. Three models can be used: affine, projective, or similarity (Helmert). This latter model was used for comparison with the proposed technique, considering the confidence interval of 95% and maximum distance of 2 pixels. When few inliers are mixed with many outliers, MSAC tends to fail. In contrast, Helmert transformation with majority voting extracts reliable matches even with few inliers, since all matches are computed in the accumulator matrix. Then the dominant matches, which converge to the same bin in the matrix, are selected. In performance tests with the proposed filtering, the majority voting technique was 50% faster than the MSAC. Figure 6 shows an example of filtering in which the MSAC failed. In this case, the same set of matches presented in Figure 6(a) was filtered by both techniques. Figure 6  To perform the image orientation by bundle adjustment with polynomials, positions of the perspective centres were obtained from the camera's GPS receiver (navigation accuracy latitude and longitude) to be used as initial approximations in the bundle adjustment. The polynomial parameters were set with weighted constraints according to the flying speed. Seven ground control points (GCPs) and eight checkpoints were also used in this procedure, as shown in Figure 5. Previously, the IOPs were calibrated as well as all image coordinates were also rectified to correct the misalignment between the sensors, as proposed by Berveglieri et al. (2019).

Results
The polynomial technique and a conventional bundle adjustment were applied with the same set of tie points of the four sample bands for comparison purposes. Two sets of tie points were tested. The first set was formed by the tie points generated by the proposed technique (tie points with the maximum number of rays), and the other was composed of common tie points (at least two radii). Table 2 presents the values of root mean square error (RMSE) calculated on the checkpoints, considering both techniques with the two set of tie points. As can be seen in Table 2, the number of rays in the tie points has an impact on the resulting accuracy. When tie points appearing in the four bands and in the adjacent cubes were used (maximum number of rays), minor discrepancies were obtained in the checkpoints. In this case, both techniques (I and II) produced similar RMSEs: less than 5 cm (1 GSD) in XY and approximately 11 cm (2 GSD) in Z. In comparison (III and IV), when conventional tie points extraction techniques were used, it cannot be ensured that corresponding points appear in the four bands and, thus, larger discrepancies were obtained in the checkpoints: approximately 7 cm (1.4 GSD) in XY and 20 cm in Z (~ 4 GSD). The increase in the discrepancy in XY can be explained by the short baseline that, in many cases, results from pairs of tie points between very close bands. This produces a weak geometry for the intersection of rays in the object space. Consequently, the error in Z has more impact, as reported in Table 2 when using common tie points.
In summary, the overall accuracy was similar between both techniques in the two comparisons presented (I with II, III with IV). However, the EOPs estimated with the conventional bundle adjustment do not produces the expected trajectory in the hypercube because they do not have a geometric constraint, which hampers the interpolation for other bands. In contrast, the polynomial technique uses a model in which observations and estimated EOPs are fitted to a mathematical model and then enables interpolation of other bands according to the parameters of the polynomial function. Figure 7 shows the effect of the adjustment in a hypercube when bands are interpolated by the polynomial technique in comparison with 25 bands directly estimated by conventional bundle adjustment. In the polynomial technique, the 25 bands were interpolated from the four sample bands used to estimate the polynomial parameters. As it can be seen, the ordinary bundle adjustment results in scattered band positions that do not follow a geometric model. Therefore, intermediate bands cannot be accurately interpolated.

CONCLUSION
In this study, a tie point generation technique, in which tie points are extracted on all sample bands was presented, tested and compared. The technique results in points with the maximum number of rays, improving the geometric robustness for the image orientation procedure.
Typically, hyperspectral images produces lower number of tie points in comparison with RGB images, due to low contrast, lower resolution, and noise. Due to these imaging characteristics and the polynomial approach, the tie point generation technique was developed with transfer functions and correlation analysis. The objective was to ensure points distributed in the images and appearing in all sample bands. In the selection step of reliable matches, used to define transfer functions, the Helmert transformation combined with majority voting technique was more efficient than the MSAC technique, when few inliers are available. Such tie points were generated to support the image orientation technique using bundle adjustment with polynomial models, which enables a better geometric fit of the polynomial models using four sample bands of 25-band hypercubes. From the estimated polynomial parameters, all 25 bands were interpolated. In contrast, the orientation of images by conventional bundle adjustment does not allow interpolation of bands from sample bands, because it does not have a geometric fitting model for band displacement.
Based on the experiments, both ordinary bundle adjustment and polynomial techniques obtained similar RMSEs. Both techniques demonstrated that the geometric robustness of the generated tie points improved the results. They achieved an accuracy of 1 GSD in planimetry and 2 GSD in altimetry. In future works, experiments will be developed to assess the generation of hiperespectral orthomosaics.