MULTISPECTRAL AND PANCHROMATIC REGISTRATION OF ALSAT-2 IMAGES USING DENSE VECTOR MATCHING FOR PAN-SHARPENING PROCESS

The pan-sharpening is a widely used operation in remote sensing image processing, this operation aims at combining an observable high spatial resolution panchromatic image with a multispectral one, to generate an unobservable image with the high spatial resolution of the former and a high spectral resolution of the latter. Generally, papers dealing with this problem omit the geometric part and suppose that these images are perfectly aligned, which is not necessarily the case for the raw imagery, where even the different bands in the multispectral imagery are misaligned. In this paper, new method for multispectral and panchromatic image registration is proposed to deal with the misalignment problem that reduces the pansharpening quality. This method called Dense Vector Matching (DVM) is based on the matching of a whole line-vector or column-vector from a reference band with the corresponding vector in a target band. DVM is applied on real data and has given acceptable results, where the QNR index of the pan-sharpening is better for images after band registration, also the registration error is reduced to sub-pixel using the proposed approach.


INTRODUCTION
Pan-sharpening, land-cover classification, change detection…etc are applications that involve the use of multispectral remote sensing images, which require that the same image position (row, column), in all bands, corresponds to the same pixel on the ground.This is not always the case for the satellite's pushbroom sensors raw images.Therefore, a band to band registration (BBR) is necessary to achieve such processing levels.The quality of obtained products (maps) depends directly on the quality of this registration (Xu et al. 2012).
When considering raw images, the multispectral bands are misaligned.Also, this misalignment is present between the multispectral and panchromatic image.This effect is principally caused by the terrain altitude variation and the fact that the detectors line of each band is physically separated in the focal plane along the track direction.This separation makes the acquisition of the same ground point to be at different times.Therefore, the imaging of the considered ground point is done at different position and with different attitude (variation of the exterior orientation parameters) in each sensor line, which creates a parallax that leads to non-registered bands.This problem is widely discussed in the literature, and two major solution categories are considered.
In the first one, "Geometry-based registration" (Radhadevi et al. 2009) or "the photogrammetric" methods are used.These ones are based on the collinearity equation to eliminate the misalignment effect.In (Radhadevi et al. 2009;Zhu et al. 2012;Jiang, Zhang, Tang, Li, and Huang 2014;Jiang, Zhang, Tang, Li, Huang, et al. 2014;Boukerch et al. 2015;Liu et al., n.d.), the provided ephemeris and attitude data are used to generate an interpolated sensor model.The reference band pixels are then projected to their ground positions by using the obtained model and digital elevation model.These methods are robust since they reproduce the imaging process, but their drawback is that they are time consuming due to the required long series of transformations.Also, another weakness occurs when the measured trajectory and ephemeris are of low quality.This induces a bad ground positioning of the pixels in the direct and back projection process, so the misalignment will remain between multispectral and panchromatic bands.Therefore, it will be necessary to refine the geometric model using Ground Control Points (GCPs).
The second category includes "image-based registration" methods that does not require the knowledge of satellite position and orientation (Radhadevi et al. 2009).Generally, in this category, some pixels on target bands are matched with corresponding pixels in a reference band to measure the registration errors (Fang et al. 2012;Ma and Zhang 2009;Pan et al. 2011;Lehner and Kornus 1999).These ponctually measured errors are then spatialized over target bands to obtain new perfectly aligned bands.
In the aim of obtaining good pan-sharpening results, it is important to have well aligned multispectral (MS) bands as well as a good multispectral to panchromatic (PAN) registration.In these investigations, a new approach, called Dense Vector Matching (DVM), is proposed to achieve the considered registration.
The DVM technique belongs to the image-based registration category and is specially designed for the images acquired by the high spatial resolution Algerian satellites Alsat-2 (A and B).
In this paper the DVM technique is used to register the MS bands (which are considered as target bands) with the PAN image (which is the reference band).The well registered MS and PAN images are then fused using the well-known "Gram-Shmidt" pansharpening technique.The main advantage of this process is to enhance the spatial/spectral resolutions without disturbing the raw image geometry that remains the same as the reference band.This allow us to use the obtained "geometrically raw pan-sharpened" image in a photogrammetric orthorectification process without any modification in the image metadata.

Dense Vector Matching (DVM)
Dense Vector Matching (DVM) is a newly proposed method to deal with the BBR problem, this method belongs to the imagebased methods that use the image analysis techniques to register the bands belonging the same multispectral image.
The image-based approaches follow a standard workflow which starts with a feature points extraction or detection, this step is followed by a matching between reference and target feature points, after finding the correct matches the registration error is estimated in these points then this error is spatialized over the image in a way we can interpolate the error on any pixel of the image.Then a new interpolated target band is calculated where the mis-registration error is removed.
For the first stage of the BBR process, which is point extraction and matching, different techniques are used in the literature.In (Fang et al. 2012) the HJ-1A/B satellite image is divided in four sub-blocks, the normalized mutual information (NMI) is used for points detection and matching.(Pan et al. 2011) used a feature points extraction by Harris operator followed by similarity measure using the correlation coefficient with pyramid strategy.In (Ma and Zhang 2009) MI combined with Genetic Algorithm (GA) is used to find matching points in 400*400 windows.In (Tilton, Lin, and Tan 2017) the registration error measurement is performed by computing the normalized mutual information (NMI) between shifted image band pairs and finding the amount of shift required (if any) to produce the peak in NMI value.Subpixel accuracy is obtained by utilizing a bicubic interpolation.
For the second stage of the BBR process, which is the error spatialization, (Fang et al. 2012) proposes the use of the elastic thin plate spline interpolation.In (Pan et al. 2011) the error measured using the found points is spatialized by Delaunay triangulation.(Ma and Zhang 2009) utilize the Multi Quadric geometric correction models to establish the corresponding relationship in positions between reference image and input image through the distances from control points.
The proposed approach of the band to band registration is initially designed for ALSAT-2 multispectral imagery to enhance the image quality, this approach combine the two stages into one operation, therefore, in the proposed dense matching each line and column in the image (consequently each pixel) will have its position correction directly after the DVM.
DVM is based on the idea that consists in taking a whole linevector or column-vector from a reference band and finds the matching (by using the non-centered cross correlation measure) vector in a target band.Therefore, each pixel, in a given target band, has its own position corrections.The advantage of taking a whole vector comparatively to square or rectangular region is to reduce the risk of no-match finding in smooth regions.
The main steps of the DVM band to band registration method are as follows: 1. Finding in all target bands for each line and for each column the registration error using the following algorithm.
Algorithm 1: finding a line/column-vector registration error Input: search domain (D), reference band (RB), and target band (TB) For each vector (line or column) in RB do:  Calculate the correlation between the RB vector and a set of TB vectors at the same position +/-D. Fit the obtained correlation coefficients by cubic function and find the subpixel Position of the Maximum (PM). Calculate the Registration Error (RE), which is the difference between the RB vector position and the obtained PM.
Output: table that contains for each line/column-vector the corresponding registration error 2. Smoothing the obtained registration error (RE) by robust version of local regression using weighted linear least squares LOESS (Cleveland 1979).
3. Creating a new band with the same size as RB, and fill it with values obtained by using bilinear interpolation from TB at the position corrected by smoothed RE.
The algorithm described above aims at registering two given bands where the misalignments are less than a value given as search domain (D), this is generally the case for the Multispectral intra-bands registration.But in the case of the inter-band registration (multispectral with panchromatic) the shift between images can be higher than this value.Thus, the DVM process is preceded by a step of higher shift elimination.This is achieved using an extension of phase correlation, which is a well-known fast image alignment technique that cover translation, rotation, and scaling (Reddy and Chatterji 1996).
Therefore, the overall workflow will be as follow: 1. Use the DVM for the intra-band registration where the MS bands (blue, green and NIR) are registered with the Reference band (red band for this case) using D = 5 and a span of 0.07 for the smoothing.This step will produce a mis-registration free MS image called COR-MS.2. The PAN image is down-sampled (using bi-cubic interpolation) to the MS resolution, so the obtained DS-PAN image will have the same size as the MS image.3. Use Phase correlation for the pre-alignment of the COR-MS with the DS-PAN to eliminate the high shift.The obtained image here is called PC-COR-MS 4. Use the DVM for inter-band registration between the PC-COR-MS bands and the DS-PAN as reference band with same parameters as before.5. PAN and PC-COR-MS are fused using the wellknown Gram-Shmidt algorithm.
This approach allow the creation of a perfectly aligned pansharpened image without any alteration of the PAN image geometry, which offer the possibility of using the calculated image with the PAN provided metadata for rigorous orthorectification.

Pan-Sharpening and its evaluation
The Gram-Schmidt (GS-based) pan-sharpening approach is one of the most used methods in the field of remote sensing images sharpening.It is based on an orthogonal decomposition of the multispectral data, and belongs to the component-substitutionbased methods category (Laben and Brower 2000).This method is used in this paper in the aim of merging the MS and the Pan images.To evaluate the quality of the fusion many algorithms are proposed in the literature like SAM, PSNR and ERGAS (Karoui et al. 2017), but their problem is the need of a validation data.To deal with this QNR (Quality with No Reference) is used in this work.The QNR is a spatial-spectral reconstruction quality index that does not require a reference data to measure the reconstruction quality of the sharpened image (Alparone et al. 2008).It includes two indices: the first one is a spatial distortion index (D1), while the second index measures the spectral distortion (D2) of the fused image, for our application the spatial distortion index (D1) is representative since we are studying the effects of displacements on the fusion.

EXPERIMENTS AND DISSCUSSION
The proposed DVM method for band-to-band registration is tested successfully on a large number of Alsat-2 scenes, and is currently used to correct the intra band registration error for the production of 1A level MS imagery.The scenes used in this work are presented in the Figure .1.All these images are given in MS and PAN products.To evaluate the performance of the proposed approach, first a fusion is done using images as received, then the misregistration is eliminated using the proposed approach and the resulting images are fused in the same manner.The QNR is calculated for both pan-sharpend images and the results are presented in Table 1.Table 1.Resulting QNR score for the fusion with and without registration.
From Table 1 it can be seen that the fusion done with images after the registration error correction gives a higher QNR score and lower spatial distortion index.From Figure 2 it can be seen that the proposed method provides a good visual quality improvement of the produced pan-sharpened image.For the last views in the figure 2 the mis-registration between the PAN and the MS is 104 pixels which provide a completely distorted image, but the shift compensation using phase correlation permit the estimation of this error then estimate the residual error using DVM.This way it is possible to enhance the quality of the image as can be seen on figure 2.
Table 2 represents the measured registration error for intra and inter band before and after the application of the proposed approach, these misalignments are measured relatively to the reference band.Shift columns represents the high shift estimated by the phase correlation, Range L and Range C are the error range in line and column directions respectively.
For the intra-band registration, the red band is considered as the reference one, the shift is very low between the MS image bands (estimated between 0 and 2 pixels).Form the table 2 it can be seen that using the proposed approach the registration error range is reduced to a sub-pixel level.
In the inter-band registration the reference band is the PAN, the maximum observed high shift is 104 pixels, this generally occurs in the first image of the strip, because the PAN detector starts recording before the MS one.Also for this case the range of the error drops from 1~2 pixels to subpixel level.

CONCLUSION
This paper deals with the correction of the intra and inter band registration error in the PAN and the MS imagery, this approach is designed for Alsat-2 the Algerian high spatial resolution satellite in the aim of enhancing the pansharpening.
This approach uses a newly proposed method for band to band registration called Dense Vector Matching (DVM), this method is based on a non-centered cross correlation technique, wherein each line-vector and each column-vector of a given target band is aligned with the corresponding vectors of the chosen reference band.
The proposed approach is applied on four real data images of 1A correction level acquired by Alsat-2 and representing different landscapes.Pan-sharpening of images after the registration error correction gives a higher QNR score, lower spatial distortion index and the visual quality of the pansharpened images is clearly enhanced.Also the registration error for intra and inter-bands is reduced to a sub-pixel level, and it has been demonstrated that the proposed approach can handle high shift values between PAN and MS bands.
The proposed approach allow the generation of a high spatial resolution multispectral image where the band to band registration error is eliminated.This approach has the advantage of keeping the raw geometry (1A) of the reference band (PAN) which offers the possibility of using this imagery in photogrammetric (rigorous) orthorectification process.

Figure 2 .
Figure 2. Visual samples from the pan-sharpened images before (Right) and after (left) the registration.

Table 2 .
Registration error (in Pixel) for intra MS bands and inter (PAN as reference with MS bands) before and after the application of the proposed method.