BLUR KERNEL’S EFFECT ON PERFORMANCE OF SINGLE-FRAME SUPER-RESOLUTION ALGORITHMS FOR SPATIALLY ENHANCING HYPERION AND PRISMA DATA

: Single-frame super-resolution (SFSR) achieves the goal of generating a high-resolution image from a single low-resolution input in a three-step process, namely, noise removal, up-sampling and deblurring. Scale factor and blur kernel are essential parameters of the up-sampling and deblurring steps. Few studies document the impact of these parameters on the performance of SFSR algorithms for improving the spatial resolution of real-world remotely-sensed datasets. Here, the effect of changing blur kernel has been studied on the behaviour of two classic SFSR algorithms: iterative back projection (IBP) and gaussian process regression (GPR), which are applied to two spaceborne hyperspectral datasets for scale factors 2, 3 and 4. Eight full-r eference image quality metrics and algorithm processing time are deployed for this purpose. A literature-based re-interpretation of Wald’s reduced resolution protocol has also been used in this work for choosing the reference image. Intensive intra-algorithm comparisons of various simulation scenarios reveal each algorithm’s best performing Gaussian blur kernel parameters. Inter-algorithm comparison shows the better performing algorithm out of the two, thereby paving the way for further research in SFSR of remotely-sensed images.


Background
High resolution (HR) images possess high pixel density, thereby offering detailed information. Their requirement is usual in computer vision tasks to improve image analysis and pattern recognition performance. Remote sensing exercises like large scale mapping need high spatial and spectral resolution, implying the need for HR. Hyperspectral images (HSI) (Goetz et al., 1985) have a large number of contiguous bands which provide much information about features but at the cost of low spatial resolution. On the other hand, multispectral images (MSI) offer a high spatial resolution. These images acquired by conventional cameras offer little spectral content. Because of innate sensor constraints and optics manufacturing technology, a compromise between comprehensive spatial and spectral content cannot be reached. Resolution improvement techniques can solve this predicament. These are fusion (Pohl and Van Genderen, 1998), interpolation (Meijering, 2002), super-resolution (SR) (De Santis and Gori, 1975) and restoration (Andrews and Hunt, 1977). A lower spatial resolution image is combined with a high spatial resolution image in the fusion process to obtain an output with high spatial resolution. However, the spectral properties may be lost, or the resultant output may have pronounced blurring (Kwan et al., 2017). In interpolation, the low resolution (LR) image is transformed into the HR space, and a function is utilized to find the missing figures. But, interpolation operators omit the highfrequency information of the LR input image (Gotoh and Okutomi, 2004). The size of the output and input images remains the same in the case of restoration. SR prevails over these limitations by retaining the spectral properties as well as * Corresponding author enhancing the spatial features of the LR dataset(s) at a larger spatial scale (Protter et al., 2008) SR is an inverse imaging problem, which reverses the degradation process initiated by the imaging model to generate the HR image from its LR image (Nasrollahi and Moeslund, 2014), i.e., Where, IHR = HR output B = Blurring operator Δ = Down sampling operator ILR = LR input η = Additive noise B is sensor dependent and provided in the technical specification document as the point spread function (PSF) (Fernandez-Beltran et al., 2017). Although theoretically represented by a Bessel function (Chen et al., 2018), B can be appropriately represented by a Gaussian function to account for the lens aberration and atmospheric turbulence during image acquisition (Fernandez-Beltran et al., 2017). Scale factor (SF) governs Δ. Depending on the number of input LR images, SR can be multiframe or single-frame. Figure 1 shows a functional classification of spatial domain SR algorithms depending on the image type (Nasrollahi and Moeslund, 2014;Fernandez-Beltran et al., 2017;Jiang, J. et al., 2020)

Choice of SR Algorithms for HSI SR and Their Evaluation
It is not possible to acquire multiple images of the same scene in many exercises or it is costly and time-consuming. There exist many case studies of extending grayscale/red green blue (RGB) multi-frame SR for real-world HSI SR. However, these algorithms succeed only in spatial enhancement and the outputs may suffer from co-registration issues or partial preservation of spectral information. Multi-frame or single-frame methods designed specifically for HSI SR require spectral response functions for end member estimations or large amount of training data which may lack labelling. Moreover, they have a high computation cost owing to requirement of parallel processing routines. Their implementation is shown only on a few benchmark datasaets such as CAVE (Yasuma et al., 2010), Salinas (Plaza et al., 2005), and Pavia (Dell'Acqua et al., 2003). Dearth of open source real-world HR references makes the validation of the generated super-resolved outputs challenging (Ghamisi et al., 2017). Only spectral angle mapper (SAM) (Yuhas et al., 1992) is used to assess spectral quality during quality metric evaluations. No detailed spectral profile examinations have been done in existing literature to support the claim that grayscale/RGB single-frame SR algorithms do not consider the spectral content of the original HSI data during the SR output generation. Also, there exist applications where application of grayscale/RGB single-frame SR algorithms is economically feasible (Fernandez-Beltran et al., 2017). Therefore, there is a need to revisit use of single-frame SR in HSI and study effect of blur kernel on SR algorithm's performance, which is largely absent in many case studies.

Objectives
The objectives of the present study are to • apply classic (grayscale/RGB) single-frame SR algorithms to real world HSI data • assess the performance of these algorithms under changing blur kernel parameters for different SF

Study Area
The area of investigation occupies an area of 38.8 square kilometres (sq. km.) in Ahmedabad, Gujarat, India. Figure 2 shows its location. It is characterized by the densely settled walled city, lands of closed textile mills, well-planned commercial, residential and educational areas. The Sabarmati river is the major natural feature in the scene.

Data Used
Tables 1 and 2 present the sensor and scene specifications of the HSI data used in this study, respectively. Following this, zero, noisy and water absorption bands are removed leaving 128 bands in the dataset. Along track destriping is also done for the removal of bad columns, which arise due to calibration differences in temporal variation and response of Hyperion detector arrays (Han et al., 2002). Atmospheric correction by the Fast Line-of-sight Atmospheric Analysis of Hypercubes (FLAASH; Adler-Golden et al., 1999) provides accurate, physics-based derivation of apparent surface reflectance (Kruse et al., 2004). The surface reflectance values are scaled band-wise to the range of 0 to 1 through division by 10000.

Sensor Name
The VNIR-SWIR stacked PRISMA data in Environment for Visualizing Images (ENVI) standard format is extracted using the R programming language package "prismaread" (Busetto and Ranghetti, 2020) along with scene metadata in separate files. It is co-registered with Hyperion using 1° affine transformation and spatial subsetted into a square patch of identical samples and lines. Bands with haziness, single and multiple columns of no data or very low radiometric accuracy, and pixels possessing a very low radiometric accuracy are removed and only 112 bands are remaining in the dataset.

SR Algorithm Execution:
The chosen classic SR algorithms: Iterative Back Projection (IBP) (Elad and Feuer, 1996;Yang et al., 2014) and Gaussian Process Regression (GPR) (He and Siu, 2011) show outstanding performance in literature in terms of quick processing speed, visual appeal of the generated HR imagery and efficient extraction of spatial and spectral properties from the input data (Yang et al., 2014;Fernandez-Beltran et al., 2017;Mishra et al., 2019). The SR algorithms are executed band-wise on a single machine with configuration: Random Access Memory (RAM) = 16 GB, Processing Speed= 2.5 GHz, Central Processing Unit (CPU)= Intel Core i5-10300H to examine its calculation productivity.
Single-frame IBP focusses on the iterative refinement of an initial guess of super-resolved image, i.e., reconstruction error between LR image and LR version of super-resolved image is minimized throughout the iterative process. The iterations continue until the maximum number of iterations or a limit in the reconstruction error is reached. The parameters are: Backprojection Kernel Window = Gaussian, size dependent on standard deviation (sigma) = 0.1 -2.1, iterations = 100, SF = 2, 3 and 4 GPR is a hybrid framework consisting of two stages. In the first stage, the input LR image is bicubically interpolated to the target spatial resolution. Each pixel in the super-resolved output is predicted by that pixel's neighbours in the interpolated output. The structural information defining the pixel's neighbourhood is used for this purpose. In the second stage, the output obtained in the first stage is deblurred. The final super-resolved output with sharper edges is produced by learning from a training set of LR and HR image pairs. This training set is obtained from the input LR image and the interpolated output. The parameters are: Patch size = 20 × 20, Overlapping Factor = 0.66, PSF = Gaussian, size = SF and calculated according to sigma, sigma = 0.1 -2.1, SF = 2, 3 and 4 It is also assumed that the original HR image of the scene area is not accessible, and hence determination of the optimal value for every parameter of each algorithm is not possible.

Performance assessment of SR algorithms:
The super-resolved image's quality is assessed using 8 full-reference quality indices for every SF and changing blur kernel parameters. These metrics are: bias, cross-correlation (CC), difference in variance (DIV), error relative globale adimensionnelle de synthèse (ERGAS), entropy, root mean square error (RMSE), relative average spectral error (RASE), and universal image quality index (Q). Reader may refer to Vaiopoulos (2011) for more information on these indices. In the absence of a suitable HR reference image, reducedresolution protocol (Wald et al., 1997) is used. According to this protocol, original HSI dataset is the ground truth for SR product, assuming that the LR version of the SR product has as much resemblance as possible to the original HSI data. Considering the ill-posed nature of the SR problem (Lugmayr et al., 2022), uniform imaging model parameters are taken: a blur kernel window size calculated according to σ = 0.1 -2.1 and scale factor = 2, 3 and 4 for generating the LR version of every superresolved output (Loncan et al. 2015).    There is not much difference in the RMSE values for x3 Hyperion and x4 PRISMA as the blur kernel parameters change. Again σ = 0.1 shows the lowest RMSE for each SF irrespective of the dataset.      Hyperion begins from σ = 0.7, peaks at σ = 1.5 and reduces slightly, whereas in x3 Hyperion the rise happens from σ = 0.7 and the rise continues with increasing kernel size and σ. x3 PRISMA rises from σ = 0.5 to peak at σ = 1.1 and overlap with x2 Hyperion at σ = 1.2, then decrease till σ = 1.5 and rise slightly thereafter.

CONCLUSION
Since this is an ongoing work, conclusions can be drawn only after compilation of the processing times and and use of peak signal-to-noise ratio (PSNR), SAM and structural similarity index measure (SSIM) for investigating impact of changing blur kernel parameters on SR algorithm's performance.