RESOLUTION ENHANCEMENT OF INFRARED THERMAL IMAGING BY PANSHARPENING ALGORITHMS

One common tool in Cultural Heritage inspections is thermal cameras, which are sensitive to the infrared part of the electromagnetic spectrum. But the resolution of these sensors is quite lower than other kinds like visible spectrum range cameras. Typically, the sensors in thermal cameras do not exceed the megapixel frontier. This limitation becomes a problem when trying to combine the information from the thermal images with data from other sensors with much higher resolution such as visible RGB cameras in the same project. In Remote Sensing, algorithms have been designed to fuse multispectral images with panchromatic images (in origin from satellite platforms) to enhance the resolution of lower resolution images with higher resolution ones. These processes are known as pansharpening. Although pansharpening procedures are widely known, they have not been tested working with thermal imaging. The first approach of merging thermal and visual spectrum images to enhance the resolution of the original thermal image involved applying the intensity-hue-saturation (IHS) algorithm (Lagüela et al., 2012, Kuenzer and Dech, 2013). These works only studied one particular algorithm and they did not include any quality study of the results. Our work contains a complete review of a bigger pansharpening algorithms’ set and provides an in-depth study of thermal imaging pansharpening, with a numerical assessment. Our research allows the use of thermal sensors with a lower resolution than other types of sensors used simultaneously in the same project.


INTRODUCTION
Thermal cameras, with sensors sensitive to the long-wave infrared part of the electromagnetic spectrum (wavelengths from 9 to 14 micrometers), are becoming commonly used tools in inspection, and, for extension, in Cultural Heritage documentation works. But, unlike other types of sensors, such as visible spectrum cameras, the resolution of thermal sensors has barely increased in recent years. The most advanced sensors hardly go beyond the megapixel boundary. The most commonly used thermal imaging cameras are based on uncooled sensors operating at room temperature called microbolometers. These microbolometers receive radiation, absorbing it and heating up, thus varying their electrical resistance. Technical limitations in the manufacture of microbolometers make their miniaturization difficult: the signal-to-noise ratio is inversely proportional to their size. It can therefore be stated that the resolution of thermal sensors, at least with this technology, will not equal that of other sensors in the short and medium term. Thermal camera manufacturers have followed different strategies to improve the resolution of their thermal images. The manufacturer FLIR with its Ultramax c technology combines several slightly different shots (16 shots per second) due to the inevitable movements and vibrations during capture. This manufacturer claims to double the resolution with this technique. Another manufacturer, Infratec, has designed a

PANSHARPENING ALGORITHMS
Pansharpening algorithms belong, within the field of Remote Sensing, to the branch of image fusion. They aim to enhance low-resolution images by using images from other sensors with higher resolution. It should be clarified that both images must show the same object with the same field of view.
There are two well-defined families of pansharpening algorithms in the scientific literature. They differ essentially in their approach to the problem: spatial or spectral.
Component Substitution (CS) algorithms are based on the transformation of the colour space of the multispectral image, disassociating spatial and spectral information. The spatial information is then replaced by that from the higher resolution image. Applying the colour space transformation in reverse, we obtain the multispectral image with improved resolution. CS algorithms are global, as they act uniformly over the entire image extent (Chang and Bai, 2018).
On the other hand, Multi-Resolution Analysis (MRA) methods use spatially invariant linear filters to extract spatial details from the high-resolution image and add them to the multispectral image (Chen et al., 2012).
Algorithms IHS, PCA, GS, BDSD, and PRACS belong to the CS category, and we selected HPF, SFIM, INDUSION and the different MTF variations from the group of MRA algorithms. All these algorithms have been computed using a MATLAB library distributed by (Vivone et al., 2015).
To perform the pansharpening processes, we designed a methodology for testing the possibilities of the algorithms. Starting from a raw thermal image in full resolution, we simulated a pseudo-multispectral image from it. Then, a low-resolution 'synthetic' thermal image was created by simulating one that would come from a lower-resolution sensor than the original. We needed to do this to compare the results of the enhancement with the original full resolution thermal image.

METHODS AND MATERIALS
Multispectral images are images composed of different bands representing different parts of the electromagnetic spectrum. The typical bands in these images correspond to colours within the visible spectrum (red, green and blue), along with other bands such as those from the near-infrared (NIR), the shortwave infrared (SWIR) or parts of the ultraviolet spectrum.
In summary, we can define a multispectral image as a set of images (usually between 3 and 15) corresponding to the same frame with different parts of the electromagnetic spectrum.
Thermal images are usually expressed with different colour masks, thus defining a false colour image (figure 1). This aids correct analysis and makes it easier for the user to interpret the information provided by the thermal image. The most common colour chart expresses lower temperatures with cool colours (blues and violets) and higher temperatures with warm colours (yellows, oranges and reds). Although this is an artificial representation of the raw values of the thermal image, it will help us to compose our pseudo-multispectral (PS-MS) image. To test the performance of different pansharpening algorithms, we have to simulate a low-resolution pseudo-multispectral image (PS-MS LR). Applying a Gaussian pyramid algorithm with ratio 4 and σ= 4/3, we achieved this low-resolution pseudomultispectral image. Pansharpening algorithms will be applied to improve its resolution and analyse its quality. Another element common to all pansharpening algorithms is the necessary panchromatic image (PAN). This image must come from a higher resolution sensor. Usually, thermal cameras, mount a RGB visible spectrum sensor next to the thermal sensor, which helps to frame correctly the scene, as thermal images sometimes lack contrast. This visible spectrum sensor has much higher resolution so it will provide us with our PAN panchromatic image, expressing the RGB image in greyscale.
We can now apply all the pansharpening algorithms under study on the PS-MS LR image, which, when fused with the PAN image, will give us the pseudo multispectral enhanced image (PS-MS HR*). This product image is composed, like the PS-MS, of three RGB bands representing the false colour and a fourth band corresponding to raw thermal(4 bands in total). For further quality analysis, this final product has been separated into two images: the corresponding false-colour image (3 bands) and the "raw" thermal image in greyscale (1 band). Now that we have defined the methodology applied, we must establish what characteristics the final products of the image fusion should have. With those in mind, we did determine the quantitative evaluation.

Quality Assessment of Pansharpening Products
3.1.1 Wald's Protocol Before further use, the images resulting from the resolution enhancement must be evaluated by quantitative quality indices. A visual evaluation is completely insufficient to check their suitability.
Among the research community, the so-called Wald's protocol (Wald et al., 1997) is accepted as establishing the essential properties that image fusion products should exhibit whenever possible. This protocol is defined in three theorems (Chen et al., 2012): 1. Consistency: any fused imageÂ, once degraded to its original resolution, should be as identical as possible to the original image A 2. Synthesis: any imageÂ fused by means of a highresolution (HR) image should be as identical as possible to the ideal image AI that the corresponding sensor, if it exists, would observe at the resolution of the HR image.
3. The multispectral vector of imagesÂ fused by means of a high-resolution (HR) image should be as identical as possible to the multispectral vector of the ideal images AI that the corresponding sensor, if it exists, would observe at the spatial resolution of the HR image.
As the original image AI was available in our research, we can comply with Theorem 2 and 3 of Wald's protocol.
The quality of the images resulting from pansharpening algorithms must be measured. A visual inspection is necessary but not sufficient. Several image fusion quality indices have been proposed to ensure the quality of this fusion.
3.1.2 Quality Indices Image fusion quality indices measure spatial and spectral distortion based on different statistics with variations between them. While some focus on spatial reconstruction, others are designed to assess spectral variation.
The following indices have been selected and calculated in our study: • RMSE: Root Mean Squared Error between the two images. It expresses both distortions (spatial and spectral). Its optimum value is 0.
• ERGAS (Erreur Relative Globale Adimensionnelle de Synthèse): Proposed by (Ranchin and Wald, 2000), it is a global statistic that expresses the quality of the final image. It measures the transition between spatial and spectral information. Its optimum value is 0.
• SAM (Spectral Map Angle): It measures the spectral distortion by the angle formed by the two spectrum vectors of both images. Its optimum value is 0.
• PSNR (Peak Signal to Noise Ratio): It describes the spatial reconstruction of the final images. If the two images were identical, the PSNR would tend to infinity.
• UQI (Universal Quality Index): Proposed by (Wang and Bovik, 2002). It estimates distortion by combining three factors: correlation loss, luminance distortion and contrast distortion. UQI values move within the range [-1,1] with 1 being the optimal value.
All these indices have been calculated from the processed images differentially. The pseudo-multispectral enhanced image (PS-MS HR*) was split in two: one image with the three false colour bands and the last band alone, which will be compared with their corresponding originals. This allows us to distinguish the quality of the transformation independently of the chosen false colour mask.

DATASETS
Two different image datasets were built in order to test the performance of the pansharpening algorithms in thermal imaging.
A terrestrial dataset was chosen as the first approach. Then a second dataset from an Unmanned Autonomous Vehicle (UAV) was built because, nowadays, these platforms are widely used in Cultural Heritage documentation studies .

FLIR ADAS Thermal Dataset
The FLIR Thermal Starter Dataset (FLIR, 2019) was originally designed to supply a thermal image and a set of RGB images for training and validating neural networks for object detection. It provides thermal and RGB images simultaneously, making it optimal for applying pansharpening methods.
The dataset was acquired via a RGB and thermal camera mounted on a vehicle (car). All pictures were taken on streets and highways in Santa Barbara, California, USA, under generally clear-sky conditions during both day and night.
Thermal images were acquired with a FLIR Tau2 (13 mm f/1.0, 45-degree horizontal field of view (HFOV) and a vertical field of view (VFOV) of 37 degrees). RGB images were acquired with a FLIR BlackFly at 1280 X 512m (4-8 mm f/1.4-16 megapixel lens with the field of view (FOV) set to match Tau2). The cameras were 48 ± 2 mm apart in a single enclosure.
The aerial vehicle was equipped with two sensors: a 4K RGB CMOS sensor with a resolution of 3840 x 2160 pixels; and an uncooled VOx microbolometer radiometric thermal infrared sensor with a pixel size of 17 micrometres. The thermal images have 640 x 512 pixels, spectral bands of between 7.5 and 13.5 micrometres, and a temperature sensitivity of 50 mK.

RESULTS
As indicated in the section 3, the thermal images have been processed following the necessary steps to obtain the pseudo multispectral image in the original resolution (PS-MS HR) and the pseudo multispectral image in low resolution (PS-MS LR). On the other hand, the RGB visible spectrum camera image was expressed in grayscale giving the panchromatic image (PAN).
Applying all selected pansharpening algorithms to the low resolution pseudo-multispectral image (PS-MS LR) in combination with the panchromatic image (PAN), the final product will be called pseudo-multispectral enhanced image (PS-MS HR*) corresponding to each of the algorithms (figures 3, 4).
The quality indices have been calculated, as mentioned above, in a differentiated way: on the one hand, the false colour image and on the other, the raw thermal image in greyscale. The indices were calculated based on the original resolution image (PS-MS HR) and the enhanced resolution image (PS-MS HR*). In tables 1, 2, 3 and 4 we show the values of the indices for each dataset.

DISCUSSION
Having analysed the results obtained, we can affirm: • The results for the false colour and the corresponding raw greyscale images are quantitatively different. The greyscale thermal images perform much better than the false  colour ones. The RMSE index of the greyscale images is similar or even lower than in other research works consulted (Mandanici et al., 2019). Therefore, greyscale images should be the ones used in subsequent processes, even in the case of applying the same or another false colour mask.
• Except for some specific values, the two families of pansharpening algorithms obtain similar values in the quality scores. Minor differences in the way the different algorithms process the data produce better results. In the family of Component Substitution (CS) algorithms, the BDSD algorithm performs better than all the others.  (Aiazzi et al., 2009).
• Spatial reconstruction is better in multi-resolution analysis (MRA) algorithms. The PSNR index is higher, reporting better reconstruction of spatial details. Again, the BDSD algorithm is the best in terms of geometric quality.
• This work allows the use of thermal sensors with lower resolution than others used simultaneously in the same project, because these pansharpening algorithms improve  Table 4. Quality indices for the raw band grayscale Thermal Pansharpened images from the Illescas UAV dataset for each pansharpening algorithm tested Figure 5. Enhanced thermal image in false color by the BDSD pansharpening algorithm, from the FLIR image dataset and homogenise the resolution. A limitation could be the resolution ratio between the thermal and visible sensors. A ratio between PAN and thermal image's resolution greater than 4 can lead to unexpected artefacts and process failure (Dumitrescu and Boiangiu, 2019).
• Although the results may depend on the false colour mask chosen to express the thermal information in the pseudomultispectral image, the validation of the pansharpening algorithms on thermal images demonstrated here highlights the interest in future developments and research to adjust the algorithms parameters to specifically adapt them to thermal images.

CONCLUSIONS
The use of certain pansharpening algorithms applied to thermal imaging had been studied in previous research. Our work contains a complete review of a large number of algorithms, and provides an in-depth study of their performance and a quantitative analysis, which has not been done to date. The availability of an accurate estimation of the quality of the products of pansharpening algorithms on thermal images will, in the future, facilitate the development of robust and reliable remote sensing systems, leading to their use in Heritage studies.