AN INTEGRATED FUSION FRAMEWORK FOR JOINT INFORMATION RECONSTRUCTION AND RESOLUTION ENHANCEMENT

In this paper, an integrated fusion framework for joint information reconstruction and resolution enhancement is proposed. In the proposed framework, an integrated variational model based on multi-source and multi-temporal remote sensing images is constructed, which is able to simultaneously achieve resolution enhancement and joint cloud and cloud shadow removal. In addition, the ground feature changes between multi-temporal scenes are comprehensively considered. Through this integrated framework, a promising cloud and cloud shadow free fused image with both high spatial and spectral resolutions is obtained. The experimental results confirm the effectiveness of the proposed method.


INTRODUCTION
Due to the technical limitations of sensors and other factors, the existing optical remote sensing images often feature a tradeoff between the spatial and spectral resolutions (Shen et al., 2016;Zhang, 2004).Accordingly, most of the remote sensing satellites provide both high spatial resolution (HR) panchromatic (PAN) images with a low spectral resolution, and low spatial resolution (LR) multispectral (MS) images with a relatively higher spectral resolution.In addition, optical remote sensing images are often obscured by clouds and cloud shadows, which seriously hinders their usability (Shen et al., 2015).
PAN/MS fusion, which is typically referred to as "pansharpening", is an effective way to improve the quality of degraded images and obtain a high spatial and spectral resolution image.It is originated in 1970s (Daily et al., 1978;Harris and Graham, 1976), to date, a variety of pansharpening methods (Alparone et al., 2015;Zhang et al., 2012) have been proposed.In general, they can be grouped into several categories: 1) component substitution based methods (Carper, 1990;Laben and Brower, 2000); 2) multiresolution analysis based methods (Aiazzi et al., 2002;Meng et al., 2016); and 3) model-based methods (Li and Yang, 2011;Meng et al., 2015a;Meng et al., 2014;Zhang et al., 2012).Among them, the modelbased methods have attracted ever-increasing attention in recent years.
Although numbers of pansharpening methods have been proposed, most of the existing methods do not consider the possible cloud contamination in the PAN and MS images.However, it should be noted that, at any one time, approximately 35% of the global land surface is covered by clouds (Shen et al., 2015).Therefore, for the cloudcontaminated optical remote sensing images, how to achieve the integration of pansharpening and cloud removal to obtain a cloud-free fused image with both high spatial and spectral resolutions is a promising and challenging task.To the best of our knowledge, very few papers have addressed this problem.Therefore, in this paper, an integrated fusion framework for joint information reconstruction and resolution enhancement is proposed.The proposed framework can simultaneously achieve pansharpening, and cloud and cloud shadow removal, obtaining promising cloud-free fused images with both high spatial and spectral resolutions.

Framework Description
In this paper, an integrated fusion framework for joint information reconstruction and resolution enhancement is proposed.The proposed framework can obtain an HR MS fused image without cloud and cloud shadow contamination, as shown in Figure 1.To obtain the desired fused image at the target time, the corresponding multi-temporal auxiliary PAN and MS images are utilized.The proposed integrated framework is based on a variational model, and the information reconstruction and resolution enhancement are integrated.Furthermore, ground feature changes between the target images and auxiliary images are comprehensively considered.Accordingly, multi-temporal radiometric normalization and thin cloud and light cloud shadow removal techniques are developed.Through the proposed framework, a final cloud-free fused image with high spatial and spectral resolutions is obtained

Method
Some notations are first introduced.We let 0 y and 0 z denote the target LR MS and HR PAN images, respectively.1 y and 1 z denote the multi-temporal auxiliary LR MS and HR PAN images, respectively.It should be noted that there may be multiple auxiliary images utilized.The desired fused image is represented as x

Multi-temporal Radiometric Normalization:
The multitemporal radiometric normalization is performed on the auxiliary MS and PAN images.It is first performed by moment matching, based on the cloud-free areas.

Variational Model Based Integrated
Framework: An integrated variational model is established for the joint information reconstruction and pansharpening.The total energy functional includes three terms, i.e., a spectral fidelity term, a spatial enhancement term, and a prior term.
A: Spectral Fidelity Term: The spectral fidelity term relates the desired fused image and degraded MS observations.It is represented as the following energy form: where S denotes the blurring matrix, D denotes the down- sampling matrix, and denotes the image masks.
Here, 0 M is the cloud-free mask, and 1 M is mask for thick clouds and dark cloud shadows.x denotes the bth band of x .In this paper, moment matching of the gradient features is introduced to further enhance the robustness.The energy functional is represented as:

C: Total Energy Functional:
The total energy functional is finally obtained by combining ( 2), (3), and the Laplacian prior term (Meng et al., 2015b).It is represented as: where 1  and 2  are the tradeoff parameters, b w is the adaptive weight to balance the spatial enhancement for each spectral band, and Q denotes the spatial activity measure.Finally, the classical gradient descent method (Zhang et al., 2012) is employed to minimize (4).

Experimental Results
The experimental results are shown in Figure 3.Among them, Figure 3   Table 1 shows the quantitative evaluation results.Four popular evaluation indices (Shen et al., 2015;Zhang et al., 2012) are employed, including the correlation coefficient (CC), the peak signal-to-noise ratio (PSNR), the dimensionless global error in synthesis (ERGAS), and the spectral angle mapper (SAM).It should be noted that due to the lack of a reference image, the evaluation was only implemented on the cloud-free region by down-sampling to the original MS image (Wald et al., 1997).It can be seen that the proposed framework obtains the best quantitative evaluation results compared to the PCA and GS methods with all the evaluation indices.On the whole, the proposed integrated fusion framework can successfully achieve the joint information reconstruction and resolution enhancement, and the final result is competitive.

CONCLUSION
This paper has presented an integrated fusion framework for joint information reconstruction and resolution enhancement.In the proposed framework, an integrated variational model based on multi-source and multi-temporal images is proposed, which can simultaneously achieve pansharpening, and cloud and cloud shadow removal.The ground feature changes between multitemporal scenes have also been considered, especially in thin cloud and cloud shadow regions.Even though, it is still inevitable to introduce some errors in thick cloud regions, because the information of the target images in these areas is completely missing, and this problem may be solved by making use of useful information from more auxiliary images.On the whole, the proposed method can yield the effective fused images, and the result is promising.

Figure 1 .
Figure 1.Schematic diagram of the proposed integrated fusion framework for joint information reconstruction and resolution enhancement.

zFigure 2 .
Figure 2. Schematic diagram of sliding window based local moment matching for thin cloud and light cloud shadow removal.(a) Target image with cloud shadows.(b) Auxiliary image.
of corresponding degraded MS images, where 0 y is the target LR MS image with thin clouds and light cloud shadows removed, and 1 y is the auxiliary LR MS image with multitemporal radiometric normalization.B: Spatial Enhancement Term:It is assumed that the desired HR MS image has similar spatial structures to the HR PAN image(Ballester et al., 2006;Wen et al., 2006).It can be represented

Figure 3 .
Figure 3. Experimental datasets and results.(a) Target MS image with clouds and cloud shadows on March, 31, 2015.(b) Target PAN image with clouds and cloud shadows on March, 31, 2015.(c) Auxiliary MS image on May 30, 2015.(d) Auxiliary PAN image on May 30, 2015.(e) PCA fusion result.(f) GS fusion result.(g) The fusion result of the proposed integrated framework.corresponding HR PAN images, and B represents the number of spectral bands.
Figure 3 (a) and Figure 3 (b), respectively, and the MS and PAN images on May 30, 2015, act as the auxiliary images.The main task is to obtain the HR MS image without clouds and cloud shadows on March 31, 2015.
(g)  shows the experimental result of the proposed integrated fusion framework with 1 e) shows the fusion result of the popular PCA (principle component analysis) fusion method, and Figure3 (f)shows the fusion result of the popular GS (Gram-Schmidt) fusion method.They are both implemented by the popular ENVI software.It can be clearly seen that the result of the proposed integrated framework has both high spectral fidelity and high spatial resolution structures.What is more, the clouds and cloud shadows have been successfully removed.In contrast, there is still cloud and cloud shadow contamination in the fusion results of the PCA and GS fusion methods.To allow a more detailed evaluation, three typical zoomed-in areas of the proposed fusion result are shown in Figure4, i.e., a region of cloud shadows, a region of clouds, and a cloud-free region.It can be seen that the result of the proposed framework is inspiring, and it can obtain an HR MS image without cloud and cloud shadow contamination.
Figure. 4. Three typical zoomed-in areas, i.e., a region of cloud shadows, a region of clouds, and a cloud-free region, of the fusion result of the proposed framework,.

Table 1
The quantitative evaluation results