MULTISPECTRAL PANSHARPENING APPROACH USING PULSE-COUPLED NEURAL NETWORK SEGMENTATION

The paper proposes a novel pansharpening method based on the pulse-coupled neural network segmentation. In the new method, uniform injection gains of each region are estimated through PCNN segmentation rather than through a simple square window. Since PCNN segmentation agrees with the human visual system, the proposed method shows better spectral consistency. Our experiments, which have been carried out for both suburban and urban datasets, demonstrate that the proposed method outperforms other methods in multispectral pansharpening. * Corresponding author


INTRODUCTION
Panchromatic (PAN) and multispectral (MS) images over the same area can be acquired in bundle by some optical satellites.However, satellite sensors are not capable of providing the narrow spectral bandwidth and high spatial resolution at the same time, since the attainable signal-to-noise ratio (SNR) constrain.One possible solution comes from pansharpening, which fuses the PAN image with the MS image to produce a synthetic image featuring both the high spatial resolution and spectral resolution.
A number of pansharpening methods have been proposed in the last two decades (Amro et al., 2011, Alparone et al., 2006).These algorithms can be divided into component substitution (CS) methods and multiresolution analysis (MRA) methods.The former class includes intensity-hue-saturation method (IHS) (Tu et al., 2001), principal component analysis method (PCA) (Psjr et al., 1991, Shah et al., 2008), and Gram-Schmidt (GS) method (Vivone et al., 2015).The MRA methods comprise decimated wavelet transform (DWT) (Mallat, 1989), undecimated wavelet transform (UDWT) (Nason et al., 1995), "à trous" wavelet transform (ATWT) (Vivone et al., 2013, Shensa, 1992), Laplacian pyramid (LP) (Burt et al., 2003), and morphological pyramids (Restaino et al., 2016).More specifically, the pansharpening methods aim to inject the spatial details of PAN image into the MS bands.Although different authors have different schemes for the injection, all these methods suggest that the injection coefficients should be estimated either locally through a fixed square window or globally over the full image.Obviously, local estimation methods, which compute the injection gain in the square window, perform better than global methods, since the reducing variance of the estimate.However, for obtaining the local injection gain, conventional methods only cluster the pixels in the square window without considering the relevance among the pixels.
In the paper, a pulse-coupled neural network (PCNN) segmentation based pansharpening (PSBP) method is proposed.The PCNN, which is a biologically inspired neural network, has been proven to be agreeing with the human visual system (Johnson et al., 1999).In the proposed PSBP method, uniform injection gains of each region is estimated through PCNN segmentation rather than through a simple square window.Experimental results show that the proposed method is feasible and efficient in solving the pansharpening problem.

PCNN Model
The PCNN is a laterally connected feedback network of pulsecoupled neurons not requiring any training.The standard neuron model is given by the following equations (Johnson et al., 1999): If we define that each pixel represents a neuron, the index (i, j) and index (k, l) refer to the current neuron and its neighborhood, respectively.I denotes the input image.F is feeding input, and L is linking input in iteration n.M and W represent linking synapse weights.Internal activity U is generated by the modulation of F and L through linking strength β.The neuron will be stimulated, when the internal activity U is greater than the dynamic threshold E. In addition, V F , V L and V E are normalizing constants, and the parameters α F , α L , and α E are the time constants.

Proposed PSBP Method
The proposed PSBP method belongs to the MRA pansharpening methods.Thus, the fused multispectral image result MSO can be defined as follows: where MSI refers to input MS image interpolated at the scale of PAN image.g is the injection gains.P and P L refer to the PAN image and its low resolution version.k denotes the channel indexes of the MS images, and K is the total channels of the MS images.The architecture of the proposed PSBP fusion method is shown in Figure 1.The detailed implementation is described as follows: (1) Interpolate the original MS image by using an even cubic kernel with 8 coefficients to obtain the MSI image (Aiazzi et al., 2013).
(2) Perform a histogram matching between the PAN image and the kth band of multispectral image, modifying the former to get the histogram matched PAN images.And then decompose it to wavelet planes using "à trous" wavelet (Vivone et al., 2013).
The decomposition level L is log2(r), which is determined by the resize factor r. Furthermore, the resize factor r means that the resolution of MS image is r times smaller than the one of PAN image.
(3) Perform the inverse wavelet transform to get P L after setting the high frequency coefficients to zeros.

EXPERIMENTAL RESULTS
The performance of the proposed PSBP method has been tested with two different datasets, which are captured by Quickbird and WorldView-2, respectively.

Datasets for Comparison
The first data set has been collected by the Quickbird sensor, which represents a suburban area of Boulder city in the United States.The PAN image of the data set has a size of 4096×4096 pixels.The MS image of the data set has a size of 1024×1024 pixels, which is composed by four bands (blue, green, red, and near infrared).The MS bands have a resolution cell of 2.8m×2.8m,while the PAN channel has a resolution cell of 0.7m×0.7m.
The other data set is the urban area of Washington in the United States, which is acquired by the WorldView-2 sensor.The size of PAN image and MS image are equal to 2048×2028 pixels and 512×512 pixels, respectively.The MS sensor is also characterized by four bands (blue, green, red, and near infrared).The resolution of the four MS bands is 2m×2m, while the PAN channel is four time higher (0.5m×0.5m) than the MS image.Since there is no reference MS available at high spatial resolution, original Pan and MS images are degraded by the scale of resize factor r according to the Wald's protocol (Wald et al., 1997).Therefore, the original MS image can be used as the reference image for testing the fused products.

Initialize PCNN Parameters
PCNN parameters and matrices are initialized by experience.F=Y F =Y U =0, E=1.The iteration number n=1.I is the input image which is normalized between 0 and 1.The connection weight matrixes M and W are given by [0.5, 1, 0.5; 1, 0, 1; 0.5, 1, 0.5].Other parameters of PCNN are shown in Table 1.GS method (e) PCA method (f) BT method (g) HIS method (h) CBD method (i) ATWT method (j) MOF method

Quality Assessment Criteria
The spectral angle mapper (SAM), the relative dimensionless global error in synthesis (ERGAS) and quality index (Q4) criteria (Vivone et al., 2017) are used for assessment.SAM represents the global measurement of spectral distortion, while ERGAS can measure the radiometric distortion between two images.In addition, both radiometric and spectral distortions can be assessed simultaneously by Q4.The ideal value of SAM, ERGAS and Q4 are 0, 0, and 1, respectively.
Figures 3(a) and 4(a) show the original low resolution MS images of Boulder and Washington datasets, respectively.Figures 3(b)-(j) and 4(b)-(j) show the fusion results of the proposed method and other methods.From Figures 3 and 4, we can see that EXP method shows better spectral consistency but poor spatial properties.In particular, MRA methods (including PSBP, CBD and ATWT) show better spatial detail properties than CS methods (including GS, PCA, BT and IHS), which benefit from proper detail extraction.
Table 2 and Table 3 compare the proposed PSBP method with other methods through the Boulder and Washington datasets.It indicates that the proposed method outperforms the other methods with less spectral and radiometric distortion.In particular, PCNN can divide the image into pieces which is coincident with human visual characteristics.Thus, the better performances are attained by adding the extracted similar detail in the uniform PCNN segmentation region.

CONCLUSION
The paper has presented a novel PSBP pansharpening method.The method uses the uniform injection gain in each PCNN segmentation piece.Since PCNN segmentation agrees with the human visual system, the proposed method shows better spectral consistency in visual analysis.Our experiments, which have been carried out for both suburban and urban datasets, demonstrate that the proposed method outperforms either classical well-known methods or some state of art methods, in the aspect of spectral and radiometric preservation.

Figure 1 .
Figure 1.The architecture of the proposed PSBP method (4) Implement the PCNN model to obtain the current output array Y[n].Here, we set dynamic threshold E equals to infinity to make sure that the neuron will not fire again if it has already fired.Y[n] represents the segmentation result in each iteration n.An example of Y[n] can be seen in Figure 2. (5) Calculate g k [n] in each iteration n according to Eq. (7) and Eq.(8), where Cov(A,B) denotes the covariance of A and B. And Std(A) denotes the Standard deviation of A. (6) Update g k and n=n+1 until all the neurons have been stimulated.(7) Calculate fusion product MSO according to Eq. (6).

1 Table 1 .
The parameters set for PCNN

Table 2 .
Comparison results of the proposed PSBP method with other methods for Boulder dataset

Table 3 .
Comparison results of the proposed PSBP method with other methods for Washington dataset