Spatial-Spectral Manifold Embedding of Hyperspectral Data

In recent years, hyperspectral imaging, also known as imaging spectroscopy, has been paid an increasing interest in geoscience and remote sensing community. Hyperspectral imagery is characterized by very rich spectral information, which enables us to recognize the materials of interest lying on the surface of the Earth more easier. We have to admit, however, that high spectral dimension inevitably brings some drawbacks, such as expensive data storage and transmission, information redundancy, etc. Therefore, to reduce the spectral dimensionality effectively and learn more discriminative spectral low-dimensional embedding, in this paper we propose a novel hyperspectral embedding approach by simultaneously considering spatial and spectral information, called spatial-spectral manifold embedding (SSME). Beyond the pixel-wise spectral embedding approaches, SSME models the spatial and spectral information jointly in a patch-based fashion. SSME not only learns the spectral embedding by using the adjacency matrix obtained by similarity measurement between spectral signatures, but also models the spatial neighbours of a target pixel in hyperspectral scene by sharing the same weights (or edges) in the process of learning embedding. Classification is explored as a potential strategy to quantitatively evaluate the performance of learned embedding representations. Classification is explored as a potential application for quantitatively evaluating the performance of these hyperspectral embedding algorithms. Extensive experiments conducted on the widely-used hyperspectral datasets demonstrate the superiority and effectiveness of the proposed SSME as compared to several state-of-the-art embedding methods.


INTRODUCTION
Currently operational hyperspectral missions, such as DLR Earth Sensing Imaging Spectrometer (DESIS) (Krutz et al., 2018), Gaofen-5 (Ren et al., 2017), Environmental Mapping and Analysis Program (EnMAP) (Guanter et al., 2009), enable the recognition and identification of the materials of interest at a more accurate level compared to the multispectral data (Hong et al., 2015) or RGB data , Wu et al., 2019. However, due to the effects of curse of dimensionality, some drawbacks are inevitably introduced with the high spectral dimensionality, possibly leading to the degradation of spectral information. As a result, the dimensionality reduction is a necessary step before the high-level data analysis is performed.
Over the past decades, a large amount of dimensionality reduction approaches have been successfully applied in many computer vision related fields, such as low-level vision analysis (Bi et al., 2017, biometric (Hong et al., 2014, Hong et al., 2016b, large-scale data classification (Hong et al., 2016c, Huang et al., 2020a, Bi et al., 2019a, Huang et al., 2020b, Bi et al., 2019b, multimodal data analysis (Zhang et al., 2019b, Zhang et al., 2019a, data fusion (Hu et al., 2019a, Hu et al., 2019c, etc. Among them, spectral manifold embedding, as a popular topic in hyperspectral dimensionality reduction (Hong et al., 2016a), has attracted a growing attention in various hyperspectral remote sensing applications, such as hyperspectral image denoising (Cao et al., 2018b, Cao et al., 2018a, land cover and land use classification (Hang et al., 2019, Hong et al., 2019d, spectral unmixing , Yao et al., 2019, target detection and recognition , Wu et al., 2020a, and multimodal data analysis . It is well known that the hyperspectral imagery is a three-dimensional imaging product by continuously scanning the region of interest (ROI) to obtain hundreds or thousands of two-dimensional images finely sampled from the wavelength nearly covering the whole electromagnetic spectrum, e.g., 300nm to 2500nm. This enables the identification and detection of materials lying on the surface of the Earth at a more accurate level compared to other optical data, e.g., RGB. In the meanwhile, high dimensional spectral signatures are also introduced some serious drawbacks. For example, high storage and computational cost, redundant information, and complex noises caused by atmospheric correlation would have a negative influence on the spectral discrimination of hyperspectral images, further degrading the performance of high-level applications, e.g., classification, detection.  Figure 1. An illustration for the holistic workflow of the proposed SSME model.
ity of low dimensional hyperspectral embedding in the spectral domain. More specifically, Hong et al. (Hong et al., 2016a) proposed robustly select the neighbouring pixels for hyperspectral dimensionality reduction. The same investigators in (Hong et al., 2017) further design a novel hyperspectral low-dimensional embedding algorithm to learn a robust local manifold representation for dimensionality reduction of hyperspectral images. Inspired by the recent success of deep learning , Hong et al.  developed a joint and progressive learning strategy to learn the low-dimensional representations by using manifold regularization techniques in each layer.
The proposed deep embedding model has demonstrated its superiority and effectiveness in the hyperspectral dimensionality reduction task.
Yet the spatial information  is less investigated by the researchers who are working in the remote sensing community in the process of hyperspectral embedding (Rasti et al., 2020). It is well known that the spatial information has been proven to be effective in the hyperspectral image classification task (Cao et al., 2020), owing to the important and reasonable assumption in hyperspectral images, that is, the target pixel and its neighboring pixels would share the same category to a great extent. We have to admit, however, that the spatial information modeling is capable of improving the discriminative ability of learned embedding representations more effectively, as the spatial structure is one of most important physically meaningful properties in hyperspectral imaging.
For the aforementioned reason, in this paper we attempt to develop a novel hyperspectral embedding approach by simultaneously considering spatial and spectral information, called spatialspectral manifold embedding (SSME). SSME not only learns the spectral embedding by using the adjacency matrix obtained by similarity measurement between spectral signatures, but also models the spatial neighbours of a target pixel in hyperspectral scene by sharing the same weights (or edges) in the process of learning embedding. Classification is explored as a potential strategy to quantitatively evaluate the performance of learned embedding representations. Extensive experiments conducted on the widely-used hyperspectral datasets demonstrate the superiority and effectiveness of the proposed SSME as compared to several state-of-the-art embedding methods in terms of overall accuracy (OA), average accuracy (AA), and kappa coefficient (κ). More specifically, our contributions of this paper can be highlighted as follows: • A novel hyperspectral dimensionality reduction approach -spatial-spectral manifold embedding (SSME) -is devised to learn the low-dimensional manifold embedding of the hyperspectral data.
• Beyond the pixel-wise spectral embedding, we propose to construct the spatial-spectral weight matrix in spectral embedding, yielding more smooth low dimensional hyperspectral embedding.
• Experimental results conducted on a widely-used hyperspectral data demonstrate the effectiveness and superiority of the proposed SSME approach.
The rest of this paper is organized as follows. Section 2 details the methodology of the proposed SSME approach with some necessary formulation derivation. Accordingly, extensive experiments are conducted in comparison with several competitive methods in Section 3. Finally, we draw a conclusion in Section 4.

METHODOLOGY
Manifold embedding, also known as manifold learning, is built on the graph embedding framework (Yan et al., 2006) by attempts to capture the underlying structure of the original data and preserve it in the latent embedding space. The embedding process mainly consists of three steps in the following.
• Neighbor selection on the spectral domain by spectrally measuring the similarities between pixels; • Adjacency matrix computation between each target pixel and its neighbouring pixels by using regression-based methods (Hong, 2019) or Gaussian kernel functions; • Calculation of embedding by solving a generalized eigendecomposition problem.
Unlike the previous manifold embedding techniques, such as locally linear embedding (LLE) (Roweis, Saul, 2000), Laplacian eigenmaps (LE) (Belkin, Niyogi, 2002), and their linearized approaches: locality persevering projections (LPP) (He, Niyogi, 2004) and neighborhood preserving embedding (NPE) (He et al., 2005), that only conduct on the spectral domain, the newly-proposed spatial-spectral manifold embedding (SSME) performs the low-dimensional embedding process from both spatial and spectral domains in a joint fashion. Similarly, SSME follows the graph embedding framework as well. Given a hyperspectral image X ∈ R D×N with D bands by N pixels, xi,j is denoted as the spectral signature located in (i, j) of the image. We then have • Spatial-spectral neighbor selection by using spatially prior knowledge and Euclidean distance based similarity measurement in spectral domains, respectively, which can be written as follows φ spa i ← [xi,j, xi−1,j, xi,j−1, xi+1,j, xi,j+1], (1) φ spa i and φ spe i denote the spatial and spectral neighbours of the target pixel xi, where the latter one can be obtained by sorting the Euclidean distances (sort).
• Spatially-induced adjacency matrix generation by computing the regression coefficients or weights between the target pixels and their spatial-spectral neighbours. The process can be formulated by where x i,k ∈ φ spe i,k represents the k nearest neighbors selected from the spectral domain, and j ∈ φ spa i is defined as the target pixel in the HSI and its neighbouring pixels, respectively. Accordingly, wi,j = [wi,1,j, ..., w i,k,j , ...], j ∈ φ spa i , where wi,0 is the to-be-estimated regression coefficients of the target pixel, and η is the tolerate error (10 −3 in our case).
With the estimated w, the affinity weights A can be obtained by using the following equation: (3) • Joint embedding guided by the aforementioned adjacency matrix by solving a generalized eigen-decomposition problem. Once the affinity matrix A is given, the final hyperspectral embedding Y = {y N i=1 } can be computed by solving the minimization problem as follows.
(4) Fig. 1 illustrates a workflow of our proposed SSMR method for extracting the low-dimensional hyperspectral embedding representations.

Data Description
To assess the effectiveness of the proposed SSME in hyperspectral embedding task, classification is selected to be a potential strategy (Gao et al., 2020). In our case, a simple but efficient classifier: nearest neighbour (NN), is used. Moreover, as a widely-used and classic hyperspectral dataset, the Indian pines scene is chosen in our experiments. It was acquired by the Airborne Visible Infrared Imaging Spectrometer (AVIRIS) sensor in the northwestern of Indiana, USA, which consists of 145 × 145 pixels with 220 spectral bands covering the wavelength from 400nm to 2500nm. There are 16 classes in the studied scene. A fixed and popular training and test sets widely used in many references  is given in Table 1, and a false-color image of the hyperspectral data is shown in the first place of Fig. 2.

Results
A false-color image of this scene is shown in the first column of Fig. 2 and the distribution of training and test sets are also given in the subsequent column of Fig. 2.
Several hyperspectral embedding baselines are selected to evaluate the quality of learned embedding representations using the different methods, such as the original spectral features (OSF), principal component analysis (PCA) (Wold et al., 1987), LE, LLE, and ours (SSME). Quantitative classification accuracies of these compared methods in terms of OA, AA, and κ are listed in Table 2, while Fig. 2 visualizes the classification maps with false-color image and the distributions of training and test sets.
Overall, the results using PCA are basically consistent with those using OSF in all indices. Also, LE holds similar embedding results assessed by means of classification tasks compared to PCA and OSF. By linearly regressing the local neighboring relationship of a target pixel, LLE performs better than the aforementioned embedding methods at an increase of about 5% OAs. As expected, our proposed SSME outperforms others dramatically by jointly considering spatial and spectral information, showing the superiority in hyperspectral low-dimensional embedding tasks. In addition, the accuracies for most of class using our proposed SSME are higher than those using other competitors, as listed in Table 2.

CONCLUSION
In this paper, we propose a novel spatial-spectral hyperspectral embedding approach, called spatial-spectral manifold embedding (SSME), for hyperspectral dimensionality reduction in remote sensing community. SSME not only utilizes the spectral information but also modals the spatial information when calculating the low-dimensional embedding. We have to admit, however, that although the SSME is capable of extracting the hyperspectral features well, yet the discriminative ability for feature representations still remains limited due to the relatively weak data fitting ability (linearized). To this end, we would like to introduce more powerful techniques, e.g., deep learning, to further enhance the representation ability in extracted lowdimensional embedding or introduce additional data sources to better guide the hyperspectral embedding, e.g., light detection and ranging (LiDAR) (Huang et al., 2019), synthetic aperture radar (SAR) (Hu et al., 2019b), multispectral data, in the future work.