3 D OBJECT CLASSIFICATION BASED ON THERMAL AND VISIBLE IMAGERY IN URBAN AREA

The spatial distribution of land cover in the urban area especially 3D objects (buildings and trees) is a fundamental dataset for urban planning, ecological research, disaster management, etc. According to recent advances in sensor technologies, several types of remotely sensed data are available from the same area. Data fusion has been widely investigated for integrating different source of data in classification of urban area. Thermal infrared imagery (TIR) contains information on emitted radiation and has unique radiometric properties. However, due to coarse spatial resolution of thermal data, its application has been restricted in urban areas. On the other hand, visible image (VIS) has high spatial resolution and information in visible spectrum. Consequently, there is a complementary relation between thermal and visible imagery in classification of urban area. This paper evaluates the potential of aerial thermal hyperspectral and visible imagery fusion in classification of urban area. In the pre-processing step, thermal imagery is resampled to the spatial resolution of visible image. Then feature level fusion is applied to construct hybrid feature space include visible bands, thermal hyperspectral bands, spatial and texture features and moreover Principle Component Analysis (PCA) transformation is applied to extract PCs. Due to high dimensionality of feature space, dimension reduction method is performed. Finally, Support Vector Machines (SVMs) classify the reduced hybrid feature space. The obtained results show using thermal imagery along with visible imagery, improved the classification accuracy up to 8% respect to visible image classification.


INTRODUCTION
The spatial distribution of land cover in the urban area is a fundamental dataset for urban planning, ecological research, change detection, disaster management, etc.Over the past decades, different types of remotely sensed data widely applied for classification of urban areas, such as LiDAR (Chehata et al. 2009), aerial visible image (Myeong et al., 2001), satellite multispectral image (Moran, 2010), hyperspectral imagery (Samadzadegan et al. 2012), etc.
High spatial resolution images have been increasingly used for urban land use/cover classification, but the high spectral variation within the same land cover, the spectral confusion among different land covers, proximity of 3D objects and the shadow problem often encounter classification performance with some challenges (Moran, 2010).Therefore, the classification of urban area is ongoing topic in remote sensing community (Wentz et al. 2014;Weng, 2012).
By recent advances in sensor technologies, different types of remotely sensed data are available from the same area.Due to incompetence of just one dataset to model all characteristic of urban area and also availability of different source of data, much attention has been recently paid to multi-sensor data fusion.Multi-sensor data fusion seeks to integrate data from different sources to obtain more information than can be derived from a single sensor (Kumar et al. 2015).The assessment of the capacity of several combination of dataset are investigated in the literatures, such as hyperspectral and LiDAR data (Liao et al. 2015), optical image ad synthetic aperture radar (Zhu et al. 2012), * Corresponding author aerial image and LiDAR data (Huang et al. 2011), optical image, thermal image and LiDAR (Brook et al. 2012).
Due to high spatial resolution of the aerial imaging sensors, they play a very important role in the discrimination of land-cover classes in complex urban area.The visible camera observes a radiation reflected from Earth's surface over a visible wavelength range, whereas the thermal camera observes radiation emitted from Earth's surface over IR wavelength range.However, thermal imagery has coarse spatial resolution and its application has been restricted in urban areas.The brightness values of the visible and thermal images correspond to Earth's surface reflectance and temperature, respectively (Liao et al. 2014).
Fusion of the aerial visible and thermal images can enhance the spatial details of the thermal image and also add the temperature information to the visible image.Consequently, fused images of a target scene from visible and thermal imaging sensors provide additional information and make classification performance better than would be possible when the sensors are used individually.
Berni et al. generate quantitative remote sensing products for vegetation monitoring by means of a helicopter-based UAV equipped with inexpensive thermal and narrowband multispectral imaging sensors (2009).Moreover, fusion of thermal and visible imagery has application in real time people and vehicle tracking from unmanned aerial vehicle (Gaszczak et al. 2011). Xin et al. propose hierarchical classification framework to fuse thermal hyperspectral and visible imagery.Firstly, feature extraction is performed and then different classes were identified successively using appropriate feature combinations.Finally, the pixel-based classification map was improved by post-processing steps (Lio et al. 2014).This paper presents the method to fuse thermal hyperspectral imagery and aerial RGB image in feature level fusion for classification of urban area.

PROPOSED METHOD
This paper evaluate the capacity of aerial thermal hyperspectral and visible imagery fusion in classification of urban area.The proposed pipeline of multi-sensor data classification procedure composed of four steps: data registration, feature extraction, dimension reduction and classification.

Data Registration
The visible data were gereferenced and registered to the thermal data but they have different resolution.Thermal imagery was upsampled to visible image by bicubic interpolation.

Feature Extraction
Feature extraction is the second step where spectral and spatial features are extracted from visible and thermal images to explain more about the scene.In this research, the following features were extracted:  PCA: spatial analysis of thermal hyperspectral imagery is carried out on first two PCs.
Table 1.GLCM descriptors formula where G is the number of grey level and the matrix element (, ) is the relative frequency with which two pixels occur within a given neighbourhood, one with intensity i and the other with intensity j (Haralick et al. 1973).
All the GLCM features were extracted from each band of the visible image and 2 PCs of thermal hyperspectral imagery.
 Morphological Profile (MP): according to high-resolution of visible image, MPs are computed using the mean visible image bands to extract spatial features.MP is computed by applying opening and closing morphological operator with different structure element size.It is widely agreed that spatial features can significantly improve the classification accuracy due to the consideration of spatial correlation between neighbouring pixels (Benediktsson et al. 2003).
 Structural Feature Set (SFS): this spatial feature is used to extract the statistical features of the direction-lines histogram.Direction lines can be defined as a series of a predetermined number of equally spaced lines through the central pixel.Six statistical features are considered, consist of length, width, pixel shape index (PSI), ω-mean, ratio, standard deviation (Table 2).are i th minimum and maximum of all direction-line respectively (Huang et al. 2007).
 Local statistics: four statistic features include mean, variance, skewness and kurtosis are extracted from each visible image band.
 Vegetation index: Due to the biophysical characteristics of green plants, the normalized ratio between the mean thermal hyperspectral bands and the red band of visible image is used as vegetation index (Lio et al. 2014).
By merging all extracted features along with original images, the hybrid feature space is generated.

Dimension Reduction
High dimensionality of the hybrid feature space pose classification process some challenges, such as correlation between some features, time consuming and complexity of computation.Therefore, dimension reduction is applied to reduce the dimension of hybrid feature space, yet it preserve the main information.PCA is well-known dimension reduction method that is applied in this paper to reduce the dimension of hybrid feature space.The intrinsic dimension of hybrid feature space is determined based Eigen value and then corresponding PCs are chosen for classification.

Classification
Support Vector Machines (SVMs) are powerful and accurate classifiers, commonly used in remote sensing community.
According to the stability of SVM, it is selected as classifier.
SVM is a learning technique derived from statistical learning theory.It is calculating an optimally separating hyperplane that maximizes the margin between two classes.If samples are not separable in the original space, kernel functions are used to map  1) is given by: where the constants α i are called Lagrange multipliers determined in the minimization process.SV corresponds to the set of support vectors, training samples for which the associated Lagrange multipliers are larger than zero.The kernel functions compute dot products between any pair of samples in the feature space.Gaussian Radial Basic Function (RBF) is a common kernel which is used in this paper and it is defined by (3). (3) In the proposed method, the classification module plays an important role in evaluation of the fitness function where SVM is trained by training data and trained SVM is evaluated by testing (unseen) data.
It is notable that SVM parameters have strong effect on its performance.SVMs parameters are: (i) Regularization parameter (C) that determines the trade-off between minimizing the training error and minimizing the complexity of the model; (ii) Kernel parameters: bandwidth for Gaussian kernel functions (σ).In order to improve the performance, grid search is applied before classification process to determine the best value for these parameters.

Dataset
The potential of the proposed method, implemented on two airborne datasets acquired at different spectral ranges and spatial resolutions: 1) a coarser-resolution LWIR hyperspectral image and 2) fine-resolution visible imagery.The data were acquired and were provided for the 2014 IEEE Contest by Telops Inc., Québec, Canada.They cover an urban area, including buildings, roads and tree (Liao et al. 2014).The spectral variability of objects in the same class, spectral similarity of some different classes and proximity of 3D objects in urban area, make it complex scene for classification and appropriate for assessment of the proposed method.
The LWIR data were acquired using the Telops Hyper-Cam, an airborne long-wave infrared hyperspectral imager.It consists of 84 spectral bands in the 7.8−11.5 µm wavelength region with approximately 1m resolution.The VIS data consists of uncalibrated data with 0.2m spatial resolution.

Feature Space Generation
Generation of feature space is performed by processing both thermal hyperspectral imagery and visible image.
Visible feature space consists of original RGB bands (3 features), SFS features on each band of VIS (18 feature), MP on mean visible bands (12 feature), GLCM descriptors on each visible band (24 features), HSV transformation (3 features) and local statistics on each band of visible image (12).Therefore the visible feature space with 72 spectral and spatial features is generated.
Thermal hyperspectral image has 84 bands.PCA transformation is applied on thermal hyperspectral imagery and 2 first PCs are selected to use for texture analysis.GLCM descriptors apply on both PCs and 16 features are generated.Consequently, the thermal feature space compose of 102 descriptors.
Finally, feature stacking is performed to form feature space.
Visible and thermal feature space in addition to vegetation index are merged to construct the hybrid feature space consists of 175 features.

Dimension Reduction
According to high dimensionality of the hybrid feature space with 175 features, dimension reduction is required to decrease the computational time and complexity.The intrinsic dimension of hybrid feature space is 9 based on the Eigen value.Therefore PCA transformation is applied on the feature space and corresponding PCs are chosen for the classification.

SVM Classifier Results
To evaluate the classification performance, several accuracy criteria are computed.Kappa coefficient and Overall Accuracy (OA) are common criterion in classification accuracy measurement which is applied in this paper to compare the classification accuracy of thermal image, visible image and fused image.Moreover Khat is used to compare each class accuracy.
SVM classifier is applied to classify visible, thermal and hybrid images.Referring to effect of SVM parameters on classification performance, grid search is used to estimate the SVM parameters.The effect of regularization and kernel parameters is shown in Figure 2. Figure 2 shows that SVM performance is strongly affected by the kernel parameter and less so by the regularization parameter.Table 3 presents the obtained results of grid search for visible image, thermal image and the proposed hybrid image.The obtained results clearly showed the superiority of the proposed algorithm in that the classification accuracy was reached.As Table 3 shows, the classification accuracy of the proposed method improve classification accuracy of visible and thermal imagery, 8% and 9% respectively.More detail about the each class accuracy is shown in Figure 3.
dimensional space, able to separate the data from classes 1 and -1 with a maximum margin.w is a weight vector, orthogonal to the hyperplane, b is an offset term and  is a mapping function which maps data into a high dimensional space to separate data linearity with a low training error.Maximizing the margin is equivalent to minimizing the norm of w. thus by solving the following minimization problem, SVM will be trained:   (.() + ) ≥ 1 −   and   ≥ 0,   = 1, … , (1)where C is a regularization parameter that imposes a trade-off between the number of misclassification in the training data and the maximization of the margin and i  are slack variables.The decision function obtained through the solution of the minimization problem in Equation (

Figure 3 .
Figure 3. Khat measurement for each classPer class analysis depicts the superior performance of the proposed method in almost all classes, and for red roof and tree, there is not meaningful difference with visible and thermal

Figure
Figure 4. Classification map

Table 2 .
SFS features formula   is length of directional line, c denotes central pixel, H(c) is directional line histogram of central pixel () = { 1 (), … ,   ()}, a is constant value,   is length of i th direction line,   is standard deviation of spectral value in the i th direction line and and

Table 3 .
Results of grid search