LEVEL FUSION OF ORTHOPHOTO AND LIDAR DATA USING CONFUSION MATRIX INFORMATION FOR LNAD COVER CLASSIFICATION

Automatic urban objects extraction from airborne remote sensing data is essential to process and efficiently interpret the vast amount of airborne imagery and Lidar data available today. The aim of this study is to propose a new approach for the integration of highresolution aerial imagery and Lidar data to improve the accuracy of classification in the city complications. In the proposed method, first, the classification of each data is separately performed using Support Vector Machine algorithm. In this case, extracted Normalized Digital Surface Model (nDSM) and pulse intensity are used in classification of LiDAR data, and three spectral visible bands (Red, Green, Blue) are considered as feature vector for the orthoimage classification. Moreover, combining the extracted features of the image and Lidar data another classification is also performed using all the features. The outputs of these classifications are integrated in a decision level fusion system according to the their confusion matrices to find the final classification result. The proposed method was evaluated using an urban area of Zeebruges, Belgium. The obtained results represented several advantages of image fusion with respect to a single shot dataset. With the capabilities of the proposed decision level fusion method, most of the object extraction difficulties and uncertainty were decreased and, the overall accuracy and the kappa values were improved 7% and 10%, respectively.


INTRODUCTION
The performance of land cover classification using LiDAR data and Aerial imagery data separately has previously been analyzed and it was shown that superior results were achieved using LiDAR data (Jakubowski et al., 2013).However, simultaneous use of several remote sensing data from different sensors or methods of integration may be appropriate.In other words, the data obtained from different sources, each with aspects of the value that can be used together and are complementary (Esteban, Starr et al. 2005).Multiple sensors may provide complementary data, and fusion of information of different sensors can produce a better understanding of the observed site, which is not possible with single sensor (Simone, Farina et al. 2002).
Fusion of multiple data sets can be performed on signal, Pixel, feature and decision level (Pohl and Van Genderen 1998).In signal level fusion, signals from multiple sensors are combined together to create a new signal with a better signal-tonoise ratio than the input signals.In pixel level fusion, the information from different images, pixel by pixel, are merged to improve detection of objects in some tasks such as segmentation.Feature level fusion consists of merging the features extracted from different images.In this level of fusion, features are extracted from different sensors and combined to create a feature vector for classified using a classifier method (Abbasi, Arefi et al. 2015).
In decision level fusion, different datasets are combined at a higher level of integration.In this level of fusion, first the data from each single sensor is separately classified, then fusion consists of merging the output from the classification (Goebel and Yan 2004, Dong, Zhuang et al. 2009, Du, Liu et al. 2013).
Singh and Vogler investigated the impact of urban land use litter to detect classes of integration of Landsat imagery and lidar data began.By applying the supervised classification method showed that most like to merge the two data classification accuracy up to 32% compared to the separate use of lidar data has increased (Singh, Vogler et al. 2012).
Kim and colleagues used data integration aerial imagery and lidar ground cover gave way to improve classification accuracy.(Kim 2016).Gerke and Xiao proposed a combination of lidar and aerial image classification and automatic detection of effects used in urban areas.The method combines lidar data and aerial image of two dimensional geometry and spectral data liar image to extract four classroom buildings, trees, vegetation and land without vegetation land was used.The new classification method was introduced to the use of geometric and spectral information during the process of classification was defined (Gerke and Xiao 2013).
Another method for automatic extraction of buildings using lidar integration and optical image was presented by Li Wu.This procedure was completely data-driven, and also was suitable for any form of buildings (Li, Wu et al. 2013).

Bigdeli proposed a hyperspectral image classification system to integrate multiple fuzzy decision model based on lidar data
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W4, 2017 Tehran's Joint ISPRS Conferences of GI Research, SMPR andEOEC 2017, 7-10 October 2017, Tehran, Iran Decision template (DT.In this way, the characteristics of each data source separately extracted and the optimal properties were selected among them.Then, each of the data sources was classified separately using support vector machine.The output of this classification integrated using a Bayesian algorithm (Bigdeli, Samadzadegan et al. 2014).
Gilani, proposed a graph based algorithm for combination of multispectral images and airborne data lidar.The results showed enhancement in performance and improvement in the accuracy of the reconstruction and building recognition process (Gilani, Awrangjeb et al. 2015).Rastiveis Presented a Decision level fusion of lidar data and aerial color imagery based on Bayesian theory for urban area classification.This classification is performed in three different strategies: (1) using merely LiDAR data, (2) using merely image data, and (3) using all extracted features from LiDAR and image.The results of these classifiers were integrated in a decision level fusion based on Naïve Bayes algorithm (Rastiveis, 2015).
For the abovementioned researches, therefore, using color imagery with lidar data together in decision level of fusion may improve accuracy of classification in urban areas.In this study, we proposed a new decision level fusion of lidar data and orthophoto to improve the classification results.For this purpose, the paper will go on with describing the details of the proposed method.After that the study area would be introduced and, then, obtained results from the implementation of this method will presented.

PROPOSED METHOD
In this paper, an automatic object extraction system based on the integration of high resolution aerial orthophoto and LiDAR data is provided.Figure 1 shows the flowchart of the proposed algorithm.
Figure 1.Flowchart of the proposed method for automatic object extraction from LiDAR data and orthophoto As shown in Figure 1, after extracting proper features, both input data are classified through support vector machine classifier.Integrating all the extracted fetures from both sensors, another another classification are performed.Finally, according to the output from each of the classification and their error matrices, the results are integrated in a decision level fusion system to obtain the final classification result.The details of the proposed method are described in the following sections.

Data Preparation
In this step, as can be seen from Figure 1, different features are extracted from Lidar and ortho image data.These features must contain useful information to improve accuracy of classification process.
Intensity of Lidar as the primary attribute from lidar data recorded the reflectance on the surveyed area (Axelsson 1999).
It is useful to discriminate asphalt areas from grassland or buildings in classification process (Buján, González-Ferreiro et al. 2012).Therefore, the intensity and is used as a feature in the classification of Lidar data.Besides, Normalized Digital Surface Model (nDSM) is extracted using LiDAR raw points cloud through Equation 1.
nDSM= DSM -DTM (1) It should be noted that to simplify the fusion process the data set should have the same spatial resolutions.Therefore, a irregular format of the nDSM and the Intensity of LiDAR data are generated using interpolation techniques with the same resolution of the orthophoto.
From the orthophoto, three visible channels of Red, Green and Blue were considered as feature vector for the classification process.
Although different features may be used in classification process, however, increasing the number of features results more complexity in computations.Therefore, the most easyaccess features for both data set are considered.

SVM Classification
There are a lot of classification algorithm that have been introduced during the last decades.In this paper, Support Vector Machine (SVM) classification algorithm is used to classify the data sets.Support vector machine is a machine learning method that is widely used for data analyzing and pattern recognition.
The algorithm was invented by Vladimir Vapnik and the current standard incarnation was proposed by Corinna Cortes and Vladimir Vapnik.Classifying data has been one of the major parts in machine learning.The idea of support vector machine is to create a hyper plane between data sets to indicate which class it belongs to.The challenge is to train the machine to understand structure from data and mapping with the right class label, for the best result, the hyper plane has the largest distance to the nearest training data points of any class (Pal 2006).In this step, both data sets are classified using the aforementioned feature vectors.

Feature Level Fusion
In the orthophoto classification dataset, building was difficult to extract appropriately due to the similar spectral features with roads.But if only with orthophoto, it was not able to provide sufficient information.Meanwhile, in the lidar classification dataset, water and grass extraction also met problem caused by lidar data acquisition.Therefore, by integrating lidar and orthophoto, the problems could mitigate and the results of classification could be improved.
Here, all layers used in the previous classifications are integrated together as the input data of another SVM process.Orthophoto, including three spectral bands Red, Green, and Blue, and lidar derived Intensity image and nDSM image are employed in the classification process.Finally, combining the previous classification results, the final classification are obtained.In this case, decision for each pixel are made according to the results of the overall classification from ortho photo, lidar data and the feature level fusion results.So that, first all pixels that the three classifications were agree about belonging them to the same class are remaind without changing.The pixels that the classifications were disagree were labeled with regard to the overall accuracy and omission, commission error is a class act.In this case, the priority of a class in the final image is selected based on the lowest error rate or highest overall accuracy.Thus, all the pixels are checked and a decision is made on each.

EXPERIMENTAL RESULTS AND DISCUSSIONS
In this study, the standard data set from IEEE data fusion contest 2015 were used.The details of the data set are described in the following section.After that the obtained results from testing the algorithm using this data set are presented.

Data Set
The imaging data were acquired on March 13, 2011, using an airborne platform flying at the altitude of 300 m over the urban and the harbor areas of Zeebruges.The data were collected simultaneously and were georeferenced to WGS-84(IEEE, 2015).
The point density for the LiDAR sensor was approximately 65 points/m², which is related to point spacing of approximately 10 cm.Both the 3D point clouds and the resulting digital surface model (DSM) are provided.The color orthophotos were taken at nadir and have a spatial resolution of approximately 5 cm (IEEE, 2015).The ground truth of this data set five different land cover classes; in our proposed method these classes are Building, Grassland, Ground, Car and Tree.These features from LiDAR data, orthoimage and Ground trouth are displayed in Figure 2 (a-c).

Results and Experiment
In the first step of the proposed method, feature spaces on orthophoto and Lidar data produced independently.Generate regular Lidar data with 5 cm spatial resolution, intensity image and nDSM.These features from LiDAR data are displayed in Figure 2(d-e).In the next step, classification based on SVM is applied on orthophoto and Lidar data.This step by selecting the appropriate training and set dependent parameters were implemented.Five classes of "Buildings", "Cars, "Grass", "Ground" and "Tree" were considered.The main function of radial is the kernel used.Gamma parameter 1, parameter 100 penalty for lidar data and ortho photo by 0.333 and 100 has been set.In Figure 3 the outputs of the classifications are shown.
After separately classifiying the orthophoto and the Lidar data, another classification based on fusing five extracted features were performed.Figure 4 represented the output of this classification.Also, figure 5 shows the output of the weighted majority voting method on these three categories.This method uses the correctness of each class as a weight in decision making.The Error matrix of each category in Table 1 to 5 is shown.The final classification was made from the decision level fusion method which is shown in Figure 6.As can be seen from Figure 7, the decision level fusion improved the kappa coefficient and overall accuracy of the three previous classifications.Also, it can be observed from Table 5 that the Buildings and the Ground classes, in relation to other classes, have been classified more accurately.Accuracy grass class in the other photo is higher of the other data.The building class in the lidar data and accuracy class of ground and tree in the feature fusion image data had the maximum accuracy in separately classification of the datasets.With the capabilities of the proposed decision level fusion method, most of the object extraction difficulties and uncertainty were decreased and, the overall accuracy and the kappa values were improved 7% and 10%, respectively.

CONCLUSION
In this paper, a new decision level fusion approach for the integration of high-resolution aerial imagery and Lidar data to improve the accuracy of classification in the city complications was proposed.The overall accuracy of the classification complications urban area in five different classes, for orthophoto 70.95%, 80.31% for lidar data, 84.51% at feature level and for the final image of integration in decision level 87.28% were obtained.The obtained results showed 7% and 10% improvement in the overall accuracy and the kappa values, respectively.the weighted majority voting is a fusion method at the decision level.also the results are compared with that method.Although the results were promising, however, more tests considereing different data ses, features, number of classes are suggested.
separately applied to each data.Then results of single classifiers are fused through a multiple classifier system classifier fusion is successfully applied on various types of data to improve single classification results.Classifier fusion can improve classification accuracy in comparison to a single classifier by combining results of classification algorithms on different data sets.The possible ways of integrating the outputs of classifiers in a decidedly fusion depend on what information can be obtained from the discrete members.In our proposed method, based on classification results in three lidar data, orthophoto and integration features, overall accuracy and omission, commision error for each class final image is created from the classification.

Figure 4 .
Figure 4.The output of the SVM classification from feature level fusion.

Figure 5 .
Figure 5.The output of the weighted majority voting classification.

Figure 6 .
Figure 6.The output of the final classification using Decision Level Fusion.

Figure 7 .
Figure 7.Comparison between classification results

Table 3 Confusion
Matrix from SVM classifiers feature level.

Table 4 .
Confusion Matrix from SVM classifiers decision.

Table 5 .
Confusion Matrix from Weighted majority voting.In Tables 6 (a-b) user and producer accuracies of each category for all performed classifications are shown, and Figure7depicted the resulted overall accuracy and kappa coefficient.