Quality Assessment of Land Cover Classification Products Based on the Fuzzy–AHP Synthesis Model

As one of the critical products National Geographic Conditions Monitoring Achievements (NGCMA), land cover dataset has great value for the practical needs of China natural resources investigation and monitoring. For providing This paper proposed a fuzzy-AHP synthesis model, for assessing the quality of land cover classification products, based on the current qualifications of inspection and acceptance as well the characteristics of the inspection process of this type of results. The model is practiced on a data set that is the inspection results from 27 surveying areas in northern China. By comparing the results to the results from minimum method (MinM) and weighted average methods (WAM), this study pay attention on giving the analysing of the reasonability and feasibility of this model on practical uses and to provide a technical reference for quality evaluation of the new-type of fundamental surveying and mapping products.


INTRODUCTION
Land Cover dataset (LC) is the one of the most critical products of National Geographic Conditions Monitoring Achievements (NGCMA). Considering the practical needs of China natural resources investigation and monitoring, this product contains comprehensive attributes that aims to correctly reflect the types and natural properties of the features on the ground surface. Undoubtedly, the higher the quality of LC products, the stronger the support will be for analysing the changes of national geographic conditions (Haipeng et al, 2018, Wenjun et al, 2019. Therefore, it is of great significance to assess the qualities reasonably.
According to the current-used quality inspection standards of surveying and mapping results, the quality evaluation adopts the multi-factors evaluation method based on quality elements, that is, on the basis of quality element score, the MinM, WAM and others are used to determine the quality score and quality grade of unit results. In practice, due to the fuzziness of the quality of surveying and mapping results, the evaluation methods mentioned above based on the rigid quality concept have the problem of information loss, resulting in rough evaluation results and weak objectivity (Li et al, 2015, Yanwei and Jianya, 2004, Jingtong et al, 2008 Fuzzy-AHP is a synthetic model that is centred on fuzzy logic and introduce the multi-criteria decision making (MCDM) method to assess the quality of the products. On the one hand, analytical hierarchy process (AHP) is widely used in MCDM as it can unravel the unstructured problems and sort out the stakeholders' preferences into structured hierarchies (Islam et al, 2020). On the other hand, fuzzy theory takeovers the binary concept that it can process uncertainty and imprecise information and problems more sophisticatedly. Integrating fuzzy theory with AHP provides a synthetic method to eliminate the ambiguity between the border of two criteria and the inconsistency of linguistic and numerable criteria and support decision-makers to give interval judgements (Mosadeghi et al, 2013, Mosadeghi et al, 2015.

* Corresponding author
There were lots of Chinese researchers applied fuzzy theory in evaluating the qualities of digital geographical products (Mangguo and Xiaohong, 2013, Pan et al, 2013, Yuru et al, 2014 and the researches on the application of fuzzy theory in quality assessment on GIS products were also circumstantial enough (Shengwu, 2004). However, the analyses of the results were relatively weak and none of them has set foot in LC products.
Hence, hoping to provide the technical supports for the quality assessment of new types of surveying and mapping geographic information products, this paper proposed a Fuzzy-AHP model aiming to give a reasonable and effective quality assessment for LC products. Based on the previous practical works, a representative dataset is picked up for case the study.

METHOD
The assessment consists of three parts including factors identifying, fuzzy modelling and defuzzification ( Figure 1). Factors and corresponding weights were defined considering the intrinsic features of the LC product. For fuzzy modelling, four grades membership were built according to the standard of quality assessment grade (GB/T 18316-2008) for data normalising. Sequentially, the normalised data were organised in to a matrix through pair-wise comparison. Although the evaluations are processed under the fuzzy circumstance, the results are going to be defuzzied in the final to exact scores in order to provide the clear reference to decision-makers.

Identification of factors and weights
There are 7 quality elements that are considered in the currently used specifications for LC product, in which, the elements of 'Coordinate system' and 'Time accuracy' are special ones that even if there is only one mistake in either of the two factors, the entire sheets of maps would go unqualified. In other words, only these two elements get totally correct is there an opportunity to go further evaluation. So, the two elements are excluded from the factors set that is defined as showing in Table 1.  Table 1 Identified factors and weights values.

Development of grade membership function
The grade membership function has the feature of monotonous at the beginning and the end while crosses between the adjacent grades, which means to create the buffer zones between grades to eliminate the 'hard-border' problem. Practically speaking, the settings of the position and range of buffer zones are subjective that always rely on the products characteristics and the experience of experts. This study set a buffer zone of 10 scores and define the grades set in 4 levels as "Excellent", "good", "qualified", "unqualified" according to the standard of quality assessment grade (GB/T 18316-2008). The function is showed as followed (Eq.1) The fuzzy relationship matrix is generated through calculating each factor in the function.

Analytic hierarchy process (AHP) and normalisation
AHP proposed by Saaty (1988) is one of the commonly used approach for multi-criteria evaluation and has adopted in fuzzy logic. It uses pair-wise comparison to estimate the relative importance of each attribute. In this case, the fuzzy relationship matrix and the weights matrix were compared and the relative importance matrix (Eq. 2) were obtained.
In the matrix , each grade was normalised to a value between 0 and 1 indicating the contribution of that grade to the product quality.

Defuzzification
In order to get a unique value other than a range of scores, defuzzification is applied to the fuzzy set. Many studies have clarified that the defuzzification values would have influences on final results, while practically, the calculating approach of that values are varied depending on the different applications and much rely on expert's experience (Saaty, 1988, Jing et al, 2019 In this paper, the mean value method (Jing et al, 2019), which is easy to calculate and easy to apply, is adopted. The defuzzification values are depicted in Table along with their corresponding grade levels and the ranges of scores of that levels.  Table 2 The defuzzification values

Overview of the case
The quality evaluation model and method proposed in this paper is examined with a case study of the LC monitoring results from 27 county-level survey areas in northern China. The assessment results will be compared with the results obtained from WAM and MinM to explain the scientific and rationality of Fuzzy-AHP model on LC quality assessments.

Results analysis
The data and results are shown in appendix. The result illustrates that the Fuzzy-AHP model can evaluate the quality of LC products effectively and more reasonable than other two methods. The scores obtained from Fuzzy-AHP are in the mid of the other two ( Figure 2) and smoothed out the influences of the unduly high or low scores of factors to overall results. As can be seen from Table 3, the qualities that are evaluated by Fuzzy-AHP as "excellent", "good", "qualified" are 11 sheets, 13 sheets and 3 sheets, respectively. Comparing with that of WAM (14 sheets of "excellent") and MinM (7 sheets of "qualified"), Fuzzy-AHP method is more refined on LC qualities evaluation.   Table 3 The number of map sheets that are evaluated by three methods in each grade level.
Specifically, the high precision of classification is the fundamental factor for achieving excellent quality of LC products therefore the highest weight of the factor of "classification accuracy" (0.6). Taking the 3 sheets of lowest scores as examples (Table 4), in the test field No.15, the scores of "classification accuracy" is 64.4, while that of "collection accuracy" is 80. In this case, the overall score calculated by WAM, on the one hand, is still higher than 75 so that got the "good" grade level, which is lack of reasonability because of the importance of the factor of "classification accuracy". On the other hand, the overall score calculated by MinM (64.4) is far too low when the scores of other three factors are much higher than 90, which is not able to reflect the influences of every factors on the overall quality of the product. In a nutshell, Fuzzy-AHP method is of more realistic and objective for LC quality assessments, especially when the lowest score occurs on the classification accuracy.  Table 4 When the scores of "Classification accuracy" are the lowest, the comparison of the final scores between the three methods.

Uncertainty analysis
By comparing the results that were calculated by three methods, some abnormalities can be observed that the results calculated by Fuzzy-AHP method were lower than that of MinM in the test field No. 4,6,10,11,17,25. Through specific analysis, it can be learnt that when of the scores of each factor are all higher than 85 and the "classification accuracy" has the lowest scores in the meanwhile, the final result of Fuzzy-AHP would be lower than that of MinM. In addition, the higher the scores of "classification accuracy", the bigger the difference between the Fuzzy-AHP scores and the MinM ones.
The abnormalities could be from the process of defuzzification. During the process, the proper defuzzification values were sought to exact the fuzzy intervals to the precise scores (Shengwu, 2004), however, as what was mentioned before, the final results could be varied by different settings of the defuzzification values. Sometimes, the values are determined based on expert's experience (Saaty, 1988), while those can also be carried out through a large batch of testing in this topic.
But surely, those lower scores have shown no influence on the final quality grade evaluations. Consequently, under the circumstance of obtaining high score of every factor, it could be necessary to have a further discussion on the defuzzification values, for example, taking relative importance of every factor into consideration.

CONCLUSION
This paper proposed a fuzzy-AHP synthesis model, for assessing the quality of LC classification products, based on the current qualifications of inspection and acceptance as well the characteristics of the inspection process of this type of results. One the one hand, the experiment got fair and reasonable results and showed that Fuzzy-AHP method was able to eliminate "hardborder" problem when confronting the integration of different factors so that removed the extreme high and low scores for assessment significantly. On the other hand, the uncertainties in Fuzzy-AHP were mainly from the defuzzification process and thus a further discussion on the defuzzification values should be brought forward under the circumstance of obtaining high score of every factor.