A STUDY ON PRODUCING HIGHLY RELIABILE REFERENCE DATA SETS FOR GLOBAL LAND COVER VALIDATION

Validating the accuracy of land cover products using a reliable reference dataset is an important task. A reliable reference dataset is produced with information derived from ground truth data. Recently, the amount of ground truth data derived from information collected by volunteers has been increasing globally. The acquisition of volunteer-based reference data demonstrates great potential. However information given by volunteers is limited useful vegetation information to produce a complete reference dataset based on the plant functional type (PFT) with five specialized forest classes. In this study, we examined the availability and applicability of FLUXNET information to produce reference data with higher levels of reliability. FLUXNET information was useful especially for forest classes for interpretation in comparison with the reference dataset using information given by volunteers.


INTRODUCTION
The Global Change Observation Mission-Climate (GCOM-C1) is progressing along with the Japan Aerospace Exploration Agency (JAXA)'s activities, and the second-generation global imager (SGLI) on GCOM-C will be launched in 2016 fiscal year.SGLI covers a wide spectrum from 380 nm to 12 um with spatial resolutions from 250 m to 1 km, polarimetry, and forward/backward simultaneous observation capabilities.Global land cover products have been planned to be produced by the International Geosphere-Biosphere Programme (IGBP) class scheme (Loveland, 1997) with the SGLI 250m spatial resolution data to derive gross primary production (GPP) estimations.GPP estimation studies require highly accurate land cover products whose class scheme is based on the plant functional type (PFT) since the estimating error of GPP products is dependent on the accuracy of land cover products.Therefore, validating the accuracy of land cover products using a reliable reference dataset is important.
Reference datasets for validation methods are produced using a large amount of high spatial resolution satellite data (Olofsson, 2012 andStehman, 2012), and by collecting information on the land cover state using a broad network of experts with local knowledge (Mayaux, 2006 andBontemps, 2015).Recently, land cover information collected by volunteer such as that obtained in the Degree Confluence Project (DCP) was used to validate land cover products (Foody, 2013 andIwao, 2006).They use land cover information as ground truth information for producing reliable reference data.The information includes a location (latitude and longitude), photographs of landscapes, and brief descriptions of the land state (DCP, 2015).DCP is a volunteer-based project that aims to collect land cover information at the point of latitude and longitude integer degree intersection in the world.In our previous study (Soyama, 2015), a method was developed to produce high-quality reference data, using the information from DCP to validate global land cover products based on the IGBP land cover class scheme in 500m spatial resolution.The DCP data is collected from a uniform spatial and temporal spread of potential sampling points.However, the DCP includes limited useful vegetation information to produce a complete reference dataset based on the PFT with five forest classes: evergreen needleleaf forest, evergreen broadleaf forest, deciduous needleleaf forest, deciduous broadleaf forest, mixed forest.GPP estimation studies (Sasai, 2011 andThanyapraneedkul, 2012) use the FLUXNET database (FLUXNET, 2015).FLUXNET contains information about tower locations and site characteristics.Thus, we can expect more comprehensive and detailed vegetation information from this resource that can be used to produce a more accurate reference dataset.
In this study, we examined the availability and applicability of FLUXNET information to produce reference data with higher levels of reliability based on our method developed in previous study.Reference dataset, which is produced using the method are applied to MODIS IGBP global land cover products.Finally, a discussion about the use of FLUXNET information to validate global land cover products is presented.

Data used for interpretation
FLUXNET is a global network of micrometeorological tower sites that use eddy covariance methods to measure the exchanges of carbon dioxide, water vapor, and energy between terrestrial ecosystems and the atmosphere.FLUXNET contains information about tower locations and site characteristics.In this study, information of Ameriflux, Asiaflux, Ozflux, Canadian Carbon Program (CCP) and Infrastructure for Measurements of the European Carbon Cycle (IMECC) were used.In total, 409 flux-site points were selected.As information for interpretation of reference data, Coordinates (longitudes, latitudes), photographs and Site Characteristics (Investigator Provided Type) from FLUXNET web site were used.After checking those information in FLUXNET web site, we found that there is a little error information for coordinates and lack of Investigator Provided Vegetation Type in web site description.Therefore, we also use site additional information on network: Ameriflux (Ameriflux, 2015), Asiafux (Asiaflux, 2015), CCP (CCP, 2015) ,IMECC(European Fluxes Database Cluster, 2015) and Ozflux (Ozflux, 2015).
Figure 1 shows the distribution of 409 confluences on Google Earth examined in this study.In this study, reference data are interpreted based on the IGBP class scheme at a spatial resolution of 500 m, which is the double space scale of SGLI sensor of 250m spatial resolution.Google Earth was used to validate the accuracy of the class interpretations at 500 m resolution.

Global land cover map
Land cover type 1 (IGBP global vegetation classification scheme) of the MODIS land cover type products (MCD12Q1) in 2007 was used.

Interpretation of reference data
Method of producing reference data is mainly the same as our previous study.Interpretation of reference data was conducted by two interpreters.Their method of interpretation was the following procedure.Firstly, label was interpreted from characteristics (Investigator Provided Vegetation Type) on FLUXNET website (FLUXNET, 2015) and additional site information on website for each fluxnet.For Ameriflux, Asiafux and Ozflux, detailed land cover condition information helped accurate interpretation.Two interpreters can line up as many as four candidate land cover classes with variable reliability.Secondary, label interpreted using fluxnet information was checked at a spatial resolution of 500 m using Google Earth.The interpreters identify a flux site point at 500 m spatial resolution as one of four land cover types i.e., woody land, herbaceous land, mixel or non-vegetated (hereafter simple land cover types), by cross-referencing the location with Google Earth data according to the rule determined in our previous study (Soyama, 2015).
Quality level of reference data is determined by assigning each label to two interpreters and combining their common interpretations with four land cover types.

Quality level of reference data
The quality level of reference data was determined using the number of agreement between labels by two interpreters, the reliability level of labeled classes assigned using four simple land cover types and the number of candidate class nominated according to the rule determined in our previous study (Soyama, 2015).
The conditions of the three quality levels are defined as follows: Quality level 1: the two labels agreed upon by two interpreters, as well as IGBP classes interpreted by two interpreters are considered to have high reliability of labeled class assigned using four simple land cover types, and they nominate one candidate class.Quality level 2: Agreement by interpreters on two labels, considered to have high reliability.
Quality level 3: Agreement by interpreters on two of the assigned labels, considered to have high or medium reliability.

Agreement of candidate class-1 between two interpreters
The kappa coefficients of the agreements between the two interpreters on candidate class-1 for all data is good to fair (kappa coefficient = 0.781).Based on the kappa coefficients, the two interpreters appear to apply the same rules for data interpretation.In our previous study using DCP information, the interpreted reference data by IGBP classification, grassland and cropland were recognized more often than the other classes by the two interpreters.The total percentage of five forest types was not so high comparing grassland and cropland.This may be due to poor forest landscape visibility with many DCP data points and it is not easy to reach point of a dense forest for volunteers.On the other hand, dense forest sites of FLUXNET are selected by specialists for each research objectives.

Reliability of labels based on simple land cover types
For the simple land cover types for all the data, the agreement between two interpreters from ranged from good to fair (kappa coefficient = 0.775).Table 2 shows the reliability of the two interpreters' assigned labels on simple land cover types, based on the IGBP's definition of reliability levels (shown in Table 1 of our previous study paper (Soyama, 2015)).The FLUXNET sites were assigned to high-reliability classes with a 90% and 97%, respectively, whereas low-reliability classes are assigned at 3% and 0% of the cases, respectively.Although there were some incorrect interpretations of the simple land cover types, the IGBP classes' interpretations by the two interpreters were considered to be highly reliable.

Quality of reference data for FLUXNET
The number of reference datasets selected based on three quality-level conditions was 192, 335 and 370 in quality-level order.About half of total flux sites used in this study satisfied the condition of quality level 1.Most flux sites (82%) satisfied the condition of quality level 2. FLUXNET information was considered to be efficient in interpretation in comparison with the result using DCP information.
Table 3 shows the percentage of quality level-1 reference data to all points for each flux network.There is characteristics information on web site of flux networks except IMECC.In particular, on the Asiaflux network database on web, there are useful information for interpretation; domestic species of overstory and understory, canopy height, age of the vegetation and fetch of the land condition.On that of the Ameriflux, there are dominant species composition, canopy height, vegetation type, land history and research topics.Oz-flux has useful information on 'Site Description' pages.Most of CPP sites are included on Ameriflux sites.In case of IMECC, there is only IGBP class name, whereas the percentage of quality-level-1 reference data to all points for each flux network was a little lower than other flux network.
Table 3.The percentage of quality-level-1 reference data to all points for each flux network.

Comparison of results between the MODIS IGBP land cover products and reference data quality level
Table 5 shows the validation results of the MODIS IGBP global land cover products, with the reference datasets selected based on three quality-level conditions as well as the number of quality-level reference data.One-pixel columns show the overall accuracy and the kappa coefficients for One-pixels.Nine-pixels columns display the result of Nine-pixels.Both the overall accuracy and kappa coefficients of quality level 1 are the highest of the three quality-level reference datasets.For quality level 1, agreement between two interpreters ranged from good to fair (kappa coefficient = 0.620, 0.725).Accuracy decreases with the quality level.This outcome is the same for validation with reference data using DCP information in our previous study.

SUMMARY
Our final aim is to produce high-quality reference data set for validation of global land cover product in GCOM-C.Validation needs a reliable reference dataset produced with information derived from ground truth data.Recently, the amount of ground truth data derived from information collected by volunteers has been increasing globally.However, it is difficult to produce forest reference data in high-quality, since the DCP includes limited useful vegetation information to produce a complete reference data of forest classes.In this study, we applied the method of producing reference data sets with quality-level information to FLUXNET information.FLUXNET information was useful especially for forest classes for interpretation.

Figure 1 .
Figure 1.Distribution of 409 confluences on Google Earth examined in this study.MODIS/Aqua Surface Reflectance 8-Day L3 Global 500 m data for all bands was used to examine the seasonal changes in the vegetation classes.

Table 2 .
There are less the low or the middle reliability class points by V2.V2 lined up multiple candidate land cover classes for their flux sites.The low or the middle reliability classes by V1 are 42 points in which similarly V1 lined up multiple candidate land cover classes except 2 sites.This may that the low or the middle reliability, determined by two interpreters, was due to an erroneous interpretation of the simple land cover type.The reliability of the two interpreters' labels based on simple land cover types.

Table 4 .
The percentage distribution of IGBP classes of qualitylevel-1 reference data (192 points) for each flux network.

Table 5 .
The validation results of the MODIS IGBP global land cover products with quality-level reference data.

Table 6 .
The validation results of the MODIS IGBP global land cover products with quality-level reference data for each fluxnet.
. A global land cover validation data set, II: Augmenting a stratified sampling design to estimate accuracy