Accuracy Dimensions in Remote Sensing

: The technological developments in remote sensing (RS) during the past decade has contributed to a significant increase in the size of data user community. For this reason data quality issues in remote sensing face a significant increase in importance, particularly in the era of Big Earth data. Dozens of available sensors, hundreds of sophisticated data processing techniques, countless software tools assist the processing of RS data and contributes to a major increase in applications and users. In the past decades, scientific and technological community of spatial data environment were focusing on the evaluation of data quality elements computed for point, line, area geometry of vector and raster data. Stakeholders of data production commonly use standardised parameters to characterise the quality of their datasets. Yet their efforts to estimate the quality did not reach the general end-user community running heterogeneous applications who assume that their spatial data is error-free and best fitted to the specification standards. The non-specialist, general user group has very limited knowledge how spatial data meets their needs. These parameters forming the external quality dimensions implies that the same data system can be of different quality to different users. The large collection of the observed information is uncertain in a level that can decry the reliability of the applications. Based on prior paper of the authors (in cooperation within the Remote Sensing Data Quality working group of ISPRS), which established a taxonomy on the dimensions of data quality in GIS and remote sensing domains, this paper is aiming at focusing on measures of uncertainty in remote sensing data lifecycle, focusing on land cover mapping issues. In the paper we try to introduce how quality of the various combination of data and procedures can be summarized and how services fit the users’ needs. The present paper gives the theoretic overview of the issue, besides selected, practice-oriented approaches are evaluated too, finally widely-used dimension metrics like Root Mean Squared Error (RMSE) or confusion matrix are discussed. The authors present data quality features of well-defined and poorly defined object. The central part of the study is the land cover mapping, describing its accuracy management model, presented relevance and uncertainty measures of its influencing quality dimensions. In the paper theory is supported by a case study, where the remote sensing technology is used for supporting the area-based agricultural subsidies of the European Union, in Hungarian administration.


INTRODUCTION
Dozens of algorithms operate to obtain different outcomes for countless applications of remote sensing data.A great part of their production chain are low documented or standardised.Some procedures have well-known quality dimension metrics like Root Mean Square Error (RMSE) or confusion/error matrix, etc.Yet numerous techniques flourish without any information on the level of trust of their outcomes.Quality of procedures like image classification are usually described by statistics of a confusion matrix, however the error budget may be a cumulative sum of errors originating from previous procedures.Another interesting question arises from the fact that we usually assume that external sources like reference data, training sample for different procedures contain no error.However the scale of uncertainty should limit the use of the end products.Still no common, standardised measure for uncertainty estimations exists for image processing performed by the general user community as welldefined as by data providers.For this reason, the current paper goes into the detail of the interconnections between the remote sensing data quality * Corresponding author models, the features of the processing, the metrics and their supporting validation, verification and calibration techniques.The paper is intended not to be pure theoretical, in contrast, practical example is taken to demonstrate feasible methods.In land use and rural agricultural case studies we demonstrate the spatial data quality descriptions in fitness for use assessment.The paper is organized as follows.Remote sensing data quality management with terminology and methodology is presented in Section 2. Section 3 introduces the accuracy dimensions and their relevant metrics.The supporting use case is described in Section 4 and some conclusions close the paper.

REMOTE SENSING DATA QUALITY MANAGEMENT
In the past decade, data quality in remote sensing (RS) has come a major issue of Earth Observation (EO) and geomatics.A very diverse community reaching from academia, RS industry, government agencies and product end-users have developed fundamentally different understandings of the subjects.Up to now the uncertainties in remote sensing data products has been a challenging task with no global solution.Various and conceptually different approaches have been developed in RS communities to manage the validation problems of RS data products.Several methods try to answer the urging question: how good is the RS data set?To draw a typical example, governmental agencies dealing with large EO datasets try to understand the most relevant user requirements of administration and based on it develop an official remote sensing data quality model.Other key leaders of the RS community are typically space agencies focusing on verification and documentation of data capturing and their processing chain from the sensor's perspective.Data providers delivering RS Data Quality (RSDQ) information to general user community by standardised quality models.Yet these quality measures are often misused or ignored due to lack of technical and theoretical knowledge.As a result, given a specific application a decision based on a heterogeneous RS data sets with published uncertainties from the producer's side, often results in false assumptions that data is perfect, error-free on the end-users' side.
Generally, the RSDQ is impossible to determine by exact and simple definitions from the user's perspective, since varies across diverse user requirements.End users look for a solution to evaluate the data set as "fitness for use" for a certain application, but up to now there is no globally, generally applicable solution for that.This paper reviews the RS Data Quality management model based on the production lifecycle phases (Before production, Production, After production).The sequential phases after Jakobsson and Giversen can be summarised as follows: Before productionconceptualize: Quality documentation focusing on the specification of the Data Quality Model (DQM).
The main goal of this phase is to define quality requirements of the costumer.Based on the ISO 19000 standard family the concept of an applicable quality model can be defined (ISO 19157).The usual components of the DQM are the DQ Criteria describe the RS Quality Dimensions, while the DQ Metrics describe the quantitative, qualitative values and the DQ Checking procedures describe the quality evaluation process.
Productioncollect, process: In this phase the certificate process is focusing on the documentation of the process history.The goal of the quality management is the inspection whether data meet the given specification or not.During the production phase the ISO 9000 quality management principles should be applied.Stakeholders have to be certificate for quality management process, are required to have recorded lineage, production methods etc.The most important factor is the traceability of the complex production chain procedure.
After productiondeliver & use: After the production phase quality evaluation should be performed, too.The quality documentation in this phase is focusing on metadata test reports.
The main goal of the evaluation reporting is to measure conformance to predefined quality requirements (ISO 19157).

Terminology
The RS and the GIS community does not apply a common accepted terminology for quality management.The understanding of different terms varies in the devise community reaching from academia, producers to the user side.Different concepts are named identically, or similar concepts use different names.
According to common definitions, the terminology of data quality element is as follows: • Usability: Usability is the degree of adherence to a specific set of data quality requirements.Usability shall be used to describe specific quality information about a dataset's adherence to a particular application or requirements -ISO 19157 • Spatial resolution: defines the level of spatial detail depicted in an image and is related to the smallest ground object that can be distinguished as a separateentity in the image - (Kemp 2008) • Temporal resolution: considers how often a sensor obtains imagery over a particular area or interval between successive acquisitions - (Kemp 2008).
• Spectral resolution: refers to the specific wavelength intervals that a sensor can record and is determined by the number of bands acquired and their width, measured in micrometers (μm) or nanometers (nm) - (Kemp 2008).
• Radiometric resolution: is the amount of energy required to increase a pixel value by one quantization level or count - (Kemp 2008).
• Fitness of use: geospatial and RS datasets are widely used in many applications including critical decision support systems.
The goodness of the dataset, called the Fitness of Use (FoU), is used in the analysis and has direct bearing on the quality of derived information from the dataset that ultimately plays a role in decision making for a specific application - (Shekhar, Xiong Xiong 2008)

RSDQ methodology -Uncertainty and Data Quality Metrics
The general objective of using RS data, in other words the general lifecycle scheme of RS is to register and semantically interpret the natural and built-up environment by RS data.Ignoring the uncertainty, undermine the reliability of the decisions based on RS data.The most user oriented interpretation of the uncertainty problem can be defined by sorting the different types of uncertainties based on the observed, measured and processed object classes like in Fisher et al. 2006.
Well defined objects: • Error -> probability: errors may exist, and its magnitude is then proportional to how much is known about them.Uncertainty can be conveniently represented using probability theory, which provides a sound theoretical basis for assessing and propagating uncertainty.
Poorly defined objects: • Vagueness -> Fuzzy set theory: Vagueness occurs when the identification of the presence of a phenomenon is a matter of degree.To represent uncertainty in spatial data, many models suggested the introduction of broad boundaries replacing crisp ones in the case of regions and lines.The fuzzy sets theory can manage the vagueness, describe the internal structure of the uncertain geometry with membership functions.
• Ambiguity • Discord -> Expert opinion, Demster-Shafer theory: Discord occurs when completely different naming conventions are used.The Dempster-Shafer belief theory is used as the basis for this approach in which the FoU is represented as a range of possibilities and integrated into one value based on the information from multiple sources.The two central ideas of the Dempster-Shafer belief theory are: (a) obtaining degrees of belief from subjective probabilities for a related question, and (b) Dempster's rule for combining such degrees of belief when they are based on independent items of evidence.For a given proposition, and given some evidence, a confidence interval is derived from an interval of probabilities within which the true probability lies within a certain confidence.
• Non-specificity -> Endorsement theory, Fuzzy set theory: Non-specificity arises when the boundary condition that allows the identification of a phenomenon as being present is not well defined.
Various researchers have used multiple realizations of the identification the phenomenon to generate fuzzy and probabilistic extents of the phenomenon

ACCURACY DIMENSIONS IN REMOTE SENSING
Figure 1.Remote sensing data quality context (compiled after C. Batini's idea) Because RSDQ management is a complex area and the current paper does not allow a comprehensive description, we focus on the most frequently used dimensions.These quality dimensions in RS context are an implementation of a generic quality dimension taxonomy (Fig. 1).No single dimension can be represented without its suitable metric.RSQDs have to be interpreted with respect to the data sources and their processing steps, the latter can be named as lifecycle phases.
The current section aims at covering the most common dimensions, starting with the numeric descriptive resolution cluster, followed by the accuracy cluster.All metric values are in SI units or dimensionless.

Resolution Dimensions
The most common quality related dimensions in RS environment are the resolution features, which are mainly associated to the RS sensors.These dimensions fundamentally influence the whole lifecycle and obviously the obtained products.Elements of the cluster with a short definition and their metrics are presented in Table 3.

Accuracy Dimensions
The accuracy quality cluster is a multilevel feature group containing similar categories as in the resolution cluster.The main subgroups are the precision and the accuracy dimension elements.
Because of the heterogeneity of this cluster, some new metrics arise (percent, RMSE or confusion matrix based features).

Dimension metrics: basic terms and their definitions
The previous Section 3.2 has an overview of the accuracy dimensions.Since they are usually presented by numeric values in daily RS practice, their common understanding is crucial.
Literature study emphasises the weakness of their definitions, although many of them have the same or very similar meaning.
The following collection has the goal to systematically explain the occurring terms and their definitions.Synonyms are also given in the description.
The root mean square error (RMSE) is the accuracy measure of the differences between the actual (true) and measured values.
It can be computed theoretically as where MSE is the Mean Square Error, which is the expected value (    E ) of the square of the differences for the measured (M) and ground-truth (GT) values.
In practice, as an example in spatial accuracy measure RMSE is defined for coordinates in X and Y directions as (3) The confusion matrix, also known as error matrix is a specific The rates are often given in percent.
A multiclass confusion matrix can be like the following:  The measures are the same as in binary case, but formulas are extended by the increased number of classes.Accordingly, user's accuracy is computed as the right classified pixels divided by all pixels belonging to the certain class.
In multiclass case two further measures are frequently used: average accuracy and Kohen-kappa.Average accuracy (AA) is defined as the average of all producer's accuracy, while Kohenkappa has a complicated definition.See details in (Cohen, 1960).
Because of the spreading of artificial intelligence (neural networks, deep learning), the confusion matrix has four forms as one can compute for the (1) training set, (2) validation set, (3) test set and (4) all analysed pixels.

CASE STUDY
To illustrate the findings of this paper the case study related to the area-based agricultural subsidies of European Union (EU) will be introduced in this section.The application presented is part of the Integrated Administration and Control System (IACS), which plays a key role in the implementation of the Common Agricultural Policy (CAP) inside the EU.The technical processes of subsidy management heavily rely on the utilization of remote sensing data, supported by sophisticated photogrammetric procedures.The case study is the Hungarian Land Parcel Identification System (LPIS) which is principally a large GIS spatial database serving both administration and mapping procedures.It is the exclusive reference database system of area-based subsidies.Therefore, it must be as up-to-date as possible to efficiently support application and control processes.In Hungary, LPIS database is built-up from physical blocks, and is updated in three different procedures both regularly and occasionally.Primarily, in the framework of a three-year regular updating cycle, about one third of the country's territory is systematically updated every year.Moreover, farmers and the Hungarian paying agency i.e., the Hungarian State Treasurycan make requests for modification in case of any difference is detected between the LPIS data and the real/true cultivation structure (Figure 2).Orthophotographs play the most important role in the development and the systematically updating of LPIS.In the past years about one third of the country has been captured by aerial photographs, and the orthophotos generated from them serve as a basis of systematic update of the LPIS database.Remote sensing proved to be the fastest and the most objective solution to successfully control the area-based agricultural subsidies.Most member states of the EU use remote sensing as a control method, because in the processes of IACS a large amount (at least 5%) of dossiers must be controlled within a very short time period, and the cost of classical in-field checks is relatively high in relation to remote sensing.Considering this and the necessity of objectiveness, the EU Commission motivates member states in legal way to use satellite and airborne images in the control of subsidies of agricultural parcels.The total number of submitted claims in Hungary is between 170 000 and 200 000.During control with remote sensing (CwRS), among others, declared parcels must be measured to check whether their area is correct.High resolution (HR) image time series are used to determine cultivated crops on parcels, while the accurate area measurement is carried out using very high resolution (VHR) images or latest orthophotographs corresponding to the technical specifications.

State acceptance of aerial photographs and orthophotographs
The first quality evaluation procedure in the LPIS lifecycle is related to the quality control of captured aerial photos and the orthophotos generated from them.In Hungary, all the remote sensing data acquisition that were partially or fully financed from public funds must be passed to the polity organization responsible for geodesy and GIS (i.e., Government Office of the Capital Budapest).Data have to pass through the state acceptance procedure and will become the part of official state remote sensing database afterwards.The orthophotos captured for the purposes of LPIS updating and Control with Remote Sensing (CwRS) take the largest part within all the orthophotos in Hungary.Therefore there is a close relationship between state acceptance and the above mentioned two agricultural projects.
In the state acceptance procedure solely the quality of remote sensing data and its pre-processing is examined in details.Within the control of aerial mapping processes, state acceptance deals with the quality control of orthoimage production processes.State acceptance is preceded by an appropriateness check, in which the presence and completeness of documentation related to orthoproduct is confirmed.Consequently, successful state acceptance is confirmed by a certificate and a detailed technical description of the results.These steps ensure the traceability of quality indicators.
The state acceptance procedure of aerial photographs requires several documents, data and statements.These include, but are not limited to: • the permission for aerial remote sensing, and if applicable, the permission for border crossing; • flight plan and other auxiliary data related to the flight, e.g., meteorological data and flight reports; • the valid calibration record of the camera, Global Navigation Satellite System (GNSS) and Inertial Measurement Unit (IMU); • the full documentation of internal quality control of data provider; • the copy of remote sensing data to be accepted, in standard, open data format, without compression.
The evaluation checks whether aerial photographs is conform with the documentation provided and meet all the readability, completeness, radiometric and geometric criteria.
Having accepted the aerial photographs, it can proceed to the next step of state acceptance procedure for orthophotographs generated from them.In this phase, several documents must be submitted to complete this procedure: • the records of ground control points, independent check points and tie points; • the detailed technical documentation of aerial triangulation and orthophoto generation; • the detailed technical description of the generation and amendment of Digital Terrain Model; • the copy of digital orthophoto sections.
Based on this documented indicators, the state acceptance consists of the evaluation of ground control points and check points, aerial triangulation, digital orthophotographs, mosaic lines, radiometric correction, colour equalization, Digital Terrain Model.

The validation of remote sensing images used as area measurement tools
The correctness of area measurement from control with remote sensing (CwRS) of area-based subsidies fundamentally depends on the spatial accuracy of very high resolution orthoimages (aerial or spaceborne).The JRC established an "area measurement tool validation method", which is designed to determine the inherent tool error (accuracy) and is part of Common Technical Specifications (Loudjani et al., 2016).Beside orthoimages, its domain includes GNSS as well.It is important to emphasise that the resulting measurement error is related to the complete method of measurement and not only to the device (images or GNSS in our case).Therefore, although its result considerably depends on the spatial resolution and accuracy of orthoimagery, it is also determined by the methodology of remote sensing data processing, primarily by of parcel delineation phase.The validation method has been built based on ISO 5725 family of standards, "Accuracy (trueness and precision) of measurement methods and results" (ISO, 1994).
The quality of parcel delineation on orthoimagery as a measurement tool can be characterizedassuming there is no biasby its reproducibility limit.This is a parameter used to determine the technical tolerance, which is an inevitable value when determining the acceptability of a declared parcel.
During CwRS, a decision is taken on each declared parcel using technical tolerance.The declared area is accepted if its difference from the measured area does not exceed technical tolerance.Otherwise, it results in or underdeclaration of the area. .

Accuracy management model in land cover application
The quality of the RS based land cover application introduced in the previous section can be evaluated by various quality dimensions.In our paper we limit our analysis to the relevance and to the degree of uncertainty for accuracy dimensions only.
To be able to compare the relevance of each dimension we set up a relative scale between 5 and 1 indicating high and low relevance as well as high and low uncertainty for each QD. Figure 3 summarises the results of the analysis.

Figure 3. Relevance and uncertainty of accuracy dimensions in LPIS land cover application
Geometric precision is an important aspect in classification procedures and is a well measurable dimension.Both spatial precision and accuracy are highly relevant quality dimensions in most applications of RS and is a well-defined, easy to measure metrics.Radiometric precision and accuracy both play a significant role in classification processes however the latter accounts for lower relevance if classes show a high separability in intensity space.The evaluation of both dimensions is part of sensor calibration procedures performed by sensor builders and space industry, measured during in-flight and pre-flight calibration phases.They provide sophisticated quality evaluation models, their procedures are well documented and attached to RS data.Spectral precision is an important issue in classification procedure, the separability of classes showing very similar reflectance properties can strongly depend on the correctness of spectral boundaries (e.g. to allow the separation of different crops with similar reflectance properties).This dimension is measured likewise the previous radiometric precision and accuracy by sensor builders, and is a well-defined quality measure.Spectral accuracy crucially depends on atmospheric conditions at the time of acquisition, and is further influenced by many other distortion factors like shadows, illumination conditions etc.Both relevance and uncertainty is high for this dimension.Temporal precision is relevant to land cover classification but since the mapped phenomena (mostly vegetation) is not a highly varying procedure in time, this dimension is not a key factor of the application.Temporal precision is again measured and documented by the senor operator, thus has low uncertainty.For the same reason temporal accuracy has similar significance however it is more uncertain than precision.
Classification accuracy plays a key role in land cover application, for this reason it's relevance is high just like its uncertainty.the same rule applies for semantic accuracy; relevance is high but results can be very uncertain.Figure 3 indicates that the most relevant and highly uncertain dimensions are spectral accuracy, thematic accuracy, semantic accuracy.

CONCLUSION
In this paper we addressed the problem of the remote sensing data quality (RSDQ) especially from the accuracy dimension point of view to cover the lifecycle phases of production.The data quality concept was introduced after Batini's scheme: RS lifecycle phases -RS data sources -RS quality dimension triangle.The RS data quality management process and the basic terminology of RS data quality dimensions and metrics was described.A simplified remote sensing accuracy management model was applied in the case study of agricultural land cover application.RSDQ is a complex issue that the ISPRS working group has recently started to investigate.The complexity of the issue could be studied by robust evaluation methods in the future due to the heterogeneous user requirements and application areas.No general solution exists for RSDQ model.Metadata interoperability is still a future perspective.Yet this publication can contribute to the growing awareness and understanding of uncertainty in data.It can assist to clarify the importance of error measurement by supporting the general user community with dimensions and their metrics to evaluate this complex problem of data quality.As a result, end-users can arrive at a conceptual understanding that their products are not error-free and decisions can be taken accordingly.

Table 3 .
Resolution dimension cluster

Table 4 .
Accuracy dimension cluster table that allows visualization of the performance of a classification.The matrix is an excellent base to derive further quality measures.Two basic types exist: the binary and the multiclass confusion matrix.The binary confusion matrix visualizes, how a single class (C) and its complementary (not C) were classified in comparison to the true ordering.The rows of the matrix contain the classified (sometimes called as predicted or estimated or output) results, while the columns present the actual (true or target) belongings.The original table has been extended by the goodness and error measures computed from the original table values (the filled values in the matrix are only illustration):