IEEE PROJECT 4001 – STANDARDS FOR CHARACTERIZATION AND CALIBRATION OF HYPERSPECTRAL IMAGING DEVICES

Hyperspectral imaging (HSI) systems have been invaluable tools for over two decades, but there are few authoritative standards that characterize these systems or define the data and metadata they produce. Manufacturers calibrate instruments and report specifications differently and, in some cases, the same term has different definitions among HSI programs. To address these inconsistencies, the Institute of Electrical and Electronics Engineers (IEEE) Geoscience and Remote Sensing Society (GRSS) sponsored Project 4001 (P4001), a Hyperspectral Working Group under the auspices of IEEE’s Standards Association. Since its inception in 2018, the IEEE P4001 Working Group has been working to specify testing and characterization methods for HSI device manufacturers, as well as recommend data structures and terminology for HSI products. P4001 focuses on the ultraviolet through the shortwave infrared spectral range (~250 to 2500 nm) and prioritizes camera technologies that are in widespread use. Many aspects of the standard will have wider applicability with respect to camera technology and wavelength range, and updates will expand the range of technologies and topics covered. Industrial, laboratory and geoscience use cases are informing the development of the standard. Utilization of the P4001 HSI standard will lead to HSI systems with consistent characterization and calibration criteria, as well as interoperable data products with a common lexicon for data and metadata. 1. THE NEED FOR HYPERSPECTRAL IMAGING STANDARDS 1.1 Proliferation of Hyperspectral Systems In recent years, the number of hyperspectral imaging (HSI) devices has grown, as has the market for these devices. Hyperspectral imagers are smaller and lighter as manufacturers take advantage of new technology and innovative designs. In addition, the price of hyperspectral systems has decreased, making the technology more cost-effective for a wider range of applications. It is beyond the scope of this manuscript to describe all the practical applications of hyperspectral imaging devices, as there are too many to list. A few of the major application areas are agriculture, medicine, environmental science, manufacturing, and defense. The number of HSI platform types is growing. Traditional benchtop, handheld, airborne, and satellite platforms continue to be relevant, and now unmanned aerial vehicles (UAVs) and small satellites may host HSI devices. Finally, hyperspectral imaging datasets are available to the masses via internet-accessible public domain data repositories. For example, the U.S. Geological Survey’s Earth Resources Observation and Science (EROS) Center archive contains HSI datasets collected by NASA’s Hyperion sensor, which launched in November 2000 and was decommissioned in March 2017. NASA’s Airborne Visible Infra-Red Imaging Spectrometer (AVIRIS) is also available online as well as AVIRIS NextGeneration data from the National Ecological Observatory Network (NEON). 1.2 The Value of Standards With so many hyperspectral imaging devices available, a potential HSI user must first decide which device supports a specific application. Unfortunately, there is no standard set of quality and performance metrics for HSI devices, which makes device comparison problematic at best. Each hyperspectral instrument manufacturer may have its own unique terminology, specifications, and characterization methods. In addition, there are few standards for the measurements and metadata in a hyperspectral dataset, i.e., the data structures for the resulting measurement data. The two HSI standards for datasets encoded under the National Imagery Transmission Format (NITF) are the 2011 Implementation Profile for Tactical Hyperspectral Imagery (HSI) Systems (NGA.IP.0006, 2011) and the 2019 Spectral NITF Implementation Profile (SNIP) (NGA.STND.0072_1.0_SNIP, 2019), which updated and augmented the 2011 HSI standard. Both standards are tailored to a specific use case, i.e., remote sensing of the Earth. Standards for the terminology, characterization, testing, and data structures of HSI devices are needed so that one can (A) compare HSI devices to determine which device is suitable for a particular application; (B) determine if the data structure produced by an HSI device contains the measurements and metadata needed for a particular application; and (C) determine whether the data structure and its format are interoperable with existing tools, techniques, and systems. Also, standards for The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-M-3-2021 ASPRS 2021 Annual Conference, 29 March–2 April 2021, virtual This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLIV-M-3-2021-43-2021 | © Author(s) 2021. CC BY 4.0 License. 43 dataset content and format are needed so that HSI libraries can be mined by the widest possible user community without requiring customized software for each sensor system. 2. SPECTRAL CAMERA TYPES Spectral cameras collect data across the electro-optical (EO) spectrum, typically ranging between ultraviolet light and longwave infrared radiation. The quality and applications differ with camera type. The four main scanning techniques are spatial, spectral, snapshot and spatio-spectral (Figure 1). Figure 1. Spectral camera types. 2.1 Spatial Scanning Cameras Also known as pushbroom or line scanning spectral imagers, these devices collect data in a line imaging mode, like machine vision line scan cameras, using a two-dimensional detector system in space and wavelength (x, ) (Figure 1A). Sequential collection of spatially contiguous (x, ) data results in a hyperspectral image with dimensions (x, y, ). Pushbroom images are ideal for remote sensing and on-line production applications. 2.2 Spectral Scanning Imagers A spectral scanning imager uses tunable wavelength filters to sequentially collect two-dimensional images in (x, y) over a range of wavelengths (Figure 1B). These imagers commonly use Liquid Crystal Tuneable Filters (LCTF) or Acousto-Optic Tuneable Filters (AOTF). This spectral camera type can achieve high quality spatial resolution and is ideal when the sample under study is static. 2.3 Snapshot Spectral Imagers Also known as framing or non-scanning imagers, these devices simultaneously record both spatial (x, y) and spectral () information of a scene (Figure 1C). There are a few forms of these types of imagers. One common type is the “Computed Tomography Imaging Spectrometer (CTIS), which uses a twodimensional kinoform dispersing element to simultaneously collect data in (x, y, ), which is then processed to construct a three-dimensional image (Okamoto et al., 1993). 2.4 Spatio-Scanning Imagers Spatio-scanning imagers comprise of linear spectral filters in front of the sensor and require either a moving camera platform or moving samples like pushbroom systems (Figure 1D). These instruments usually have reasonable spectral resolution and are potentially lower cost in the visible spectrum compared to pushbroom imagers. The resulting data frames require processing to construct the three-dimensional (x, y, ) image. 3. IEEE HYPERSPECTRAL STANDARDS INITIATIVE


Proliferation of Hyperspectral Systems
In recent years, the number of hyperspectral imaging (HSI) devices has grown, as has the market for these devices. Hyperspectral imagers are smaller and lighter as manufacturers take advantage of new technology and innovative designs. In addition, the price of hyperspectral systems has decreased, making the technology more cost-effective for a wider range of applications.
It is beyond the scope of this manuscript to describe all the practical applications of hyperspectral imaging devices, as there are too many to list. A few of the major application areas are agriculture, medicine, environmental science, manufacturing, and defense.
The number of HSI platform types is growing. Traditional benchtop, handheld, airborne, and satellite platforms continue to be relevant, and now unmanned aerial vehicles (UAVs) and small satellites may host HSI devices.
Finally, hyperspectral imaging datasets are available to the masses via internet-accessible public domain data repositories. For example, the U.S. Geological Survey's Earth Resources Observation and Science (EROS) Center archive contains HSI datasets collected by NASA's Hyperion sensor, which launched in November 2000 and was decommissioned in March 2017. NASA's Airborne Visible Infra-Red Imaging Spectrometer (AVIRIS) is also available online as well as AVIRIS Next-Generation data from the National Ecological Observatory Network (NEON).

The Value of Standards
With so many hyperspectral imaging devices available, a potential HSI user must first decide which device supports a specific application. Unfortunately, there is no standard set of quality and performance metrics for HSI devices, which makes device comparison problematic at best. Each hyperspectral instrument manufacturer may have its own unique terminology, specifications, and characterization methods.
In addition, there are few standards for the measurements and metadata in a hyperspectral dataset, i.e., the data structures for the resulting measurement data. Standards for the terminology, characterization, testing, and data structures of HSI devices are needed so that one can (A) compare HSI devices to determine which device is suitable for a particular application; (B) determine if the data structure produced by an HSI device contains the measurements and metadata needed for a particular application; and (C) determine whether the data structure and its format are interoperable with existing tools, techniques, and systems. Also, standards for dataset content and format are needed so that HSI libraries can be mined by the widest possible user community without requiring customized software for each sensor system.

SPECTRAL CAMERA TYPES
Spectral cameras collect data across the electro-optical (EO) spectrum, typically ranging between ultraviolet light and longwave infrared radiation. The quality and applications differ with camera type. The four main scanning techniques are spatial, spectral, snapshot and spatio-spectral ( Figure 1).

Spatial Scanning Cameras
Also known as pushbroom or line scanning spectral imagers, these devices collect data in a line imaging mode, like machine vision line scan cameras, using a two-dimensional detector system in space and wavelength (x, ) ( Figure 1A). Sequential collection of spatially contiguous (x, ) data results in a hyperspectral image with dimensions (x, y, ). Pushbroom images are ideal for remote sensing and on-line production applications.

Spectral Scanning Imagers
A spectral scanning imager uses tunable wavelength filters to sequentially collect two-dimensional images in (x, y) over a range of wavelengths ( Figure 1B). These imagers commonly use Liquid Crystal Tuneable Filters (LCTF) or Acousto-Optic Tuneable Filters (AOTF). This spectral camera type can achieve high quality spatial resolution and is ideal when the sample under study is static.

Snapshot Spectral Imagers
Also known as framing or non-scanning imagers, these devices simultaneously record both spatial (x, y) and spectral () information of a scene ( Figure 1C). There are a few forms of these types of imagers. One common type is the "Computed Tomography Imaging Spectrometer (CTIS), which uses a twodimensional kinoform dispersing element to simultaneously collect data in (x, y, ), which is then processed to construct a three-dimensional image (Okamoto et al., 1993).

Spatio-Scanning Imagers
Spatio-scanning imagers comprise of linear spectral filters in front of the sensor and require either a moving camera platform or moving samples like pushbroom systems ( Figure 1D). These instruments usually have reasonable spectral resolution and are potentially lower cost in the visible spectrum compared to pushbroom imagers. The resulting data frames require processing to construct the three-dimensional (x, y, ) image.

Goal
The IEEE Project 4001 (P4001) aims to standardize the characterization and calibration of ultraviolet through shortwave infrared (250 to 2500 nm) hyperspectral imaging devices.
Operating in accordance with a four-year charter, the P4001 Working Group was formed on May 14, 2018 with an end date of December 31, 2022.

Oversight
The IEEE Geoscience and Remote Sensing Society (GRSS) Standards Committee, which is closely aligned with the GRSS Standards for Earth Observations (GSEO) Technical Committee, oversees the P4001 effort and operates under the IEEE Standards Association policies and procedures. The GRSS Chair is a member of the P4001 leadership team and actively participates in P4001 meetings. One or more liaisons to IEEE also participate in P4001 activities.

Membership
International experts comprise the P4001 membership. Currently there are 34 voting members and over 232 observers. IEEE membership is not required to join, but P4001 elected officers must be members of the IEEE Standards Association.
There are three levels of participation: observers, non-voting members and voting members. Observers may attend meetings but shall not participate in discussion or voting. A non-voting member must attend one meeting and ask the P4001 Chair for non-voting membership. Non-voting members may participate in discussion but return to observer level by missing four consecutive meetings. Voting membership requires participants to attend two of the last four meetings and request the Chair for voting membership.

Methodology
The P4001 Working Group is developing and formalizing a standard that will support the hyperspectral community over the long term, ensuring stability for years to come. The time required for P4001 to develop the HSI standard, as well as the standard's success, depends in large part on the strength of the consensus reached by the hyperspectral community. The stronger the consensus on the scope of P4001's efforts and the resultant formalized results, the more likely that the P4001 standard will benefit a large percentage of the hyperspectral community for many years.

Organization
The P4001 Working Group holds meetings for the general membership every four to six weeks. The work is managed by three subgroups, which meet every one to two weeks: Terminology, Characterization & Testing, and Data Structures (Figure 2.) These subgroups necessarily interact. For example, the Data Structures Subgroup requires terminology and definitions for metadata fields, which the Terminology Subgroup will supply.

Terminology Subgroup:
The goal of the Terminology Subgroup is to collate and define physics-based terminology to establish methods for testing and calibration that accurately convey product specifications. Written definitions and welldefined equations (where possible) that objectively and unambiguously define these terms is critical for success.
Initial work has focused on collating terms for review. In order to align with the efforts of the Characterization & Testing Subgroup, the terms are divided into four categories: • Spectral Terminology • Spatial Terminology • Signal/Radiometric Terminology • System and Operational Terminology.

Data Structures Subgroup:
The product of a hyperspectral image collection, i.e., the measurements and metadata, is ultimately analyzed by users for specific applications. The goal of the Data Structures Subgroup is to create conceptual and logical data models of HSI data structures independent of product format, leveraging the physics-based terminology and characterizations developed by the Terminology and Characterization & Testing Subgroups. These models may then be used to develop data product standards in one or more formats.
Some manufacturers create datasets in proprietary formats, and few HSI communities have reached consensus on the content, structure, and format for HSI data. Nor has consensus been reached on the processing chain, i.e., the processing that takes place, after collection of the EO signal, that produces analysisready data (ARD), i.e., data that can be exploited for a particular application.
The Data Structures Subgroup is addressing these issues by developing sample data flows and data structures for three use cases: • Laboratory use case • Industry use case • Geoscience use case, i.e., remote sensing of the Earth.
This approach reveals the measurements and metadata that are common to all three use cases, as well as the data specific to a particular use case.
For example, spectral radiance measured by a HSI device is usually processed to create a hyperspectral data-cube, i.e., a hypercube, free from detector and optical effects (Figure 3). The input signal is at-aperture spectral radiance. The image sensor produces raw digital numbers (DN), at which point a raw image dataset may be created. The raw DNs are then corrected for detector and optical effects, and a hypercube is created, with spectral radiance in relative DNs, i.e., proportional to spectral radiance. The processing chain may then store these results, which in the geoscience use case is a relative non-rectified radiometric product. For applications that need measurements in absolute radiometric units, the relative DNs are radiometrically calibrated so that the measurements are in a physical unit of spectral radiance, such as W m -2 sr -1 µm -1 . The ARD created by the processing flow may then be exploited in several ways. Figure 3 is an example of one possible sequence of exploitation operations for the geoscience use case. Spectral analysis may be performed on the at-aperture absolute radiometric non-rectified product to classify materials in the scene, or spatial analysis may be performed to classify objects. More often in the geoscience use case, the effects of the atmosphere between the ground and the sensor are mitigated as much as possible so that the measurements represent groundleaving radiance, and measurements are converted to percent reflectance to minimize artifacts from the illumination source, i.e., the sun. Spectral and spatial analyses of the resultant ground-leaving reflectance product often yield more accurate data than can be achieved from the at-aperture absolute radiometric non-rectified product. A high-level conceptual data model of a hyperspectral dataset for all use cases is shown in Figure 5. The core of the dataset is the hypercube with dimensions x, y, and . Spatio-temporal metadata (i.e., where and when the data were collected) are also necessary, as are sensor properties, such as the number of bands, spectral smile, detector bit depth, etc. The surrounding environment, whether man-made as in the laboratory, or natural for geoscience data, is also needed, as are collection properties such as view geometry, integration time, and collection time.

Characterization & Testing Subgroup:
While imperfections in the optics, sensors and electronics degrade image data, the performance characteristics specified in this standard are in terms of the output image and therefore independent of the internal operation of the camera. Since hyperspectral imagers differ in the degree of processing applied to the output data, the standard covers three levels of calibration and correction: radiometrically calibrated cameras, optical and sensor-effect corrected cameras and uncorrected cameras.
Over 50 performance characteristics have been identified spanning a breadth of terms related to distortions, enclosed energy, coregistration, interdependence, stray light, timing, calibration, and more.
A set of test procedures have been considered to verify camera specifications meet the P4001 standard. From radiometric, spatial, and spectral resolution tests to spatial geometry and spectral calibration, as well as a stray light test and spectralspatial interdependence estimation, members of P4001 have devised a comprehensive list of procedures to act as guidelines for manufacturers to follow.

COLLABORATION WITH OTHER STANDARDS DEVELOPMENT ORGANIZATIONS
A guiding principle of at least one standards developer is to "invent as little as possible" (Eckstein and Durdall, 2019) and thus leverage existing standards as much as possible. This approach has multiple advantages. Leveraging mature standards lessens the risk of issues and errors in the new standard. Incorporation of existing standards lessens the workload while simultaneously producing a standard that is already interoperable, at least in part, with existing tools and techniques. In terms of a standard dataset design and format, interoperability is key to minimizing the cost of incorporating new data products into an existing enterprise because costs may prevent a program from adopting a standard.

European Machine Vision Association 1288
In this spirit, the P4001 Working Group actively collaborates with several other SDOs. For example, the European Machine Vision Association (EMVA) 1288 is a standard that simplifies the comparison of cameras and imaging sensors used for machine vision applications (EMVA 1288(EMVA , 2021. The P4001 Working Group plans to leverage EMVA 1288 whenever applicable, such as for fundamental testing standards, especially for the "snapshot" hyperspectral systems. EMVA 1288 can also help P4001 make sure that hyperspectral system test requirements are simple, essential, and consistent.

The International Organization for Standardization
Some of the International Organization for Standardization (ISO) standards are essential to the geoscience-related hyperspectral imagery. For example, the ISO/TS 19159-1:2014 standard addresses calibration and validation of airborne and spaceborne optical imaging sensors, including geometric, spectral, and radiometric calibration (ISO/TS 19159-1:2014, 2021. The ISO 19115 series for geographic information metadata is also applicable and widely used by the remote sensing community.

IEEE Standards Association Projects
The P4001 Working Group also collaborates with other projects in the IEEE Standards Association, such as P4005, which is developing a standard and protocols for reflectance spectroscopy of soil (IEEE P4005, 2021).

National Center for Geospatial Intelligence Standards
The P4001 Working Group is collaborating with the USA National Center for Geospatial Intelligence Standards (NCGIS) because of the hyperspectral standards developed under its auspices by the Research Directorate of the USA National Geospatial-Intelligence Agency (NGA) (Eckstein and Durdall, 2020). The Spectral NITF Implementation Profile (SNIP) version 1.0 addresses the geoscience use case and was ratified by the USA Geospatial Intelligence Standards Working Group (GWG) and then the Joint Enterprise Standards Committee (JESC) in 2019. SNIP version 1.1 is currently in development. The P4001 Working Group is considering working with the SNIP to bound image specifications.

The Spectral NITF Implementation Profile (SNIP)
The SNIP is a physical data product standard, i.e., it specifies the content requirements, plus optional and conditional elements, for both multi-and hyperspectral NITF-encoded datasets, down to the bit and byte level (Eckstein and Durdall, 2020). SNIP-conformant datasets contain the complete suite of measurements and metadata necessary for all expected exploitation functions in the spectral, spatial, and temporal dimensions, as well as other aspects of an actual dataset. Metadata fields are described in detail, with definitions, relevant character sets for encoding field values, and the allowed ranges of such values.
As shown in Figure 6, a SNIP dataset may contain many elements. At its core, however, are a few required elements: a spectral image segment, a text segment indicating that the SNIP was used to create the data product, and a few tagged record extensions (TREs) providing the minimum essential metadata:  If the SNIP is used to encode a hyperspectral dataset, then a quick-look image segment, i.e., a one-or three-band image derived from the spectral image segment, is required. The quick-look image may also be a spatially downsampled version of the spectral image segment.
The SNIP specification document is located on the NSG Standards Registry, at https://nsgreg.nga.mil/doc/view?i=4879.

CALL FOR PARTICIPANTS
Ultimately the success of this endeavor depends on the participation from the hyperspectral community as it will affect the adoption and benefit of the standard itself. As the full draft of the P4001 standard nears completion, the P4001 leadership invites the hyperspectral community for input and feedback.