MULTIVIEW ANALYSIS OF MIXED PIXELS IN THE FRACTION AND REFLECTANCE DOMAINS FOR UNDERSTANDING SUB-PIXEL TOPOGRAPHIC STRUCTURE

The spectral mixture analysis (SMA) plays a vital role in spectral data analysis and extraction of subpixel information. However, this technique provides only quantitative information regarding the materials’ abundance fractions within the pixel. On the other hand, the Bidirectional Reflectance Distribution Function (BRDF) indicates that sub-pixel topography affects the surface’s directional reflection to a large extent. Unfortunately, despite the high importance of the BRDF effect and the SMA in remote sensing, only very few research works addressed their mutual influence. Thus, in this work, we propose a study that addresses this mutual influence and suggests an approach for extracting sub-pixel topographic information from mixed pixels. For this purpose, we conducted two multiview imaging experiments under controlled conditions using artificial mixed surfaces. Each surface type is made of two materials and has a varying structural pattern. Then we measured the BiConical Reflectance Factor (BCRF) of each surface from various viewing zenith angles. Next, we applied spectral unmixing to estimate the abundance fraction of three endmembers (EMs) in each surface’s pattern. Finally, we tested the relationship between the sup-pixel topography and the fraction variation vs. the multiple imaging directions. The first experiment results showed that multiview spectral measurements allow the separability between surfaces combining the same materials’ composition but with different sub-pixel structural arrangements. Moreover, such separability is more accurate in the fraction space than in reflectance space. Besides, and most importantly, the second experiment revealed exciting outcomes regarding the relationship between the sub-pixel topographic feature and the variation of the EM fraction vs. the imaging viewing direction. Specifically, we showed a high correlation between the EMs’ fractions and the height of a repetitive element within the sub-pixel topography with a determination coefficient that reaches 0.89.


INTRODUCTION
Hyperspectral imaging provides essential information regarding material surfaces' chemical and physical properties. Such valuable information is widely used in many interdisciplinary studies and civil and commercial applications (Carrasco et al., 2003) at various small and global scales (Hussain et al., 2013). For example, rich knowledge about the earth's surface is available for scientific research using remotely sensed spectral images. However, this data type is usually acquired under different conditions and from various sensors with different spectral, temporal, and spatial resolutions. Thus, extracting information from spectral data is a challenging task. Two main effects that increase the complexity of interpreting the data are the spectral mixture and the Bidirectional Reflectance Distribution Function (BRDF). Mixed pixels contain more than one landcover/material type within their area and always exist in spectral images due to the typical low spatial resolution. Therefore, a helpful way to analyze mixtures is through applying spectral unmixing that allows for extracting subpixel information from mixed pixels. In particular, spectral unmixing allows estimating the fractional abundances corresponding to distinct materials, so-called endmembers (EMs) (Kizel and Benediktsson, 2020;Kizel and Shoshany, 2018;Shoshany et al., 2011). Usually, the fraction estimation follows a preliminary step for EM determinations. Different approaches are available for this step. However, the most common approach relies on the automatic extraction of the EMs from the data. Well-known algorithms such as N-FINDR (Winter and E. Winter, 1999) and Vertex Component Analysis (VCA) (Nascimento and Dias, 2005) exploit a multidimensional geometrical feature of the data. A simplex is formed by all data points in N-dimensional space (N -number of wavelengths) such that the pixels containing the purest EM are located at the extreme points. The VCA algorithm is fast and suitable also for data without pure pixels due to low spatial resolution (Bioucas-Dias et al., 2012). On the other hand, the BRDF effect bears an undesired variability in the spectral measurements under various geometrical specifications in data acquisition. Therefore, researchers have mainly focused on eliminating the variability caused by BRDF (Jia et al., 2020) through inverting models to characterize the captured surfaces (Zhang et al., 2018). However, the BRDF effect combines the material's properties and roughness (Shoshany, 1991). Thus, analyzing the variability caused by the BRDF effect can provide valuable information regarding the sub-pixel characteristics of the surface. Nevertheless, this information is only geometrical regardless of the material composition within the pixel. On the other hand, spectral unmixing provides sub-pixel information regarding this composition, but only quantitatively. This work addresses the mutual influence of the spectral unmixing and the BRDF effect and studies their combination to extract sub-pixel topographic information from mixed pixels.

METHODS
Our main objective is to study the mutual influence of spectral unmixing and the BRDF effect to extract sup-pixel topographic information from mixed pixels. For this purpose, we created artificially hand-made mixed pixels with various sub-pixel topographic properties. In particular, we used a background made of one material and placed cubes made of another material on it. Then, we created two sets of surfaces with a repetitive internal topography containing two materials: 1. Set #1: To create different sub-pixel topography, we arranged the cubes in four different arrangements varying by rotation and the distance between the cubes. Figure 1 presents the repetitive pattern in the four arrangements. 2. Set #2: In this set, we focus on a single sub-pixel topographic property, precisely the height of the cubes. Thus, we created six surfaces with the exact arrangement of the cubes on top of the background but at varying heights, as Figure 2 presents.
We used each set of surfaces to create a dataset of mixed pixels measured from multiple viewing angles as follows: 1. First, we acquire a spectral image of each surface from different viewing angles. 2. Then, we create mixed pixels from each pattern by sampling a region of interest (ROI) representing the repetitive part of the pattern and averaging its pixels.
Then, to study the mutual influence of the spectral mixture and the BRDF, we analyze each dataset as follows: 1. Extract the EMs from the pixels of each pattern. For this purpose, we examined the VCA and N-FINDR algorithms. Currently, we use the original images of repetitive patterns (before averaging) for the EM extraction for more accurate results. 2. Apply the spectral unmixing algorithm to estimate fractional abundances for each mixed pixel. 3. Analyze the relationship between the fractions and the subpixel topographic properties of each pattern. We conducted the proposed study in two phases. In the first phase, we used Set #1 to examine the separability between the different patterns by analyzing the spectral reflectance and the fractions obtained from the various viewing directions. Then, we used Set #2 in the second phase to focus on a single subpixel topographic feature and estimate it by analyzing the fraction variation vs. the imaging viewing angle.    Figure 3 presents the study's workflow. Besides, in the following subsections, we detail each of its steps.

Multiview Imaging
To understand the mutual influence of the BRDF and the spectral mixture, we created two datasets of mixed pixels with the previously described internal topographic patterns. For this purpose, we captured spectral images of each pattern from different viewing angles. The setup specifications, e.g., the number of acquired images, the camera positions, and the light source, vary between the two datasets. We acquired hyperspectral images under controlled conditions, including camera positions, light source, patterns, and calibration panel. The patterns in each scene have been captured from multiple viewing angles. We acquired the two datasets in a dark room using the Specim IQ hyperspectral camera. This camera employs a push-broom line scanner mounted behind the optical lens with an internal rotating mechanism. The obtained images have a size of 512 512  pixels and 204 spectral bands within 400-1000nm. The camera's field of view is 31 31    , and provides a spatial resolution of 1mm at a 1m distance. Besides, we used the C12 silent halogen lamp provided by Hedler (https://www.hedler.com). To retrieve the reflectance values in each spectral image, we divided each pixel's radiance by the radiance of the BaSO4 (barium sulfate) calibration panel that we placed within the image's frame. Under these specifications, we created two datasets as follows: and camera coincide, and the camera's shadow appears on the scene. Thus, we located the camera, in this case, near the light source with a slight deviation from the principal plane. After selecting and averaging the repetitive pattern area in each image, Dataset #1 includes 20 mixed pixels, five per pattern.
Dataset 2: The mixed surfaces are blue foam cubes arranged on recycled mat paper sheets in this dataset ( Figure 2). We use the same cubes' arrangement in all patterns but with five different 1,2,3,4,5 [cm], arranged symmetrically with a 1 cm distance between each row and column. Figure 4(a) presents one of the surfaces with a polygon representing the repetitive pattern. We located the camera at a 1m distance from the sampled surface at seven view angles [ 60 , 45 , 20 ,0 ,20 ,45 ,60 ]

Mixed pixels
To create mixed pixels from each surface, we first select the ROI representing the repetitive pattern from each image. Then, we sampled these ROIs manually using MATLAB 2021b. The sampled ROI represents the fundamental component of an infinite surface with a repeating pattern. Therefore, it simulates structures captured at a lower resolution or from a distance. Eventually, the spectral mixture is calculated by averaging the reflectance values of all pixels within the ROI as follows: Where, R and i R are the reflectance spectra of the obtained mixed pixel and the i-th pixel, out of n pixels with the ROI, respectively. The wavelength  indicates a spectral band.

EM extraction and Spectral Unmixing
First, we automatically extracted the EMs from each pattern's pixels (i.e., different viewing angles). For this purpose, we examined the VCA and N-FINDR algorithms. Three EMs were determined: background, cubes, and shaded areas. Then, we applied the unmixing for all mixed pixels. The main objective of the unmixing process is to estimate the fractional abundance of each EM within the pixel. Given the linear model, the light from the captured surface is received as a weighted mixture of the EMs. Thus, we represent a mixed spectrum as follows: is a systematic noise (assumed to distribute as mean-zero Gaussian function), and , db are the number of EMs and spectral bands, respectively. Additionally, the estimation of fractional abundance vector, â , is subject to sum-to-one and nonnegativity constraints, i.e., The unmixing process estimates a vector of fractional abundances corresponding to the EM set by solving an optimization problem that minimizes the spectral distance between the measured and reconstructed spectra. Different methods for unmixing vary mainly according to the metric used as an objective function and the type of optimization. We examined the two methods: Vectorized Projected Gradient Descent Unmixing (VPGDU) algorithm (Kizel et al., 2017), and SUNSAL (Bioucas-Dias and Figueiredo, 2010) sparse unmixing by variable splitting and augmented Lagrangian. The VPGDU algorithm is robust to amplitude variation, as may be caused by illumination changes. This robustness is an advantage over the SUNSAL and the classical unmixing techniques such as Fully Constrained Least Squares FCLS (Heinz and Chang, 2001). On the other hand, SUNSAL is fast and computationally light.

EXPERIMENTAL RESULTS AND DISCUSSION
We intended to understand whether spectral mixture analysis combined with a multiview imaging can assist in understanding sub-pixel topography within mixed pixels. Thus, we designed two experiments using the previously described Dataset #1 and Dataset #2. In both sets, the hand-made mixed pixels include background cubes and a mutual shadowing effect. Thus, during the unmixing process, we consider that each mixed pixel has three EMs. Moreover, we hypothesize that the mutual shadow and the obstructed parts of cubes or background are the most important aspects to investigate in this study. Therefore, multiview variation of these EMs will be addressed in the following two experiments.

Experiment 1: Understanding the separability between sub-pixel topography patterns
Analyzing mixed-pixel data captured from different viewing angles has the potential to determine pattern variation and accordingly provide valuable information regarding their subpixel topography. In this experiment, we used the mixed pixels from Dataset #1. Recall that this set has 20 mixed pixels, each five corresponding to a particular pattern we captured from five different viewing angles. First, automatically extracted the EMs for each pattern. Figure 5 presents the result for the examined four patterns in Dataset #1. Then, we estimate the abundance fractions of the EMs in each mixed pixel. The results in Figure 5 show that the extracted EMs for the background and the cube vary between the different patterns. However, all the patterns are made of the same two materials. Thus, this variation occurs only due to the sup-pixel topography variation between the patterns. Therefore, the extracted EMs from the different patterns can be the first indicator for the subpixel topography within mixed pixels.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France  Next, we observed the obtained reflectance and EMs fractions for each pattern from the different viewing angles. The plots in Figure 6 show that the results vary between the different patterns. Again, since all patterns are made of the same two materials, this variation indicates a variation in the subpixel topography.
Then, for a quantitative assessment of this indication, we measured the separability between the patterns in the reflectance and the fraction domains as follows: Given two signals, 1 s and 12 , ss we compute the Euclidean distance (ED) between them as follows: Then, to compute the separability between two patterns, we compute the ED between them in all spectral bands and EM fractions. Next, to allow comparison between the two domains, we normalized the ED values in each domain by computing as follows: where, ED and * ED are the mean and normalized ED values. Finally, we compute the separability metric between the two patterns in the spectral domain as the ED with the maximal value over all the spectral bands. Similarly, the separability metric in the fraction's domain is the ED with the maximal one over all the EMs. Table 1 summarizes the results that show that the separability varies between the patterns according to their internal topographic variation. The separability is mainly higher for pairs that vary in rotation and spacing between the cubes. Besides, except for the case A-C, the separability in the fraction domain is higher. Thus, we believe that the variation in the fraction values over the different viewing angles is a better indicator for the sub-pixel topography.

Experiment 2: estimating topographic sub-pixel information
Encouraged by the first experiment's results, we focused on one sub-pixel topographic feature in this experiment. In particular, address the variation in cubes height within the mixed pixels. Thus, we use the mixed pixels from Dataset #2. As mentioned before, the patterns with five different heights of 1,2,3,4,5[cm] are arranged similarly with a 1cm distance at five rows and five columns in this set. In this way, the representative part of the infinite pattern is at the center of a pattern Figure 4(a)). First, we automatically extracted three EMs for each pattern. Figure 7 presents the results. The shape of the EMs from the different patterns are highly similar. However, the magnitude of the extracted EMs for the cube varies significantly between the different patterns. To test the relationship between EM magnitude and the sub-pixel topographic feature, we plot the cube's heigh vs. the reflectance in the spectral band 464 nm and compute their correlation. Figure 8 presents the results that show a correlation of 0.89 between the cube's EMs reflectance magnitude and the cube's height. Then, we tested the correlation between statistical features of the EMs fraction and the cube's height in each pattern. In particular, we tested how the mean and maximal values of the cube and shadow EMs over the different viewing angles vary as a function of the cubes. Figure 9 shows the data plots and correlation coefficients. Together with those in Figure 8, these results significantly indicate that analyzing the measurements of mixed pixels from various viewing angles assets in extraction information regarding the sup-pixel topography.

CONCLUSIONS
We presented a study addressing the mutual influence of the spectral mixture and the BRDF effect. In particular, we studied the option of extracting information regarding the sub-pixel topography within mixed pixels by analyzing multiview images The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France of such pixels from various viewing directions. We hypothesized that the sub-pixel topography of a mixed pixel influences its BRDF. Thus, the variability of the fractions vs. the imaging viewing angle should indicate some of this internal topography. To test this hypothesis, we conducted experiments with data sets of artificially created mixed pixels that involve different patterns of a sub-pixel topography. The results support our assumptions and clearly show a correlation between the sup-pixel topography of mixed pixels and their fraction variability when observed from different directions. These promising results indicate that spectral measurements can help extract topographic information from surfaces with internal patterns more minor than the image resolution. However, we should invest many efforts to achieve this objective. Thus, we will examine the proposed study on natural mixed surfaces in future work.