GEOMETRIC QUALITY EVALUATION METHOD FOR MULTI-VIEW SATELLITE IMAGES IN 3D RECONSTRUCTION

The quality of the 3D model reconstructed using multi-view satellite image depends on the quality of the image. To evaluate the geometric quality of the satellite image, we proposed a method to evaluate the geometric distortion for satellite images and defined the deviation coefficient as a metric to evaluate the bending degree of a curve. After projecting a ground grid consisting of straight lines into the image space, the geometric distortion of the image can be evaluated quantitatively by calculating the deviation coefficients of the projection trajectories. Experiments have been carried out with three datasets obtained by JiLin-1, GaoFen-2, and WorldView-3 respectively. The results show that the proposed method can used to evaluate the geometric quality of satellite images effectively, and this evaluation method will be useful in image selecting in 3D reconstruction using multi-view satellite images. * Corresponding author


INTRODUCTION
In recent years, with the rapid development of high resolution optical remote sensing satellites, satellite image resolution, spectrum and data acquisition capabilities have been continuously enhanced. Large-scale 3D reconstruction based on high-resolution satellite images has gradually become possible (Ozcanli et al., 2015). A famous commercial high resolution satellite constellation is the DigitalGlobe satellite constellation. The WorldView-3 belong to the DigitalGlobe satellite constellation is the representation of the high-resolution satellites and its resolution is up to 0.31m. This provide a good foundation for 3D reconstruction. At the same time, the resolution and width of frame video satellites have also been improved (Li et al., 2018). For example, the resolution and the width of Video Satellite 03 in JiLin-1 constellation is better than 0.92m and 11km*4.5km respectively. This provide a new type of satellite image for 3D reconstruction.
To promote the research of 3D reconstruction using highresolution satellite images, IARPA conducted the IARPA Multi-View Stereo 3D Mapping Challenge and released the corresponding IARPA MVS3DM dataset in 2016. This dataset contains 50 WorldView-3 panchromatic, visible, and near infrared (VNIR) images and 1 short wave infrared (SWIR) image. The area imaged is near San Fernando, Argentina . Johns Hopkins University Applied Physics Laboratory (JHU/APL) introduced this dataset in detail and maintains a web site that provides download links to this dataset.
In 2018, IARPA released the dataset used for the Creation of the Operationally Realistic 3D Environment (CORE3D) program (Brown et al., 2018;Leotta et al., 2019). The CORE3D program aims to develop technology that generates accurate 3D object models with real physical properties automatically. The CORE3D dataset include 26 WorldView-3 PAN and MSI images over Jacksonville, FL; 43 WorldView-3 PAN and MSI images over Omaha, NE; 35 WorldView-3 PAN and MSI images over UCSD, CA; 44 WorldView-2 PAN and MSI images over UCSD, CA. This will further promote the research on 3D reconstruction using satellite images.
Researchers have made great efforts to reconstruct 3D models from satellite images. The quality of the 3D models reconstructed from satellite images is affected by various factors (Aguilar et al., 2019;Ozcanli et al., 2015), including type of cover objects (Poli and Caravaggi, 2013), sensors for satellite imaging Poli et al., 2015), 3D reconstruction software and algorithms (Facciolo et al., 2017;Qin, 2017;Shean et al., 2016), and so on. For the 3D reconstruction of a specified area, as the type of cover objects is fixed, the reconstruction quality mainly depends on the image acquisition (sensors for satellite imaging) and image processing (3D reconstruction software and algorithms). In order to improve the quality of 3D reconstruction using satellite images, it is necessary to filter the satellite images before 3D reconstruction, which has been proved to be very effective for satellite images 3D reconstruction (Facciolo et al., 2017). In fact, the quality of satellite image data obtained by different satellites varies a lot, it will seriously damage the reconstructed 3D models if all the images are used for reconstruction without preprocessing. If the number of satellite images is more than enough, the quality of the 3D model can be improved by selecting the best ones for 3D reconstruction.
By modifying the Satellite Stereo Pipeline (S2P) developed by De, et al (De Franchis et al., 2014), Facciolo, et al (Facciolo et al., 2017 proposes a solution that can reconstruct high quality 3D models from multi-view satellite images. The completeness of the 3D models reconstructed by the modified method can reached 79%, and the method wins the first place of the IARPA Multi-View Stereo 3D Mapping Challenge. Facciolo's improving work mainly includes selecting image pairs according to the image consistency and image pairing angle and modifying the registration and fusion algorithms for 3D point clouds. In the follow-up studies, Gong, et al (Gong and Fritsch, 2018) proposed a method to generate 3D point clouds and digital surface models (DSMs) from multi-view satellite images. They select images according to the incidence angle and acquisition date. After epipolar resampling, dense matching and triangulation, the 3D point clouds and DSMs are acquired. The IARPA MVS3DM dataset has been used to demonstrate the improvement of their method in the 3D reconstruction.
As described above, in order to improve the 3D models, image selecting are usually implemented before reconstruction according to the incidence angle, pair forming angle, and acquisition date. In this paper, we propose a new method to evaluate the geometric quality of images that can be used for satellite image selecting in 3D reconstruction. First, we establish a ground grid consist of a series of special straight lines in the ground space; then, we project the ground grid into the image plane using the satellite sensor model (rational function model); Finally, the geometric quality of satellite images are evaluated by analyzing the projection trajectories of the ground grid.

Rational Function Model
First, we introduce the rational function model (RFM) that will be used in our method. The RFM describes the relative positional relationship between an image and the ground object. The forward form of the RFM is used to describe the transformation from ground space to image space (Tao, 2002;Tao and Hu, 2001): coordinates and ground coordinates, and the range is from -1 to +1. The normalization of the coordinates is computed using the following equations: where ijk a are the coefficients of the polynomial. These coefficients together with the offset values and the scale values are called the rational polynomial coefficients (RPCs).

Project a Ground Grid into Image Space
To analyze the geometric quality of a satellite image, we proposed a method as shown in Figure 1. First, a ground grid consist of a series of special straight lines (such as plumb line, horizontal line, diagonal line, etc.) is established in the ground space limited by the offset values and the scale values in the RPCs. Then, the ground grid is projected into the image space using the RFM. Finally, the geometric quality of the satellite image is obtained by analyzing the geometric characteristics of the projection trajectories.
The method used to evaluate the geometric quality of satellite images By choosing straight lines in the range box defined by RPCs appropriately, the geometric quality of satellite images can be analyzed comprehensively. In fact, due to the existence of imaging distortion (Toutin, 2004), the projection trajectories of the ground lines are no longer straight lines, but curves. The geometric quality of satellite images can be evaluated by analyzing the distortions of the projection trajectories.

Analyze the Projection Trajectory
As there is no metric that can be used to evaluate the distortions of the projection trajectories, we define a metric to evaluate the bending degree of the projection trajectories.
Definition 1. For a curve, we fit it to a straight line. The deviation coefficient  of the curve can be calculated as follows: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) Where, max R is the maximum of the fitting residual (the maximum distance of all the points on the curve to the fitted line), min R is the minimum of the fitting residual (the minimum distance of all the points on the curve to the fitted line, usually a negative number), and L is the length of the fitted line. The units of these three variables are pixel. The deviation coefficient describes the ratio of the maximum distance of the curve deviates from the fitted line to the length of the fitted line. When the curve is shorter than 1  pixels, the error introduced by approximating the curve as a straight line is less than 1 pixel.

DATASETS
We chose three datasets consist of multi-view satellite images for experiments, including a dataset obtained by JiLin-1, a dataset obtained by GaoFen-2, and the IARPA MVS3DM Challenge dataset. The JiLin-1 dataset consists of 9 images with different viewing angle obtained by the 03 video satellite belong to the JiLin-1 constellation. The imaging area is New York, and the ground sample distance (GSD) is 0.92m. The GaoFen-2 dataset consists of 7 images with different viewing angle obtained by the Gaofen-2 satellite. The imaging area is Beijing and the GSD is about 0.8m. The IARPA MVS3DM Challenge dataset includes different kind of data as introduce above. We just use the panchromatic images in the dataset for experiments. The GSD of these panchromatic images is about 0.31m.
The information of the three data sets is shown in Table 1, and the coverages of the three datasets are shown in Figure 2.

Overview of the Dataset
Since the multi-view satellite image datasets contain a lot of images, we visualized the imaging direction angles of each image before the quality evaluation experiments. The imaging direction angles of the images in the three sets of data sets were obtained and visualized using a polar coordinate respectively. The visualization results are shown in Figure 3. Every number beside an "x" represents a satellite image. The polar angle represents azimuth angle and the polar radius represents the incidence angle. The units of these two variables are degree. As the number of the images in the WV3 dataset was so large, we selected nine images with uniform distribution for the next experiments, as shown in Figure 3 (d).

Quality Evaluation of the Near Nadir Image
Generally, the geometric distortion of a satellite image is minimal when the incidence angle is zero degree (the nadir image). Therefore, we selected an image whose incidence angle is closest to zero degree from each dataset for the first evaluation experiment. The evaluation experiment was conducted using the method described in the second section. The selected images are JL1-2, GF2-3, and WV3-19.
First, we chose 5 along-track horizontal lines, and 5 cross-track horizontal lines, and 4 diagonal lines in the range box defined by the RPCs to establish a ground grid. Then, the ground grid was projected into the image space. Finally, the residual is obtained by fitting (in a least-squares sense) the projection trajectories into straight lines. The results are shown in Figure 4.
It can be seen from Figure 4 that the projection trajectories projected form the straight lines in the ground space are no longer straight due to the geometric distortion of the satellite imaging.
For the JL1-2 image, the distortions in the along-track direction are similar with the distortions in the cross-track direction. Both the distortion directions in the along-track direction and the cross-track direction are stable. This is because the sensor of the 03 video satellite is a frame camera, and the imaging performances in the cross-track direction and along-track direction are relatively consistent. For the GF2-3 image and WV3-19 image, the distortions in the along-track direction are different with the distortions in the cross-track direction. The distortion directions in the cross-track direction appear to be stable. But the distortion directions in the along-track direction vary irregularly. This is because the sensors of the GaoFen-2 satellite and the WorldView-3 satellite are linear push-broom cameras. The cross-track direction is consistent with the direction of the linear array sensor, and the distortions in the cross-track direction are relatively stable and consistent. The distortions in the along-track direction are mainly caused by the disturbance of the satellite platform, and the distortions in the along-track direction are appear to be unstable. To analyze the distortion quantitatively, the deviation coefficients of the projection trajectories are calculated respectively. The results are shown in Table 2, Table 3 and  Table 4. "Curve n" represents the projection trajectory. "Cross DC" represents the deviation coefficient of the projection trajectory projected from the cross-track horizontal line. "Along DC" represents the deviation coefficient of the projection trajectory projected from the along-track horizontal line. "Diagonal DC" represents the deviation coefficient of the projection trajectory projected from the diagonal line.
For the JL1-2 image, the distortions in the cross-track direction are greater than the distortions in the along-track direction as presented in Table 2. This may be due to that the number of pixels in the cross-track direction (12000) is greater than the number of pixels in the along-track direction (5000). The longer the array, the more likely it is to be distorted. For the GF2-3 image, the distortions in the cross-track direction are greater with the distortions in the along-track direction and the distortions in the diagonal direction are the worst as presented in Table 3. For the WV3-19 image, the distortions in the crosstrack direction are close to the distortions in the along-track direction and the distortions in the diagonal direction are the worst as presented in Table 4.
The experiments in this section show that the method presented in this paper can be used to analyze the geometric distortion of satellite image effectively. The geometric distortion of a satellite image in different directions can be analyzed by fitting the corresponding projection trajectories into straight lines, and the bending degree of the projection trajectories can be quantitatively analyzed by calculating the corresponding deviation coefficients.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)

Quality Evaluation of the Datasets
To evaluate the geometric quality of the datasets, we analyzed all the images in the datasets using the proposed method. We calculated the maximum deviation coefficients of the three types of projection trajectories in each image. The results are shown in Table 5, Table 6 and Table 7  The results show that the geometric quality of the image obtained by the same satellite is uneven. The maximum deviation coefficient between different images may differ by several times. By selecting the images with small maximum deviation coefficients for 3D reconstruction, the quality of the reconstructed 3D models can be improved.
Comparing the results of the three datasets, it can be found that the distortion in JL1 dataset is the smallest and the image quality is the best. This may be benefit from the better imaging stability of the frame camera. For the push-broom satellite images, the overall distortion of the WV3 dataset is smaller than that of the GF2 dataset, and the corresponding image quality is better. This is consistent with the results in the previous section.
The experiments in this section demonstrate that the proposed method can be used to evaluate the geometric quality of images effectively, and this can be used in image selecting in 3D reconstruction using multi-view satellite images.

CONCLUTIONS
In this paper, we proposed a method to evaluate the geometric quality of images in multi-view satellite image datasets. By calculating the deviation coefficients of the projection trajectories projected from the ground grid, the geometric distortion of the image can be evaluated quantitatively. The experimental results of three datasets show that the evaluation method proposed in this paper can be used to evaluate the quality of the satellite image effectively. By evaluating the geometric quality of the image in the multi-view satellite image dataset, we can select the best images for 3D reconstruction to obtain high-quality 3D models.
By evaluating the geometric quality of images obtained by the different satellites, the proposed method can also be used to evaluate the imaging performance of different satellites. Since the distortion of an image with incidence angle of zero degree is minimal, we can also evaluate the imaging performance of a satellite by evaluating the geometric quality of a near nadir image.