LINE OF SIGHT AUTO-CALIBRATION FOR CO3D ROLLING SHUTTER MATRIX DETECTOR

The accurate calibration of the line of sight (LOS) is very important for a high-resolution imaging satellite, especially in a system as Co3D, whose mission is to produce 3D elevation models. A calibration error immediately translates into an altimetry error. Co3D has a matrix and the reading of the lines is done in rolling shutter, each line is read at a different date. So, the geometry of the image is affected by the dynamic perturbations of the attitude during the acquisition time. The objective of the study is to propose a calibration method accurate to 0.1 pixel at 90 percentile. The images needed for calibration must only require one or two satellite passes over the same site, and be compatible with the pointing agility. The method must be robust to residual errors in attitude knowledge. The assumptions of the simulations have been modified in this article to keep the performance of Co3D confidential. * Corresponding author


INTRODUCTION
The general principle of the method is to estimate the LOS parameters and the attitude correction function of the satellite by a least squares method by minimizing the differences between the theoretical measurements and real measurements obtained by correlation between images. LOS are modeled by a bi polynomial function, and we assume that the attitude restitution of the satellite has errors. If the problem is poorly constrained, during the system resolution, LOS and attitude correction parameters can be correlated. In such a case the system converges, but the calibration will be wrong because the LOS model may have been mixed with attitude errors. The aim is to achieve self-calibration, with only the images acquired by the satellite being used to estimate viewing directions. It will nevertheless be possible to use less precise reference images to calibrate a global bias. In order to reduce the time needed for the calibration campaign, the method should not require more than two satellite passes. It is necessary to take into account the agility of the satellite, i.e. the speed of rotation to change the roll, pitch and yaw angles between two acquisitions. The choice of geometric configurations is a compromise between the number of images and the diversity of pointing.

Simulation diagram
The study was conducted by image simulation in order to fully control the geometry of image acquisition and the errors injected into the geometric model. It is composed of the following steps.

Measurements
The measurements are composed of Ground control points (GCP) and Tie points (TP). Operationally these measurements will be obtained by image correlation, in the study they are simulated from the true model.

Ground control points
A GCP is a point in the image whose geolocation is known. They are obtained by correlating the Co3D satellite images with reference georeferenced images. For a given site, we estimate to between 10 and 20 GCPs with a precision of 2m (altimetric and planimetric). These points are not numerous and not precise, but they are useful in the resolution to calibrate an absolute bias of the satellite attitude.
For each GCP, the measurement is composed of 5 values: the geolocation (lat, lon, alt) and the position in the image (raw, col)

Tie points
A Tie point is a detail on the ground that is seen in several images, we do not necessarily know the geolocation of the point, but we know its coordinates (row, column) in each image. TPs are obtained by correlating Co3D images with each other. We estimate to obtain 5000 points, with a correlation accuracy of 0.1 Pixel These numerous and precise points will participate, during resolution, to calibrate LOS and time-dependent relative attitudinal errors. TPs are in the form of N-uplets, i.e. the same ground point will be seen by N images. The following graph shows, for the 18acquisitions configuration, the size of the N-uplets depending on images intersections.

Geometric configuration
This step is very important, it represents the largest part of the work of the study. It consists of determining the geometry of the acquisitions (the number, the footprint and their orientation) which will allow the resolution system to de-correlate estimated parameters. In addition, operational constraints must be respected, by minimizing the number of satellite passes, so that the calibration campaign does not take too long, and take into account the satellite's guidance capacity. Three configurations are detailed bellow, to show the impact on calibration performance.
TPs measurements do not allow direct observation of the satellite attitude and LOS. We only have access to differential measurements.
For a configuration where 2 images are shifted in row and column without rotation (without yaw) All TPs will have the same pixel offset. The date difference of the pixels will also be constant, we will only observe the effect of the attitude (AttSat1(t) -AttSat2(t+dt)) with a constant dt. The observation context gives more information to the system because the pixel offset and the pixel dating will not be constant. This information helps to remove the correlation between attitude and LOS

True model and known model
The true model corresponds to the reality of the satellite state: true LOS, true attitude, true DEM (digital elevation model). The known model corresponds to the knowledge we have of the satellite state. It therefore contains errors compared to the true model.

Line of sight
The LOS error is modelled by a bi polynomial function of order 3 (2*16 parameters) The following graph illustrates the shape of the error induced by each polynomial coefficient.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France For the simulations, the total distortion error is around 10 pixels, below is an example of distortion error for a test case.

Satellite attitude
As the images are read with a rolling shutter, each line of the image is acquired at a different date.
If the attitude law of the satellite is not well known, an error will be made in the interpretation of the measurements. The known attitude error, for each acquisition, is modelled by a low frequency on roll and yaw whose amplitude is about 0.2 pixels. When solving the problem, the attitude error is estimated by a 12-node piecewise spline function. As the yaw error has little impact on the measurements, only roll and pitch will be estimated. For each acquisition, there are therefore 24 attitude parameters to solve

DEM
As the satellite moves, the images are acquired at different angles. An error in the knowledge of the DEM will cause an error in the measurement of the tie points. The error of the known DEM compared to the true DEM is modelled by a 2dimensional high frequency function with an amplitude of +/-30 m.

Resolution method
The method is based on a least squares resolution using the levenberg marquardt algorithm. System parameters are the coefficients of the bi-polynomial LOS correction function and the parameters of the spline attitude correction functions for each image. The number of parameters to be estimated is therefore 32 + 24 * nbImages. The measurements are given by the GCPs and the TPs. They represent the difference between the measurement given by the known model and the true model. For a GCP, if there is a model error, the geolocation of a pixel will not match the correct geolocation. For a TP, if there is a model error, the geolocation of the pixels in each image will be different when by definition it should be the same.

System result analysis
When the system converges, it will always give a solution for the system parameters. But there is no guarantee that this solution is the correct one. In our case it may not give the true viewing directions. In a simulation-based case, it is easy to check whether the resolution has worked, just check that the model corrections (LOS and attitude) found by the system match the errors injected into the simulation.
In a real calibration case, we cannot use this validation method, because we do not know the real LOS and the real attitude. But we can get information by analysing the residuals of the measurements and the covariance matrix of the system after resolution.

TPs and GCPs residuals
When plotting the TP residuals with respect to rows or columns of the image, one should see only white noise. If a signal appears, this can have several causes: • The correction model is not adapted to the real error, e.g. the attitude error is higher frequency and cannot be corrected with a 12-node spline function.

•
The system has found a local minimum and not the absolute minimum. The following graphs show the measurement error residuals of the tie points, the row errors are in red and the column errors are in blue. Each point on the graph corresponds to a TP, the X axis represents the TP line in one of the images

Covariance Matrix
With the covariance matrix of the system parameters, one can calculate the correlation matrix of the parameters as well as the norm of the covariance of each parameter. The correlation matrix can show us whether the LOS parameters are correlated with the attitude parameters. If the parameters are correlated, it means that the system is not constrained enough. An attitude error can have the same effect on the measurements as LOS error. Even if there is no signal in the measurement residuals, the viewing directions given by the system resolution will be false because they contain satellite attitude errors. In such a case, the solution is to add images with a different geometric configuration, which will provide additional constraints that will allow the system to remove correlations.

RESULTS
In order to obtain meaningful results, a Monte Carlo of 200 cases was performed for each configuration. For each case, the attitude error and LOS error are different, as well as the distribution of TPs and CGPs.

Satellite attitude correction
Even if the final objective is not to correct the attitude of each acquisition, we check if the system was able to correct the attitude. The following graphs show only for one test case and for one image (it is not a statistic) the attitude corrections for the roll and pitch axes.

Line of sight correction
The graphs below represent the statistical error (CE 90) of the norm of the difference between the true and the refined LOS, in the field of view (FOV). It can be observed that in all cases the los is better estimated at the centre of the FOV than at the edge. This is due to the fact that there are more measurements and constraints at the centre than at the edges. The bi-polynomial function is therefore better estimated at the centre

5-acquisitions configuration
It requires only one satellite pass over the same site. One central acquisition, four acquisitions shifted of 1/3 of the size of the of the central image footprint (roll, pitch). Each image has a rotation (yaw) of 1.5 degrees with respect to the previous acquisition. Figure 13: 5-aquistitions configuration Figure 14: 5-aqu attitude correction The system failed to correctly estimate attitude errors, and even less so for pitch errors. Note, this acquisition corresponds to the bottom image, there is no TPs in the first third of the image. It is therefore not possible to correct the attitude of the beginning of the acquisition. The residual LOS errors are smaller than the initial knowledge errors, but they are much stronger than our specifications. This configuration does not constrain the system sufficiently for LOS calibration

9-acquisitions configuration
It requires only one satellite pass over the same site. The 5 acquisitions of the previous configuration and 4 acquisitions shifted by 1/4 of the size of the central image footprint, each image has a rotation (yaw) of 1.5 degrees with respect to the previous acquisition The attitude is slightly better estimated than in the previous configuration. It can be seen that the attitude is less well corrected on the pitch axis. This is due to the fact that the 9 images are acquired on the same orbit, the epipolar axis of each image pair is orthogonal to the pitch axis. Pitch errors are therefore less observable as they are correlated with DEM errors. The LOS estimation errors are lower than in the previous configuration, but still higher than the specification

18-acquisitions configuration
It requires two satellite passes over the same site. The first pass is identical to the 9-acquisitions configuration, the second pass is also identical with a rotation (yaw) of 90 degrees.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France In this two-orbit configuration and with a 90-degree rotation for nine shots, the satellite's attitude is very well corrected, both on the roll and pitch axis. The LOS estimation errors are consistent with the specification

CONCLUSION
The study has shown there are geometric acquisition configurations that allow for the removal of correlations between LOS and satellite attitude and allows for compliance with the calibration specifications The method will be used for co3D LOS when it is in flight.
The study could be continued to find a new configuration that would allow the calibration to be performed in a single satellite pass, thus reducing the duration of operations. It will also be possible to find a way to do without GCPs, so that reference images do not have to be used.
The method is adaptable to other satellites with a matrix detector, although the geometric configuration will have to be adapted to the agility of the platform. As the method works for a rolling shutter matrix, it should be tested whether it can also be used for push broom type sensors.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2022