A NOVEL TECHNIQUE FOR PRECISION GEOMETRIC CORRECTION OF JITTER DISTORTION FOR THE EUROPA IMAGING SYSTEM AND OTHER ROLLING-SHUTTER CAMERAS

Abstract. We use simulated images to demonstrate a novel technique for mitigating geometric distortions caused by platform motion (“jitter”) as two-dimensional image sensors are exposed and read out line by line (“rolling shutter”). The results indicate that the Europa Imaging System (EIS) on NASA’s Europa Clipper can likely meet its scientific goals requiring 0.1-pixel precision. We are therefore adapting the software used to demonstrate and test rolling shutter jitter correction to become part of the standard processing pipeline for EIS. The correction method will also apply to other rolling-shutter cameras, provided they have the operational flexibility to read out selected “check lines” at chosen times during the systematic readout of the frame area.


Summary:
We use simulated images to demonstrate a novel technique for mitigating geometric distortions caused by platform motion ("jitter") as two dimensional image sensors are exposed and read out line by line ("rolling shutter").The results indicate that the Europa Imaging System (EIS) on NASA's Europa Clipper can likely meet its scientific goals requiring 0.1 pixel precision.The method will also apply to other rolling-shutter cameras.
Background: Remote imaging has been a key tool for planetary investigation since the earliest days of the space age, but the technology has evolved greatly.Each new type of sensor (film, vidicon, charge-coupled device or CCD, and active pixel sensor or APS) has conferred improved capabilities but created new processing challenges.
Our study is part of the design effort for EIS [1], which consists of two cameras with different optics but identical APS detectors and readout electronics: the Wide Angle Camera (WAC) with 48°x24° field of view (FOV) and the Narrow Angle Camera (NAC) with 2.3° x 1.2° FOV.Selection of an APS was motivated in part by its ability to read out individual detector lines in any order.This allows the EIS cameras to operate as pushbroom sensors.The WAC has miulti-line stereo and color capabilities, similar to the High Resolution Stereo Camera on Mars Express [2].The NAC has color but its FOV is too small for useful pushbroom stereo.Geometric distortion of pushbroom-mode images will be corrected by the same approach [3] used for the Mars Reconnaissance Orbiter High Resolution Imaging System (HiRISE) [4] by using overlapping lines at different downtrack positions.Matching of features in the overlap areas yields time differences of the camera pointing, from which the absolute pointing history (apart from an overall pointing bias) can be modeled.The main difference is that whereas the intervals between lines are set by the focal plane design for HiRISE, for EIS they can be chosen anywhere on the APS chip.
The EIS cameras' large unfiltered area (1536 lines by 4032 samples) can be used to obtain two-dimensional "frame" images.Unfortunately, these lines must be read out sequentially over a period of 26 ms, not simultaneously, so these images are also affected by jitter.The specification for pointing stability of Clipper is ±25 µRad/s, leading to an estimate of 4.5-15 µRad (a negligible fraction of a WAC pixel but 0.5-1.5 NAC pixels) during readout.The detailed spectrum of pointing variations is currently unknown but could extend up to 700 Hz, so distortions could range from a slow drift to ~18 cycles across the image in any combination.Pixel-level distortion is acceptable for geologic studies but would interfere with geodetic and limb-topography observations with the NAC that require ~0.1 pixel precision.We therefore undertook to develop and demonstrate a technique for modeling and correcting rolling-shutter jitter distortion.If such correction cannot be done, the precision requirements for geodesy and limbs would have to be relaxed.A redesign of the electronics to permit true frame imaging has been judged prohibitively costly in terms of mass and complexity.
Approach: Our approach makes use of the flexibility of APS readout.The systematic (line by line) readout of the frame area is periodically interrupted to read some detector rows ("check locations") one or more additional times, resulting in "check lines."These are matched to the corresponding locations in the systematic readout and the resulting time-differences of pointing are analyzed to produce a model of the pointing history (as in the pushbroom case, the absolute pointing bias cannot be determined from the differences).Because the expected motions are small, it is convenient to model them as translations in the image plane rather than as rotations of the camera.
The location and timing of the check lines has a strong impact on how the jitter is modeled.For example, "shadowing" the systematic readout (reading some lines a second time shortly after each is read systematically) would produce a data set similar to that used for pushbroom mode, and a similar modeling approach could be used.If a single check location is used (not a disadvantage, since at any given instant all detector lines move together) the analysis is particularly simple because its pointing differences relative to the systematic read-out time are identical to the pointing history (apart from a constant offset).Although reading a small number of check locations makes the analysis slightly more complex (their delta-pointing histories must be adjusted to create a single pointing history), it has several advantages.First, each check location is read out less often, resulting in longer exposures and higher signal to noise ratio (SNR).Second, if the image texture at a single line is highly directional, the estimates of sample and line components of the jitter will be correlated.This is likely to be a severe problem for limb images, where the main "feature" is the limb itself, but could also be an issue for non-limb images because of the prevalence of parallel linear features on Europa.Checking several locations can break this correlation of the sample and line estimates.
Proof of Concept: To evaluate the approach outlined above, we implemented software to simulate jittery images by resampling a "truth" image as well as to model and correct the distortions.
Software.Steps in the simulation and processing were implemented as individual applications that are compatible with the USGS ISIS3 system [5].
• Create, from user input, a text file defining the "true" jitter to be applied to the image.An arbitrary number of harmonic terms with arbitrary frequencies and independent amplitudes and phases in sample and line can be specified.
• Create, from user input, a text file defining the schedule for reading out the systematic image lines and check lines, allowing any number of check locations and repeat readings.
• Resample the truth image according to the jitter and schedule definitions, placing the systematically read image in one file and the check lines in another.
• Match each check line to the corresponding neighborhood of detector lines in the systematic image, yielding a list of delta-pointing values in samples and lines for known pairs

2188.pdf 49th Lunar and Planetary Science Conference 2018 (LPI Contrib. No. 2083) of readout times. Normalized cross-correlation (NCC [6]
) is used to identify the best whole pixel alignment of systematic and check imagery, and the correlation is then interpolated in the neighborhood around this offset location to estimate the optimum to a fraction of a pixel.Additional enhancements to this procedure are described below.
• Fit a polynomial model of the pointing history (independently in sample and line directions) to the match data.Such polynomials are first fit separately for each check location, then their biases are adjusted to place them on a single curve and the model is re-estimated using all check lines.
• Resample the distorted systematic image, using the modeled pointing history to place pixels at their undistorted locations, as described by the fitted model.
Test Data.A 210 m/pixel mosaic of Galileo images of Europa was selected as the truth image.This is the same mosaic used in our simulations of the impact of illumination angles on EIS stereo [7].To simulate a limb image, the portion of the mosaic outside a circular arc was set to zero.
Results: We selected three check locations well distributed across the image field, each checked 20 times.The jitter was approximately a full cycle over the duration of the readout, with arbitrary phases and different amplitudes (near but not exactly one pixel) in sample and line.
The 0.5 pixel root-mean-square (RMS) difference between the true and matched displacements in our first tests was disappointing.Matching errors of 0.2 pixel are commonly achieved in stereo mapping [7], and errors for matching two versions of the same image should be even smaller.Matches also appeared to cluster near a few preferred values, suggesting that "pixel locking" could be affecting the results.This is a well-known problem in which area-based subpixel matches tend to be biased toward whole-pixel offsets [8] We therefore implemented two published strategies for mitigating pixel locking.The first simply involves enlarging the images by smooth interpolation (in our case by 4x) so locking occurs at a level inside the original pixels [9].The second averages the standard subpixel match location with one computed after interpolating one of the images to displace it by 0.5 sample and line.The "half-pixel locking" of the second estimate cancels the pixel locking of the first [10].
We eventually traced the poor early results to our failure to search a large enough range of offsets for the optimal displacement, causing the subpixel step to fail.With this problem corrected, even the basic matching strategy achieves the desired precision of ~0.1 pixel (Table 1) but the anti-locking approaches yield even smaller RMS errors and are thus valuable.The larger errors for the limb image may result from the limb mask not being adequately anti-aliased.Note that these are errors between individual matches and the true displacements of the respective lines.Because the jitter model is fitted through many such estimates, its deviation from the true jitter is even smaller.We have yet to explore the behavior of our correction method for higher frequency jitter, but we have performed a simplified calculation that indicates it will likely be successful.In this we defined a jitter function with 9 oscillations during the readout, half the maximum expected on Clipper.Rather than simulating and matching images, we simply took 30 equally spaced samples of the true jitter and added Gaussian noise with a standard deviation of 0.2 pixel.From the noisy data points we were able to fit a model jitter function with only 0.06 pixel RMS error relative to the true one, indicating that the matching precision and number of samples suffice even for high frequency jitter.
Remaining Work: Considerable work remains to incorporate rolling-shutter jitter correction into the EIS uplink and downlink pipelines and test it with real data.Our programs to define the jitter and distort a truth image will not be needed, but the program to schedule check lines will be used and the flight operations team will use our schedule files to build commands to the camera.We are presently combining the line-matching and jitter-fitting programs into a single ISIS3 application that will be used to model the jitter in flight images.The jitter correction step will be incorporated into the camera model for EIS, which will correct jitter distortions immediately before correcting optical distortion in projecting from image to ground space, and add back the distortions immediately after adding optical distortion when projecting from ground to image.The correction will be used "on the fly" in geodetic control of images by bundle adjustment and projection to make image mosaics.It can also be used to project images to the ground and back into a distortion-free "ideal" version of EIS, yielding a distortion free but otherwise geometrically "raw" image that can be used by other software, e.g., for stereo mapping, that does not perform jitter correction on the fly (cf.[11]).
Because the precise nature of jitter on Europa Clipper cannot be predicted, considerable experimentation will be required in the commissioning and early operations phases of the mission.We will collect images with a generous number of check lines and with various onboard sources of vibration turned off singly and in combinations to the extent possible.This will enable us to determine which instruments, etc., need to be inactive during NAC imaging and the minimum number of check lines needed for jitter correction.
Conclusion: Matching a small number of check lines obtained throughout the systematic readout provides a viable basis for correcting geometric distortions caused by jitter.The achieved accuracy meets the requirement for EIS geodetic and limb topography imaging.We see no reason the technique would not also apply to any rolling-shutter sensor with similar flexibility in scheduling the readout.