IMAGE MOTION COMPENSATION – THE VEXCEL APPROACH

Motion compensation in general and forward motion compensation in particular was an important milestone in aerial imaging when presented for film-based camera systems in the late 90 of the last century. It focused on the forward motion compensation to enhance the image quality when flight speed and image scale produce such motion blur even at short exposure time. Another development and milestone in aerial photogrammetry, the active mount, contributed as well to reduce motion blur. When digital aerial cameras replaced the film-based camera systems in the first decade of the 21 century, forward motion compensation (FMC) could be implemented as an electronic feature of the CCD sensors, namely the time delayed integration (TDI) feature, which worked fine and did not require a mechanical component. Not all cameras could make use of that but large format frame cameras like DMC and UltraCam were able to compensate forward motion blur exploiting this feature of the electronic sensor component. Since CMOS sensors were replacing the CCD sensor component of digital aerial cameras there was a need to implement the FMC mechanism by another solution. One approach was based on a mechanical device able to move the sensor along the flight path of the aircraft like it was the approach for film cameras. At that time Vexcel Imaging decided to develop a more versatile solution based on software and without any additional mechanical part in the camera body. This solution was designed to not only compensate for a uniform compensation to the forward motion but also for angular motion blur and for different scales in one and the same image. This is especially important for the oblique viewing direction of a camera when foreground and background of an oblique scene show different scales in one and the same image. The need to compensate for motion blur is evident when large scale aerial imaging is required, and best image quality is expected. Motion blur is caused from the speed of the aircraft over ground, the image scale, and an angular component – the angular motion blur caused from turbulences if they exist. The magnitude of the forward motion blur can be estimated when multiplying aircraft speed and image scale and exposure time (e.g. speed over ground 75 m/sec, scale 1/10000 and exposure time 0,001 seconds leads to 0,0075 mm or 7,5 μm in the image). Different image scales result in different magnitude of motion blur. This is evident for oblique camera systems. 1. IMAGE MOTION BLUR 1.1 Magnitude of motion blur Motion blur has been a well-known effect in photography and the desire to reduce it exists if sharp images are required. In the field of aerial photogrammetry this is always the case. Thus, several mechanical and electronical solutions to reduce motion blur have been developed for aerial cameras and are widely used. However, the focus was on the removal of forward motion blur, which exists if an airborne platform serves as the carrier of the camera and moves at a specific speed over the object on the ground. The magnitude of the motion blur caused from a translatory movement can be calculated by known image scale, exposure time value (Tv) and aircraft speed over ground (SoG). At a scale of 1:10000 and a speed of 75 m/sec the magnitude of the motion blur is 7,5 μm * Tv. With an exposure time of 2 milli-seconds the magnitude of the blur is 15 μm and therefore about 4 Pixels on a 4 μm pixel sensor. The image looks blurry and accurate photogrammetric processing is impossible. The method to overcome this motion blur is well known as the Forward Motion Compensation (FMC). It works fine for an explicit average image scale but cannot handle different scales as they exist in oblique images or nadir images as well. No improvement of FMC can be expected if the camera is gyrated, and angular motion blur is effective. Even if the camera is operated from an active mount and such angular motion is detected by gyros or a precise IMU, some amount of blur still remains when larger turbulences exist. The magnitude of angular motion blur depends on the amount of angular motion during the exposure and the position in the image. Thus, angular motion blur is inhomogeneous and needs to be handled as such. The example of the angular motion blur caused by a roll angle of 5 deg/sec and at a 100 mm lens results into a cross track displacement of 17 μm in the image center and 26 μm at the left and right image edge during the expose time of 2 milli-seconds. In the case of an angular motion perpendicular to the vertical (yaw or kappa) the motion blur has even different orientations when comparing different locations of an image. It is obvious, that forward motion and additional angular motion implies a specific pattern of motion blur. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2022 XXIV ISPRS Congress (2022 edition), 6–11 June 2022, Nice, France This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLIII-B1-2022-333-2022 | © Author(s) 2022. CC BY 4.0 License. 333 1.2 Implication of varying magnitude of motion blur From basic photogrammetric knowledge and given parameters of the photo mission one can easily understand that a global solution for motion blur does not exist and therefore forward motion compensation is not enough to compensate for all camera movements and the effects of motion, the motion blur. The specific pattern of the magnitude of motion blur can be derived from known parameters of the camera movement and the path of the platform which is used for the photo mission. Knowledge about the object by means of a digital surface model is needed as well. This information allows to design a software-based removal of any motion blur in the image, may it be nadir or oblique and may the motion be translatory or rotatory. As the varying magnitude of motion blur for one shot position is a specific characteristic of oblique aerial camera systems, we show examples from the UltraCam Osprey 4.1 camera product of Vexcel Imaging. The camera is equipped with CMOS detector arrays and consists of the mapping grade nadir component with 20,5 k Pixel cross track and 14,0 k pixel along track and four oblique camera heads equipped with 150 MPixel detector arrays at an image format of 14,2 k by 10,6 k Pixels. The pixel size of all detector arrays of the UltraCam Osprey 4.1 is 3,76 μm by 3,76 μm. Figure 1a: UltraCam Osprey 4.1, nadir and oblique camera system by Vexcel Imaging GmbH, Graz, Austria. The camera offers a 280 MPix mapping grade nadir component and four 150 MPix oblique camera heads. Figure 1b: Footprint of UltraCam Osprey 4.1. The oblique camera heads are inclined by 45°. 2. MOTION COMPENSATION 2.1 AMC – Adaptive motion compensation The need to compensate for motion blur is evident when large scale aerial imaging is required, and best image quality is expected. Motion blur is caused from the speed of the aircraft over ground, the image scale and the angular component – the angular motion blur caused from turbulences if they exist. Different image scales result in different magnitude of motion blur. This is evident for oblique camera systems. Figure 1b illustrates the situation at an UltraCam Osprey 4.1 flying at a speed of 100 m/sec and at an altitude of about 1065 m above ground level. Figure 2a: Magnitude of Image Motion and corresponding motion blur at different image positions of nadir and oblique images: H_agl: 1065 m, GSD: 5 cm, SOG 50 m/sec, Tv = 2 msec, Image motion Nadir 7,5 μm eq. 2,0 Pixel, Oblique near 10,3 μm eq. 2,76 Pixel, Oblique far 5,9 μm eq. 1,58 Pixel The magnitude of the angular motion blur depends on the situation in the air, turbulences may happen and cause larger movements of the camera, which could not be compensated by an active mount. Taking a roll angular rate in consideration, this causes a motion blur perpendicular to the flight line. The magnitude is angular rate multiplied by focal distance and exposure time (e.g. tan 5° per second multiplied by 80 mm focal length and by 2 msec exposure time leads to 14 μm image blur). The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2022 XXIV ISPRS Congress (2022 edition), 6–11 June 2022, Nice, France This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLIII-B1-2022-333-2022 | © Author(s) 2022. CC BY 4.0 License. 334 An example of strong angular motion blur is well visible in Figure 3a and Figure 3b. Figure 2b: Magnitude of Image Motion: Angular motion blur (Yaw) at different image positions Figure 3a: Part of large-scale oblique BWD image (# 19103) of a flight mission (London, GSD 3,5 cm). The image has been processed with adaptive motion compensation (AMC). Figure 3b: Street details of one large scale oblique image (cf. Figure 3 a). Strong angular motion blur at about 11 pixel exists in the raw image data (cf. left image) and this was successfully compensated by AMC (cf. right image) Figure 3c: Angular motion of the camera registered from the IMU (# 19103, London, GSD: 3,5 cm). d_omega = -5,5 °/sec, d_phi = 8,2 °/sec, d_kappa = 0,06 °/sec. Read-out interval is 5 msec, the exposure corresponds to position 11. 2.2 Mathematical Background The mathematical background of the adaptive motion compensation (AMC) technology is the so-called deconvolution or inverse convolution of an image. Knowing the point spread function (PSF) a good approximation of the deconvolution kernel can be estimated. The blurred image b that the sensor records is given as where models the effect of the optical system on the motion blur, including the (mechanical) shutter, is the latent image that needs to be reconstructed, and denote the start and end of exposure, respectively, and where H is the transformation operator (homography) that is used to describe the motion trajectory of each pixel within the latent image . Additionally, the sensor records random white noise, denoted as in (1). Proper numerical integration and operator discretization of (1) leads to a system of linear equations of the form The operator A basically models the point spread function (PSF) of each pixel of the discrete latent image L. If the PSF is assumed to be constant within given image patches, e.g., after suitable domain decomposition, the linear operation in (2) simplifies to


Magnitude of motion blur
Motion blur has been a well-known effect in photography and the desire to reduce it exists if sharp images are required. In the field of aerial photogrammetry this is always the case. Thus, several mechanical and electronical solutions to reduce motion blur have been developed for aerial cameras and are widely used. However, the focus was on the removal of forward motion blur, which exists if an airborne platform serves as the carrier of the camera and moves at a specific speed over the object on the ground. The magnitude of the motion blur caused from a translatory movement can be calculated by known image scale, exposure time value (Tv) and aircraft speed over ground (SoG). At a scale of 1:10000 and a speed of 75 m/sec the magnitude of the motion blur is 7,5 µm * Tv. With an exposure time of 2 milli-seconds the magnitude of the blur is 15 µm and therefore about 4 Pixels on a 4 µm pixel sensor. The image looks blurry and accurate photogrammetric processing is impossible.
The method to overcome this motion blur is well known as the Forward Motion Compensation (FMC). It works fine for an explicit average image scale but cannot handle different scales as they exist in oblique images or nadir images as well.
No improvement of FMC can be expected if the camera is gyrated, and angular motion blur is effective. Even if the camera is operated from an active mount and such angular motion is detected by gyros or a precise IMU, some amount of blur still remains when larger turbulences exist. The magnitude of angular motion blur depends on the amount of angular motion during the exposure and the position in the image. Thus, angular motion blur is inhomogeneous and needs to be handled as such.
The example of the angular motion blur caused by a roll angle of 5 deg/sec and at a 100 mm lens results into a cross track displacement of 17 µm in the image center and 26 µm at the left and right image edge during the expose time of 2 milli-seconds. In the case of an angular motion perpendicular to the vertical (yaw or kappa) the motion blur has even different orientations when comparing different locations of an image. It is obvious, that forward motion and additional angular motion implies a specific pattern of motion blur.

Implication of varying magnitude of motion blur
From basic photogrammetric knowledge and given parameters of the photo mission one can easily understand that a global solution for motion blur does not exist and therefore forward motion compensation is not enough to compensate for all camera movements and the effects of motion, the motion blur.
The specific pattern of the magnitude of motion blur can be derived from known parameters of the camera movement and the path of the platform which is used for the photo mission. Knowledge about the object by means of a digital surface model is needed as well.
This information allows to design a software-based removal of any motion blur in the image, may it be nadir or oblique and may the motion be translatory or rotatory.
As the varying magnitude of motion blur for one shot position is a specific characteristic of oblique aerial camera systems, we show examples from the UltraCam Osprey 4.1 camera product of Vexcel Imaging. The camera is equipped with CMOS detector arrays and consists of the mapping grade nadir component with 20,5 k Pixel cross track and 14,0 k pixel along track and four oblique camera heads equipped with 150 MPixel detector arrays at an image format of 14,2 k by 10,6 k Pixels. The pixel size of all detector arrays of the UltraCam Osprey 4.1 is 3,76 µm by 3,76 µm.

AMC -Adaptive motion compensation
The need to compensate for motion blur is evident when large scale aerial imaging is required, and best image quality is expected. Motion blur is caused from the speed of the aircraft over ground, the image scale and the angular componentthe angular motion blur -caused from turbulences if they exist.
Different image scales result in different magnitude of motion blur. This is evident for oblique camera systems. Figure 1b illustrates the situation at an UltraCam Osprey 4.1 flying at a speed of 100 m/sec and at an altitude of about 1065 m above ground level.

Figure 2a:
Magnitude of Image Motion and corresponding motion blur at different image positions of nadir and oblique images: H_agl: 1065 m, GSD: 5 cm, SOG 50 m/sec, Tv = 2 msec, Image motion Nadir 7,5 µm eq. 2,0 Pixel, Oblique near 10,3 µm eq. 2,76 Pixel, Oblique far 5,9 µm eq. 1,58 Pixel The magnitude of the angular motion blur depends on the situation in the air, turbulences may happen and cause larger movements of the camera, which could not be compensated by an active mount. Taking a roll angular rate in consideration, this causes a motion blur perpendicular to the flight line. The magnitude is angular rate multiplied by focal distance and exposure time (e.g. tan 5° per second multiplied by 80 mm focal length and by 2 msec exposure time leads to 14 µm image blur).   . d_omega = -5,5 °/sec, d_phi = 8,2 °/sec, d_kappa = 0,06 °/sec. Read-out interval is 5 msec, the exposure corresponds to position 11.

Mathematical Background
The mathematical background of the adaptive motion compensation (AMC) technology is the so-called deconvolution or inverse convolution of an image. Knowing the point spread function (PSF) a good approximation of the deconvolution kernel can be estimated.
The blurred image b that the sensor records is given as where models the effect of the optical system on the motion blur, including the (mechanical) shutter, is the latent image that needs to be reconstructed, and denote the start and end of exposure, respectively, and where H is the transformation operator (homography) that is used to describe the motion trajectory of each pixel within the latent image . Additionally, the sensor records random white noise, denoted as in (1). Proper numerical integration and operator discretization of (1) leads to a system of linear equations of the form The operator A basically models the point spread function (PSF) of each pixel of the discrete latent image L. If the PSF is assumed to be constant within given image patches, e.g., after suitable domain decomposition, the linear operation in (2) simplifies to L + i.e., the application of the spatially variant operator A can be replaced by a convolution with a motion blur kernel a that describes a constant Point Spread Function. Several mathematical solvers for (2) or (3) are well established, e.g., one can apply Richardson-Lucy algorithms or optimization techniques which are based on Total-Variation regularization in order to find a solution of the given discrete problems.
The illustration below shows an example. The unblurred image upper left was blurred by a simple motion blur kernel at an angle of 20° from the x-axes (cf. Fig 4, upper right). The lower left image is the result from deblurring at the proper angle of the motion blur and the image on lower right shows the poor result, using a wrong motion blur direction. The example shows that proper settings for the deconvolution is critical and need to be tuned to the optimum to receive good results. The basic solution which is implemented in the AMC procedure does follow this concept. The individual point spread function of each image position is estimated from known angular and translative movements of the camera by making use of IMU and GNSS observations. Specific effects caused from the camera system itself are known from the camera calibration.

Benefits of the Adaptive Motion Compensation (AMC)
One major benefit is the ability of the concept to handle any kind of motion blur, may it be forward motion along the flight path or any other direction, such as an angular movement of the camera. Another benefit of the AMC solution is its adaptability to the varying magnitude of the blur in the image, depending on the different object scale or image position. This is obvious when oblique images are processed. Foreground and background of such images are different in scale by design. If angular motion blur needs to be removed, the effect of the blur depends on the distance of the image position to the rotation axis. And finally, a benefit of AMC is the fact that it is a software-based solution. There are no mechanical parts which may degrade or cause malfunctions.

Availability of Adaptive Motion Compensation (AMC)
The AMC technology was developed for Vexcel's UltraCam Sensors of the 4 th generation and is available through the latest UltraMap Software. The requirements for a proper solution are simple and do not differ from standard modern flight environment. That means the camera management must offer GNSS and IMU recordings to estimate the Point Spread Function and to compute the correct deconvolution kernel.

RESULTS FROM AERIAL MISSIONS
In this section we show results from flight missions at large scale and illustrate how AMC is able to recover blurred image information.

Heavily blurred Images
A large-scale flight mission by UltraCam Osprey 4.1 at a ground sampling distance of 3.5 cm at the nadir image was used to illustrate the excellent performance of AMC. The nadir image (cf. Figure 5) shows some motion blur due to suboptimal flight conditions. Small structures like tiles on the sidewalk and road markings look unsharp. After applying Adaptive Motion Compensation (AMC) the image looks sharp and fine structures are visible. Another example of an oblique image illustrates how effective AMC works against angular motion compensation. The figure below shows building facades, which are blurred due to angular movement of the camera when the image was taken (left side of Figure 6) and the reconstructed image after applying AMC (right side of Figure 6).

Evaluation of a photogrammetric flight mission
We analyze the effect of Adaptive Motion Compensation (AMC) to a photogrammetric flight mission and compare results of the Aero-Triangulation and the solution of the Bundle Adjustment. This experiment shows how the quality of tiepoint-matching, and the overall quality of the bundle adjustment can be improved by AMC (cf. Figure 7 and Table 1).

Stability of intrinsic parameters
The Lvl0 data of the flight mission were used to compute a complete aero-triangulation and results were derived from one set with AMC enabled and one set without applying AMC. The results were used to analyze the overall quality of the adjustment by sigma_o and the stability of the camera parameters (Principal Point Position and Principal Distance).
The entire block of images contains 304 shot positions. Each shot position includes the mapping grade nadir image and four oblique images. The nadir serves as the photogrammetric backbone of the block and thus it makes sense to analyze results from the aero-triangulation of the nadir image.  Table 1: Result of the flight mission without AMC and with AMC applied during the processing. The overall geometric quality is illustrated by sigma_o and improved when applying AMC. The camera parameters were adjusted as well and remained almost unchanged. The maximum difference was detected for PPA_x at a magnitude of 1 µm.

Results from Exterior Orientation Parameters
The result of the aerial-triangulation by means of quality parameters of the least squares bundle adjustment were analyzed for three processing scenarios. One and the same set of raw image data (the so-called Lvl0 data) was processed 3 times. For one set no FMC or AMC was enabled for the image processing, the other set contains forward motion compensation only and the final set was processed enabling AMC. It is also noteworthy to mention, that FMC is included in the AMC processing.    Figure 10) show the improvement of the AMC technology. The maximum difference of 3,3 (1/1000 deg) at omega Max Error corresponds to 1,9 Pixel in the oblique image.
The evaluation of the results from this dataset focuses on overall geometric quality (represented by sigma_o values) and the result of the IMU residuals. Table 2 shows how pose results are improved by correct motion compensation. The quality measures from the roll angle omega are improved by 16% (no correction vs. AMC) and 5% (FMC vs. AMC).
Differences in RMS Residuals in the lateral displacement are small in x and y (below 1%) and slightly larger in the vertical direction 5% larger for FMC compared to AMC and 17% larger when no compensation was applied compared to AMC.

CONCLUSIONS
Excellent image quality is highly desired in photogrammetry. Not only the visual appearance but also the ability to support an accurate photogrammetric workflow is evident. One important attribute of high-quality images is sharpness. This needs to be supported to enable a first-class photogrammetric production. The software based Adaptive Motion Compensation was developed by Vexcel Imaging to improve the quality of UltraCam images and to handle both, forward motion, and angular motion. Different scales in the image are considered as well. This is significant in the case of oblique images. We show the benefit of this new method and give examples from large-scale flight missions. Results from aero-triangulation and bundle adjustment experiments show the improvement of image measurement quality. the stability of the principal camera parameters and the improvement of the EO solution.