OPTICAL BLUR DISTURBS – THE INFLUENCE OF OPTICAL-BLURRED IMAGES IN PHOTOGRAMMTRY

Photogrammetric processes such as camera calibration, feature and target detection and referencing are assumed to strongly depend on the quality of the images that are provided for the process. Consequently, motion and optically blurred images are usually excluded from photogrammetric processes to supress their negative influence. To evaluate how much optical blur is acceptable and how large the influence of optical blur is on photogrammetric procedures a variety of test environments were established. These were based upon previous motion blur research and included test fields for the analysis of camera calibration. For the evaluation, a DSLR camera as well as Lytro Illum light field camera were used. The results show that optical blur has a negative influence on photogrammetric procedures, mostly automatic target detection. With the intervention of an experienced operator and the use of semi-automatic tools, acceptable results can be established.


INTRODUCTION
Photogrammetry and Structure from Motion (SfM) Applications strongly depend on 2D imagery. Usually these images are acquired with digital cameras allowing for digital processing. These 2D images can be then processed using appropriate software to 3D models of objects or used for the localisation of objects or camera platforms (Park et al., 2012;Toyoura et al., 2014;Wu et al., 2011). These calculations are assumed to depend strongly on image quality so that blurred imagery is often manually excluded from the processing to prevent their assumed negative influence.

Blur -influence, detection and correction
The problem of blur in photogrammetric procedures was addressed in previous research, with a focus on motion blurred imagery (Sieberth et al., 2014a(Sieberth et al., , 2014b. The previous research demonstrated that motion blur can influence photogrammetric procedures such as the automatic or semi-automatic detection of coded and un-coded targets, the detection of feature points with subsequent feature matching and the calibration of cameras. However, it was also shown, that direct and manual operator intervention can help mitigate effects and generate acceptable results (Sieberth et al., 2014a). The basis for this research were unmanned aerial vehicles (UAV, UAS) which can contain blurry image sets due to motion of the camera platform, caused either by voluntary flight movements or involuntary platform vibrations by the engines (Prasad et al., 2015). The incidence of such motion-blurred images is reduced by mechanical and electronic gimbals, which stabilise the camera footage. Although blurred images can often be prevented, some degraded images may remain making it necessary to isolate them. There are a variety of algorithms which aim to detect blurred images and some even correct for blur (Alvarez-Gila et al., 2019;Shi et al., 2014). Unfortunately, many methods do not concentrate solely on motion-blur but also optical blur arising from incorrect focus or insufficient depth-of-field. Correct focusing and an appropriate aperture setting can largely prevent this problem. However, especially in close range applications and indoor settings the focus to aperture setting does not allow for a large enough depthof-field. This can cause partially blurred images in areas that still belong to the object of interest, which can then disturb subsequent photogrammetric procedures.

Light Field Cameras
A way to solve the problem of optical blur was developed by Ng et al. (2005), who developed the Lytro light-field camera. Lightfield cameras do not just record light intensity received by the image sensor, like usual DSLRs, but also the directions which light rays travel in space (Adelson and Berger, 1991;Georgiev, 2008). This additional information allows an image to be refocused or even the view point of the camera to be altered after image acquisition, within certain limitations (Ng et al., 2005;Perwass and Wietzke, 2012;Wang et al., 2015). Also a depth map can be estimated, which could be beneficial for photogrammetric applications (Wanner and Goldluecke, 2014;Yu et al., 2013).
To allow for the recording of the light field the Lytro Illum camera (Version 2.0.0(42)) adopted in this study, uses an array of lenticular lenses located in front of the image sensor and in addition to the main camera lens, (Knight et al., 2012;Ng et al., 2005). While the main lens focuses the light on the lenticular micro-lens array, the array further distributes light onto a number of sensors placed under each lens (Ng et al., 2005). The distribution of the light is dependent on the direction, which the light arrived at the camera. However, the accuracy of direction depends on the size of the lenses on the micro lens array as well as the number of pixels available for each micro lens (Jeon et al., 2015).
It has been shown previously that plenoptic cameras can be used in photogrammetric applications (Sieberth et al., 2018;Zeller et al., 2017Zeller et al., , 2014. In the research presented in this paper, images acquired with a Lytro Illum camera will be used to explore the influence of optical blur and manually changing the aperture on normal photogrammetric procedures.

Aim
The aim of this paper is to test whether and how much optical blur influences normal photogrammetric procedures. This will be achieved by comparing a test scene captured using a standard DSLR camera and a Lytro Illum camera. In another test, both camera types were used for camera calibration, to analyse the influence of optical blur on the calibration procedure.

METHOD
To analyse whether and how much photogrammetric procedures are influenced by optical blur it was decided to use two different camera systems, a standard digital single lens reflex camera and a Lytro Illum. Both systems were used in two different scenarios, one close range scene, derive from a staged forensic scenery (Sieberth et al., 2018) (Figure 1 (a)), another for camera calibration (Figure 1 (b)). The camera calibration setup was based on the configuration presented in Sieberth et al. (2014a), which analysed the influence of motion blur upon photogrammetric procedures. The two other tests presented (Sieberth, et al. 2014a), regarding coordinate calculation and target size, were not conducted in time for this conference publication.
(a) (b) Figure 1. (a) The forensic scene representing a real case photogrammetric application. (b) A camera calibration field with Siemens star in the centre and dotted targets on a black pole as focus point in front of the calibration field.

Scene Setup
Forensic scene: The staged forensic scene was set up in an indoor room, containing table and a clothed mannequin (Sieberth et al., 2018). Within the scene, three measurement crosses were distributed evenly within the working volume, each providing two distances of known length between the coded target points.

Camera Calibration Scene:
The camera calibration scene was based upon a white planar wall with nine targets sheets, each with six coded targets and a Siemens star located in the centre. A single column of dot targets was placed at a distance of 0.86m in front of the wall. The camera was located 2.04m from the dot target column and therefore 2.9m from the wall (Figure 2).

Photogrammetric image acquisition
Forensic scene: The Forensic scene was captured using both a DSLR and Lytro camera. The DSLR camera, (Nikon D700) was equipped with a 50mm lens (Nikkor 50mm f/1.8 G). The procedure was carried out using aperture f/4.5 and fixed focus, and then acquiring 58 overlapping close range photos of the scene (Sieberth et al., 2018). Then, the Lytro Illum was set to a focal length of 50 mm and focused once on the scene and then 58 overlapping images were acquired in similar fashion to the DSLR procedure.

Camera Calibration Scene:
For camera calibration, image acquisition involved five positions around the test field. The first included a central position perpendicular to the planar wall, where the DSLR camera (Nikon D750) was set up with a 50mm lens (Nikkor 50mm f/1.8 D). The camera was focused not on the test field itself, but on the dot targets 0.86m in front of the test field. With the smallest aperture set (f/22), the test field was wholly in focus due to the large depth of field. This was evaluated with reference to the central Siemens star, to ensure sharp images. After acquisition of this initial image, the aperture was gradually opened, causing the test field to blur optically, although not the dot targets in front of the field (Figure 3). After reaching the widest aperture and most blurred image, the aperture was again closed and the camera moved to the remaining four positions around the test field. From these four positions, images with three roll rotations were acquired, necessary for subsequent camera calibration.
A similar procedure was adopted for the Lytro camera. However, as the camera has no aperture control, the opening of the aperture could not be performed. The camera set up at the centre position was just used to acquire one image, focused on the dot targets. Then the camera was moved to the remaining four positions around the test field and images with three roll rotations were similarly acquired.
(a) (b) Figure 3. Image detail of DSLR camera calibration setup with part of the Siemens star at the top and a coded target in the bottom right. (a) sharp image (b) only dot targets are in focus, due to small depth of field.

Post-Processing
Forensic scene: The 58 images from the DSLR camera were processed in a software package called Darktable, to create two sets of images. One consisted of the full pixel count of the DSLR, whilst the second consisted of a reduced pixel count equal to the count of the Lytro Illum camera (2450x1634 pixels). The 58 overlapping images of the Lytro camera were processed in the dedicated Lytro software to allow for adjustment of the aperture. The aperture was set to f/16, f/8, f/4, f/2 and f/1. The change in sharpness from f/16 to f/2 was barely visible to the human eye, only the change to f/1 being discernible. The 58 images were exported with each aperture setting, creating five image sets. Subsequently these were imported into Agisoft Photoscan (Agisoft, 2018) along with the DSLR sets. In a first step Agisoft's "Detect Markers"-tool was applied, with the marker type being "coded circular", with a tolerance setting of 50, and a maximum residual setting of 5 pixels.
In another test the f/1, f/2 and f/16 set of the Lytro camera were processed in Agisoft, with the camera alignment set to "high". After successful camera alignment, a survey staff in the scene provided a scale. An Operator then manually measured the targets of the three crosses, repeated for each of the three aperture settings. This whole procedure was then repeated for all 58 images. Distances calculated by Agisoft for those crosses were then compared to the known distances between the measurement crosses.

Camera Calibration Scene:
The post processing of the camera calibration scene also required using the dedicated Lytro Software to create the centre calibration image with aperture settings similar to those adopted for the DSLR. For the remaining four positions, the aperture was set to f/16, the most smallest aperture possible with the Lytro camera. Photomodeller software was used for calibration, as used prior (Sieberth et al., 2014a). Five sharp images of the surrounding camera positions were imported and 54 targets were automatically detected and marked in each image. This was performed twice, for both DSLR and Lytro camera images. Then one centre image was added, coded targets detected, referenced and camera calibration performed. After this the camera calibration parameter were documented, and the centre image replaced by a new centre image, repeating the procedure for all aperture openings.
In detail, the target detection was performed with the "sub-pixel target tool" mode. In a first instance the tool was applied to the whole image. If successful, all 54 coded targets were automatically detected and referenced. With degrading image quality, the detection and referencing became less successful. Some targets would be detected but not referenced, as the code became increasingly unreadable. Here two procedures were applied, at first, the camera calibration calculation was performed using only the fully automatic referenced targets. Then, in a second step the unreferenced targets were manually identified and another camera calibration calculated. With further degradation of image quality the automatic tool applied to the whole image did not only fail with referencing the targets but also with detecting the targets and even rejected some of the detected and referenced targets during the camera calibration calculation. To resolve this challenge, the "sub-pixel target" tool was applied gradually. Initially, it was applied to a group of six targets, and if this failed, the tool was applied singly to the target with code, and separately without code (Figure 4). Ultimately, manual referencing became necessary.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition)

RESULTS
Forensic scene: Unsurprisingly, increased optical blur causes a reduction in the number of automatically detected targets. For the Lytro images set with aperture f/16, the algorithm was able to detect six targets, whilst just two targets were detected in the most blurry image with aperture f/1 ( Figure 5). In comparison, it was possible to detect 22 targets using sharp DSLR image and even with a reduced pixel count Agisoft was still able to detect 17 targets out of 27. The distances measured in the f/16, f/2 and f/1 dataset showed discrepancies of under a millimetre for all datasets (Table 3). Also, the pixel error of the marked targets was below one pixel (Table 4). However, it is apparent that blur affected the Operator's ability to mark targets.

Targets Distance [mm]
Discrepancie [mm] f  Camera Calibration Scene: it was also possible to see that automatic target detection and referencing gradually decreased for both the DLSR and Lytro cameras. While there were no problems with the detection of targets with small apertures, difficulties increase with blur ( Figure 6 (a)). With an aperture opening of f/10, automatic detection and referencing of targets is no longer possible. It is evident that detection works significantly better for the Lytro dataset, where fully automatic detection and referencing was possible up to an aperture of f/8 (Figure 6 (a)).
The results of the camera calibration appear to be similar throughout most of the blurred images. However, this starts to degrade especially in the RMS around an aperture of f/2.5 for the DSLR camera (Figure 8 (a)). For the Lytro, a change can be seen in the width and x coordinate of the principal point around f/2.8 but only for fully automatically processed images. When Operator input was required, results are like those achieved with the calibration of sharp images (Figure 7 (b)).
(a) (b) Figure 6. Calibration Scene: Detection of targets gradually worsens and requires more operator input.

DISCUSSION
These results confirm the common experience that blur disturbs photogrammetric processing procedures. Automation greatly suffers when degraded images are used and can negatively influence the user experience (Park et al., 2012;Toyoura et al., 2014). Automatic target detection, as performed in both scenes, becomes unsuccessful. In the forensic scene even the original sharp Lytro images have difficulty with target detection. This might be due simply to the lower pixel count (4Mpx) than the DSLR images (12Mpx). However, even the reduction of the pixel count of the DSLR images to the same level as the Lytro camera resulted in the detection of significantly more targets. Considering the high contrast scene documented in this test, it is plausible that the poorer contrast and dynamic range of the Lytro camera might influence the ability to detect targets (Figure 9).  Although automatic target detection is not always successful, it remains possible to manually mark targets. Even with significant blur the operator was able to mark targets and successfully process. Manually measured targets even return similar results to sharp images in both the image and object space (Table 3 & 4). To reduce the amount of operator intervention blur resistant markers could be used (Prasad et al., 2015). These are especially in real time applications such as augmented reality scenarios useful (Toyoura et al., 2014). Also in non real time photogrammetric applications they could safe both, time and money. The camera calibration scene confirms these findings. The automatic detection of targets decreases gradually with increasing image blur and significantly increases the Operator intervention required. This is similar to the findings made by Sieberth et al. (2014a). Furthermore, these results confirm previously made findings that direct and manual operator intervention can help mitigate the impact of blur on subsequent calculations. Semi-automatic tools, such as the "sub-pixel target" tool provided by Photomodeller can greatly support this. However, these need to be applied precisely to the target and is clearly time-consuming. The DSLR calibration test shows that blur does not affect calibration initially but then quality decreases gradually. Judging from the image width, principal point position and RMS, this threshold appears at an aperture of f/2.5 for the DSLR. At this point the target detection is so compromised that just 13 targets were detected out of 54, which was insufficient for automatic processing. The targets detected just with the centre dot started to become unreliable and multiple re-detection of the same target often resulted in a variety of different marker positions. With the Lytro dataset the influence of optical blur is weaker than the DSLR. This can be explained by the much lower pixel count and different geometric resolution. The larger pixel area is not strongly influenced by the small changes that the aperture openings cause to the optical light paths. Overall, the results remain similar to the findings of Sieberth et al. (2014a), which analysed the impact of motion blur on photogrammetric procedures. However, in this previous study circular targets there were blurred in a linear direction, allowing a precise centre estimation in the direction orthogonal to the motion. This approach is not possible with targets that are optically blurred. While the work described here aimed to replicate the original tests (Sieberth et al., 2014a) with optical blurred images, there remain some important additional tests to be conducted, which the authors aim to perform in the future. One of these is to analyse the influence of optical blur on the calculation of 3D positions of marked points, to establish whether it is possible to use blurred images with a known camera calibration. An additional question is to identify some form of threshold to quantify the degree of tolerable target blur before successful detection fails.

CONCLUSION
In conclusion, it can be stated unequivocally that photogrammetric procedures are influenced by optical blur. However, it is only automated procedures that appear to be affected and successful processing can be achieved if manual intervention is conducted and provided that sufficient time and fortitude remains available.  Table 4).