ACCURACY INVESTIGATIONS OF IMAGE MATCHING TECHNIQUES BY MEANS OF A TEXTURED DUMBBELL ARTEFACT

: In the last few years, photogrammetric methods for 3D surface reconstruction at close range have increased significantly in importance. On the one hand, this is due to the increased performance of the systems and on the other hand to the improved quality (accuracy, completeness) of the created point clouds. In order to verify the accuracy of various area probing methods, the German VDI guideline 2634 part 2 and 3 is applied. However, the high-precision test reference objects existing so far consist of diffuse textureless surfaces, so that passive methods, like image matching, cannot be compared with active methods (e.g. structured light systems). In order to make this possible, a certified textured dumbbell with an accuracy of better than 10 µm is presented in this paper, with the aim to examine the suitability of the textured dumbbell artefact for close-range photogrammetric 3D surface reconstruction. Furthermore, the accuracy level of a structured light system, Structure from Motion (SfM) and Multi-View Stereo Method (MVS) is verified and compared with each other.


INTRODUCTION
The photogrammetric area based acquisition of free-form surfaces takes a major role in close-range applications. For this purpose, active methods, such as structured light systems, and passive methods, such as multi-view matching or digital image correlation (DIC), can be used. In both areas, developments can be observed that lead to an improved performance of the methods and an increased quality of the reconstructed surfaces. To improve these developments and investigate different methods, different objects or scenes are usually captured. The resulting surfaces can then be compared with the previously acquired reference. The reference should be 5-10 times more accurate than the methods under investigation, so that different methods can be used (Luhmann et al., 2019). Strecha et al. (2008) and Remondino et al. (2014), for example, use a laser scanner to investigate Structure from Motion (SfM) and Multi-View Stereo (MVS) approaches. A well-known benchmark from the computer vision community, for the investigation of the quality and performance of different MVS approaches, is presented by Seitz et al. (2006). Instead of a laser scanner, a structured light system is used to create reference data from real close-range sceneries. Although the orientation data and images are free available for this benchmark, it is difficult to reproduce the used scenarios. Therefore, artificial test objects are used to solve this problem. Kersten et al. (2016), for example, use a so-called Testy, that consists of spherical bodies and planes for the investigation of low-cost structured light systems, which were previously referenced by a high-end structured light system. Wenzel et al. (2013a) also use this method for the investigation of MVS approaches. However, in some applications, like underwater welding tests (Nietiedt et al., 2019a) or in dynamic applications given in wind tunnels (Nietiedt et al., 2019b), it is not possible to acquire reproducible reference data with an accuracy level up to 5 times higher. To overcome that restriction, it is possible to use * Corresponding author simulation or tests according to industrial standards and transfer the results to the application. An accepted guideline for the verification of optical 3D-measuring systems with multiple view area-based scanning is the VDI 2634 part 3 (VDI 2008). It has to be noted that the verification procedures of VDI 2634 enables retraceability to the SI metre unit and characterisation of wellaccepted parameters such as the length measurement error. For this purpose, a dumbbell and plane normal is positioned at least seven different positions in the measuring volume and recorded with the measurement system. From the reconstructed objects, quality parameters can be derived which describe the absolute accuracy level. The application of the guideline is described for example by Eiríksson et al. (2016) and Finke and Bartelt (2010). However, reference objects are used, which are usually characterised by a homogeneous texture that is optimised for structured light. Therefore, the results of matching methods that depend on a natural heterogeneous texture cannot be compared equivalently with the results of passive methods. To solve this problem, an artificial texture can be applied by a projector, spray (Przybilla et al., 2010) or by a thin foil (Luhmann et al., 2008). However, both variants can lead to a difficult reproducibility and to a corrupted reference accuracy. These restrictions show, that a high accuracy reference object with a natural heterogeneous texture is necessary to be able to evaluate active and passive methods in the same way for applications with high accuracy requirements. For this purpose, a textured dumbbell bar and a first investigation of different methods are presented in this paper.

REFERENCE OBJECT
The dumbbell bar used to evaluate the methods according to the guideline VDI 2634 part 3 is shown in Figure 1. It is characterised by two textured spheres with a diameter of 75.727 mm and 75.584 mm respectively and a total length of 199.498 mm. The dumbbell reference has previously been calibrated by a certified testing laboratory with a coordinate measuring machine (CMM). It shows uncertainties for the sphere centroid distance of 1.7 µm and for the diameters of 2 µm each.

Figure 1. Textured dumbbell bar
The range of radial deviations of the two spheres to the normal sphere is approximately 10 µm. These high accuracies allow for the verification of high-end structured light systems. A summary of the parameters is given in Table 1 The diffusely scattering spheres are characterised by a heterogeneous texture in addition to their high accuracy. For the description of the texture, different parameters can be used (see Luhmann et al., 2019). However, the properties of the texture has not yet been conclusively investigated. The heterogeneous surfaces of the spheres generate high grayscale gradients, which, according to Schneider (1991), allow for a good image matching.

Workflow
The investigation of the different methods with the help of the presented dumbbell bar is based on the VDI guideline 2634 part 3, which describes the verification of optical measuring systems that use multiple views. Thus, the determined results represent the accuracy of the entire measurement process (calibration, correspondence analysis, registration and fusion). The dumbbell bar is arranged at various positions in the measurement volume. The entire dumbbell is acquired using several views, which are fused differently depending on the algorithm. For the investigation of passive approaches (SfM and MVS) the same image sets are used. Possible negative influences of the image acquisition are thus contained in all resulting point clouds. However, the determination of the required orientation parameters is performed individually for each method (see chapter 4). The resulting point clouds are reduced to the reconstructed dumbbell bar, whereas the determination of the required sphere parameters is performed individually for each sphere. These are determined by means of a least-squares adjustment, in which the coordinates of the surface points are taken into account with a uniform weight. Possible outliers are identified using a RANSAC method. A maximum of 3 per thousand of the points are declared as outliers and excluded from the adjustment.

VDI Guideline
The determined sphere parameters can be used to calculate the following quality parameters: • Probing error PS (size): The probing error PS describes the deviation of the determined sphere diameter Da to the reference diameter Dr.

PS = Da-Dr
(1) • Probing error PF (form): The probing error PF describes the range of radial deviations of the surface points to the best fit sphere.
• Sphere spacing error SD: The sphere spacing error SD describes the difference between the calibrated distance Lkr and the measured distance Lka. The measured distance results from the reference ball centre coordinates k1 and k2.
In addition to these criteria, the reconstructed sphere surface is visually examined for any special detail and density of the reconstruction.

Investigated methods
For a first study with the textured dumbbell bar, three different systems/procedures were investigated. The methods represent the common range of photogrammetric area based scanning techniques in close-range applications.

SmartScan HE C8 M-650:
SmartScan HE C8 M-650 is a structured light system from AICON, which consists of two cameras and a projector. The measuring volume is 525 x 400 x 330 mm with a resolution of 0.159 mm. An absolute accuracy of 42 µm according to VDI 2634 part 3 is achieved (AICON, 2020).

SURE:
SURE is a software from nFrames, which can be used for 3D reconstruction. The implemented Multi-View Stereo approach, based on Semi Global Matching, is described by Wenzel et al. (2013b). The required interior and exterior orientations are calculated in advance by a bundle adjustment using AICON 3D Studio. The used image sets and acquisition procedure are described in chapter 4. The settings in SURE (version 3.0.11) are set to default, only the processing quality was set to the best resolution "ultra".

Metashape:
Metashape is a commercial Structure from Motion software from Agisoft. It includes a self-calibration method within an automatic orientation process up to the threedimensional reconstruction of the captured surface. Metashape (version 1.6.0.9925) is used with the settings "high" for alignment and dense cloud processing.

DATA AQUISITION
The experiments were run in the laboratory of the Institute for Applied Photogrammetry and Geoinformatics. The average room temperature is 20° C, so temperature dependent corrections do not have to be considered. For the experiments a frame is used, which is shown in Figure 2. With the support of the frame it is possible to determine the required orientation parameters for MVS with high accuracy, independent of the experimental environment. The frame consists of coded and uncoded targets. In addition, some areas consist of a random texture. The coordinates of the uncoded and coded targets are determined by bundle adjustment. Two independent scales are introduced which are certified with an accuracy of 2 µm. The quality of the reference frame can be checked by an independent bundle of images. The same distances on the reference frame are used as scales as in later investigation (see Figure 3 red). The absolute accuracy level can be estimated over independent distances, which also have an absolute accuracy of 2 µm. The comparison shows that deviations of up to 31 µm can be expected over a distance of 510 mm (room diagonal of the reference frame).

Figure 2. Light tent and used dumbbell positions
For the investigation of the three presented procedures, the dumbbell bar is measured in three different positions, which are also shown in Figure 2. At each position, two independent data sets are acquired, in total six data sets are available for the investigation.

Digital single-lens reflex
The images are captured with a DSLR Nikon D850 and a 24 mm NIKKOR lens with the specifications shown in Table 2.
Parameter Value Sensor size 35.9 mm x 23.9 mm Sensor size 8256 px x 5504 px Pixel pitch 0.0044 mm Focal length 24 mm Table 2. Nikon D850 specification A light tent and several LED spotlights are used for a homogeneous background and uniform illumination. Each dataset has the same imaging configuration as shown in Figure 3. The imaging configuration consists of three rings, each with a distance of about 15 cm to each other. A tripod is used for image acquisition. The average distance to the dumbbell bar is 68 cm, resulting in an object resolution of 0.12 mm. The reference frame and the dumbbell bar are located on a rotary table that is rotated by 10° for each image acquisition. Additionally, rolled images are acquired to ensure the significant determination of the interior orientation parameters. Each image set consists of 117 images in total, which are stored in an 8-bit TIF format. Hence, the imaging configuration is characterised by a high degree of overlap so that high precision can be expected. A self-calibrating bundle adjustment is used for SfM and MVS.
As scale constraints, three distances are taken from the reference frame calibration (Figure 3 red) and are introduced to each adjustment. The bundle adjustment is performed for each data set individually using the same images as for matching. An analysis of the results indicate that all calculated image sets show similar statistics and the differences in interior orientation parameters between Metashape and AICON 3D Studio are very small. To investigate the impact on the matching, the interior orientation from AICON in Metashape was used. A Comparison of the resulting quality parameters shows that the differences are smaller than the measurement noise. The object coordinates calculated in AICON have a standard deviation of 4 µm.

Structured light system
For data capture with the structured light system the light tent is not used. Instead, the measurement system is guided around the dumbbell bar for a complete acquisition. On average, each data set consists of approx. 17 individual scans which are taken both horizontally and vertically. The individual scans are then merged by an ICP algorithm. The average precision of the merging lies in a range of 15-30 µm. A classification of the scan areas with the highest deviations is not possible. Therefore, no precise statement can be made about the merging in the area of the dumbbell bar. For the calibration of the system a calibration plate is used (Figure 4). It is placed at nine positions in the measurement volume which allows to determine the orientation parameters. Due to the black box system design, a quality analysis of the calibration is not possible. After the measurement, the point clouds are again reduced to the dumbbell. However, the resulting surfaces cannot be used directly, since only the export of meshes is possible. Therefore, in an additional step the vertices are extracted using CloudCompare, which may have an impact to the quality. On this basis the calculation of the individual sphere parameters is performed. To make all experiments comparable, the standard settings and the same calibration are used for the structured light system. Figure 5 shows the resulting point clouds for each examined technique. For the MVS and SfM method (middle and bottom) more outliers can be detected, which can be eliminated with simple cut out techniques. In all point clouds deviations occur at the interfaces to the mounting device and at the aluminium bar (see red boxes in Figure 5). These can be attributed to the object or the object resolution on the one hand and to the respective reconstruction algorithm with respect to the scene configuration on the other hand. In order to minimise the influence of these deviations, these areas are eliminated from the calculation to an acceptable extend. After filtering, the point clouds of the single spheres consist of an average of 150,000 (SmartScan), 200,000 (SURE) and 370,000 (Metashape) points, which are introduced into the sphere adjustment. For SmartScan, the point cloud contains the smallest number of points, which leads to unreconstructed areas at the bottom. However, this was to be expected since SmartScan outputs surface meshes represented by less individual points. The point cloud density is also different between Metashape and SURE. These effects can be caused by the different matching and filtering approaches. Other differences can be seen when considering the a posteriori standard deviation of unit weight 1 (s0). While the s0 in SmartScan averages 17 µm, the data sets with SURE are characterised by an average s0 of 22 µm and those with Metashape of 30 µm. Despite the differences, the internal accuracy level can be considered high for all methods. In addition, the span of residuals (parameter PF) can be expected to be lowest for the structured light system and highest for Metshape. However, it is not possible to draw a direct conclusion about the absolute accuracy level from the s0 because there is no comparison to an absolute reference. Table 3 shows the determined values for the quality parameter PS which describes the differences between the measured and the calibrated diameters. The values marked in bold represent the values which show the smallest deviation for each method. The values in italics show the largest deviation of each method. For the SmartScan, the deviations are between 2 µm and 34 µm and thus have the largest range. In contrast, the values for SURE are very homogeneous and range from -19 µm to -26 µm. With Metashape, the deviations between -9 µm and -24 µm scatter slightly more. It is interesting to note, for both passive methods, that all deviations are negative. The measured sphere diameter is therefore always smaller than the calibrated diameter. A possible explanation may be an undiscovered scaling effect. However, it should be noted that all deviations are smaller than the absolute accuracy of the reference frame. Therefore, an evaluation and analysis of the deviations is difficult and shows the importance of a high accurate scale bar information.  Table 3. Quality parameter probing error PS Despite these effects, all investigated methods reach a high accuracy level. Besides the advantageous imaging configuration, calibration and the heterogeneous texture of the dumbbell bar the results seem to prove the correct implantation of the applied software packages.

Probing error PF
The PF values of the investigated methods are listed in Table 4 It can be seen that the 12 values of each measurement system fit well together. Differences can be seen between the respective methods. The structured light system is characterised by the lowest ranges, whereas the values with Metashape are almost of factor 2 higher up to 358 µm. This confirms the expectation that The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B2-2020, 2020XXIV ISPRS Congress (2020 Metashape results with a higher noise level. In comparison to the s0, the values of the quality parameter PF are very high, which is due to the definition of PF. PF describes the absolute worst-case scenario, where only one outlier can decrease the quality level dramatically. To get a better view about the distribution of the deviations, Figure 6 shows the color-coded radial deviations (residuals) from the best-fit sphere. Similar effects occur in all data sets of the three different methods, so that the figures shown are valid for the entire measurement series. As already be seen in Table 4, the smallest deviations can be noticed for the SmartScan (top). In addition, no systematics can be detected in the distribution of the deviations. This is different for the data sets of SURE (middle) and Metashape (bottom).
Here, the largest deviations can be observed at the interfaces to the mounting device and the aluminium bar. In addition, increased deviations occur on the top of the spheres, which is not true for the SmartScan acquisition. As a possible reason, the used imaging configuration can be taken into consideration. Although this configuration consists of several closed-loop structures, no images could be taken from the top view due to the light tent. Consequently, the domes are only reconstructed from oblique views, which may lead to the increased deviations. This shows the importance of an object-specific and procedure-specific image configuration. When comparing the residuals of SURE and Metashape, the same systematic effects can be observed, but they are much more significant in Metashape. This could be due to the different processing (calibration, matching and filtering) steps. It has to be stated that the images are taken with respect to a comparable analysis using different methods. The scene structure with the reference frame and a limited homogeneous light distribution has a negative influence on the reconstruction using Metashape. This leads to higher deviations at the given parts of the dumbbell reconstruction.  Table 5. Quality parameter sphere spacing error SD

Sphere spacing error SD
The smallest deviation is 9 µm or -9 µm for all methods. The largest deviation can be observed using the structured light system with 42 µm. For SURE, it is -25 µm instead and for Metashape -19 µm. The signs of the deviations are conspicuous for all measurement systems. While with SmartScan all deviations are positive, the deviations with SURE and Metashape are again negative. However, the values of the parameter are below the accuracy level of the used scales, so that a detailed analysis is hardly possible. Despite of this effect, the values of the quality parameter are very small and show the high potential of passive measurement methods. In addition, the SmartScan also reaches a high accuracy level and confirms the accuracy level of the manufacture.

SUMMARY AND OUTLOOK
In this paper a textured dumbbell bar is presented which is characterised by a very precise calibration. Furthermore, the dumbbell bar consists of a natural heterogeneous texture, so that image matching methods can be investigated on an equal level as with active methods for surface reconstruction. For a first investigation according to the guideline VDI 2634 part 3, Structure from Motion and Multi-View Matching methods are compared with the high-end structured light system SmartScan. For this purpose, a reference frame is used in addition to the described dumbbell. Thus the two passive methods can be calibrated and orientated with very high accuracy using the same reference targets.
The investigations demonstrate that very high accuracies can be achieved with all three methods and that the sphere spacing errors are less than 42 µm. However, it is noticeable that with the passive methods all deviations are negative. This could indicate an undetected scaling effect. However, the deviations are smaller than the uncertainty of scale calibration, so that an exact analysis is difficult. When looking at the visualised residuals, a systematic effect could be detected for passive methods. The systematic is characterised by an accumulation of high deviations on the top of the domes and at the interface to the aluminium bar. The effect can be attributed to the imaging configuration, since no top views could be used for matching due to the light tent.
In spite of these limitations, it can be shown that with all methods the dumbbell bar can be successfully reconstructed with high accuracy which shows the potential of photogrammetric areabased scanning methods. From this can be concluded that the investigated passive methods achieve the same level of accuracy as the high-end structured light systems, if the surface provides a sufficient texture and a well-configured imaging configuration is used.
After the first successful application of the dumbbell bar in combination with the reference frame, various further investigations are possible. Due to the available orientation information and images, it is possible to extend the investigation to methods like OSGM (Bethmann and Luhmann, 2017), Multi-Photo-LSM (Gruen, 1996) and others. Furthermore, the use of the reference frame allows for the determination of the accuracy level in different environments and shows various accuracy level of different photogrammetric area-based scanning methods.