ACCURACY INVESTIGATIONS OF HAND-HELD SCANNING SYSTEMS USING DIFFERENT DUMBBELL ARTEFACTS

The use of hand-held 3D scanning systems is becoming increasingly widespread in both the industrial and cultural sectors, and, particularly in the cultural sector, the systems are sometimes used by non-specialists. At the same time, the surface of cultural objects has a wide variety of textural properties. It is therefore of great importance that the measuring systems meet qualitative standards. For this purpose, the Creaform GO!Scan, Artec Eva and Artec Spider are compared in absolute terms of accuracy using a calibrated textured spherical dumbbell in accordance with the VDI 2634 guidelines. Both GO!Scan (sphere spacing error SD = 0.03 mm) and Eva (SD = 0.03 mm) meet the expected accuracy, while the Spider (SD = 0.3 mm) does not. Furthermore, a relative comparison is made with dumbbell bars with different texture properties. The probing error form (maximum range of residuals to the best-fit sphere) was able to reveal the effects of different texture properties on surface noise for both the Artec Eva and the Artec Spider. The Spider had larger margins for the metal sphere, while the Eva had larger margins for the textured sphere. In the case of the GO!Scan, texture properties did not matter. Furthermore, the scanners were tested on a cultural reference object. The quality standards of the VDI examination were achieved on average for every scan system, but this cannot be guaranteed for complex object areas.


INTRODUCTION
The acquisition of free-form surfaces plays a major role in photogrammetric close-range applications. In addition to passive methods such as multi-view image matching or Structure-from-Motion (SfM) approaches, hand-held 3D scanning systems are frequently used. Even though hand-held 3D scanning systems were developed primarily for industrial applications, they are increasingly being used for the digitisation of cultural objects (Luca et al., 2019;Siu, 2021;Kalinowski et al., 2022). The advantage of modern scanners is the fast acquisition and postprocessing, thus a completed 3D model is available soon after scanning. As their operation is becoming easier, and these systems are being used by untrained operators, reliability is of high importance.
In several studies in the cultural heritage field, hand-held 3D scanning systems are only compared with other methods (e.g. SfM) without using reference objects (Allegra et al., 2017;Barszcz et al., 2021), while for the verification of industrial measuring systems, calibrated reference objects are mandatory, with the reference 5-10 times more accurate than the methods under investigation (Luhmann et al., 2019). With independent and high accuracy reference objects, the complete measuring workflow can be evaluated and certified, including traceability to the SI unit metre. This includes the measurement system components as well as the complete evaluation process up to the final product of the 3D model (mesh or point cloud). An accepted guideline for the verification of optical 3D-measuring systems is the VDI 2634(VDI 2008. Eiríksson et al. (2016) and Finke and Bartle (2010), for example, describe the practical application of * Corresponding author this guideline. Furthermore, it is possible to evaluate the accuracy of different photogrammetric methods and systems using highly accurate calibrated reference objects, as shown by Nietiedt et al. (2020). This guideline can also be applied in different environments, for example underwater (Kalinowski et al., 2021). Kersten et al. (2018) carried out a major investigation into the accuracy of different hand-held 3D scanning systems based on VDI 2634. In addition, reference objects that are close to practice were examined. These included the well-known Testy reference, a bust and a wheel hub. It was found that not all systems could achieve the accuracies specified by the manufacturer. In addition, it should be mentioned that all the reference objects used had a similar white surface texture. However cultural objects in particular exhibit many different surfaces and textures. This paper presents a high-accuracy investigation of several hand-held 3D scanning systems using dumbbell artefacts with different textures. The accuracy investigations are based on the VDI 2634 part 3 guidelines. Since not all reference objects are calibrated, a relative comparison is also carried out. Furthermore, the reliability of the measuring systems is examined with the help of a practical cultural object, a human skull. The results are particularly important for the area of cultural heritage, as here in particular, the scanning systems have to be able to handle a wide variety of textures and surfaces. The aim is to examine whether different textures lead to different accuracy results. Figure 1 shows the hand-held 3D scanning systems used and reviewed. The technical characteristics of the Creaform GO!Scan, Artec Eva and Artec Spider are listed in Table 1

Creaform GO!Scan Spark
The hand-held 3D scanning system Creaform Go!Scan Spark (Figure 1, left) is a white light scanner with a total of four cameras. Three of them are monochromatic and responsible for tracking geometry and the projected pattern. For the projection of 99 white stripes, a white light projector is installed. The fourth camera is an RGB camera for colouring the texture. According to the manufacturer, the hand-held scanner measures with an accuracy of up to 0.05 mm and a volumetric accuracy of 0.050 mm + 0.150 mm/m. This specification refers to the measurement of the diameter of a calibrated dumbbell bar reference object. The scanning area is 390 x 390 mm with an average acquisition distance of 400 mm and a measuring rate of 1,500,000 measurements/second. The resolution depends on the selected positioning method. The main difference between these methods is in the information used for positioning (determining the current external orientation). Positioning exclusively by targets allows a resolution of up to 0.2 mm. If the positioning is estimated by geometry (ICP), the resolution is limited to a maximum of 0.5 mm (Ametek, 2021).

Artec EVA
The Artec Eva structured light scanner consists of a white LED projector and two cameras, one for tracking the projected speckle pattern, and a second RGB camera used solely for recording colour textures. As per the manufacturer's specifications, the Eva measures with an accuracy of 0.1 mm + 0.3 mm/m and a resolution of up to 0.2 mm. The Eva uses hybrid geometry and texture tracking and does not require targets. The working volume has a depth of 40 -100 cm with a field of view of approximately 375 x 260 mm at the midpoint (where the measurement accuracy is at its highest). The measuring rate is up to 16 frames and 18 million points per second (Artec 3D, 2022).

Artec Spider
The Artec Spider uses a similar target-free measuring system to the Eva, though with a higher resolution and accuracy, and correspondingly smaller measuring volume. The system consists of a blue LED projector and three cameras for capturing the pattern, plus one RGB camera for texturing. As per the manufacturer's specification, the Spider has an accuracy of 0.05 mm + 0.3 mm/m and a resolution of up to 0.1 mm. The scanning volume has a depth of approximately 170-300 mm with a field of view of approximately 135 x 105 mm at the midpoint. Measuring rate is up to 7.5 frames and 1 million points per second. (Artec 3D, 2022)

REFERENCE OBJECTS
The reference objects used for the investigations are shown in Figure 2. There are three different dumbbell bars with varied texture properties. The sphere parameters were determined and certified in a testing laboratory of the company ISM3D (National Accreditation Body, Spain) using a coordinate measuring machine (CMM). The uncertainty of the sphere diameters is 2 μm each. An uncertainty of 1.7 μm is given for the distance between the centres of the spheres. The radial deviations of a nominal sphere are approx. 10 μm in each case. In addition, one aluminium dumbbell bar with white texture (Figure 2, B) and one steel dumbbell bar with metal texture (Figure 2, C) was used. Unfortunately, these dumbbell bars are not calibrated with the same high accuracy as the textured spheres, but nevertheless, a relative comparison is possible with these objects. Especially the VDI-parameter probing error (see section 4.2), which describes the maximum noise on the object surface, can be analysed with these different textured objects. The dumbbell bar with metal texture has two spheres, each 50 mm in diameter and with a distance of 150 mm between the centres. The dumbbell bar with white texture has two spheres, each 63.5 mm in diameter with a distance of 258 mm between the centres. In summary, various reference objects were used with different sizes and texture properties. Furthermore, a cultural object -a human skull (Figure 2, D) -is applied as a reference object. The fringe-projection system AICON SmartScan provides the ground truth of the skull with an absolute accuracy of 42 µm.

General workflow
The investigations of the hand-held 3D scanning systems are carried out using the different reference objects. With the help of VDI guideline 2634, an independent and absolute evaluation and comparison is possible. VDI 2634 Part 3 describes practical acceptance and reverification methods for the evaluation of the accuracy of optical 3-D measuring imaging systems based on area scanning. In contrast to Part 2, here the sensor is moved in relation to the object. Quality parameters are defined to assess the accuracy of the measurement system, and the entire recording and evaluation process is included in the calculation of these parameters. This includes the arrangement of sensor positions, data processing (calibration, correspondence analysis, registration and fusion) and data provision (manual editing) as well as the external conditions. The reference objects are scanned as completely as possible from several viewing or scanner positions. Furthermore, the reference object is placed at different positions within the measurement volume. The data fusion is done via transformation in a uniform object coordinate system by the geometry of the object (ICP). In order to achieve the highest accuracy with Creaform GO!Scan, the transformation is also carried out via reference targets that are not pre-determined. The sphere surfaces of each sphere are extracted from the calculated point clouds in order to derive the quality parameters. For this purpose, a fitted sphere with a free radius is determined by least-squares adjustment. Using RANSAC, possible outliers are determined so that a maximum of 3‰ of the points are not considered in the adjustment.

VDI quality parameters
The following quality parameters can be used to specify the optical measurement system and allow a comparison of the presented hand-held 3D-scanning systems: • Probing error PF (form): The quality parameter PF describes the range of the radial distances of the measuring points from a calculated fitting sphere.
• Probing error PS (size): The quality parameter PS describes the difference between the fitted diameter Da and the calibrated reference diameter Dr of the sphere.
• Sphere spacing error (SD): The quality parameter SD describes the difference between the measured length Lka and the calibrated reference length Lkr of the sphere centres.

DATA PROCESSING
Data acquisition has been conducted with an average room temperature of 20°C, so no temperature correction has to be considered. Since the measuring instruments should not be forced out of their capabilities, the set-up had to be adapted for each scanner accordingly. The Artec Eva and the Creaform GO!Scan have similar characteristics in terms of measuring volume. Figure 3 shows a reference frame with a size of 450 x 300 x 250 mm, that defines a measuring volume for the experiments. In the frame, the dumbbell bars are placed in three different positions, with every position being recorded three times. Since the Creaform scanner requires targets for the highest accuracy, the reference frame is also equipped with targets for one set-up. Table 3 gives an overview of the experimental set-ups chosen for every reference object.  Table 3: Overview of the set-ups.
Since the Artec Spider sensor and measurement volume is too small for the reference frame, the dumbbell bars are recorded in only one position. The processing of the calculated point clouds is similar for each system. If possible, a complete point cloud is already generated during the recording. Otherwise, the individual parts of the entire point clouds must be transformed into a common object coordinate system. The fine registration is done via ICP. The data processing has been done with the manufacturer's software. VX Elements 9.1 was used for the Creaform GO!Scan and Artec Studio 15 for the Eva and Spider. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France

RESULTS
To evaluate the hand-held 3D scanners, an absolute comparison is first carried out according to VDI 2634 part 3 using the textured spherical dumbbell. Then a relative comparison is made using the probing error PF. Finally, the measurement quality is compared using a skull as a complex cultural object.

Absolut comparison according to VDI 2634
For an absolute comparison of the hand-held 3D scanning systems the parameters of VDI 2364 part 3 already presented were calculated. Figure 4 shows the results of the quality parameter probing error size, which represents the deviation between measured and calibrated sphere radius. Due to the large amount of data, the minimum and maximum deviation as well as the (absolute) mean value are shown for each measuring system. The mean deviation for the GO!Scan is 0.049 mm (targets) and 0.032 mm, which is within the specified measuring accuracy of the system of 0.05 mm. It is noticeable that the maximum deviations in both cases are larger than the specified measurement accuracy. It is clear that the use of targets increases reliability. If no targets are used, the deviations vary more, e.g. from 0.003 to 0.07 mm with sphere 1. Furthermore, it is noticeable that the deviations of the GO!Scan are always negative when targets are used. If no targets are used, the deviation is always larger. This leads to the conclusion that the use of targets has an influence on the scanner's internal scale calibration. However, since the results are within the range of measurement accuracy, it is negligible. The average deviation of the Artec Eva is also within the specified measuring accuracy of 0.1 mm. The maximum deviations are also slightly higher here with 0.11 and 0.108 mm. The average deviations of the Artec Spider are 0.07 mm, which is above the specified accuracy of 0.05. Also the minimum deviation is above the specification at 0.058 and 0.067.

Probing error size (PS)
In summary, the Creaform GO!Scan and the Artec Eva achieve the specified accuracies, while the Artec Spider does not.   Table 4: Quality parameter sphere spacing error (SD) of the textured dumbbell bar. Maximum, average (absolute) and minimum deviation.

Relative comparison with different texture properties
Since not all reference objects are currently calibrated, a relative comparison of the probing error form (PF) is carried out below. The probing error form indicates the maximum range of the radial deviation (residuals) of an ideal sphere. This statistical parameter thus describes the maximum measurement noise on the surface of the sphere. In contrast to absolute values such as the reference length, a highly accurate calibration is therefore of less importance. Figure 5 shows the maximum, minimum and mean values of probing errors PF for all reference objects with different surface textures. For the Creaform GO!Scan, the average range for all surfaces is 0.2-0.26 mm when using targets and 0.25-0.36 mm without targets. Apart from a few outliers, it is clear that the texture properties have no influence on the maximum measurement noise. The Artec Eva shows a slightly higher mean span of 0.8 mm with the textured sphere in contrast to the metallic with 0.5 mm and white with 0.4 mm. The Artec Spider is also in this range, but shows a very high average range of 0.95 mm on the metal sphere surface. This may be due to the fact that the recording distance is so short and reflections are therefore stronger. The white sphere has the smallest range of 0.17 mm, while the textured sphere has a range of 0.37 mm. It is interesting to note that the Artec Spider has the lowest average deviation for the white sphere of all the scan systems. It shows that the potential of the measuring device is strongly dependent on the texture properties. Figure 6 shows an example of all radial deviations to the best-fit sphere. As expected, there are areas with lower deviations and areas with higher deviations that are shadowed or more difficult to reach due to the experimental setup. However, the noise in the textured sphere is striking. There are slight, random deviations along the entire surface of the sphere. Since this is similar for all scanning devices, it is assumed that it is associated with the contrast of the bright and black areas on the textured sphere. This becomes clear when taking a closer look at the mesh and texture image of the data of the Artec Eva (Figure 7). There is an unevenness, and prominent black areas of the texture can also be seen as deviations in the geometry on the mesh surface. (At this point it should be mentioned that the spherical dumbbell does not have any visible roughness.) This explains the higher mean deviation ( Figure 5) of the Artec Eva with the textured sphere. The residuals of the GO!Scan show the same irregularities and are also visible in the 3D reconstruction. However, they are below the maximum measurement noise defined by PF and therefore not significant. At this point it can be concluded that the texture has an influence on the measurement system. With the GO! scan, the influence is below the probing error form and therefore has no effect on the accuracy -the texture has no significant influence on the quality in this case. However, with the Artec Eva and Spider the probing error varies depending on the texture property, and the texture has a significant influence on the accuracy.

Comparison of a cultural object
Finally, the measurement accuracy is examined using a cultural object. For this purpose, a human skull is available that has both simple (e.g. forehead area) and complex (e.g. teeth) object areas.  Table 5: Mean deviations of cultural object.
As already mentioned, the reference measurement was carried out with an AICON SmartScan with an absolute accuracy of 42 µm. Table 5 shows the mean deviation of a cloud-to-mesh comparison. This is -0.011 mm for the Creaform GO!Scan with targets and is therefore very low. If no targets are used, the mean deviation is more than three times as large. It is noticeable that the deviations are negative when using targets, as with the absolute VDI comparison. The average deviation of the Artec Eva is higher at 0.099 mm. The largest average deviations are found in the Artec Spider with 0.2 mm and lie outside the specified measurement accuracy. Figure 8 shows the radial deviations compared to the reference. The smooth, simple object areas, such as the forehead, can be reconstructed with small deviations. Slightly higher deviations occur in complex object areas such as the teeth. This occurs with all scanning systems. It is also noticeable that the Artec Eva has lower deviations in the area of the teeth compared to the Creaform GO!Scan. In general, the highest deviations in the GO!Scan and Spider appear in the same area, perhaps due to increased shininess in this area.

SUMMARY AND OUTLOOK
In this paper, three different hand-held 3D scanning systems were investigated. The investigations of the Creaform GO!Scan, the Artec Eva and the Artec Spider were carried out using different reference objects: three spherical dumbbell bars with different texture properties and a practical cultural object.
Using the textured spherical dumbbell, an absolute comparison according to the guideline VDI 2634 part 3 was carried out. The Creaform GO!Scan and the Artec Eva were able to achieve the specified measurement specifications. The sphere spacing error SD here is 0.03 mm on average. The Artec Spider could not meet the specifications. SD is 0.3 mm on average, although a volumetric accuracy of 0.05 + 0.3 mm/m (0.087 mm at 124 mm reference length) is specified.
Furthermore, a cultural object was used as a reference object. The mean deviations of the GO!Scan (0.011 -0.040 mm) and Eva (0.099 mm) are within the expected accuracy. The mean deviation of the Artec Spider is 0.2 mm, above the specified accuracy. All systems have higher deviations in complex areas.
In general, it becomes clear that on average, the suitability values determined by VDI 2634 can be met. However, this cannot be guaranteed for complex object areas.
In addition, a relative comparison was made using the maximum radial range (residuals) to the best-fit sphere, which defines the maximum measurement noise. The different texture properties do not seem to have any influence on the probing error PF using the GO!Scan. The range is given for every reference sphere in a similar order of magnitude of 0.2 -0.36 mm. With the Artec Eva it is slightly higher with the textured dumbbell at 0.8 mm compared to metal (0.5 mm) and white (0.4 mm). With the Artec Spider, the radial deviation is greatest with the metal sphere at 0.95 mm. This can be attributed to the fact that the reflections are stronger due to the short recording distance. When visualising the residuals of the textured spheres, systematic deviations on the sphere surface become apparent, which are very likely due to the heterogeneous texture, as a visual comparison verifies. These deviations do not occur with the white sphere and metallic sphere and are visible with every scanning system. With the GO!Scan, the residuals lie within the PF values. The properties of the texture therefore have no influence on this quality parameter, although it is visible. With the Artec Eva, the textured sphere has the highest PF value, which is due to the texture. It is assumed that it is particularly associated with the contrast of the white and black areas on the textured sphere.
In summary, it becomes clear that different textures can have an influence on the results of different measurement systems and can be determined with the help of the VDI guideline. At the same time, very minor irregularities below the PF quality parameter cannot be detected, as is the case with the Creaform GO!Scan. However, different texture properties can be detected with the help of the residuals even if they do not have a significant influence on the accuracy.
In the future, a full accuracy study with calibrated reference bodies and even more different textures would be of great interest. Then another VDI quality parameter length measurement error E could be included, which considers the measurement noise.
Especially in the cultural field, with its complex objects and heterogeneous textures, testing of further parameters would be advantageous to enable a quick qualitative classification of the 3D digitisation.