COMPARATIVE GEOMETRICAL ACCURACY INVESTIGATIONS OF HAND-HELD 3 D SCANNING SYSTEMS – AN UPDATE

Hand-held 3D scanning systems are increasingly available on the market from several system manufacturers. These systems are deployed for 3D recording of objects with different size in diverse applications, such as industrial reverse engineering, and documentation of museum exhibits etc. Typical measurement distances range from 0.5 m to 4.5 m. Although they are often easy-to-use, the geometric performance of these systems, especially the precision and accuracy, are not well known to many users. First geometrical investigations of a variety of diverse hand-held 3D scanning systems were already carried out by the Photogrammetry & Laser Scanning Lab of the HafenCity University Hamburg (HCU Hamburg) in cooperation with two other universities in 2016. To obtain more information about the accuracy behaviour of the latest generation of hand-held 3D scanning systems, HCU Hamburg conducted further comparative geometrical investigations using structured light systems with speckle pattern (Artec Spider, Mantis Vision PocketScan 3D, Mantis Vision F5-SR, Mantis Vision F5-B, and Mantis Vision F6), and photogrammetric systems (Creaform HandySCAN 700 and Shining FreeScan X7). In the framework of these comparative investigations geometrically stable reference bodies were used. The appropriate reference data was acquired by measurements with two structured light projection systems (AICON smartSCAN and GOM ATOS I 2M). The comprehensive test results of the different test scenarios are presented and critically discussed in this contribution.


INTRODUCTION
Portable 3D scanning systems are increasingly used today in many different applications due to their high flexibility, portability and efficiency.These hand-held 3D scanners are able to scan large-scale objects/models by recording hidden areas in conjunction with terrestrial laser scanner on one hand or scan small artefacts on the other hand.Typical application fields of these 3D scanning systems are mostly limited to close range, i.e. for measuring tasks with distances from under one meter up to some few meters.In close range areas measuring systems such as structured light systems and image-based systems are already available, which are strong competitors for hand-held 3D scanning systems due to their high accuracy.However, it is a challenge today to select a suitable 3D scanning system for a concrete application due to the current technological variety of available 3D scanning systems.These scanning systems are used in different fields of applications such as architecture (Maxwell, 2017), cultural heritage (Ouimet et al., 2015), biology (Friedman et al., 2015), forensic science (Larsson & Letalick, 2013;Wieczorek & Gorawska, 2017), orthopaedics (Salleh et al., 2017;Dessery & Pallari, 2018), construction (Senthilvel et al., 2017) and many others.In the recent years the market for hand-held 3D scanners extended increasingly in different price segments (EUR 500 to EUR 50,000).
Therefore investigations of the metric quality and of the functionality of hand-held 3D scanners were necessary for contributing to understand, to evaluate and to assess the range of possible applications for these portable measuring systems.In the last two years investigations of hand-held 3D scanners were already published.First investigations of 3D consumer sensors based on the PrimeSensor (Microsoft Kinect) concerning its repeatability and absolute accuracy were published by Boehm (2014).Allegra et al. (2017) investigated different hand-held 3D scanning systems for cultural heritage applications, while Lachat et al. (2017) made recommendations for the field use of the Freestyle 3D hand-held system for the scanning of chiselled and formed stone blocks.First comparative investigations on the geometrical accuracy of different hand-held 3D scanning systems were already published by Kersten et al. (2016aKersten et al. ( /2016b)).Following these investigations geometrical accuracy tests of current systems such as Creaform HandySCAN 700 and Mantis Vision F5-B were published by Przybilla et al. (2018).For the investigations presented in this contribution the following current hand-held 3D scanners were tested using the same procedures and the same test and reference bodies as published in the previous investigations at HafenCity University Hamburg: four systems from Mantis Vision (F5-B, F5-SR, F6 SMART and PocketScan 3D), Artec Spider, Creaform HandySCAN 700 and the FreeScan X7 from the Chinese company Shining 3D.

PocketScan 3D:
The PocketScan 3D (size 120×60×30 mm) is the lightest (0.25 kg) and smallest hand-held 3D scanner for the close-range scanning market (Fig. 1) of the Israeli enterprise Mantis Vision (www.mantis-vision.com).As all measuring systems of this manufacturer this scanner consists of two components: a video camera with two channels (colour and depth) and a projector.The PocketScan 3D records video through a depth and a colour sensor while projecting a structured light pattern (infrared).However, the camera must be attached to a Windows OS based tablet, laptop, or workstation via USB 3.0 for viewing, data processing, and storage.Accuracy is listed as 1 mm at 0.5 m range and 2 mm at 1 m range with a working range of 0.25-1.0m.

F5-SR and F5-B:
The Mantis Vision F5 is a structuredlight hand-held scanner with a measuring range of 0.5-4.5 m (MV F5-B) respectively 0.3-0.8m (MV F5 Short Range).The sensor hardware consists of two modules: a video camera and a projector, which is integrated in a grab handle.The projector emits infrared light on the object (proprietary pattern), which is captured as coded light by the video camera.The triangulation algorithm calculates a point cloud with 500,000 points/sec.The point density in XY is 1.6 mm @ 0.5 m distance for each image.Because of the low dependence on ambient light, the system is usable both in darkness and in daylight.Wrona (2014) and Zhang et al. (2015) describe diverse applications of the scanner.Larsson & Letalick (2013) tested the performance of the Mantis Vision F5 sensor and investigated the serviceability for military-forensic applications.Both measuring systems are not today any longer manufactured, since they are replaced by the new F6 SMART.

F6 SMART:
The F6 SMART (Fig. 1, centre) is on the market since 2017 as the newest portable scanner of Mantis Vision.The 3D hand-held scanner (weight 1 kg) is for scanning medium and large objects in the range from 0.5 m to 4.5 m with an accuracy of 0.1-0.2% of the scanning distance.The IR light allows working in any lighting conditions from complete darkness to daylight.The F6 SMART 1.3 megapixel RGB camera captures high-quality images during scanning of 640,000 points per second and Echo smart algorithm convert those images to a photorealistic textured mesh.The coloured 3D point cloud is shown on the screen in real-time mode, which offers immediate control for completeness of the scanned object.The scanner is equipped with an internal battery, so that no external power source is needed in the field service.F6 Smart is able to operate in tough environments and under difficult light conditions.

Artec Spider
Artec Spider (Artec 3D, Luxembourg) is a hand-held 3D scanner, which was developed particularly for CAD users, to scan small items with complex surface structure, sharp edges and thin ribs with 7.5 photos and 1 million points per second respectively.The system needs a preheating time of approx.30 minutes and works   with a linear field of view between 90×70 mm² and 180×140 mm².The measuring range is between 0.17 and 0.35 m.The Artec Spider uses structured light technology (speckle pattern) with blue LED as a light source and a colour camera with 1.3 megapixels for the texture mapping of the objects.In combination with the Artec Studio software the system is (according to the manufacturer) "a desktop tool for designers, engineers and inventors of each art".Sample applications of this system have been published by Adams et al. (2015), Friedman et al. (2015) as well as Inzerillo et al. (2015).

Creaform HandySCAN 700
The HandySCAN 700 has been introduced as the newest generation of hand-held 3D scanning systems from Creaform in May 2014 as "portable 3D measuring solutions and 3D engineering services".Creaform was founded in Lévis, Québec, Canada in May 2002 and is now a part of AMETEK Ultra Precision Technologies.The portable 3D scanner is equipped with power supply, USB 3.0 cable, calibration board, USB stick, positioning targets and a notebook computer with the software VXelements.
The resolution of the sensor is 0.050 mm, while the scanning area is 275 mm × 250 mm with a depth of field of 250 mm.Two principle cameras, integrated at the front of the sensor on top of each other, acquire 60 images per second.Using seven laser crosses (plus one extra line for difficult accessible areas) as a light source, the system is able to provide 480,000 measurements per second to generate the point cloud for meshing.The sensor position is determined in real-time by spatial resection using retro-reflective targets in object space.Ouimet et al. (2015) present the use of the former system for the documentation of masonry sculptural elements of the Canadian Parliament Buildings.Starosta (2016) investigated the operational capability of the 3D scanner Handy-SCAN 700.

Shining 3D FreeScan X7
The FreeScan X7 (Fig. 1) is an "ultra-portable laser hand-held 3D measurement system with only 0.95 kg", which is available on the market since 2017.The scanner is attached to a powerful computer with the software Shining3D.Additionally the scanner is equipped with a power pack, USB 3.0 cables, targets, a licence dongle and a double-sided calibration board.The system works very similar as the HandySCAN 700 according to the principle of a photogrammetric stereo system.The technical specifications (Tab. 1) and the handling of the measuring system correspond to those of the HandySCAN 700.Only the scanning range of the sensor is with 8 m twice as large as with the HandySCAN 700 according to manufacturer´s specification.Due to its handy form the scanner is flexible to complex and narrow environments and it operates in both indoor and outdoor scenarios.

TEST AND REFERENCE BODIES
For the benchmarking test the following test and reference objects were used (Fig. 2): a bust of Einstein from gypsum (height of 160 mm), a wheel hub from cast irons with the dimensions 232×120×232 mm 3 and a so-called "Testy"(height of 380 mm) from the Institute for Computer Science of the Humboldt University in Berlin (Reulke & Misgaiski 2012) and geometrically-stable reference bodies from the Bochum University of Applied Sciences: a cross-shaped body with steel spheres (max.distance 450 mm of five spheres with a diameter of 65 mm) and a planar granite slab (size 300×300 mm²).

DATA ACQUISITION
The measurements with the different hand-held 3D scanners were carried out in August (Mantis F5-SR, F5-B, Mantis PocketScan 3D, Artec Spider and HandySCAN 700) and in October 2017 (Mantis F6 SMART and FreeScan X7) in the Photogrammetry & Laser Scanning Lab of HafenCity University Hamburg.The reference measurements were already conducted during the first investigations of hand-held 3D scanners in January 2016 using the two structured light systems GOM ATOS I 2M and AICON smartSCAN.Since strong reflecting, mirroring, black or shining surfaces cause problems for the scanning systems; the spheres of the cross-shaped reference body as well as the wheel hub were sprayed with lime spray before scanning, in order to generate matt and thus diffuse scattering surfaces of the shining metal bodies.The scanning conditions in the laboratory were the same for each 3D scanner.The 3D scanning systems were used in the op-erator´s hand during the object scanning process by moving them slowly and steadily over and around the test object to have a smooth and fluent scanning.
Before starting the scanning of the test bodies both photogrammetric stereo systems HandySCAN 700 and FreeScan X7 were calibrated on the basis of the calibration boards provided, while for the Artec Spider and the Mantis Vision systems the appropriate latest calibration files were selected in the control software.
For the scans of the HandySCAN 700 and FreeScan X7 the test and reference bodies and the close proximity were equipped with circular, self-bonding and retro-reflective targets (diameters about 11 mm).The positions of the scanning systems are computed by triangulation of the targets in real time (resection in space).The number and the distribution of the targets depend on size and shape of the object to be scanned, whereby the minimum distance between targets should amount to 20 mm and the maximum distance 100 mm according to the manufacturer´s specifications.
The hand-held 3D scanning systems of Mantis Vision scanned around and across the object in a distance of 0.5 to 1.5 m (depending upon optimal distance for each system).The sensors (F5-SR, F5-B, PocketScan 3D) were attached to a Microsoft Surface 4 via an USB 3.0 cable and controlled with the software Kapla Vision.The scanned data (current frame and the picture of the infrared camera) were shown in real time on the Microsoft Surface.The F6 SMART was attached to a laptop via an USB 3.0 cable using the software Echo.During scanning with the Mantis F5-B scanner possible signal losses will only be recognized in the data post processing, however, this could be corrected by manual registration of the scans.The data frequency of this system is 10 fps (frames per second) and a typical entire object scanning takes one minute and supplies about 15 million points.These Mantis Vision systems work without control points (targets), but a sufficient amount of spatial structure has always to be present in object space to support real-time scan registration.
The Artec Spider was attached to a tablet with the software Artec Studio 9. Before starting the scans, the system needed a warming up phase of about 30 minutes.The position and orientation of the scanner is calculated for each scan (7.5 fps) using the geometry of the object and its surroundings.For the full and complete data acquisition of a test body several scans were necessary.Therefore jerky movements, for example, should be avoided, since it caused a loss of registration and requires a restart of the scan procedure to guarantee smooth and complete scan registrations.

DATA EVALUATION AND RESULTS
To evaluate the data from the diverse measurement systems standard file formats (OBJ, PLY, STL) had to be processed.All systems used generated 3D models (meshes) on the fly through triangulating point clouds using the system software.
Three reference bodies (Testy, wheel hub and the bust of Einstein) had been measured in detail and at high precision with the structured light systems and afterwards the modelling had been carried out with Geomagic Studio 2012.The ATOS system generated the reference data set for Testy, while the wheel hub and the bust of Einstein were measured with the smartSCAN.
The guideline VDI/VDE 2634, part 2 and 3, is an accredited standard for acceptance tests (verifying the specified accuracy) and reverification (to ensure long-term compliance) of optical measurement systems based on area scanning (VDI/VDE 2002, 2006).Using the framework of well-defined test scenarios, suitable test objects (artefacts) are employed to determine quality parameters.
Following the guidelines, tests were executed using the crossshaped body HSBO with spheres and the granite slab.The derivable quality parameters are: • The quality parameter probing error PS (size) arises from the difference between the measured diameter and the diameter of the calibrated sphere.• The quality parameter probing error PF (shape) is the range of the radial distance between the measured points and a best-fit sphere.The best-fit sphere is determined according to the least-squares method with free radius.• The sphere-spacing error SD is determined from the difference between the measured and calibrated values of the distance between the centres of two spheres.The measured distance is derived from the measured values obtained from multiple area-based probings.The limit, SD, for the permissible three-dimensional sphere-spacing error is the quality parameter sphere-spacing error.It is determined as a length-independent quantity and shall be observed within the entire measuring volume specified.• The quality parameter flatness measurement error RE, is the range of the signed distances of the measurement point from the best-fit plane calculated according to the leastsquares method.
To evaluate the datasets and calculate the quality parameters Geomagic Studio was used.

Cross-shaped reference body with spheres
Fig. 3 represents the determined quality parameter probing error PF (shape), which is an indicator for the measuring noise of each system of the seven examined scanning systems.The volume size of the cross-shaped reference body with spheres corresponds to the measuring volumes of the most examined 3D hand-held scanners.In contrast the deviation behaviour of the measuring systems F5-B and F6 SMART is described in only a small part of this measuring volume.The characteristic curves of the systems HandySCAN 700, FreeScan X7 and Artec Spider show only very small deviations (up to only 1.5 mm), while all Mantis Vision systems used exhibits clear worse deviation behaviour.On the other hand the probing error PS (size) of Mantis F5-B as well as of HandySCAN 700, FreeScan X7 and Artec Spider is very small with only slight deviations up to 0.6 mm (Fig. 4).As expected the two Mantis Vision systems PocketScan 3D and F6 SMART show a little higher average deviation with 4.33 mm and 1.94 mm, respectively (Tab.2).
The parameter sphere-spacing error SD is used to check the capability of the measuring system for the length measurement from several different single views.Due to averaging effects, the probing error is not completely included in the sphere-spacing error (VDI/VDE 2006).The systems HandySCAN 700 and FreeScan X7 as well as for Mantis F5-B and F6 SMART obtained very good results with average values of less than 0.25 mm (Tab. 2).In this test scenario the two Mantis Vision systems (F5-B and F6) surprisingly show very small deviations, while the Artec Spider varies strongly between -2.4 mm and +1.7 mm (Fig. 5).
As expected the deviations of PocketScan 3D are the highest (on the average 2.9 mm) compared to all other scanning systems tested.A scale error is to be assumed as the main reason for this behaviour.

Reference body granite plate
The quality parameter flatness measurement error RE shows the high quality of the two photogrammetric stereo systems HandySCAN 700 and FreeScan X7 (Fig. 6).Although the systems produce a significant smaller number of points (in comparison to the reference system smartSCAN), the present result of these two hand scanners reflects clearly the existing high potential.The natural visual texture of the granite slab meets the requirements of the measuring procedure of these systems, so that the good results are also affected by the texture of the reference body.On the other hand the higher scanning noise of the Mantis Vision systems has already been validated by the probing error PF (shape) (Fig. 4).The Artec Spider has significantly the largest deviation of all examined systems for this quality parameter.That is to be justified rather by the surface texture of the granite slab as through general scanning noise of the system, since the probing error of the Artec Spider is quite low (see to Fig. 3 and  4).

Test bodies Testy, Einstein bust and wheel hub
The results of the 3D comparisons using the three test bodies are summarised in Table 3, while the average deviations are illustrated in colour in Fig. 7 and 8.The achieved average deviation between 10 and 110 m shows that the HandySCAN 700 obtains high accuracies in comparison to the reference system respectively, which even correspond to that of the structure light systems for the test body Einstein bust.The results using these two 3D hand-held scanners correspond to the results of the previous investigations (Kersten et al. 2017), so that thereby a high repeatability (precision) is confirmed.The span, which is calculated from the difference between the average negative and positive deviations, is between 0.07 mm and 1.4 mm for the seven examined 3D hand-held scanners.Systematic deviations are visible in yellow/orange (PocketScan 3D and F6 SMART) and blue/light blue (F5-B and F5-SR) for the Mantis Vision systems at the three test bodies (Fig. 7 and 8).The distance deviations of the PocketScan 3D are systematically positive for the test bodies Testy and Einstein, which proved the fact that the instrument scale was not precisely determined.However, the 3D hand-held scanners Artec Spider, HandySCAN 700 and FreeScan X7 as well as Mantis F5-B and F5-SR show only small deviations (span) up to 0.26 mm on the Einstein bust (Tab.3 and Fig. 8).Systematic deviations at the wheel hub, shown in light blue, might have been caused by an incorrect scan registration of front and rear face due to the complex geometry of this test body (Fig. 8).However, it cannot be perfectly determined, whether the applied lime spray has any systematic influence in the 3D comparison.Despite complex geometry all hand-held scanners, except the Artec Spider, could completely scan the wheel hub, which succeeded only with few systems in previous investigations (Kersten et al. 2016a).Nevertheless, the deviations achieved in this investigation are clearly higher than the accuracy of the examined hand-held scanning systems specified by the manufacturers.

CONCLUSION AND OUTLOOK
In this contribution the results of geometrical accuracy tests of seven hand-held 3D scanners Mantis Vision PocketScan 3D, Mantis Vision F5-SR, Mantis Vision F5-B, Mantis Vision F6 SMART, Artec Spider, Shining 3D FreeScan X7 and Creaform HandySCAN700 are presented.The test results are only valid for the specific model of each system investigated, i.e. it does not represent a universal statement about the respective product series of the hand-held 3D scanning system.It could be shown that the evaluated systems achieved different accuracies for the test bodies in 3D comparison to the references system and for the determination of parameters following the guideline VDI/VDE 2634 using stable reference bodies.The two photogrammetric stereo systems Shining 3D FreeScan X7 and Creaform Handy-SCAN 700 obtained the best results, which were not far away from the reference system used in some tests.The high accuracies of these systems are essentially obtained using signalised targets in object space and by sensor calibration accomplished just before object scanning.The Artec Spider achieved results in the 3D comparison of the test bodies, which were significantly better than in the first investigations (Kersten et al. 2016a), which is ensured due to the use of the correct calibration file during the current investigations.However, a complete scanning of complex objects such as the wheel hub did again not succeed with this system.The manufacturer's accuracy specifications could not be kept for all systems examined in all test scenarios.The system evaluation following the guideline VDI/VDE 2634 is suitable for an evaluation of the efficiency and for a comparison with earlier investigations of hand-held 3D scanning systems.However, in the future the reference and test bodies must be adapted for systems with larger scanning volume such as the Mantis F6 SMART, in order to be able to present informative results for the respective scanning volume.
The quality parameters probing error and sphere-spacing error, which are determined following the guideline VDI/VDE 2634, part 2 and 3, show for the PocketScan 3D, that the instrument scale was not accurately determined and/or the sensor is probably not geometrical stable due to the mechanical design.Therefore, field or lab test procedures and/or simple self-calibration routines carried out by the user are meaningful and recommended for all handheld scanning systems.
The handling of the systems in principle is simple; however, the data recording requires slow, homogeneous movements -around and over the object scanned -and appropriate experience of the operator to permanently continue the automatic real time registration of each scan.Therefore additional geometrical elements placed in object space significantly support the registration of the scans for the Mantis Vision systems and the Artec Spider.The scanning speed is quite high for all systems evaluated using only few minutes for object scanning.In the future it is planned to carry out further geometrical accuracy tests regarding repeatability for scanning systems already tested and further evaluations of other hand-held 3D scanners such as Faro Freestyle 3D and Artec Leo.

Figure 2 .
Figure 2. Test and reference bodies for the investigations of the hand-held 3D scanning systems -f.l.t.r.Einstein bust, wheel hub, Testy, cross-shaped body with spheres (centre) und granite slab (right)

System Scan method Scan speed [pts/s] Range [m] Accuracy [mm] Weight [kg] Price [EUR]
1 Depending on scan distance, 2 No longer directly sold, price for pre-owned system (Status Jan. 2018), SL/SP = Structured light/Speckle pattern, SP = Stereo photogrammetry

Table 1 .
Selected technical specification of the examined hand-held 3D scanning systems (manufacturer´s data)

Table 2 .
If necessary an optimized use of the software parameters by an experienced user could still permit another result with smaller deviation.Quality parameter according to VDI/VDE 2634, part 2 and 3 The deviations of the Mantis F5-B are on the average inexplicably quite high with 10.5 mm (see also Tab. 2).The software Kapla Vision offers the possibility to reduce the measuring noise with a slider in the data evaluation of the Mantis systems.