Compact handheld fringe projection based underwater 3D-scanner

: A new, fringe projection based compact handheld 3D scanner for the surface reconstruction of measurement objects under water is introduced. The weight of the scanner is about 10 kg and can be used in a water depth of maximal 40 metres. A measurement field of about 250 mm x 200 mm is covered under water, and the lateral resolution of the measured object points is about 150 µm. Larger measurement objects can be digitized in a unique geometric model by merging subsequently recorded datasets. The recording time for one 3D scan is a third of a second. The projection unit for the structured illumination of the scene as well as the computer for device control and measurement data analysis are included into the scanners housing. A display on the backside of the device realizes the graphical presentation of the current measurement data. It allows the user to evaluate the quality of the measurement result in real-time already during the recording of the measurement under water. For the calibration of the underwater scanner a combined method of air- and water-calibration was developed which needs only a few recorded underwater images of a plane surface and an object with known lengths. First measurement results obtained with the new scanner are presented.


INTRODUCTION
Underwater 3D recording is a photogrammetric application field for more than 40 years (Höhle 1971, Moore 1976) which gets more and more importance.Application fields are the documentation of sunken objects like boat parts or ship wrecks (Korduan at al. 2003), the documentation of archaeological sites (Roman et al. 2010), the surface measurement and inspection of industrial facilities such as pipeline systems (Tetlow et al. 1999) or the measurement of biological objects such as fishes (Harvey et al. 2003, Dunbrack 2006, Costa et al. 2006) or coral reefs (Bythell et al. 2001).
Although fringe projection is a standard technique in the industrial quality management, rapid prototyping, archaeology and cultural heritage preservation, underwater applications are still seldom.This is probably because of the quite small area which can be sufficiently strong illuminated by the fringe projector.However, only in the past few years first 3Dmeasurement systems based on fringe projection technique were developed.
Structured laser light has been used for underwater photogrammetry since the nineties of the past century.Different techniques based on laser projection have been introduced (Tetlow and Allwood 1994, Moore 2001, Tan et al. 2005, Narasimhan and Nayar 2005).Massot-Campos and Olivier-Codina recently introduced a laser-based structured light system for single-shot 3D reconstruction.
First experimentation using a stereo camera setup and fringe projection technique for structured illumination was performed by Bruno (Bruno et al. 2011).There the suitability of the measurement principle for underwater 3D application is shown, and potential applications are discussed.
Zhang (Zhang et al. 2011) introduces the same time a setup using one camera and one projector for underwater 3D measurements.His contribution is the proof of the suitability of the measurement principle.Bianco (Bianco et al. 2013) compares passive photogrammetric methods with active structured light projection techniques for underwater 3D reconstruction, especially with respect to the reconstruction quality in dependence on the water turbidity.
A complete scanning device based on fringe projection technique which can be practicably used has not yet been introduced.Our scanner is the first one using this technique.Compared to classical photogrammetric underwater applications, the measurement field is very small (about 250 mm x 200 mm) and allows the measurement of only small objects or parts of larger ones.However, a precise measurement with a very dense point distribution is possible.
The measurement accuracy of underwater 3D reconstruction is usually considerable lower than in air measurements.This is due to a number of worse conditions.In water, usually considerable more light is necessary to illuminate the measurement object than in air.That's why extensive structured light such as fringe projection has not been used so far for the underwater 3D reconstruction of larger objects.The different refraction of the vision rays at the interfaces between the different media air, glass (or acrylic), and water, different geometric conditions are present than in air.This must be considered at the calibration of the systems and the calculation of the 3D measurement data.
Sedlazeck and Koch give an extensive survey and error analysis for the usage of different camera models for underwater photogrammetry (Sedlazeck and Koch 2011a).Calibration techniques are proposed e.g. by Li (Li et al. 1996) or Shortis (Shortis et al. 2000).However, these methods are quite effortful.Additionally, the long term stability of such optical underwater systems might be low as reported by Harvey (Harvey and Shortis 1998).Hence, it is desired to find a simple and robust methodology for underwater use.A method which would be performed in air and could be applied under water would be most ideal.Works by Fryer (Fryer and Fraser 1996) and Lavest (Lavest at al. 2000, Lavest et al. 2003) deal with that topic and explain the change of the intrinsic camera parameters including distortion.
Several authors (Maas 1995, Li et al. 1996, Kwon and Casebolt 2006, Telem and Filin 2010, Sedlazeck and Koch 2011b) propose the consideration of the refraction in the camera modelling and present appropriate calibration procedures and first experimental results.
In this work, first, the technical development of the underwater scanning device is described.Then a new simple calibration method for the determination of the interface distance is introduced.Finally, experiments and measurement results are presented and discussed.

Development of the 3D scanning technique
The starting point for the development of the inner parts of the scanner concerning design and frame size was the "kolibri Cordless" device (Munkelt et al. 2007) which is shown in Figure 1.However, certain functional groups had to be developed completely new because of the application field under water.This concerned a considerable more robust mechanics as well as different cameras and PC technique.First, the desired system parameters concerning measurement field size, spatial resolution, frame rate, measurement accuracy and application field were set (see Table 1).The direct connection with the control and analysis PC and the display included into the underwater housing is a complete new principle.Hence, the connection between the scanner and an external PC is omitted, which should be otherwise carried in a backpack or external case.Hereby, a better handling at the underwater use is achieved.Indeed, the scanner itself becomes a bit heavier, but this will be no disadvantage in the underwater use.The main components of the scanner which should be connected compactly for the mounting into the housing are the projection unit including lens for the structured illumination of the scene, the both cameras including lenses for the scene observation, the control and analysis PC for data processing, the display, cooling elements for the heat removal, and mechanical elements for the connection of the principal parts.
The projection unit has the task to produce and project a sequence of fringes (Gray-code and sinusoidal fringes) onto the scene.Both cameras record this fringe image sequence synchronously.From these image sequences the so called phase images are calculated, which are used for the determination of the 3D data (see e.g.Schreiber and Notni 2000).The 3D calculation is performed on the PC, which also has the task to control the image recording and to realize the user interface shown on the display.The measurement results are also indicated on the display.Additional components are two laser pointers for checking the current measurement distance.Figure 2 shows two construction drawings of the scanner.
Figure 2. Construction drawing, two views

Development of the underwater housing
The underwater housing was developed by the 4h Jena engineering GmbH (4h Jena 2015).It was conceived for both fresh and salt water use.In order to sustain the water pressure a maximal diving depth of 40 m was set.The housing material is the synthetic material PA 2200.The optical windows for the cameras, the projector, and the laser beams were produced from sapphire glass, and the window for the display from polycarbonate.The plane windows for the cameras and the projector were tilted according to the directions of the optical axis in order to simplify the calibration procedure (see next section).
For the realization of a sufficient heat removal from the housing appropriate heat sinks and a base plate with cooling ribs were constructed.Additionally, a separable under water plug-in connector cable was realized for the power supply and the signal lines.In order to provide a correct handling also with diver gloves, separable inductive switching boards including control keys and interfaces to the scanner were developed (see Figure 3).Figure 4 shows a construction drawing and the scanner in the housing without back panel with the display.

Model
The principle in order to obtain 3D measurement points is triangulation of corresponding points in the two cameras as well known in photogrammetry (see e.g.Luhmann et al. 2006).However, because of the refraction of the rays at the interfaces between air, housing windows, and water, a simple transmission of the camera model would lead to considerable errors.Even if the parameters of the pinhole camera model are adapted to the underwater situation, the classical triangulation procedure would be erroneous without model extension.Hence, an extension of the typically used pinhole camera model is necessary.
However, there is one case, where the pinhole model could be applied in the same manner.This could be achieved if we violate any refraction.This is obtained using spherical so called dome ports in the housing and placing the cameras exactly so, that the projection centre coincides with the centre point of the spheres fitted to the inner and outer dome port surface.Figure 5 illustrates this situation.In practical realizations of underwater cameras this principle has been applied, e.g. by Korduan (Korduan et al. 2003).However as reported by Bruno (Bruno et al. 2011), there occurred differences in the parameters of air and water calibration, probably due to deviations of the exact camera placement.Hence, additional correction parameters describing the distortion are necessary, probably depending on the object distance.
Figure 5. Ray directions using spherical dome ports The second typical case of the housing interface is the use of plane glass (see Figure 6).Here we can observe refraction according to Snell's law which should be considered in the procedure of calculation of the 3D measurement points.Several authors have proposed appropriate extensions of the camera model in order to regard the refraction effects, e.g.Li (Li et al. 1996), Kwon and Casebolt (Kwon and Casebolt 2006), or Telem and Filin (Telem and Filin 2011).Some Authors suggest using the pinhole camera model with adapted parameters obtained by calibration using underwater images (Fryer and Fraser 1986, Harvey and Shortis 1998, Costa et al. 2006).

Additional parameters
In contrast to the direct linear ray propagation, the directions of the rays from the object points in the water change by refraction before they reach the image plane.The change of the direction depends on the glass thickness, the orientation of the cameras optical axis concerning the glass surface, and the distance of the cameras projection centre to the housing interface.
In our modelling we assume a perpendicular orientation of the camera concerning the glass surface (see Figure 6).Hence, we have the additional parameters glass thickness th and interface distance d.These parameters have to be determined for both cameras.Additionally, the refraction indices of air n a , water n w , and the window glass n g must be known.In the following we assume to know the refraction indices (n a =1.0, n w =1.334, n g =1.7) and the glass thickness th, which e.g. can be measured tactile using any precise measurement.

Corresponding point determination
The determination of corresponding points is realized using the fringe projection technique (Schreiber and Notni 2000).
The projector generates subsequently two independent sinusoidal fringe sequences, rotated by 90° to each other (see Figure 7).As a result two pairs of phase values (φ 1,x , φ 1,y ) and (φ 2,x , φ 2,y ) at each object point M observed by the cameras from the different perspectives are obtained using a phase generation algorithm (see e.g.Creath 1986).These phase values are initially wrapped, i.e. same phase values repeat periodically over the image.Unwrapping can be obtained using the additional Gray-code sequence.
Starting from camera 1 with image co-ordinates [x 1 , y 1 ] and phase values (φ 1,x , φ 1,y ) the corresponding image point in camera 2 is searched.Image co-ordinates [x 2 , y 2 ] in camera 2 are calculated with sub-pixel accuracy, based on the identical phase information (phase values (φ 2,x , φ 2,y )).The final results of this procedure are pairs of image points resulting from the same object point, the homologous or corresponding points, respectively.On the basis of the identified homologous points the calculation of the 3D co-ordinates is done by the wellknown triangulation technique.
However, phase values are only used for correspondence finding.Hence, it is not necessary to calibrate the projector.
Figure 7. Principle of phase correlation using fringe projection Typically, a four-, six-, or eight-phase-shift algorithm (see Creath 1986) in addition to a Gray-code sequence is used for phase generation.Hence, one projected image sequence consists of between 20 and 32 images, leading to a projection/recording time of about a third or a half second per scan.Unfortunately, epipolar geometry cannot be used in order to reduce the fringe code to one direction of projection because of the invalidity of the pinhole camera model.

Approaches for underwater calibration
The change of the intrinsic camera parameters for the underwater case starting from the air parameters are described by Fryer and Fraser (Fryer and Fraser 1986) and Lavest (Lavest et al. 2003).They propose an extension of the principal distance by factor n w /n a and a reduction of the radial distortion Δr according to: The first approach for the underwater calibration would be the usage of the pinhole model and to find a set of parameters for the underwater case, which approximates the real geometry of the scene best (i.e. with a minimal error in the calculation of the 3D points).This could be obtained either by performing a calibration using underwater image recordings (as proposed e.g. by Bryant (Bryant et al. 2000) or Bruno (Bruno et al. 2011) ) or performing an air calibration and transforming the intrinsic parameters and the distortion as proposed by Lavest (Lavest et al. 2003).Additionally, a translation of the projection centre can be estimated (see next section).
We applied the second option, but only for comparison to our calibration with extended camera model (see next section).The assumptions of changing principal distance and distortion are meaningful only, if the pinhole model is used as an approximation.In fact, the intrinsic parameters do not really change.
Consequently, our final approach is to use the parameters of the air calibration and describe the direction of the vision rays according to refraction and the additional parameters th and d.Hence, one task of the underwater calibration is to find the values for the additional parameters.The other one is the formulation of the correct calculation of 3D measurement points based on found corresponding image points in the two cameras.
Because we assume for simplification a perpendicular normal angle concerning the glass surface we obtain a shift l of the projection centre in direction of the optical axis and a scaling factor sf for the principal distance c in dependence on the radial distance r of the image points concerning the principal point: Equations ( 2) and ( 3) are applied at calculation of the 3D points by triangulation using the known formulas of the pinhole camera case.However, by extension of ( 2) and (3) we, in fact, obtain a ray-based camera model (RM).

Realization of the underwater calibration
The first step of our underwater calibration was the realization of an air calibration using a plane grid target (see Figure 7) providing a number of landmarks with known 3D co-ordinates (the grid plane was set to Z=0).In principle, any suitable calibration technique known from photogrammetry or fringe projection technique can be applied.
The indices of refraction for air n a , water n w , and glass (acrylic) n g are assumed to be known.The next step is the determination of the glass thickness.In our case we measured it tactile before mounting the underwater housing.If this would not be possible, e.g. the method proposed by Chang and Yang (Chang and Yang 2014) could be applied.

Calibration evaluation
First experiments with the new underwater scanner were performed in order to characterize the measurement accuracy of the device.This means the evaluation of the calibration quality in combination with the physical limits of the scanners hardware.In order to do this, we first performed air calibration.
In order to show the advantages of the consideration of the refraction effects into the calibration, we additionally performed air calibration of the scanner inside the housing and an approximation of the underwater geometry using the pinhole model (PM).
The next step was the recording of the four calibration sequences according to A1 described in the previous section.
The measurement objects were put into a rain barrel (see Figure 10), filled with fresh water.These four datasets were used to determine the interface distances d 1 and d 2 .Afterwards, we made underwater measurements of the plane, the ball bar, and a pyramid stump (see Figure 10) in different position in the measurement volume according to the VDI/VDE suggestions (VDI/VDE 2008).These measurements were used to evaluate the quality of the calibration.The edge length of the pyramid stump was determined by air measurement for comparison.
Figure 10.Pyramid stump in the rain barrel The quantities length deviation ld and flatness deviation fd were defined as follows: where dist i meas is the measured length and dist i calib the calibrated length at measurement i, and fd j is the double maximal deviation of the measured ceramic plane from a fitted plane.At least seven measurements were performed in both cases (i, j = 1, …, 7).Additionally, noise was determined as standard deviation of the measured 3D points from a small locally fitted plane.

Test measurements in a water basin
The next task was the application of the scanner in a water basin.It was handled by a diver as can be seen in Figure 11.The first measurement objects were a pipe (Figure 12) and stones (Figure 13).The described device is one of the first complete underwater 3D scanners based on fringe projection technique.The comparison to other systems (e.g.Bruno et al. 2011, Zhang et al. 2011, Bianco et al. 2013) based on fringe projection technique is difficult, because their systems were laboratory setups and the projector was not included in the camera housing.However, our scanner has a smaller measurement field.Measurement accuracy is difficult to compare, but seems to be similar.
The reported relative measurement accuracies of other photogrammetric systems, (e.g.Li et al. 1996 or Telem andFilin 2010) has the same magnitude.The main differences between photogrammetric and systems based on structured light projection are the measurement volume (photogrammetry: possibly largestructured light: small) and the measurement point density (photogrammetry: lowstructured light: dense).
Future work should be addressed to different items.First, more experiments must be performed in order to get a more robust characterization of the scanner.Second, several methods for housing parameter calibration described by other authors (Sedlazeck and Koch 2011b, Chen and Yang 2014) should be implemented and compared to our approach.This includes the determination of the interface normal, although it could be neglected for our present scanner.The third focus of our future work is addressed to experiments concerning the dependence of the measurement accuracy on the water quality (different levels of turbidity, comparison between fresh and salt water).

Figure 1 .
Figure 1.Handheld 3D sensor system "kolibri Cordless" (left), separate control and analysis PC and wrist display (right)

Figure 6 .
Figure 6.Ray directions with refraction effects using planar glass interface An alternative is the initial usage of a different camera model, e.g. the ray-based model (Wolf et al. 2007, Bothe et al. 2010) or the raxel camera model (Grossberg and Nayar 2005).

Figure 8 .Figure 9 .
Figure 8. Grid pattern (left) and ceramic plane (right)Finally, the interface distances d 1 and d 2 for both cameras had to be determined.In order to obtain this, the following algorithm A1 was applied: Underwater recording of a plane surface (see Figure8) in two different distances (minimum and maximum distance in the measurement volume)  Underwater recording of specimen with known (calibrated) length information (ball barsee Figure9) in two different positions  Definition of the error function T for the test statistic: minimal length error and flatness of the plane  Determination of the searched parameters by minimization of T

Figure 11 .
Figure 11.Underwater use of the scanner Neither upwelling nor sinking of the device should occur.

Table 2 .
Table 2 documents the obtained results.Results for the calibration evaluation The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-5/W5, 2015 Underwater 3D Recording and Modeling, 16-17 April 2015, Piano di Sorrento, Italy