STRUCTURED LIGHT BASED 3D SCANNING FOR SPECULAR SURFACE BY THE COMBINATION OF GRAY CODE AND PHASE SHIFTING

: Surface reconstruction using coded structured light is considered one of the most reliable techniques for high-quality 3D scanning. With a calibrated projector-camera stereo system, a light pattern is projected onto the scene and imaged by the camera. Correspondences between projected and recovered patterns are computed in the decoding process, which is used to generate 3D point cloud of the surface. However, the indirect illumination effects on the surface, such as subsurface scattering and interreﬂections, will raise the difﬁculties in reconstruction. In this paper, we apply maximum min-SW gray code to reduce the indirect illumination effects of the specular surface. We also analysis the errors when comparing the maximum min-SW gray code and the conventional gray code, which justiﬁes that the maximum min-SW gray code has signiﬁcant superiority to reduce the indirect illumination effects. To achieve sub-pixel accuracy, we project high frequency sinusoidal patterns onto the scene simultaneously. But for specular surface, the high frequency patterns are susceptible to decoding errors. Incorrect decoding of high frequency patterns will result in a loss of depth resolution. Our method to resolve this problem is combining the low frequency maximum min-SW gray code and the high frequency phase shifting code, which achieves dense 3D reconstruction for specular surface. Our contributions include: (i) A complete setup of the structured light based 3D scanning system; (ii) A novel combination technique of the maximum min-SW gray code and phase shifting code. First, phase shifting decoding with sub-pixel accuracy. Then, the maximum min-SW gray code is used to resolve the ambiguity resolution. According to the experimental results and data analysis, our structured light based 3D scanning system enables high quality dense reconstruction of scenes with a small number of images. Qualitative and quantitative comparisons are performed to extract the advantages of our new combined coding method


INTRODUCTION
Three-dimensional scanning is an important research in computer vision, which has different applications in various fields such as computer graphics, industrial inspection, reverse engineering, virtual and augmented reality, object recognition, robot navigation and manufacturing.The existing 3D scanning techniques are traditionally categorized into contact and non-contact techniques.The contact 3D scanning techniques have a long history, but the performance is slow and the equipments are expensive.Besides, contact techniques always need to touch the object, which limits the applications.Then, non-contact is studied to overcome these problems and has developed recent years.The non-contact techniques are classified into two categories, active and passive.In passive approaches, the scene is imaged from two or more points of view and the correspondences between them are computed.With the calibration and correspondences results, the point cloud is generated by stereo triangulation.The major problem for this passive approach is finding the correspondences in the presence of textureless surfaces since it is a sparse reconstruction based on the texture of the object (Kawasaki et al., 2008).Therefore, active approaches are studied to cope with the limitations in finding correspondences.Structured light based 3D scanning is one of these active approaches, which creating correspondences to every pixel in the image by specific codewords.Comparing with most expensive 3D scanners, structured light based 3D scanning is one of the most wildly used and low cost techniques for dense 3D reconstruction.The structured light 3D scanning system comprises a digital pro-jector, which is viewed as an inverse camera, and a digital camera.So the calibration procedure of the projector-camera stereo system is similar to the conventional stereo-vision system (Zhang and Huang, 2006).The projector is used to project certain pattern on the scanned surface and the projected image sequences are captured by the camera.Correspondences are estimated in the decoding process, which is based on the properties of the designed codeword.Paper (Salvi et al., 2010) presnets an up-to-date review and a new classification of the existing coding strategies used in active structured light 3D scanning.The classification is depending on the strategy used to create the pattern, and qualitative and quantitative experimental results are given to analysis the feasibility and accuracy of different techniques.The classification and comparison will be discussed in details in Section 2. Another problem for structured light techniques is that it requires environment with well light condition.It is assumed that the scanned surface receives illumination directly from the light source.However, indirect or global illumination effects on the surface, such as subsurface scattering and interreflections, will raise the errors when finding the correspondences.So the performance of structured light 3D scanning strongly depends on the shape and material properties of the scanned surface.In this paper, we propose a novel combination of low frequency maximum min-SW gray code and the high frequency micro phase shifting code.Our main contributions are: (1) A complete setup of the structured light based 3D scanning system; (2) A novel combination technique of the maximum min-SW gray code and micro phase shifting code, which reduces the indirect illumination effects of the specular surface and maintains sub-pixel accuracy simultaneously; (3) Qualitative and quantitative experiments are conducted to compare our method with the state of the art, which shows that our new combined coding method outperforms conventional gray code, Gupta code ensemble and micro phase shifting method.

RELATED WORK
Structured light 3D scanning has been studied for about 40 years (Will and Pennington, 1971) and the researches focus on reducing the computation time and increasing the depth resolution.The coded structured light (CSL) system is based on the projection of one or a sequence of patterns.The existing pattern projection techniques can be classified into discrete coding methods and continuous coding method (Salvi et al., 2010).The discrete coding methods are based on spatial or temporal multiplexing techinuqes.For spatial multiplexing, the codeword of specific position is extracted from surrounding features.There are three different coding strategies in spatial multiplexing techniques: De Bruijn pattern, non-formal coding and M-arrays.De Bruijn sequences contains both striped and multi-slit patterns and the position is extracted by distinguish the color of the stripes in the same window.Different De Bruijn based patterns are proposed by Boyer and Kak (Boyer and Kak, 1987), Monks et al. (Monks et al., 1992), Salvi et al. (Salvi et al., 1998) and Pages et al. (Pages et al., 2004) (Pags et al., 2005).Non-formal encoding summarizes all the techniques non-orthodox codification, including one-axis and two-axis coding methods.Forster (Forster, 2006) and Fechteler and Eisert (Fechteler and Eisert, 2008) proposed one-axis coding methods all based on De Bruijn techniques.Tehrani (Tehrani et al., 2008) used color slits patterns and is taken from two camera views.Maruyama and Abe (Maruyama and Abe, 1989) presented a two-axis coding method with the length of the slits and their position.Another solution developed by Kawasaki et al. (Kawasaki et al., 2008) is also based on stripe lengths in both horizontal and vertical directions.M-arrays was first presented by Etzion (Etzion, 1988), which are random arrays of dimensions r×v and a sub-matrix of dimensions n×m appears uniquely in the whole pattern.Different from De Bruijn patterns, M-arrays are designed in two-dimensional space, which increasing the decoding accuracy.More M-array based patterns are proposed by (Griffin et al., 1992), (Morano et al., 1998), (Pages et al., 2006) and (Albitar et al., 2007).Time multiplexing methods project successive patterns onto the scanned surface and apply coarse-to-fine paradigm to compute the correspondences of significant bit until all patterns have been projected.There are several approaches in discrete time multiplexing, for instance, temporal binary codes, temporal n-ary codes, temporal hybrid codes and shifting methods.The temporal binary codes were firstly proposed by Posdamer and Altschuler (Posdamer and Altschuler, 1982) in 1982, which is almost the earliest coded structured light method.In this sytem, a sequence of patterns with black and white stripes were projected onto the surface.The length of the codeword was given by 2 m bits with m projected patterns.Then a coarse-to-fine strategy was applied to extract the correspondences for each stripe.Gupta et al. (Gupta et al., 2012) designed binary structured light patterns that are resilient to individual indirect illumination effects.The patterns combines conventional gray codes, maximum min-SW gray codes, logical XOR-02 codes and logical XOR-04 codes by using simple logical operations and specific combinatorial technique.One kind of the temporal n-ary codes was introduced by Caspi et al. (Caspi et al., 1998).It is a color based pattern where n m stripes were coded in RGB space.Ishii et al. (Ishii et al., 2007) proposed a system, which combined temporal and spatial coding.Shifting methods in discrete time multiplexing is popular recently because that the shifting techniques can achieve sub-pixel accuracy to obtain dense reconstruction.There are several approaches for shifting methods, such as Sansoni et al. (Sansoni et al., 2000), Guhring et al. (Guehring, 2000) and Zhang et al. (Zhang et al., 2002).According to the review and comparison among the existing techniques (Salvi et al., 2010), Guhring method achieves the most accurate result.This means that locating based on a robust gray code with sub-pixel accuracy leads to better performance than using pixel accuracy (in the case of Posdamer method (Posdamer and Altschuler, 1982)).The continuous coding methods is a set of patterns with continuous variations on intensity or color in one or two axis, including phase shifting methods, frequency multiplexing methods and continuous spatial grading methods.The phase shifting methods project sinusoidal patterns onto a surface and find the correspondences based on decoding the unique value of every point.The phase shifting methods contain single phase shifting and multiple phase shifting, both of them are in time multiplexing.The phase shifting technique is firstly proposed by (Srinivasan et al., 1985).Other single phase shifting approaches are introduced in the following years, such as (Wust and Capson, n.d.) and (Guan et al., 2003).Multiple phase shifting methods use more than one frequency in phase shifting to cope with the uncertainty in phase unwrap process.Famous multiple phase shifting methods are (Gushov and Solodkin, 1991) and (Pribanić et al., 2009).For the frequency multiplexing methods, phase decoding is performed in the frequency domain rather than in the spatial domain, including Fourier transform profilometry and wavelet transform profilometry.The continuous spatial grading methods group all techniques with codeword for a given position in spatial domain.For instance, (Carrihill and Hummel, 1985) proposed a linear grayscale codeword with wedge spread through the vertical axis.Except the structured light, Kinect is a novel, economical and widely used 3D scanner.Kinect adopts a static spatial encoded IR pattern projected on the surface to obtain the correspondences.However, since Kinect uses infrared or laser camera, which is limited in indoor environment.Therefore, by using natural light, the structured light based 3D scanning still has research value and broader applications.

Structured Light 3D Scanning System Setup
Our structured light 3D scanning system consists of an Epson projector, an AVT GigE camera and a notebook computer as shown in Figure 1.This projector-camera stereo system is economical and easy to be installed.By using this measurement system, we project different kinds of patterns on the object surface and capture image sequences at the same time.We also developed a structured light 3D scanning software, which fuses the functions of image capture, calibration, decoding and 3D reconstruction together.

Calibration
Comparing with the expensive laser scanner, the structured light system is convenient to be set up and can obtain 3D reconstruction in high precision.However, this high precision depends on accurate calibration between camera and projector.In our method, we apply a simple, accurate and robust project-camera calibration proposed by (Moreno and Taubin, 2012).This calibration method does not rely on the camera calibration parameters to find the set of correspondences, which uses local homographies to reach sub-pixel accuracy.Another advantage of this Figure 1: Setup of the structured light 3D scanning system method is that the projector and camera are calibrated together with simple procedures.The calibration process can be summarized in the following steps: 1.In order to calibrate both camera and projector, we project and capture a complete structured light pattern sequence on a planar checkerboard for a number of plane orientations as shown in Figure 2. We require a minimum of three different plane orientations.2) and the decoded pixels obtained from step (3). 5. Use the local homographies from step (4) to translate corners from camera to projector coordinates.6. Camera calibration: fix a world coordinate system to the checkerboard plane and use Zhang s method to find camera intrinsic parameters using corners detected in step (2).7. Projector calibration: fix a world coordinate system to the checkerboard plane and use Zhang s method to find projector intrinsic parameters using corners detected from step (5).8. Stereo calibration: fix camera and projector intrinsic parameters computed from step ( 6) and ( 7).Use world, camera, and projector corner positions from step ( 2) and ( 5) to estimate stereo systems extrinsic parameters (boresight and lever arm).9. Minimize the total reprojection error by bundle-adjusting all the parameters, including intrinsic and extrinsic parameters.

Codes Combination
In this paper, we proposed a novel method that combines the maximum min-SW gray code and micro phase shifting together.The maximum min-SW gray code is used to reduce the indirect illumination effects of the specular surface.
The errors in a structured light system are typically caused by long-range (interreflections) and short-range (subsurface scattering) illumination.So the errors depend on the frequency of the projected patterns and the nature of indirect illumination.Some researches show that long-range effects cause decoding errors for low-frequency patterns, whereas short-range effects affect high-frequency patterns.
According to experiments, we found that the high frequency pattern will result in huge decoding errors for specular surface.We also analysis the errors when comparing the maximum min-SW gray code and the conventional gray code, which justifies that the maximum min-SW gray code has significant superiority to reduce the indirect illumination effects.However, both conventional gray code and maximum min-SW gray code can only achieves pixel-level accuracy.To achieve sub-pixel accuracy, we project high frequency sinusoidal patterns onto the scene simultaneously.But for specular surface, the high frequency patterns are susceptible to decoding errors.Incorrect decoding of high frequency patterns will result in a loss of depth resolution.Therefore, we combine the low frequency maximum min-SW gray code and the high frequency micro phase shifting code, which achieves dense 3D reconstruction for specular surface.

Maximum min-SW Gray Code
The maximum min-SW gray code is proposed by (Gupta et al., 2013).The maximum min-SW gray code is designed with large minimum stripewidth.In the study of combinatorial mathematics, it is available to generate codes with large minimum stripe-widths (min-SW).
Then the maximum min-SW gray code is generated by combining 10-bit binary gray code with the min-SW (8 pixels) proposed by (Goddyn, 2003).The code sequence is shown in Figure 3.
In comparison, conventional gray code has a min-SW of 2 pixels and maximum min-SW gray code have a maximum stripe width of 32 pixels.Thus, this kind of code is more resistant to reflective or scattering effects than the conventional gray code.

Micro Phase Shifting
The phase shifting pattern is shifted from the previous projection by a factor of 2π/N , where N is the total number of projected patterns.The algorithm for phase shifting is shown in Eq. 1.
where A p , B p = projection constants (x, y) = projection coordinates n = 0, 1, ..., N The received intensity values from the projected surface is shown in Eq. 2.
In(x, y) = α(x, y) [A + B cos(2πf φ y p + φ(x, y) − 2πn/N )] (2) The decoding process is conducted to cancel the effect of different albedo α(x, y) and extract the phase correctly.This process is shown in Eq. 3 φ(x, y) = arctan The micro phase shifting is proposed by (Gupta and Nayar, 2012).This method uses patterns so that all the spatial frequencies are within a narrow high-frequency band.All the frequencies Ω = [ω1, ..., ωF ] lie within the band ωm − σ 2 , ωm + σ 2 , where ωm is the mean frequency and σ is the width of the band.For example, when σ is 3 pixels and ωm = 16 pixels, the resulting frequency band is [14.5, 17.5].The phase recovery algorithm with F frequencies is shown in Eq. refequ:4 and Eq.refequ:5.In this case, F + 2 images is required.One pattern of micro phase shifting code is shown in Figure 4.  (Guehring, 2000).The maximum min-SW gray code and phase shifting code are projected onto the surface together.
Since the maximum min-SW gray code is very robust and vary exactly in one bit.The decoding result of this gray code is used to labeling the integer center of every pixel.The sub-pixel locations are obtained in the decoding process of the micro phase shifting patterns.Thus, the correspondences are detected by the following steps: 1. Decode the maximum min-SW gray code and get the pixel number.
2. Decode the micro phase shifting code and get the sub-pixel correspondences.3. Attach number labelling from step (1) to the sub-pixel correspondences from step (2) to obtain robust correspondences between projector and camera with high precision.

Reconstruction
With calibration and correspondences results between projector and camera, we can use the conventional stereo triangulation algorithm to generate the point cloud.The principle of stereo triangulation is illustrated in Figure 5 (Lanman and Taubin, 2009).We can see that to obtain the 3D point on the scanned surface, we should know the 2D camera pixels and the corresponding 2D projector points.For conventional gray pattern, both column and row correspondences are required to get the 2D projected image coordinates.For our 3D sinusoidal wave gray code, the accurate 2D correspondences can be extracted simultaneously just in projected pattern.Thus, our novel gray code reduces the number of images required to be projected, which can be applied into the real-time dense 3D reconstruction.Finally, we created a complete 3D model by using Smooth Signed Distance (SSD) surface reconstruction (Calakli and Taubin, 2011).
The mesh result fills the holes caused by reflection and preserves the surface details.We have used the beam model made by specular material, given in Figure 6.Our goal is to estimate the depth of the beam, which is manually measured at 37mm (ground truth).The depth measurement is performed by fitting two planes, one for the top and one for the bottom portion of the beam, and measuring the distance between these planes.
Figure 6: Beam made by specular material and its structure First of all, we compare conventional gray code, maximum min-SW gray code and phase shifting code as shown in Figure 7.We can see that the maximum min-SW gray code outperforms other method, which is robust and more resistant to reflective or scattering effects since the holes caused by reflections are filled.The mesh result of the beam is shown in Figure 8.The comparison of the mesh results between our method and maximum min-SW gray code method are shown in Figure 9.We can see that the surface obtained from our method is much smoother, which shows outperformance over maximum min-SW method, as well as the phase shifting method.In order to reduces the indirect illumination effects of the specular surface and maintains sub-pixel accuracy simultaneously, we presented an novel code that combines the low frequency maximum min-SW gray code and the high frequency micro phase shifting code.A complete setup of the structured light based 3D scanning system is provided as well.The qualitative and quantitative experimental results verified that the system we developed outperforms stat-of-the-art methods for scanning specular surfaces.The future work is designing a codeword in multiple directions.This codeword is similar to radial function, which can be both spatial multiplexing and time multiplexing.The the number of projected images can be reduced in order to achieve real-time structured light 3D scanning.

Figure 2 :
Figure 2: Example calibration images: the checkerboard used in the process (left) and the captured image which is projected by one of the gray code (right).2. Detect corners of the checkerboard for each plane orientation prior to projecting patterns.3. Decode structured light patterns into projector row and column indices.4. Compute local homography transformations between the corners computed in step (2) and the decoded pixels obtained from step (3). 5. Use the local homographies from step (4) to translate corners from camera to projector coordinates.6. Camera calibration: fix a world coordinate system to the checkerboard plane and use Zhang s method to find camera intrinsic parameters using corners detected in step (2).7. Projector calibration: fix a world coordinate system to the checkerboard plane and use Zhang s method to find projector intrinsic parameters using corners detected from step (5).8. Stereo calibration: fix camera and projector intrinsic parameters computed from step (6) and (7).Use world, camera, and projector corner positions from step (2) and (5) to estimate stereo systems extrinsic parameters (boresight and lever arm).9. Minimize the total reprojection error by bundle-adjusting all the parameters, including intrinsic and extrinsic parameters.

Figure 7 :
Figure 7: Comparison of point cloud result: conventional gray code (left-top), maximum min-SW gray code (right-top), micro phase shifting code with frequency band around 16 pixels and 5 frequencies (left-bottom), micro phase shifting code with frequency bande around 64 pixels and 5 frequencies (right-bottom)

Figure 8 :Figure 9 :
Figure 8: Mesh result of the beam (two different views)