MULTIPLE TLS POINT CLOUD REGISTRATION BASED ON POINT PROJECTION IMAGES

In this paper, an efficient and robust registration method of multiple point clouds is proposed. In our research, we assume that point clouds are acquired by Terrestrial Laser Scanning (TLS) systems, and the scanned environments have a relatively flat base plane such as the ground or a floor. Our method is based on an existing pairwise registration method based on point projection images, which can quickly register the point clouds under the above assumptions. In the method, sliced point clouds are projected onto the base plane, and a binary image with feature points is created. The registration is done by using feature points of the images based on the sample consensus strategy. In this paper, first, we improve the efficiency of the pairwise registration method by introducing height and occlusion information to the image. Then, a validity check method of pairwise registration using space-classified images is proposed to avoid exhaustive pairwise registration in the multiple point cloud registration process. Finally, an efficient multiple point cloud registration algorithm based on progressive creation of a point cloud connectivity graph using iterative rough and precise pairwise registration and the validity check method is proposed. The effectiveness of our method is shown through its application to three datasets of outdoor environments.


INTRODUCTION 1.1 Background and Objective
Terrestrial laser scanning (TLS) systems can efficiently acquire the dense and accurate point clouds of large-scale environments, and are widely used in several fields such as plants, civil engineering, architecture, and cultural assets.To obtain the point clouds which completely cover the surfaces of objects or structures in the environments, multiple scanning at different positions is required.Therefore, registration for the acquired multiple point clouds of large-scale environments is required in point cloud applications.
Various registration methods for point clouds of large-scale environments have been proposed (Aiger et al., 2008, Akca, 2003, Al-Durgham et al., 2013, Date et al., 2014, Kang et al., 2009, Nüchter et al., 2011, Poreba and Goulette, 2015, Rusu et al., 2009, Theiler et al, 2013, Theiler et al, 2015, Yang et al., 2016, Yoshimura et al., 2016).The methods can be classified by several points of view: automatic or manual, with or without markers, rough or precise, and pairwise or multiple.With the growth of TLS technologies, many point clouds can be easily obtained, therefore more efficient and robust automatic registration methods for multiple point clouds without markers are necessary for efficient scanning and point cloud applications.
In this paper, we propose an automatic efficient marker-less registration method for multiple TLS point clouds of large-scale environments.The method is based on the iterative pairwise rough registration method and the Iterative Closest Point (ICP) algorithm (Besl and McKay, 1992) for precise pairwise registration.To realize efficient rough registration, we extend a registration method based on the point projection images (Date et al., 2014) by adding height and occlusion information to the images.The multiple registration is achieved by progressive scan graph construction using pairwise rough and precise registration, without exhaustive pairwise registration.The scan graph represents neighboring relationship of the point clouds.In order to achieve progressive graph construction, an arrangement validity check method for the point cloud pair using spaceclassified images is developed.In the algorithm, the accumulated registration errors are quickly modified using the graph and resulting coordinate transformation matrices obtained by the pairwise registration.

Related Work
Many rough registration methods for point clouds of large-scale environments have been proposed (Aiger et al., 2008, Akca, 2003, Al-Durgham et al., 2013, Date et al., 2014, Kang et al., 2009, Nüchter et al., 2011, Poreba and Goulette, 2015, Rusu et al., 2009, Theiler et al, 2013, Theiler et al, 2015, Yang et al., 2016, Yoshimura et al., 2016).Generally, rough registration is required as the pre-process of precise registration by the ICP algorithm (Besl andMcKay, 1992, Rusinkiewicz andLevoy, 2001), which works well for the roughly aligned point cloud pairs.Without rough registration, the algorithm often results in incorrect registration due to the local minima of the energy function being minimized.
In order to realize efficient and robust registration, finding correspondences between the point clouds is required, and several representations and features are used.As the features extracted from 3D point clouds, for examples, 3D SIFT (Theiler et al., 2013), 4 congruent points (Aiger et al., 2008), and point feature histogram (Rusu et al., 2009) are developed.These methods are applicable to point clouds acquired from several environments or objects, but often require a high computational cost.Geometric primitives such as linear features (edges) and planes are also used in some methods (Al-Durgham et al., 2013, Poreba and Goulette, 2015, Watanbe et al., 2016).These methods work well especially for point clouds from environments including artificial structures and objects.Features from panoramic representations of the TLS point clouds are also used for finding correspondences, for example, SIFT for reflectance images (Kang et al., 2009) and skylines for range images (Nüchter et al., 2011).However, robust extraction of correspondences from point clouds acquired at distant scanner positions is difficult.2D features on planes are also useful under the assumption that the scanned environment has a common large planar region such as a flat ground or a flat floor.By aligning the planar region, 3D registration problems can be reduced to 2D problems, therefore an efficient process can be achieved.As 2D features, for example, intersection points between vertical edges and the ground plane (Yang et al., 2016, Yoshimura et al., 2016) and images with feature points (Date et al., 2014) are used.In our research, we use the image-based method (Date et al., 2014) because of its efficiency and applicability to several environments.In this paper, we extend the method to realize more efficient registration using additional height information of points and occlusions.As a precise registration method after rough registration, the ICP algorithm and its variants (Besl andMcKay, 1992, Rusinkiewicz andLevoy, 2001) are available and used in practical applications.We used the standard ICP algorithm for precise registration in our algorithm.
Multiple point cloud registration methods for precise scan data of engineering products or small objects have been developed in last few decades (Pulli, 1999, Guehring, 2001, Zhu et al., 2016), and methods for large TLS point clouds of large-scale environments have been proposed in recent years (Yang et al., 2016, Theiler et al., 2015).Yang et al. (Yang et al., 2016) proposed a method using a minimum spanning tree for representing neighbouring relationships between point clouds.The method realizes effective and robust registration, but exhaustive pairwise registration is required and loop closure is not considered.Theiler et al (Theiler et al., 2015) achieves global alignment using the scan graph, including loops.The effectiveness was shown through applications to six datasets of different scenes.However, the method also requires exhaustive pairwise registration.In our method, by using a simple validation check for arrangement of the point cloud pair, exhaustive pairwise registration is avoided.The scan graph is constructed progressively by iterative pairwise registration, and accumulated errors are efficiently modified using the graph and coordinate transformation matrices obtained by pairwise registration.
In section 2, a pairwise rough registration method using point projection images and its extension is described.In section 3, a multiple point cloud registration algorithm is described.In section 4, experimental results for three datasets are shown.

Overview
In our research, a registration method based on the point projection images (Date et al., 2014) is used as the basic pairwise rough registration method.The flow of the method is shown in Figure 1.In the method, under the assumptions that a common planar region called the base plane, such as the flat ground or a floor, exists in the scanned point clouds, first, base planes are extracted from the point clouds and the images are generated.2D registration on the plane is done by using the RANSAC algorithm (Fischler and Bolles, 1981) and hashing for the images.Finally, with the alignment of the base planes, registered point clouds are obtained.

Point Projection Image Generation
The point projection image of a point cloud is created by base plane extraction, point projection, binary image generation, and feature point extraction.Many methods for planar region extraction from point clouds have been developed, e.g.(Dong et al., 2018).In the method that we used (Date et al., 2014), the base plane is efficiently extracted from a TLS point cloud using the region growing on a 2D panorama point cloud representation based on azimuth and elevation angles (Masuda and Tanaka, 2010).Then, points within the specified height interval shown in Figure 2(a) are projected onto the base plane, as shown in Figure 2(b).Next, using a regular grid with the user-specified cell size on the plane, a binary image where the value 1 is assigned to the cells including the projected points, and 0 to the others, is created.In this method, the center of the grid is set to the scanner position.Finally, using the Douglas-Peucker algorithm (Douglas and Peuker, 1973), corners and endpoints of figures in the image are detected as feature points, as shown in Figure 2(c).Actual point projection images generated from the TLS point clouds are shown in the middle of Figures 1 and 3(a).In the figures, red cells are the one with value 1, and yellow cells indicate feature points.

Pairwise Registration Algorithm
The rigid transformation to register the feature points in the point projection images is robustly and efficiently derived by the RANSAC algorithm (Fischler and Bolles, 1981) and hashing.The registration algorithm is described as follows.In the algorithm, the point projection image of the target (fixed) point cloud is called the target image, and the term source image is used for the source (movable) point cloud.Finally, two coordinate transformations: a 3D rigid transformation to align the base planes and scanner positions of two point clouds and a 2D transformation by the MB scaled for 3D space, are applied to the source point cloud.In our implementation, the processing time of image generation and pairwise registration for two TLS point clouds containing about 10M points was less than 3 seconds on average on a PC with Intel Core-i7 3960X CPU.

Efficiency Improvement using Height Values
The method described in section 2.3 can be extended through several approaches, e.g. using multi-layers and assigning additional information to each pixel, to improve efficiency and robustness.In our research, we add height values of the points in the point clouds to the feature points in the image for improving efficiency.In the original algorithm, two feature point pairs which have similar point distances are used for calculating the coordinate transformation matrix and the consensus.By adding the matching condition for the height values, such calculations can be reduced without the loss of correct matches.
In the point projection image generation, the maximum height of the points in each grid cell is recorded in the cell, as shown in Figure 3(b), and assigned to the feature points.In the registration process, feature point pairs which have similar distances and height differences smaller than a given threshold H  are used in This method can reduce the computational cost, however, incorrect height values are often assigned, which are caused by occlusions, as shown in Figure 4(a).In some cases, this causes incorrect registrations, as shown in Figure 5(a).To solve this problem, the feature points affected by the occlusions are recognized using the following process.
First, a 2D panorama point cloud representation is created, and planar regions are detected using region growing.Then, the boundary points of two neighbouring regions are extracted.Finally, the points on one boundary, which is more distant from the scanner, are detected as occlusion boundary points as shown in Figure 4(a).Figure 4(b) shows an example of detected boundary points.Feature points including occlusion boundary points are marked as occlusion feature points.In the height comparison of feature points in the registration process, if one of the feature points is the occlusion point, the matching condition becomes that its height is smaller than the other.If two feature points are the occlusion points, we do not apply height comparisons.
The method is applied to the registration of two datasets consisting of six and seven TLS point clouds.Compared with original method, the numbers of coordinate transformation and consensus calculation were reduced to 18.7% and 25.6% on average by using the height values.Moreover, by using occlusion boundary estimation, improvement of the robustness was observed in some registration processes, as shown in Figure 5.In the multiple point cloud registration process described in the next section, the extended method is used.

Overview
In multiple point cloud registration, reliable adjacency relationships of point clouds must be determined.A graph, whose node represents each point cloud and edge represents correctly aligned adjacent point clouds by pairwise registration, is often used for representing the relationship (Yang et al., 2016, Theiler et al., 2015, Zhu et al., 2016).The graph is often called a scan graph, and we also use the scan graph to represents the adjacency relationship of point clouds.Methods in (Yang et al., 2016, Zhu et al., 2016) use minimum spanning trees as the scan graph.However, the trees cannot handle the loops, therefore the method may miss point cloud pairs which have the potential to improve the global registration accuracy, and correct modification of accumulated errors is often difficult.Furthermore, an appropriate anchor point cloud should be selected using certain criteria.Therefore, in our research, we use a graph with loops similar to (Theiler et al., 2015).
To obtain the reliable graph connectivity, the reliability of the pairwise registration results are often calculated by using distances between sampled points, feature points or scanner positions (Aiger et al., 2008, Yang et al., 2016, Theiler et al., 2015).The scan graph is constructed so that the total reliability becomes larger from an initial complete graph (Yang et al., 2016, Theiler et al., 2015).However, the methods require exhaustive pairwise registration and it can be a cause of long computational times.In our method, we crate the scan graph progressively to avoid exhaustive pairwise registration.To realize progressive graph creation, a validity check method for the arrangement of registered point cloud pairs is used.The method is based on not only points but also the spatial consistency and overlap to achieve a more reliable validity check.
In our method, the scan graph is created by iterative pairwise registration and the validity check from a randomly selected point cloud.The point clouds which have valid arrangement to the point cloud of the leaf in the graph are added as new leaves.As a result, this process creates a tree.To create a more accurate neighboring relationship of point clouds and to be able to consider the loop closure, we add additional edges into the graph.
If the spatial distance between two point clouds (scanner positions) in the graph is shorter than a given threshold, we check the validity of their arrangement.If they pass the validity check, they are connected by an additional edge.As a result, the graph has loops, and the resulting graph is not affected by the selection of initial point clouds.
Precise registration by the ICP is also done in the pairwise registration in the graph creation.The resulting coordinate transformation matrices of precise registration are stored on the edges in the scan graph.After the scan graph creation, the accumulated errors are modified efficiently.In the next section, the arrangement validity check method is described first, and then the algorithm is shown.
Figure 5.Effect of occlusion estimation

Arrangement Validity Check Method
Our validity check method is based on consistency and overlap of classified space.Therefore, we first describe space classification.In the laser scanning, the part of the 3D space can be classified into three types, as shown in Figure 6(a): the occupied space where the object's surfaces (the scanned points) exist, the free space where no object exists, i.e. the laser passes, and the unknown space.The space classification is used in some point cloud applications, such as change detections (Xiao et al., 2013) and next best view problems (Kawashima et al., 2014).To classify 3D space, the voxel is often used.Each cell is classified into either the OCCUPIED, FREE, or UNKNOWN cell.The OCCUPIED cell includes laser-scanned points.The cells between the scanner position and the laser-scanned points are the FREE cells, and the others are the UNKNOWN cells.
In our rough registration process, the image which corresponds to the slice of 3D space is used.Therefore Conversely, incorrect registration results will make collisions between OCCUPIED and FREE cells larger (Figure 7 left), and overlaps between FREE cells of both images smaller (Figure 7 right).Based on these properties, the arrangement of two point clouds can be evaluated.
In our method, the collision ratio ( , ) where The appropriate thresholds are required for a correct decision.In our research, the values of thresholds are experimentally determined through the evaluation of the collision ratio and the overlap ratio using the several registration results.The values are shown in section 4.

Multiple Registration Algorithm
The multiple point cloud registration algorithm based on the idea described in section 3.1 is shown below.Figure 8 shows an example of the states in the registration process for eight point Finally, the accumulated errors are modified using a method similar to Pulli's method (Pulli, 1999) M , is done by minimizing the difference between the current positions and target positions derived by the above calculation for a certain number of points in overlap regions of A p .In this process, finding correspondences is not required, so the matrix A M is quickly determined by the quaternion or SVD (Besl andMcKay, 1992, Arun et al., 1987).
In the process of the improvement of the accumulated errors, first, a point cloud which has a maximum number of incident edges in the scan graph, is inserted to fixed point cloud set

REGISTRATION RESULTS
Three datasets shown in Figure 9 are used in our experiments.The scan sites are the University's campus, and all datasets are acquired by a TLS system (FARO Focus 3D S120).Dataset A consists of 5 scans (point clouds), and the total number of points is 25M points.In Dataset B, 9 scans and 45M points are included.Dataset C consists of 39 scans and 326M points.The parameters used in our experiments are summarized in Table 1.The parameters are experimentally determined.
Figure 9 shows original arrangements of input point clouds (top) and registration results (bottom) for each dataset.In the experiments, all point clouds are correctly registered.Figure 10 shows the scanning positions and the created graph for Dataset C. The edges are created between close point clouds, and a scan graph with loops is created.The influence of the difference of the initial point cloud selection was confirmed by applying the method while changing the initial point cloud.As a result, all registrations were correctly done regardless of the selection of the initial point cloud.
The processing times of the registration for Dataset A, B and C are 27 sec, 83 sec, and 876 sec, respectively (CPU: Intel Core i7-5960X, RAM 64GB).The detail of the processing time of Dataset C is shown in Table 2.Although it is difficult to perform the strict comparisons with the existing methods, our method could efficiently register multiple TLS point clouds in practical processing times.

CONCLUSIONS
In this paper, an efficient multiple point cloud registration method based on the point projection images was proposed.First, we described the basic rough registration algorithm using point projection images and its extension based on height information and occlusions.In the experiments, it was shown that the extensions can reduce the number of calculation of coordinate transformation matrices and consensus to about 20% on average compared with the original method.Then, a method for the validity check of the arrangements of two point clouds using space-classified images was proposed.Next, a multiple point cloud registration algorithm using progressive scan graph construction based on the pairwise registration and validation check was described.In the experiments, our method can register all TLS point clouds in three datasets acquired at the University's campus, which include trees and buildings.Processing times were about 27 sec, 83 sec, and 919 sec for 5 scans, 9 scans, and 39 scans, respectively.Using our method, the practical registration for multiple TLS point clouds was achieved.Future work includes more efficient registration using parallel processing and automatic execution parameter adjustments based on point cloud analysis.

Figure 1 .
Figure 1.Pairwise rough registration process feature point pairs of the target image T I .The hash key is the distance between the feature points of the pair.Iterate the following steps by the userspecified numbers ITTR N .Step2 Randomly select a feature point pair S p from the source image S I , and calculate the distance S d between the points in S p .Step3 From the hash table, obtain a set of feature point pairs T P of the target image T I , which have similar distances to S d , and apply steps 4 and 5 to each pair T T p P  .Step4 Calculate a 2D rigid transformation matrix M to align points in S p and T p , and apply the matrix to all feature points in the S I .Step5 Calculate a consensus which is the number of transformed feature points C N overlapping with the feature points in the T I .If the C N is the maximum in the iteration, store the transformation matrix M as the output matrix B M .

Figure 3 .
Figure 3. Extension of point projection image , we use the image to classify the part of 3D space.Similar to three-dimensional case, each pixel in the point projection images are classified into either of three types.The OCCUPIED cells are already recognized in the point projection image creation, as shown in Figure 2. The FREE cells are identified by lay tracing from the center of the image, which is the scanner position, to each OCCUPIED cell.In our implementation, the Digital Differential Analyzer (DDA) interpolation was used to find FREE cells.The remaining cells are marked as UNKNOWN cells.An example of point projection image with space classification is shown in Figure 6(b).If two point clouds are correctly registered, there are no large changes in the scanned scene, and certain overlaps between the two point clouds exist, then following both properties for aligned space-classified images of correctly registered two point clouds can be observed.1)No collisions between the OCCUPIED cells of an image andthe FREE cells of the other image exist.2) Certain overlaps of FREE cells in both images exist.
of OCCUPIED cells of point cloud X p X FREE P = a set of FREE cells of point cloud X p If the following condition is satisfied, we decide that the arrangement of the point clouds A p and B p are valid.

Figure 8 .
Figure 8. Process of scan graph creation

Figure 9 .
Figure 9. Multiple point cloud registration results

Figure 10 .
Figure 10.Created scan graph for Dataset C . In modification of the accumulate error, simultaneous alignment of a point cloud to the adjacent point clouds is used.Let A

Table 1 .
Parameters used in experiments

Table 2 .
Computation times