USING MOBILE LASER SCANNING DATA FOR FEATURES EXTRACTION OF HIGH ACCURACY DRIVING MAPS

: High Accuracy Driving Maps (HADMs) are the core component of Intelligent Drive Assistant Systems (IDAS), which can effectively reduce the traffic accidents due to human error and provide more comfortable driving experiences. Vehicle-based mobile laser scanning (MLS) systems provide an efficient solution to rapidly capture three-dimensional (3D) point clouds of road environments with high flexibility and precision. This paper proposes a novel method to extract road features (e.g., road surfaces, road boundaries, road markings, buildings, guardrails, street lamps, traffic signs, roadside-trees, power lines, vehicles and so on) for HADMs in highway environment. Quantitative evaluations show that the proposed algorithm attains an average precision and recall in terms of 90.6% and 91.2% in extracting road features. Results demonstrate the efficiencies and feasibilities of the proposed method for extraction of road features for HADMs.


INTRODUCTION
High Accuracy Driving Maps (HADMs) are the core component of Intelligent Driving Assistant Systems (IDAS) or Advanced Driver Assistance Systems (ADASs), which can effectively reduce the traffic accidents due to human error and provide more comfortable driving experiences.Using HADMs have the advantages to (1) provide scene interpretation with suitable background knowledge, (2) amend missing information caused by occlusion, (3) supply information beyond the horizon of the sensors, and (4) assist in positioning the vehicles (Brenner, 2009).Compared with the traditional navigation maps, HADMs contain more detailed and multiple road features, posing great challenges for data capturing and processing.Fortunately, vehicle-based mobile laser scanning (MLS) systems provide an efficient solution to rapidly capture threedimensional (3D) point clouds of road environments with high flexibility and precision (Yang and Dong, 2015).Therefore, MLS systems are a promising and feasible alternative to assist in rapidly extracting highway facilities and producing HADMs.However, point clouds collected by MLS system contain multiple objects with a variety of shapes and sizes, complicated and incomplete structures, occlusion, varied point densities, all of which pose great challenges for automatic and robust extraction of road features.To address these challenges, extensive studies have been investigated to extract road surface features and roadside objects form MLS point clouds.

Studies on road surface features extraction
Road surfaces are the main factor for road traffic safety, and the extraction of road surface points are the foundation of road marking, cracks and road manholes detecting.Guan andLi (2014, 2015) identified road surface points by detecting curbs, because road curbs represent road boundaries in most environments.Others extracted road surface points by plane fitting methods, such as Hough transform (Ogawa, 2006), Random Sample Consensus (Smadja, 2010), and weighted least square linear fitting (Yuan, 2010).Road markings painted on road surfaces are highly retroreflective, thus a global or local intensity thresholding are widely used to extract road markings.Chen (2009) used a single global thresholding to detect road markings, which severely suffers from the range effect of intensity, resulting in inaccuracies.To overcome this problem, local intensity thresholding method (Guan and Li, 2014;Yu and Li, 2015) and range dependent thresholding method (Yang and Fang, 2012;Kumar and Mcelhinney, 2014) are proposed.Local intensity thresholding uses different thresholds in different segments of road surface but it is hard to determine the size of locality in which a uniform threshold can separate road marking from road surface.And range dependent thresholding method establishes the relationship between range and intensity threshold.However, range effect of intensity is too complex to be modelled by simple inverse correlation.

Studies on roadside object extraction
Automatic extraction of roadside objects is the precondition of intelligent drive and road feature inventories, and has become a research hotspot in the fields of surveying, robotics, and computer vision.Existing methods for extracting roadside objects from MLS datasets can be classified into two categories: pointwise labeling (Munoz, 2008;Babahajiani, 2015) and object-based classification (Yokoyama, 2013;Lehtomä ki, 2015;Dohan, 2015;Yang and Dong, 2015).
Object-based classification methods involve two major stages: segmentation and recognition.In terms of segmentation, Euclidean distance clustering (Yokoyama, 2013), super-voxel based method (Yang and Dong, 2015) and top-down segment method (Dohan, 2015) are proposed.After the segmentation procedure, the object candidates are recognized as two or several classes based on a set of features in the recognition procedure.The existing references for recognizing object candidates can be classified into three categories: prior information or semantic rules based methods (Yang and Dong, 2015), 3-D Object Matching based methods (Yu and Li, 2015a), and machine-learning-based methods (Himmelsbach, 2009).
Although there are numerous research on extraction objects from MLS data, MLS software and automated algorithms for extracting them are still relatively slow.To overcome these limitations, this paper proposes a novel method for multiple types of road features extraction from large-scale highway scene point clouds.The main focuses of this work include automatic detection of road surface features (e.g., road surfaces, road markings and road boundaries) and roadside objects (e.g., buildings, guardrails, street lamps, traffic signs, roadside-trees, utility poles and power lines) from MLS point clouds, and analysis of its performance of producing HADMs.And the main contributions of the proposed method are as follows: 1) Detect road markings by combining intensity gradient feature and geometric feature of scatter points; 2) Segment the unorganized point clouds into meaningful candidate objects and identify them using multi-level features (point-based features, segment-based features, object-based features and context-based features).
Following this introduction, the key steps of the proposed method will be elaborated.Then the proposed method will be validated in experimental studies, after which conclusions will be drawn.

METHODOLOGY
Fig. 1 shows the detailed workflow of our proposed method.First, the method identifies ground and non-ground points using the method of Herná ndez and Marcotegui (2009).Then, the method detects road surfaces and road markings from ground points by combination of geometric features and retro-reflective features.Meanwhile, the non-ground points are segmented into individual object candidates within a successive scheme, which includes pointwise classification, multi-rule segmentation, and adjacent segments merging.Finally, the semantic annotation framework, benefiting from multi-level features (point-based features, segment-based features, object-based features and context-based features), is developed to achieve the classification of each object candidate.

Road Surface Features Extraction
Ground points and non-ground points of each road segment are firstly identified using the method of Herná ndez and Marcotegui ( 2009).As many road features (e.g., road markings, road cracks, road manhole covers and so on) are located on the road surfaces, we propose an algorithm that first identifies road surfaces from ground points and then extracts road markings from the identified road surface data.The proposed method first segments the ground points in each road segment into several planes using the Random Sample Consensus (RANSAC) algorithm, and identifies the road surface planes according the following features:  Road surface segments are large planes at a certain distance below the 3D trajectory of the vehicle;  The normal vector of the road surface segments are approximately parallel to the Z-direction.
Then, the α-shape algorithm (Edelsbrunner and Mücke, 1994) is adopted to extract the boundaries of each road surface.After that, the proposed method extracts road markings by combining intensity gradient feature and geometric feature from the extracted road surface points.In order to reduce the effects of noise, the median filter with radius smooth r is first applied to smooth the reflected intensity of road surface points.Then, the intensity gradient of each point P is calculated according to the following algorithm.
Step 1: Retrieve the k-nearest neighbour points i Q (i = 1, 2 ...) of point P using the KD-tree index, as shown in Fig. 3.
Step 2: Calculate intensity directional derivatives between each neighbour point i Q and feature point P according to Eq.1.x - axis and y -axis respectively, n is the total number of the neighboring points.
After that, the K-means algorithm with K=2 is used to clustering intensity gradient magnitude of all points.With an assumption that potential marking points have the higher intensity gradient magnitude, the algorithm chooses points in the cluster with higher mean gradient magnitude as seed points to search for road marking points.As Fig. 3 shows, for each seed point P , we check whether there exists a point 0 P meeting the following conditions:  0 P is located at the intensity gradient direction of the seed point P ;  The distance between 0 P and seed point P ranges from predefined thresholds min d to max d ;  The intensity gradient direction of 0 P is opposite to the intensity gradient direction of the seed point P .
If such point 0 P exists, both the point 0 P and the seed point P are labelled as road marking points.Moreover, all the points lying on the line segment x G is the intensity gradient vector of x P .

2.3.1
Point Cloud Segmentation: The proposed method segments the non-ground points into individual object candidates within a successive scheme, which includes nonground points classification, multi-rule segmentation and adjacent segments merging.
First, the proposed method classifies non-ground points into three categories (e.g., linear points, planar points, and spherical points) according to the local geometric features (e.g., dimensional features (Demantke, 2011), principal direction, and normal direction) of each point.For each non-ground point pj, the proposed method estimates local geometric features according to Eq.3, and labels it into the three given primitive classes (linear point: where L j p is the label of point pj.
Then, the proposed method segments the non-ground points that share similar characteristics into connected components using the multi-rule segmentation method.This paper employs a multi-rules region growing algorithm, which exploits labels, directions and spatial connectivity attributes of neighboring points.A point ps is first chosen randomly (the seed point), and a new empty segment is created.Then, the algorithm retrieves the k-nearest neighbor points i p (i = 1, 2, ... The algorithm then loops over all the points in the temporary queue, and performs the same procedure.Once the queue is empty, a point that has not been visited yet is randomly selected as a seed, and the same operations are repeated until all points are visited.Each segment formed by the multi-rule segmentation method may be a complete object, but most of them are a part of a certain object. Finally, the adjacent segments (the minimum distance between two segments is less the distance threshold s r ) are further merged as meaningful geometric abstractions of objects.To merge adjacent segments, the proposed method calculates the minimum hull of each segment and replaces the adjacency relationships between the segments with their minimum hulls.

2.3.2
Object Recognition: After point cloud segmentation, the non-ground points are segmented into individual object candidates, and the object candidates are further classified into several classes based on a set of features that describes the size and shape of the object.The details of extracted features are described as follows:  Segment-based features (Feature Dimension: 8) In this paper, we utilize dimensional features, minimum bounding box, normal and principal directions of each segment to describe their shapes (e.g., linear segments, planar segments, or spherical segments), size (length, width, and height) and direction (vertical segments or horizontal segments) respectively.The segment-based features of traffic signs are shown in Fig. 5.

 Object-based features (Feature Dimension: 311)
The object-based features process the object as a whole for feature calculation so that are more robust to noise and varying point density compared with the point-based features (Guo, 2014).However, they ignore the shape details and require a priori segmentation of the object from the scene.Therefore, they are not suitable for the recognition of partially visible or incomplete objects from cluttered scenes.In this paper, the Viewpoint Feature Histogram (VFH) (Rasu, 2010) with feature dimension Relative Direction Relation (Feature Dimension: 2): We calculate the angular separation between the principal direction of road segment with the normal directions and principal directions of object candidates to describe the relative direction relation between the object candidates and corresponding road segment.As shown in Fig. 7b, Pr, Pg and Nt are the principal direction of road segment, the principal direction of guardrail, and the normal direction of traffic sign.α is the angular separation between the normal direction of traffic sign with the principal direction of road segment, and β is the angular separation between the principal directions of guardrail and the road segment.
Spatial Distribution Pattern (Feature Dimension: 10): Spatial Distribution Pattern describes the regularity of the object candidates in spatial distribution.For example, street lamps are often found at regular intervals along the roadside, and the spatial distribution of road markings also has regular patterns, especially in the highway scene.Therefore, we calculate the 2D distances between the object candidate and its k O -nearest neighbor objects to describe the surrounding spatial distribution of the object candidate.In this paper, we set k O as 10.As shown in Fig. 7c, Di (i = 1, 2 ... k O ) is the 2D distance between the current object and its adjacent objects.
After feature calculation, all the features are scaled into the range [0, 1].Then, support vector machine (SVM) approach with one-versus-one classification strategy (Hsu, 2002) is adopted to classify the object candidates into different classes (e.g., buildings, trees, traffic signs, street lamps, cars, guardrails, utility poles, power lines, and so on ).In addition, the LIBSVM package (Chang, 2011) is used to perform the SVM trainings and predictions.

Data description
The performance of the proposed method was checked using a highway scenes dataset.Table 1 lists the detailed description of the MLS systems and point clouds.Fig. 8 shows an overview of the MLS system and experiment scene.

Road marking extraction
Fig. 9 shows the road features extraction results of selected point cloud of testdata.Fig. 9a shows the selected point cloud, colored by the elevation of each point.Fig. 9b shows the extraction result of road surface features, in which gray, blue and white points represent the points from ground, road surface and road markings respectively.And Fig. 10 shows more details of the road markings extraction results.Fig. 10a shows the extraction result of the irregular road markings, Fig. 10b indicates the extraction results of road markings with various intensities, and Fig. 10c is the extraction result of road markings from noisy point clouds.These results demonstrate that the proposed method is immune to range effect and noise, and the irregular road markings also can be extracted.These results show that the proposed method demonstrates good performance in extracting multiple kinds of roadside objects, even the small size traffic signs, moving cars, and incomplete.

Evaluation of proposed method
The proposed method was performed on a computer with 16.0 GB RAM and an Intel (R) Core (TM) i7-6700HQ @ 2.60GHz CPU.The precision and recall are used to evaluate the performance of the proposed method for extracting road features.Table 3 lists the precision and recall values of road features extraction from the test data set.The proposed method achieves good performance in extracting multiple type of road features with an average precision and recall of (90.6%, 91.2%).The precision and recall of road features extraction results decline slightly with the sharp decrease of point position accuracy, point density, and signal to noise ratio.These evaluation results demonstrate that the proposed method is robust to varying point densities, point position accuracy, and signal to noise ratio of the point clouds.

CONCLUSION
Automatic and accurate extraction of multiple types of road features is a precondition for producing HADMs and other transportation-related applications.This paper has proposed a road features extraction method for large-scale highway scene point clouds.The main contributions of the proposed method are extraction of road markings by combining intensity gradient features and geometric features, and recognition of roadside objects using multi-level features (point-based features, segment-based features, object-based features and context-based features).Comprehensive experiments were undertaken to evaluate the accuracy of the proposed method.And the experiments have demonstrated that the application of the multi-level features increased the roadside objects recognition accuracy, and considerable improvements were achieved.Most of the recognition errors were caused by undefined objects that mixed with more than one object (under-segmentation) or contained only a part of a certain object (over-segmentation).The investigation of advanced segmentation methods (Dohan, 2015) should be a future research topic.And also, the future work will focus on incorporating additional features such as spectral information and crowdsourcing data into object segmentation and recognition scheme.

Figure 1 .
Figure 1.Overview of the proposed framework

Figure 2 .
Figure 2. The detailed map properties of the HADMs

Figure 3 .
Figure 3.The diagram of road marking extraction coordinates values of on the oPP are labelled as road marking points.The mathematical description of the above conditions are as following: seed point, 0 P is the point with opposite intensity gradient direction to P , t P is a test point which may locate at the line segment 0 PP , min d and max d are width threshold of road markings, p k ) of ps using the kd-tree.Neighbors that satisfy one of the following conditions are added to the segment and to a temporary queue.


Point-based features ( Feature Dimension: 33) In terms of point-based features, the Fast Point Feature Histogram (FPFH) (Rasu, 2009) with feature dimension d F = 33 is used as the point-based features, due to its efficiency and discrimination.In addition, the average of the FPFHs of all the points is estimated and used to represent the shape of object candidate.The average of the FPFHs of different objects are shown in Fig. 4. (a) (b) Figure 4.The FPFHs of different kinds of objects: (a) the FPFH of traffic sign, (b) the FPFH of utility pole.The segment-based features of (a) traffic sign and (b) car.

dV
= 308 and minimum bounding box to describe the shape and size (length, width, and height) of each object candidate.The VFH of different objects are shown in Fig. 6.(a) (b) Figure.6.The VFHs of different kinds of objects: (a) the VFH of traffic sign, (b) the VFH of utility pole. Context-based feature (Feature Dimension: 15) To distinguish objects in different classes but sharing similarity in geometry, we introduce three kinds of context features, including relative position relation, relative direction relation, and spatial distribution pattern.The Context-based features of different kinds of objects: (a) Relative Position Relation, (b) Relative Direction Relation, and (c) Spatial Distribution Pattern.Relative Position Relation (Feature Dimension: 3): We utilize the 2D distance from the object candidate to road boundaries and the height difference between its lowest point and the road surface to describe the relative position of the object candidate with the road.As shown in Fig. 7a, Dlamp1, Dlamp2, Dcar1, and Dcar2 are the 2D distances of the street lamp and car to the road boundaries, respectively; and Hlamp and Hcar are the height differences from their lowest points to the road surface.

Figure 8 .
Figure 8. Overview of the MLS system and point cloud Dataset MLS system VMX-450 Number of laser scanners 2 Maximum valid range (m) 800 Maximum measurement rate 1100K Maximum line scan rate 400 Number of Points (million) 167.8 Length (km) 6.0 Point density (points/m 2 ) 204 Point position accuracy (cm)0.5-0.8Signal to noise ratio (%) 0.5 Table1.Descriptions of the MLS system and point cloud.
Figure 10.Details of the road surface features extraction results: (a) extraction result of irregular road markings, (b) extraction result of road markings with various intensities, and (c) extraction result of road markings from very nosiy point cloud.Details of the roadside objects extraction results: (a) small size traffic signs, (b) moving cars, and (c) incomplete street lamps, traffic signs and cars.

Fig. 9c illustrates
Fig. 9c illustrates the segmentation result of the non-ground points, in which each object is dotted in one color.Finally, the semantic annotation framework, benefiting from multi-level features (point-based features, segment-based features, objectbased features and context-based features), was used to classified object candidates into different classes.Fig. 9d illustrates the classification result of object candidates, with different colors standing for points from different kinds of objects.And Fig.11 shows more details of the roadside objects extraction results.Fig.11a shows the extraction result of the small size traffic sings, Fig.11 b indicates the extraction results of the moving cars, and Fig.11cis the extraction results of incomplete objects, including traffic signs, street lamps and cars.These results show that the proposed method demonstrates good performance in extracting multiple kinds of roadside objects, even the small size traffic signs, moving cars, and incomplete.

Table 2 .
Table 2 lists the time performance of the proposed method for road features extraction, including the time costs of each step.It takes about 12.6 minutes to extract road features from MLS point clouds covering 6 km and containing 167.8 million points.The time costs show that the proposed method has high time efficiency in road features extraction, particularly for the large data volumes MLS point clouds.The high time efficiency of the proposed method in road features extraction is attributable to the proposed multi-threading parallel process strategy.Time performance of the proposed method.

Table 3 .
Precision and Recall of the proposed method for road features extraction.