AUTOMATIC ROAD MARKINGS EXTRACTION, CLASSIFICATION AND VECTORIZATION FROM MOBILE LASER SCANNING DATA

: To meet the demands of various applications such as high deﬁnition navigation map production for unmanned vehicles and road reconstruction and expansion engineering, this paper proposes an effective and efﬁcient approach to automatically extract, classify and vectorize road markings from Mobile Laser Scanning (MLS) point clouds. Firstly, the MLS point cloud is segmented to ground and non-ground points. Secondly, several geo-reference images are generated and further used to detect road markings pixels under an image processing scheme. Thirdly, road marking point clouds are retrieved from the image and further segmented into connected objects. Otsu thresholding and Statistic Outlier Remover are adopted to reﬁne the road marking objects. Next, each road marking objects are classiﬁed into several categories such as boundary lines, rectangle road markings, etc. based on its bounding box information. Other irregular road markings are classiﬁed by a model matching scheme. Finally, all classiﬁed road markings are vectorized as closed or unclosed polylines after reconnecting the breaking boundary lines. Comprehensive experiments are done on various MLS point clouds of both the urban and highway scenarios, which show that the precision and recall of the proposed method is higher than 95% for road marking extraction and as high as 93% for road marking classiﬁcation on highway scenarios. The ratio is 92%


INTRODUCTION
Recent years, the development of Light Detection And Ranging (LiDAR) and Mobile Mapping Technology (MMT) made it feasible for us to achieve accurate 3D perception of the world with high efficiency.Various novel applications such as high definition navigation maps and highway reconstruction and expansion engineering are expecting for more detailed 3D data with semantic information to get involved.High-definition maps with higher accuracy and richer detailed road information can assist unmanned vehicles to achieve high-precision positioning, perceive the driving environment ahead, and accomplish accurate decision-making and control.Road marking, as one of the most important detailed road element, is an important reference for traffic control and vehicle navigation.Methods that can automatically detect, classify and model the road markings from point cloud data with high accuracy, robustness and efficiency are urgent to be proposed.

Related works
The research in literature related to road marking operation from mobile laser scanning (MLS) data can be roughly divided into detection (extraction), classification and modeling (vectorization).Recently, Machine Learning or Deep Learning is also applied on these topics.For each topic, typical related works would be generally reviewed as following.
Extraction: Lately, Various methods have been proposed to extract the road markings from MLS point cloud.Most of the * Corresponding author studies firstly extract the road surface based on profile image or curb features (Yang et al., 2013) (Yu et al., 2017).Then different kinds of thresholding algorithms based on reflection intensity are exploited either for the 3D point cloud or the 2D geo-referenced feature image to extract the road markings which are made of highly reflective materials.Those 2D algorithms are generally more efficient but less meticulous compared with their 3D counterparts.(Yang et al., 2012) proposed a reflection intensity correction scheme by transforming intensity inversely proportional to the distance.Then the point cloud is segmented into blocks and the blocks' geo-referenced images are binarized to get the road marking pixels.(Guan et al., 2014) proposed a multi-threshold reflection intensity segmentation method to extract the markings, which is more robust to scenarios with complex intensity phase.The road surface is divided into small stripes and the thresholding is done inside each stripe.(Yu et al., 2017) followed the thresholding method of (Guan et al., 2014) 's work but improve its effect by acting a spatial density filtering.(Kumar et al., 2014) made use of binary morphological operations and prior knowledges to complete and refine the road markings detected by thresholding.(Soilan et al., 2017) used a Gaussian Mixture Model (GMM) to roughly separate road marking points from other road surface points.Then an adaptive thresholding based on Otsu method is applied on the intensity image to extract road marking pixels robustly.(Wen et al., 2019) adopted a modified U-net to accurately extract road markings from a regularized intensity image.Some other studies are directly based on scan lines or points.(Yan et al., 2016) proposed a scan line based intensity analysis for road marking points extraction while (Ma et al., 2018) used pesudo road profile instead.(Yang et al., 2017) devised a local binary descriptor based on point cloud neighborhood intensity and intensity gradient, and then utilized random forest to classify road surface points.
The challenge of road marking detection now lies on the joint improvement of efficiency and accuracy.Besides, how to directly extract road markings without road surface extraction should also be considered.
Classification: (Yu et al., 2017) utilized trajectory and road arrangement to classify large markings and applied Deep Boltzmann Machines to classify small size irregular markings.(Soilan et al., 2017) achieve the classification by adopting a two-layer feedforward neural network taking road marking images' geometry based feature or Histogram of Oriented Gradients as input.As for arrow markings, the classification is accomplished through the correlation calculation between target and model arrow images.(Cheng et al., 2017) address the classification issue by using a manually defined decision tree based on marking geometric features.(Wen et al., 2019) took advantage of Convolutional Neural Networks (CNN) to achieve classification.
The more detailed, accurate and efficient semantic segmentation approach of road markings retaining the position and direction information would be the main focus of today's research.
Complement and Modeling: (Yang et al., 2012) used the Hough Transform to fit the boundary lines and dashed center lines.According to (Soilan et al., 2017), a GIS layer for road inventory could be created and updated efficiently.However, the inventory is based on images, thus having limited accuracy.(Wen et al., 2019) harnessed a Generative Adversarial Network (GAN) together with the context information to complete the road markings.Nevertheless, there is basically no studies on the accurate and efficient modeling and vectorization of classified road markings up to now.

Contributions
To address the aformentioned problem, in this paper, a novel automatic road marking extraction, classification and vectorization method is proposed.Extraction and refinement is achieved by image and point cloud thresholding.Classification is accomplished by exploiting geometric feature and a point cloud model matching scheme.Vectorization is based on the classification and localization result of the previous step.
The main contributions of the proposed method are as follows: • The proposed method generates several geo-reference images from the point cloud in accordance with its intensity, elevation and point density.Image processing approaches such as median filter, Sobel operator and max entropy thresholding adopted to guarantee the road marking pixels could be detected form the image without the extraction of road surface beforehand.
• The proposed method constructed a hierarchical scheme for road marking classification.Regular road markings are firstly classified based on its bounding box information.Then irregular road markings are classified using a model matching strategy assisted by a novel registration method so that even some incomplete markings can be classified correctly.
• The proposed method efficiently vectorize the road markings based on their bounding boxes information and the alignment result of model matching, which make up the gap of vectorization on road marking detection topic.

Ground Filter
Once the point cloud data is imported, a ground filtering algorithm introduced in (Yang et al., 2016) is adopted to segment the point cloud into ground and non-ground points, as shown in Fig. 2(b).This step is necessary for the automatic extraction of road markings since non-ground points from cars, trees and traffic signs would disturb the detection of ground elements.

Geo-referenced Image Processing
In this step, the MLS point cloud is divided into several blocks along the trajectory of the laser scanner.Both the non-ground and ground points are projected onto the horizontal plane and then rasterized with a resolution calculated from the average point spacing.For each block, three grayscale geo-referenced images (elevation, intensity and point density image) are generated.For each grid, the grey value of the three images are calculated as the average Z value of non-ground points, the average reflection intensity of ground points and the number of ground points respectively.Median filtering is conducted for denoising and the Sobel operator is used to get the slope (elevation gradient) image from elevation image and the intensity gradient image from intensity image.After that, the max entropy thresholding (Pun, 1980) is applied to generate the slope binary image and the point density binary image.
Those pixels with higher slope and point density than the threshold are assigned with the value 1. Next, only those pixels whose corresponding pixels' value is 0 in the slope binary image and 1 in the point density binary image are retained in the intensity gradient image.Through the operation, road pixels are roughly extracted while off-road pixels with larger slope and smaller point density are excluded.Then the road intensity gradient image is further binarized by max entropy thresholding and those pixels with the value 1 are roughly regarded as the Though the reflection intensity of road markings would vary a lot with different range, angle of incidence and road material, the gradient of intensity for a single block of the point cloud is roughly the same for road marking boundaries so that a single threshold is enough for the detection of marking boundaries.
The max entropy thresholding method unsupervisedly select the threshold that achieves the priori maximization of the posterior known entropy of the image histogram.It is found that, among the common thresholding algorithms such as Otsu and GMM, max entropy thresholding perform the best for this task.
After that, the closed roadmarking boundaries are filled by region growing and the candidate roadmarkings pixels have been detected, as shown in Fig. 2(c).

Road markings point cloud segmentation and refinement
In this step, candidate road marking point clouds are retrieved from the candidate road marking pixels of the geo-referenced images for all the blocks.
as shown in Fig. 2(d), The fused point cloud is further segmented into isolated objects through the Connected Component Analysis (CCA) using the conditional Euclidean clustering method.Those connected components with less point number than the threshold calculated from average point spacing are filtered.
For each isolated road marking point cloud, we firstly apply the Otsu thresholding (Otsu, 2007) according to the point intensity and keep the points with higher intensity in order to refine the road marking and get rid of the Sawtooth effect of geo-referenced images.The threshold is calculated as Eq.1, in which pRM and pRS are the probabilities of road markings and road surface points providing G as the threshold, respectively while µRM and µRS are their mean value.
To minimize the effect of noise points, the Statistic Outlier Remover (SOR)(Radu , Steve, 2011) is adopted to remove the geometric outliers.Other irregular road markings are further classified by a model matching scheme.Firstly, the boundary points of each unclassified road marking is extracted using the alpha-shape algorithm.For each typical dataset such as highway, urban motorway and urban auxiliary road, several typical clear and complete road markings (as shown in Fig. 4) contour point clouds are saved as models for matching.Next, the boundary points are subsampled with a predefined point spacing.Then the 2D Binary Shape Context (BSC) (Dong et al., 2017) descriptor is generated for each point according to the spatial arrangement of its neighbor points.After that, we apply the Iterative Global Similarity Points (Pan et al., 2018) algorithm to do the coarse-to-fine registration of the target and model road marking contour point clouds.Using this algorithm, both the feature and geometric distance are involved in the iterative alignment procedure.The registration is done for each model point cloud and the overlapping ratio of the target and the model is estimated.If the highest overlapping ratio is less than 60% , the target would be elinminated from the candidate road marking list, or the registered model with the highest overlapping ratio would be regarded as the target's category.An example is shown as Fig. 5, in which the leftmost road marking has the highest overlapping ratio after the registration, thus giving rise to both the correct classification and the accurate localization.
Then an object context corrector is adopted.For each classified road marking, its category might be corrected according to the type of its K nearest neighbor road markings.The final classification result is then determined, As shown in Fig. 2(e).

Road markings serialization and vectorization
In this procedure, broken boundary lines are serialized and reconnected under a predefined distance between the end points and a angle between the vector of two broken boundary lines.
After that, all classified road markings are vectorized as closed or unclosed polylines, As shown in Fig. 2(f).For the long boundary line, points are sampled with the distance calculated from the local curvature.More points would be sampled when the curvature is larger.The sampled nodes are connected as a polyline for vectorization.For irregular road markings classified by model matching, the position and direction of the closed polyline models are determined by the alignment result.
To save the time for model matching, the number of types should be as small as possible so that we only saved models with one direction (rightward) in the model repository.Then the true direction of road marking can be judged according to the final transformation matrix of the model.Giving the origin position and direction, the closed polyline model is generated according to the specification of road marking models which is determined by the road grade and government regulation.Three main layers (Boundary line layer, rectangle marking layer and arrow marking layer) are also generated, and each of the classified road markings are rendered by their corresponding layers.

Experimental setup
Experiment Platform: The experiments are implemented with a 16 GB RAM and an Intel Core i7-7700HQ @ 2.80GHz CPU.The proposed road marking extraction, classification and vectorization method is implemented in C++ with the assist of point cloud library (PCL) (Radu , Steve, 2011) and OpenCV.

Datasets description:
The performance of the proposed algorithm was verified on four MLS point cloud datasets of both the urban and highway scenarios (Highway 1, Highway 2, Urban road 1 and Urban road 2).The characteristics and challenge of these datasets are listed as following.There's a complex toll area in Highway 1 dataset.Two highway exits are involved in Highway 2 dataset.Road surface of Urban road 1 dataset is seriously occluded by the street trees.Urban road 2 dataset has considerable categories of road markings and it's the longest one among the four datasets with over six kilometers in length.

Results, evaluation and analysis
Qualitative evaluation: As shown in Fig. 6 and Fig. 7, for both the highway and urban road, the proposed method successfully detects the road markings point clouds and classified them into several typical categories.However, there exists some typical false detection and misclassification.As shown in Fig. 6 #1, two dashed rectangles are detected as part of their adjacent boundary line because of the connectivity between them.Fig. 6 #3 demonstrates the tool area of the highway.Since the road is relatively broader here, the point density is not enough for the right boundary lines to be detected.As for the urban road scenario, part of the boundary lines are occluded by street trees as well as parking and moving vehicles.Besides, some marking-like redundant objects of the road surface are also detected and classified by the proposed method.
The road marking point cloud extraction quality, classification effectiveness and vectorization accuracy for both the highway and urban road scenarios are evaluated separately as following.

Road marking extraction evaluation:
The extraction quality is evaluated on one randomly selected block for each dataset.For each experimental block, ground truth road marking points (True Positive and False Negative points) are selected and segmented manually using the software CloudCompare.Road marking points detected by the proposed method (True Positive and False Positive points) are compared with the ground truth to get the overlapping points (True Positive points).Then the accuracy assessment mechanism of road marking extraction is based on the paradigm of recall, precision and F1-score, which can be calculated as Eq.2, Eq.3 and Eq.4.The result of the experiment are shown in Table 1.It is noticed that the F 1−score for road marking extraction of the proposed method is higher than 0.96 for highway and 0.92 for urban road.precision = T P T P + F P (2) Road marking classification evaluation: To evaluate the classification effectiveness, confusion matrices Table 2 and Table 3 are computed for the Highway 2 dataset and Urban road 1 dataset, in which all kinds of road markings exist.The ground truth number of each category of the road markings is manually counted from all the detected road markings of the whole dataset.It should be mention that this experiment is done after the object context correction and before the serialization and reconnection of broken boundary lines.Rectangle road markings consists of the dashed center lines and the zebra crossing.'Arrow' 1 to 4 represents the leftmost four type of road markings in Fig. 4 and 'Diamond' means the pedestrian warning road marking.
It is noticed that the overall classification precision and recall is as high as 0.92 for Highway 2 dataset and 0.85 for Urban road 1 dataset.Most of the misclassifications take place on rectangle road marking and boundary lines.It's challenging for the proposed method to distinguish short broken boundary lines from dashed center lines.Even though the context corrector is adopted, some ambiguous cases still exist.Compared with highway scenario, urban roads are more challenging considering the enrichment, incompleteness and irregularity of road markings, thus giving rise to the better performance of the proposed method on highway dataset.
Road marking vectorization evaluation: Fig. 8 demonstrate the part of the vectorized model on Highway 1 Dataset.
For each dataset, fifty corner points of road markings are manually selected and their global coordinates are recorded to be compared with their corresponding coordinates in vectorized models.As shown in Table 4, the Root Mean Square Error (RMSE) calculated from aforementioned fifty pairs is exploited to evaluate the vectorization accuracy.

Time performance analysis:
A time efficiency evaluation is also conducted with regard to the runtime in each step of the algorithm.As shown in  trade-off, to speed up the whole procedure, larger grid size and estimated point spacing should be set.
Application on dense ALS data: Further experiments are done on Airborne Laser Scanning (ALS) point clouds with relatively high density ( > 50 points per m 2 ) and the results demonstrates similar performance as MLS point clouds.It should be mentioned that only the intensity and elevation geo-referenced images are utilized to detect road marking pixels.

CONCLUSION
Road information is more and more important for various rising applications, such as the high definition map for self-driving vehicles, road reconstruction engineering and road asset monitoring.However, the accurate, efficient and robust detection and interpretation of road information such as road markings is challenging.
This research focuses on the whole process of extraction, classification and vectorization of road markings from mobile laser scanning point cloud.
The road marking pixels are efficiently detected from geo-referenced images without the prior knowledge of road surface.Then road markings point clouds are segmented and refined under a 3D framework.Classification of road markings are achieved by using the bounding geometric feature and a model matching strategy based on point cloud registration.Finally the classified road markings are serialized and vectorized.
In future work, we will endeavor to apply deep learning on the workflow.Candidate road marking pixels will be detected through a UNet or Fully Convolutional Networks from the intensity, elevation and point density geo-referenced images.Besides, we can also use the neural network to accomplish the recovery of incomplete road markings caused by occlusion and the wrong extraction and filtering of previous operations.

Figure 1 .
Figure 1.First-person perspective demonstration of the proposed method

Figure 2 .
Figure 2. Workflow of the proposed method road marking boundary pixels.The procedure of georeferenced image processing is shown as Fig.3.

Figure 3 .
Figure 3. Georeferenced Image Processing, (a) Road MLS point cloud block, (b) Elevation image of non-ground point cloud, (c) Slope image of non-ground point cloud, (d) Slope binary image of non-ground point cloud, (e) Reflectance intensity image of ground point cloud, (f) Intensity gradient image of ground point cloud, (g) Road intensity gradient image, (h) Point density image of ground point cloud, (i) Point density binary image of ground point cloud, (j) Road intensity binary image corresponding to the road markings pixels.

Figure 4 .
Figure 4. Template of arrow road markings for model matching

Figure 5 .
Figure 5.An example of the classification by model matching based on registration

Figure 8 .
Figure 8.The DXF vectorized model together with the original point cloud of Highway 1 Dataset.Road marking corner points are manually selected from the original point cloud for vectorization accuracy evaluation.

Table 5 ,
T 1, T 2, T 3 and T 4 stands for the consuming time of data import and ground filtering procedure, image processing procedure, point cloud processing procedure, and the classification procedure.T is the total consuming time.It is found that it's more time consuming on urban road datasets.This is due to the larger size of model repository of urban road, resulting in the longer consuming time on model matching.Since efficiency and accuracy are usually a Figure 6.Road markings point clouds detection and classification result for Highway 1 dataset

Table 1
. Road marking point cloud extraction evaluationFigure 7. Road markings point clouds detection and classification result for the Urban road 1 dataset GT / Prediction Boundary line Rectangle Arrow 1 Arrow 2 Total Precision

Table 2 .
Confusion Matrix for road markings classification for Highway 2 dataset

Table 3 .
Rectangle Arrow 1 Arrow 2 Arrow 3 Arrow 4 Diamond Total Precision Confusion Matrix for road markings classification for Urban road 1 dataset