AUTOMATIC EXTRACTION OF ROAD MARKINGS FROM MOBILE LASER-POINT CLOUD USING INTENSITY DATA

: With the development of intelligent transportation, road’s high precision information data has been widely applied in many fields. This paper proposes a concise and practical way to extract road marking information from point cloud data collected by mobile mapping system (MMS). The method contains three steps. Firstly, road surface is segmented through edge detection from scan lines. Then the intensity image is generated by inverse distance weighted (IDW) interpolation and the road marking is extracted by using adaptive threshold segmentation based on integral image without intensity calibration. Moreover, the noise is reduced by removing a small number of plaque pixels from binary image. Finally, point cloud mapped from binary image is clustered into marking objects according to Euclidean distance, and using a series of algorithms including template matching and feature attribute filtering for the classification of linear markings, arrow markings and guidelines. Through processing the point cloud data collected by RIEGL VUX-1 in case area, the results show that the F-score of marking extraction is 0.83, and the average classification rate is 0.9.


INTRODUCTION
With the development of intelligent transportation, automatic driving and urban emergency response, the accuracy, comprehensiveness and timeliness of road information are required in both data processing and real time location based on service systems.The road's high-precision geographic information contains more detailed, levelled road information, such as markings, symbols and other related objects in road scene.Up to now, mobile mapping system integrates laser scanner, GNSS receiver, inertial measurement unit (IMU), and other related sensors to collect data in an efficient and low-cost way.The three-dimensional (3D) point cloud describes the surrounding environment derived from MMS, can also be used for road's high-precision information extraction.High-precision map, which is a typical type of high-precision geographic information data, has attracted much attention recently due to its potential usage in L3-L4 level automatic driving and Advanced Driver Assistance System (ADAS).Compared with traditional navigation map or topographic map, high-precision map contains road markings, traffic signs, traffic lights, curve curvature, pavement slope and other detailed lanelevel information.Among these different road's objects, the road markings play an important role in regulating vehicle driving, providing positioning reference and supporting behaviour decision-making, and is one of the most important element of a high-precision map.
Currently, most of the methods for extracting road markings from MMS point cloud mainly contain three steps, which consist of road surface segmentation, marking extraction and marking classification.The road surface segmentation can remove point cloud of the trees, buildings and pavement, which are irrelevant to road marking.It doesn't only decrease the amount of processing point cloud, but also it decreases the complexity and provides a reference plane for other objects simultaneously.The methods of the road surface segmentation are divided into two groups which are based on boundary feature detection and plane feature detection.The former method of the road surface segmentation was done through identifying the sudden change of point cloud attributions such as slope, point density, altitude, and reflectivity in boundary.There are some papers related to this method.For instance, Wang et al. (2012) constructed k-d tree for point cloud data, and recognized road curb by comparing altitude between laser points and adjacent points through k-d tree.Herná ndez and Marcotegui (2009) projected point cloud to the horizontal plane to generate the depth image and extracted the road boundary based on the height difference.Kumar et al. (2013) generated feature image according to the altitude, reflectivity and pulse width of point cloud, and identified road boundary by Gradient Vector Flow (GVF) Snake models.Xu et al. (2017) divides the point cloud into voxel, then defined the intensity, gradient based on point density in three-dimensional space, and the curb was identified by spatial gradient characteristics.Wang et al. (2015) projected the distance of the point's normal vector to the point cloud's dominant normal vector into a hyperbolic tangent function space, and constructed the saliency map to extract the boundary from the salient points.However, it is timeconsuming to process the point cloud in 3-D directly or to generate feature image of the point cloud.Some methods identify road boundary through analysing the geometrical characteristics of the road cross section, including the way of using scan lines and the pseudo scan lines which projects a certain thickness of the point cloud slice to the cross section (Fang and Yang, 2013;Guan et al., 2015a).The methods based on plane feature detection extract the road surface by identifying the flat area in the point cloud.For instance, Lam et al. (2010) used Random Sample Consensus (RANSAC) to fit plane for the point cloud of local region respectively, and then linked the local road surface.Yan and Zhang (2007) distinguished pavement and nonpavement through fuzzy clustering of point cloud's normal vectors.The growth algorithm is also used for road surface extraction by other authors (Cheng et al., 2017;Elberink and Vosselman, 2009).Similar to methods based on boundary detection, it is possible to identify the points of approximate linear distribution on the road section by the least square method (Yan et al., 2016;Yuan et al., 2010).Many of these methods can obtain reliable results in the segmentation of standardization hard pavement.
It is mainly based on the intensity difference of point cloud to extract the road marking from the laser point cloud.Because the reflectivity of the asphalt is low while the road marking has high reflectivity, therefore it is possible to extract road marking through the intensity information of the point cloud.The methods of extracting road marking information are also roughly divided into two groups.One is to transform the point cloud into feature image, and then utilize existed image process methods to extract marking (Kumar et al., 2013;Guan et al., 2014;Guan et al., 2015b;Yang et al., 2012).The other is to extract road marking directly from 3-D point cloud (Yan et al., 2016;Yu et al., 2015).Yang et al. (2012) preliminarily extract marking through isolating points, which have high intensity value, and filtering out points, which lies above the ground.Then they used semantic knowledge, such as shape and size, to extract marking from intensity image generated from the remaining point cloud data.The intensity of the laser echo signal attenuates with the increasing of the signal propagation distance and the incidence angle.It leads to the fact that the intensity of the point cloud is uneven.Therefore, it is difficult to obtain accurate and complete road marking results by using the single threshold image segmentation directly.Guan et al. (2014) noticed that the further away from the laser scanner, the lower density of the point cloud on the road surface.They proposed a method of multi-threshold segmentation based on the density of points.Firstly, they converted point cloud into intensity image by IDW interpolation.Then, the Otsu method was performed to compute intensity segmentation threshold based on different point density and the image was binarized with the computed thresholds.Finally, the morphological filter was applied to further modify the result.Cheng et al. (2017) proposed an intensity calibration model based on scanning angle, and used large-size high-pass filter to decrease intensity variance caused by the roughness degree of the road surface.Yu et al. (2016) divided the point cloud into several parts along the vertical direction of the vehicle trace and applied the Otsu algorithm to each part.Apart from Otsu algorithm, there are also other methods of determining intensity segmentation threshold.Kumar et al. (2014) proposed a method of determining intensity threshold values based on range to the scanner.Guan et al. (2015b) determined threshold through weighted neighbouring difference histogram and used multiscale tensor voting to extract road marking.Yan et al. (2016) used dynamic window median filter to smooth the intensity of point cloud, then the marking's edge is determined according to the gradient changes of intensity on the scan line.Finally, the extracted non-marking points were removed according to the characteristic of the marking.As we can see, for the majority of marking extraction methods, the critical procedure is to deal with the problem of uneven intensity.Some of those methods calibrate the intensity or use the multithreshold partition processing.However, it is not easy to acquire the accurate calibration model and the methods of region division are often affected with experience.
Marking classification is mainly based on the prior information such as geometric features, or combines with machine learning algorithm such as the neural network.For example, Soilá n et al. ( 2017) built the 47 dimensional geometric feature vector which including area, bounding ellipse, shape and pixel distribution to describe markings, and preliminarily classified markings into three types of rectangular, arrows, and other types using neural network.Finally, rectangle and arrow markings were further subdivided by semantic information and template correlation calculation.Yu et al. (2015) used semantic knowledge to classify the large size marking, and used the deep learning and principal component analysis to identify the small marking.Cheng et al. (2017) proposed a hierarchical classification tree, and used geometry parameters such as approximate width, area, orientation and minimum bounding rectangle to classify marking into zebra crossings, the broken lines, solid lines, road arrows, and the non-road arrows.Overall, most of current studies focused on marking classification based on a single object feature.It is still inadequate on studying the role of semantic knowledge, such as position relationship between different individuals and different categories, in the process of marking classification.
The main content of this paper include using boundary identification algorithm based on scan lines to extract road surface, using a local adaptive threshold segmentation algorithm based on integral image to extract the marking from uncalibrated intensity image.In addition, the extracted discrete marking points were clustered into individual marking and a series procedures such as template matching, feature attribute filtering were used to identify common linear marking, arrow marking and guideline marking.The detailed description of extracting road marking is arranged in the section 2. Section 3 introduces data processing of case area according to the proposed methods and analyses some factors that affect the results.Section 4 discusses some of the deficiencies and the direction of future improvement.

METHODOLOGY
In this paper, the marking information extraction includes three steps: road surface extraction, marking extraction and marking classification.Figure 1 is the flowchart of proposed method.

Point cloud Sclice
Step Most of the MMS collect point cloud data in the way of single line profile scanning and there is no echo signal in the sky area.
Therefore, there will be a distance leap between the adjacent points in the edge of two scan lines.The scan line can be extracted through identifying this distance leap.Let Pi(Xi,Yi,Zi) be the point cloud coordinate, Subscript i be the scanning order of point, di be the distance between Pi and Pi+1 and Ld be the distance parameter.When the Equation ( 1) is satisfied, the point Pi is marked as the key point and let Keyk=i.Let Ln and Lr respectively represent the minimum number of points that the single scan line owns and the minimum length of scan line.When the Equation ( 2) and ( 3) is satisfied simultaneously, the Keyk is marked as the leap point and let leapj=Keyk.leapj is the order number of edge points lying on the scan line.Finally, the scan line point is extracted by Equation (4).Lrcan be set to 1/2 of road width, Ld can be set to 1/3 of road width.Ln has a large relation with the density of the scan line points, and can be determined according to experiential Equation ( 5).
1 1 , , The structuralized road has the sudden change of geometric features such as elevation and slope at the boundary.Therefore, the pavement point can be separated by identifying the location of this sudden change.Let dhi be the height difference between adjacent points on the scan line, dxyi be the plane distance, dsi be the spatial distance, and slopi be the slope (see Figure 2-(a)).Then the three threshold parameters are cH, mS and fH.cH is related to point interval of road boundary protrusion (e.g., kerbstone, baffle, etc.).mS decides angle between protrusion and the pavement.fH reflects the degree of elevation fluctuation of the road surface.When one of the Equation ( 6), ( 7) or ( 8) is satisfied, Pi is marked as the key scan line point and let ScanKeyk=i.Then, the Equation ( 9) calculates the number of non-key points between adjacent key points.Because most scan line points are non-key points on the road surface, the non-key points group which owns the most number can be deemed as road surface points.In order to remove incorrect results which only include a small amount of points.Define a threshold minN.When the number of points is larger than minN, road point cloud extracted from scan line is effective, as shown in Equation (10).

Road Marking Extraction
The extraction of road marking is realized by image segmentation according to the intensity difference between asphalt pavement and marking paint.The process includes three steps of point cloud rasterization, binary segmentation and noise filtering.
Firstly, intensity image of road surface is generated through IDW interpolation with the radius of RIDW and the pixel size of Rpixel.
In order to avoid the black holes of null pixel, RIDW is supposed to be larger than the points interval.Usually let Rpixel be approximate to point interval and RIDW be 3Rpixel to 4Rpixel.
Then, the adaptive threshold segmentation based on integral image proposed by Bradley and Roth (2007), which does not need intensity calibration or subarea processing, is applied to segment marking.The integral image refers to the image that each pixel value is the sum of top left rectangular area of original image and it is convenient to calculate pixel value sum of arbitrary rectangular area of original image by integral image.
The adaptive threshold of each pixel is determined by the average of its surrounding pixels' grey value.Gaussian filter is applied to process intensity image, and let the original pixel be pi (xi,yi), corresponding grey value be I(xi,yi) and integral image be intI(xi,yi).We define two parameters s being background radius and t being segmentation coefficient.Then the sum of the grey value of pixels which lie in a rectangular area, the centre of which is pi and the length of which is 2s+1 (see Figure 3), is calculated as follows: The image boundary shall prevail when the rectangular area is beyond it.The binary image is as follows: There still will be some noises after image binarization.Most of these noises are isolating small plaques.We filter out the noise through removing the plaques, the number of the connected pixel of 8 neighbourhoods of which is less than Nnoise.Generally, the greater the value of s, the bigger the plaques noise is, and so is the value of Nnoise.

Classification of Road Marking
Extracted marking from the image segmentation is still point level or pixel level.It should classify markings into specific categories for obtaining the object level information.In order to reduce the mutual impact between different types of marking and improve processing efficiency, we firstly distinguish the linear marking, which has obvious characteristics and large proportion.
Then the arrows, guidelines and other markings were picked out from the rest of markings.In order to facilitate the use of 3D point information and keep the accuracy, the pixel marking were mapped into the point cloud.

Geometric Feature:
Because of the most single markings are not connected.The Euclidean distance clustering is applied to marking point cloud with the radius of r.Then the geometric feature attributions based on the minimum bounding rectangle (MBR) of each cluster is defined as follows.
Size (l,w).The size of marking is approximately represented by the length (l) and width (w) of its MBR.Length-width ratio (γ).γ=l/w.Length-width ratio reflects the degree of approaching the square.Centre(c).The centre of MBR represents the location of marking.Area of MBR(mA)).Area of marking (pA).The area of marking is calculated by its image pixel area.Rectangularity (η).η=pA/mA.Rectangularity represents the similarity between the shape of point cloud cluster and its MBR.Orientation (T).The azimuth angle determined by the connecting line of the two short side centre of MBR.included angle(α).The angle between the T and the trajectory of MMS.

Linear Marking Classification:
Linear marking includes broken line and solid line.Since each type of road marking complies with the national standard size, the linear marking can be identified by comparing cluster size with the standard size.Let lengthbrokenlineand widthbrokenline be the standard size of broken line and β1,β2 and β3 be three parameters that limit the geometric feature difference.The broken line is determined by the following discriminant.
The value of β1, β2 and β3 are related to the density of the point cloud and the integrity of the extracted marking.Generally, the values can be 0.1, 0.2, and 0.6.Classification of solid line is similar to broken line, but it can't be a strict limit on l, because the solid line's length is always different.The solid line can be specified by the following discriminant.
β4 is the limit parameter of length-width ratio which can be 7 in general.

Classification of Arrow Marking:
The shape determines the arrow marking's category, and it is difficult to reliably distinguish arrows based on MBR for the size similarity between different arrow markings.In addition, the description of point cluster's shape becomes complicated and unreliable because of the possible defect error in the segmentation of marking.We did not use the shape descriptors, but directly compared marking image with standard template and judging belonging categories by their pixel differences.Template matching is a simple and easily conducted image recognition method whose result is sensitive to matching position and direction.If matching for each pixel in each direction, it will be a time-consuming operation.However, the same type of marking image has the similar shape and the similar skeleton, which preserves the topology, position and direction of the original image.What's more important is that the number of pixels in the skeleton is greatly reduced compared to the original image.Figure 4 is the template of skeleton and marking.
Firstly, the skeleton of marking image generated from point cluster is extracted by thinning algorithm using mathematical morphology.Then, the iterative closest point (ICP) algorithm is applied to match marking skeleton with all of template skeletons.Moreover, matching the marking image with the template which has the highest skeleton matching rate.In fact, the matching position and direction of marking image determined by skeleton matching is not optimal.It need to make small traversal search on the basis of it (see Figure 5).Let Dadjust be the search range and Aadjust be the search angle.Move the image up and down at the speed of one pixel and calculate the matching rate by the method of Cheng et al. (2017)  Where TP, FN and FP are the number of correct pixels, defect pixels and overflow pixels.The best matching position is the place acquired the max Fscore.In the best matching position, the marking image is rotated clockwise and counter clockwise at one-degree resolution, and the Fscore is calculated again.The max Fscore at this moment is the final result of matching.The matching is successful if the Fscore is greater than the setting threshold.The matched image is consistent with the category of the template.Generally, let Dadjust be 10 pixels and Aadjust be 10 degrees.
In this paper, the limit coefficient of β6~β10 is 0.1m, 20, 0.12, 80, 0.1m 2 respectively.Then the filtered guideline marking is clustered with the radius of 1.5m, because defects in the segmentation may lead one guideline to be divided into several parts.

Case Area and Data
In order to verify the validity of the proposed method, the four point cloud datasets were collected in case area, 2nd ring line, Wuhan, China on April 18,2017 (see Figure 6).The total of points were 120 million at the length of around 10 km.The road of case area is two-way four to six lanes with the width of 15 m, and the pavement condition is good with complete and clear road markings.The collection parameters are shown in Table 7.

Result of Marking Extraction
The parameter s and t have significant influence on the segmentation results.We selected four 60 m data samples from case area and generated intensity image with the RIDW being 0.2 m and Rpixel being 0.05 m (see Figure 8).The manual annotation result is assessment reference.Then the intensity image is processed with adaptive threshold method and Fscore is calculated by different parameters of s and t (see Figure 9).Figure 10 is the extracted marking of sample area with their own max Fscore.Table 11 is the evaluation of extracted marking of samples.By analysing the segmentation parameters of the sample data, the adaptive threshold segmentation of all case data is determined by s=12 and t=1.25.Table 12 shows the comparison with the other methods referenced in Cheng et al. (2017).(

Result of Classification
We classified the extracted marking of case data by the proposed method.Table 13 was the evaluation of broken line and solid line.Extraction rate is the ratio of the length of extracted linear marking to the total.
Table 14 was the evaluation of arrows.The total number of arrows were 55, detected number were 46 and error detection number were five.completeness correctness F_score 0.84 0.9 0.87 Table 14.Evaluation of arrows of case area There were eight guidelines in the case area.All of those were detected and none of error detected.However, classified guideline might exist defect or overflow for segmenting and clustering error.So we compared detected guideline to manual annotation pixel by pixel and Table 15 was the evaluation of guideline of case area.completeness correctness F_score 0.98 0.93 0.95 Table 15.Evaluation of guideline of case area

CONCLUSIONS AND DISCUSSIONS
This paper uses a method of extracting road marking information from point cloud based on intensity.Firstly, the paper identifies the boundary from the scan line and segment out the road surface, and then local adaptive segmentation method based on integral figure is used to extract marking from uncalibrated intensity image.Finally, the extracted marking points are clustered into marking objects, and common linear markings, arrow markings, and guidelines are identified through template matching and feature attribute filtering.The processing result of the experimental data shows that the F-score of marking extraction is 0.83, the average marking classification rate is 0.9.In the comparison to the existing methods of road marking information extraction, the proposed method directly apply adaptive threshold segmentation to extract marking from intensity image, which avoids complexity in the process of intensity calibration and area partition.Then, this paper also presents an efficient and convenient algorithm of arrow classification based on skeleton matching.Certainly, there are also some improvements to be done.For instance, the parameters s and t are not adaptive in the process of adaptive threshold segmentation.The categories and the number of identifying markings are not so much.The classification process only utilize simple prior information, failing to adopt more advanced and more complicated semantic knowledge.However, for marking information extraction, both the global adaptive threshold segmentation and semantic knowledge worth exploring.More focus will be placed on the relevant question in the future.

Figure 1 .
Figure 1.Flowchart of proposed method

Figure 2
Figure 2. (a) Geometric features of scan line, and(b) example of extracted road surface Figure 3. Adaptive threshold segmentation

FigureFigure 5 .
Figure 4. Marking template and skeleton template Figure 6.Case area Figure 8. Intensity image of sample area

Table 7 .
Point cloud collection parameters

Table 11 .
Evaluation of extracted marking of samples.

Table 12 .
Comparison with the other methods.