FAST DRAWING OF TRAFFIC SIGN USING MOBILE MAPPING SYSTEM

Traffic sign provides road users with the specified instruction and information to enhance traffic safety. Automatic detection of traffic sign is important for navigation, autonomous driving, transportation asset management, etc. With the advance of laser and imaging sensors, Mobile Mapping System (MMS) becomes widely used in transportation agencies to map the transportation infrastructure. Although many algorithms of traffic sign detection are developed in the literature, they are still a tradeoff between the detection speed and accuracy, especially for the large-scale mobile mapping of both the rural and urban roads. This paper is motivated to efficiently survey traffic signs while mapping the road network and the roadside landscape. Inspired by the manual delineation of traffic sign, a drawing strategy is proposed to quickly approximate the boundary of traffic sign. Both the shape and color prior of the traffic sign are simultaneously involved during the drawing process. The most common speed-limit sign circle and the statistic color model of traffic sign are studied in this paper. Anchor points of traffic sign edge are located with the local maxima of color and gradient difference. Starting with the anchor points, contour of traffic sign is drawn smartly along the most significant direction of color and intensity consistency. The drawing process is also constrained by the curvature feature of the traffic sign circle. The drawing of linear growth is discarded immediately if it fails to form an arc over some steps. The Kalman filter principle is adopted to predict the temporal context of traffic sign. Based on the estimated point,we can predict and double check the traffic sign in consecutive frames.The event probability of having a traffic sign over the consecutive observations is compared with the null hypothesis of no perceptible traffic sign. The temporally salient traffic sign is then detected statistically and automatically as the rare event of having a traffic sign.The proposed algorithm is tested with a diverse set of images that are taken in Wuhan, China with the MMS of Wuhan University. Experimental results demonstrate that the proposed algorithm can detect traffic signs at the rate of over 80% in around 10 milliseconds. It is promising for the large-scale traffic sign survey and change detection using the mobile mapping system.


INTRODUCTION
The detection of traffic signs is an important part of intelligent transportation system.Based on a series of images we get from the Mobile Mapping System (MMS), we focus on finding and identifying the location of potential traffic signs.
Based on the previous EDLine and EDCircle algorithm, a new and fast method is proposed which aims at fitting the circles or ellipses in the images and identifying the possible traffic signs in the images.
Many researchers have made many efforts in recent years to accomplish the traffic sign detection.An algorithm to detect circle shapes in real images based on Harmony Search algorithm was proposed by Erik Cuevas et al (Cuevas et al., 2012a), who assume that The algorithm uses the encoding of three points as candidate circles (harmonies) over the edge-only image.Meanwhile, Erik Cuevas also put forward an algorithm based on artificial immune systems which aims for multiple circle detection (Cuevas et al., 2012b).The core of their idea is Using a combination of three non-collinear edge points as parameters to determine circles candidates.These two papers advocate the basic principles to find circles in the images.But in the same time, there are also many researches about traffic signs detection.Ming Liang et al (Liang et al., 2013) in Tsinghua University present a model consisting of two modules.The first is for ROI(region of interest) extraction and the second is for recognition.It validates if an ROI belongs to a target category of traffic signs by supervised learning.J. Stal-lkampa and his team also put forward their idea that Benchmarking machine learning algorithms for traffic sign recognition was the best-performing one in the state-of-the-art machine learning algorithms (Stallkamp et al., 2012).At last the basic algorithm of our methods, EDCircle, which we will explain in details in latter part.The core idea of it is a fast and parameter-free circles detection method (Akinlar and Topal, 2013).Therefore, in this paper we present two innovations over the existing methods: Firstly, the model we put forward is the combination of the primary feature of color and the secondary feature of shape.As you can see, nowadays many traffic sign detection methods are based on machine learning which takes a lot of time to process.So based on the simple EDcircle, our method performs more effectively.Combined with shape detection and color analysis it becomes much more quickly and precisely.The second point is the coarse prediction and set up the bond among the consecutive frames.The same object tends to maintain consistent colors and remains in the same spatial area in contiguous frames.So we come up with an improved coarse prediction method based on Kalman filter (Weng et al., 2006) to predict the same object in the consecutive frames.Differing from the traditional matching based on correction coefficient, it behaves much more quickly.If there are some traffic signs appearing in the video at the same time, we can quickly separate them and lock each one automatically among the following frames, which makes it much more easier to analyze its meaning without confusing them.

Calculate the Color Histogram of Traffic Sign
Traffic signs have certain color combinations and regular shapes.colors are firstly taken into considerations to extract the regions of interest (ROIs) from an image.In this way we can concentrate on the area where there is a high possibility to have a potential traffic sign rather than going through the whole image.It saves a lot of time.So the main points here are: 1)what color should be chosen and how to choose them to reflect the features of traffic signs in the image without leaving out any one?
2)how to extract the area from the candidate color points?
For the first question: how to choose the proper color, in this research w'd like to take color histogram as a solution.Obviously it's also essential to decide which color space should be taken into consideration.Compared with original RGB color space, HSV has its own advantages.In different conditions such as different lighting intensity or complicated background or different photographing angles, it's much more visual to describe the same color while RGB can't.Based on the priori knowledge of several colors used int traffic sign, we set up a HSV color index for the standard traffic signs we usually take in the streets.
which A represents the HSV index of certain kind of standard traffic sign and ai stands for the certain color.For each kind of traffic sign we can set up an index for them.Every ai is taken with certain threshold which we will discuss the best threshold in details in later part.Then we count the sum for each ai to establish the corresponding color histogram.The picture above is the color histogram we get for our target.a1and a2 are the main components anda3 can be regraded as the noise(background)color.The color element a k with the highest P(possibility) is regarded as the Feature Color AI and this is the color we will use below.
According to the feature color we got,we can find all the pixels which are in this color.But they still seem random and irregular.
Here comes to the second problem:how to extract the region from the found color points?There is no doubt that these points must have spatial features.Based on clustering, we divided the points into several classes in terms of the distance among the points.In the same way it's essential to talk about the details.At first we assume the first point as the first class.Then we calculate the distance between the next point and all the central point of existing classes.If the minimum distance exceeds the threshold(after many experiments we find the proper value as d = (W + H)/6, which W,H stands for the width and height of the image) then it belongs to a new class.Otherwise it belongs to the nearest class.After all the points have been classified, we check all the classes whether it has enough points(consider the scale of image we choose 0.1*N as the threshold) and the classes which are large enough can be kept.As long as we found the proper classes, it's easy to extract the corresponding ROI from current image knowing the extreme points.Eventually, we find the ROIs and cut down the processing time.From the figure below it's easy to Figure 2: The Interest Areas Extracted By Our Algorithm find that the bottom region is the error ROI we extract in color analysis.Some improvements can be proposed to make it extract precise ROI.Thus, the best threshold will be discussed in the later part.

Draw the Edge of Traffic Sign
Edge drawing can be divided into four different steps: 1).Firstly, process the image by some filter such as the Gaussian to filter the noise in the image.2).Secondly, compute the gray scale gradient of each pixel.In this part we can use many mature methods such as Sobel or Prewitt etc. 3).And some pixels which have the local maximum gray gradient in the neighborhood are set as the special points(SP) in the image.And these points mean the pixels have very high probability of being part of an edge.4).After we have confirmed all the special points, we get to connect all the SPs dot by dot.From the step 2 we can get the gradient magnitude and its corresponding direction.To be faster, we only choose horizontal or vertical neighborhood for a pixel.Starting from a SP, Edge drawing get its neighbor pixel's direction and go on(meanwhile record the coordinates of the path).And it will stop when the next dot's direction is different or the next dot is another SP.So in this procedure we can quickly get the path of the edge.And after all the SPs has been checked, the edge segments of this image could be got.
According to our experiment, after the edge detection, there are many finely edges which severely affected the following procedure in the relatively complicated images.So a simple way is chosen to erase these meaningless edges to speed up the processing rate.Depending on the length of the edges, we choose to keep or remove them.Longer it is, more information it will take.Assume that there is a N*N image and an edge whose length is L. If L*L/2 ≥ N*N/100,we think it is a meaningful edge.Otherwise, just abandon it.

Fit Traffic Sign Shape
After the image de-nosing, the chains of pixels calculated by EDdrawing algorithm (Topal et al., 2010) can be used.With the chains of contiguous pixels, in this part, we mean to split these chains to one or more straight line segments with EDLine algorithm (Akinlar and Topal, 2011).According to our first idea, use a certain number of pixels in the sequence and fit a line to these pixels by using the least squares fitting methods.Then compute the distance between this current line and the next pixel.If the distance exceeds a threshold, finish this turn and create a new line segment.If not, add this pixel to the previous ones, fit a new line with the updated pixels to get a more accurate fitting line and loop again (add the next point).The algorithm stops until the distance exceeds a threshold or this chain of pixels has been traversed and processed.
Figure 5 shows the results processed by our algorithm.Figure 6 shows the algorithm used to extract the line segments.found that many little circle arcs are only divided into 2 line segments.So with this new definition more little circles and details are kept.Here, there are another two main limitations to detect the arcs: the first one is the angle between the two consecutive lines and the second one is the ratio of the two segments length.
Given the lines making up the edge segment, calculate the angle between these two consecutive lines and their turning direction by the coordinates.If at least two lines turns in the same direction and the angles among them dont exceed a certain threshold, then calculate the ratio of these two (or more) segments length.
If the ratio is close to 1, they may form an arc.Subsequently, by coordinates in the consecutive lines and the least squares fitting methods, fit the arc and add it to the list of arcs.To be more targeted and faster, we further divide the arcs into circle-candidate arcs and ellipse-candidate arcs by using FitCircleArcs at first to judge the arcs.If the error doesnt exceed the threshold, add it to the circle-candidate arcs list.If not use FitEllipseArcs to check whether it belongs to ellipse-candidate arcs list.Eventually, split all arcs to these two specific types of arcs.Ultimately, all arcs are classified.

Find Circles based on clustering
Following the last step, there are two lists of arcs (circle-candidate arcs and ellipse-candidate arcs).It's obvious that all the circles are combined with the arcs in the lists.And we consider that longer the radius of an arc is, higher the chance of it belonging to a circle (or ellipse) will be.So in this part two assumptions are put forward to shorten the fitting process: Firstly, cluster all the circle-candidate arcs according to the similar center point.Secondly, for every circle cluster, sort the arcs based on descending order of the radius of each arc in the cluster to prepare for the secondary cluster.After these two steps, we can fit all the circles in the image without walking over and computing every arc in every loop which save a lot of time compared with the previous algorithm.For a certain circle-cluster, arcs can be extended under several criterions.Given Arc A1 to be extended, the candidate arcs radius should be within 25% of A1's.
Then check the distance between A2's end-point and A1's beginpoint which shouldnt exceed double radius and the fit error by  Here from many observations, we find that if there are more than one arc in the perfect circle, its better to join the arc from a specific direction rather than randomly .So in our algorithm we define all arcs' directions to be anticlockwise.When all the candidate arcs has been found, if A1's central angle covers at least over 50% of the circumference of its corresponding perfect circle, we make A1 a circle candidate fitted by the least squares fitting methods.If not, the arc is left for ellipse-candidate arcs.And all remaining arcs will be computed by FitEllipseArcs.If within the threshold, they will be added to ellipse-candidate arcs.

Find Ellipses based on clustering
It's obvious that all the steps above can only find the nearly perfect circles.However, in many frames we get, the perfect-circle traffic signs will show as ellipses.So we still try to fit ellipse perfectly.From the ellipse-candidate arcs we get in last step, we can also cluster them into different parts according to similar center point.Then we reorder the arcs based on descending order of the major axis of each arc in every cluster.Given Arc A1 to be extended, the candidate arcs major axis should be within 25% of A1's.Then check distance between A2s end-point and A1s begin-point and it should not exceed double major axis and the fit error shouldn't be large.This can be regarded as the secondary cluster.Then A2s central angle is added to A1s and substitute A1s end-point with A2s end-point.Here from many observations, we find that if there are more than one arc in the perfect ellipse, its better to join the arc from a specific direction rather than randomly .So in this part we still define all arcs directions to be anticlockwise.After all the candidate arcs has been obtained, if A1s central angle covers at least over 50% of the circumference of its corresponding perfect ellipse, A1 is regarded as an ellipse candidate fitted by the least squares fitting methods.Here, we use similar methods used in last step to fit ellipse.So we wont show the same parts.

Fast Traffic Sign Detection and Prediction
After all the details about color and shape have been mentioned, it's rational enough to combine them together.For the first two frames we got from the MMS, it's necessary to go through the whole image to extract the interest area.We can record its corresponding central point and boundary.Subsequently, we only need to detect the edges and fit circle or ellipse in the interest areas.In this way, it sharply decrease the quantity of data needed for edge detection and shorten processing time.And we deem that it's rational to regard the ellipse or circle fitted in this way as real the traffic signs.Becasue it's nearly impossible to find a confusion item which is same as traffic sign both in color feature and shape.
However, we still can't absolutely confirm it.To be more precise we use a statistically coarse prediction as the double-check to reconfirm the existence of traffic sign.The event probability of having a traffic sign over the consecutive frame can be detected statistically.So here the proposed method based on Kalman filter has been taken into consideration.
where X(k + 1|k) represents the prediction results in k+1 state(means for the next frame).X (k|k) stands for the optimal result in k state(means the present frame).U (k + 1) represents the controlled quantity in k+1 state while A,B are the parameters of the model.And the A,B can be calculated out from the first frame in the video sequence.
where P (k + 1|k) represents the corresponding covariance of X(k+1-k) while P (k|k) stands for the covariance of X(k-k).Q stands for the system covariance.And equation(3)(4) express the prediction for this consecutive system.
(5) here X (k + 1|k) stands for the optimal result in k+1 state and X (k|k) stands for the optimal result in k state.Z(k + 1) means the observation value(initial values we use the central points of ellipses in present image.If in the area we can't fit a ellipse or a circle we use the central points of interest area.In following loops we use the optimal result) while Kg(k + 1) is the Kalman Gain which can be calculated as: After these, we have got X (k + 1|k), the optimal result in k+1 state(which means the prediction coordinates of corresponding central points in the next frame).Then we have to compute the P (k + 1|k + 1) just like below: In every loop(from kth frame to k + 1th frame),we can get a optimal estimated result(predicted central point of the traffic sign in the next frame).Considering the effect of accumulative error, we mean to correct the error in every loop.Started from optimal estimated center,a relatively big area will go through the color analysis and extract a color core.Then based on this color core, we run over color analysis and shape detection in a relative large area.If there is an accepted result in the area, then we regard this frame as a success otherwise it's a failure.Then if it fails to find a traffic sign, the searching area will be expanded in the next frame and the observation value stays invariable.If it succeed finding one, the area won't change and the observation value will be replaced as the new central point.Thus in a consecutive frame sequence, we can count the sum of success and failure.
However, there is a problem needed to be discussed.Fristly, as we know, Kalman filter is quite unstable at first because of the ini- where P is the optimal estimated central point computed from last frame.A is the color core got in the first window.Then set A as the center and move the window to A, it will cover more color information and behave much better.These are the line charts of X,Y coordinates calculated by our Kalman filter,where red line stands for the measured value while blue line represents the prediction value.And as you can see,the prediction result is stable and relatively precise.In this video there are total 90 frames in the sequence containing the traffic sign and we pick out all the error frame(can't detect a circle, detect multi circles or lost the main part)as the failure.The failure counts up as 2 where the X changes sharply(the peaks in the chart).So the Confidence can be computed as Con = 1 − (f ailure/total)and Con=88/90=97.8%.So after this double check is finished and we can confirm the existence of traffic sign.We test our coarse prediction in a video and its result turns out to be figure 14.

EXPERIMENTAL RESULTS
To measure the performance of our algorithm, a diverse set of images are taken in Wuhan, China with the MMS of Wuhan University.It includes traffic signs of different types and colors.And the accuracy and detection speed are presented in this section.

Color Analysis
At first, the choice of color space is a really problem.Salient features should be precise and clear enough.The experiments about RGB and HSV color space are shown below: Because of the Figure 15: Different HSV Threshold Results different threshold,there are a lot of difference shown from the results.In terms of HSV, we only take H,S(represent the color and its purity)into consideration.Such as 01 and 02,the values of H both are 0to30 330to360,a relative narrow interval for color red.But they differ from S,01 is 0.28 while 02 is 0.5,which means the red found in 02 is purer than 01.So it's clear that there are lots of error points in 01 while the color points detected in 02 are incomplete.Then we test 03 and 04(exchange the S of 01 and 02,while H is 0 to 60 300 to 360).So we can draw the conclusion that larger the interval is,more error points will be found out.And S should be higher as much as possible,which means the red we find will be more close to real red.Then it comes to RGB.Because we can't stimulate the color directly in RGB with the way we analyze the color in our mind.It's quite blind for us to choose the restrictions.At first, we test 01(R120 G120 B120) and results are quite terrible and meaningless.Therefore we add the R value and decrease G,B value step by step such as 02(R125 G115 B115) 03(R135 G105 B105) 04(R145 G95 B95).As the images reflect, less and less points will be found out.In 04 nearly half of the sign has been left out.Therefore we should take a medium value for RGB space and 02 is our best choice.But it's still hard to judge which space is better.However,in videos, the traffic signs in frames are often affected by different illumination intensity and its color will become quite different.Figure 15 shows the results in low illumination while figure 16 in a higher intensity.Figure 01   And what threshold should be chose to divide the segments?In this part we have tried a lot of experiments.And we pick some salient cases shown in Table 1.where SumP means all the points detected under this threshold while SumACP stands for the sum of pints belonging to the main class.Classification Accuracy is equal to SumACP /SumP which shows the effect of noise.And we precisely got the sum of feature color points is 4435 and the ColorIntegrity calculated as SumACP /4435 illustrates the error classified points ratio.And it should be as close as possible to 100%.After comparing all the cases and we can find case 5 is better.
So it demonstrate that the most proper value of H should be 10-30/300-330 while S is 0.23-0.28.It may differ under different situations.

Shape Analysis
After color analysis the behaviour of shape detection should also be evaluated.And we use two video sequences as the sample data.
As figure 19 illustrates,there are still some ellipses can't be detected in the frame.It's obvious that the eccentricity of an ellipse will be quite large from a large photographing angle.We believe that it's the large curvature that divide the curve to more segments leading the algorithm to abandon them.It's one of the weakness in our algorithm and we are still trying to improve its performance and acquire corresponding largest photographing angle.Combined with the geometrical information from MMS, we can even compute out the limitation angle to fit the ellipse.In terms of figure 20, in more complicated situation, our algorithm doesn't behave that well.It may miss the main part or may detect many wrong circles.We think there are two possible explanation: 1.The prediction area is not big enough to cover the traffic sign.Thus only parts of the edge have been detected.And they are

Frame Sequence Analysis
In this part we mean to present its performance in a consecutive frame consequence.And it turns out to be quite precise and fast.
In the first sequence it takes nearly 10 milliseconds per frame.Thus, it proves to be fast enough to be real-time algorithm.The traffic sign have been locked and tracked at the beginning.But it performs not that well from a larger angle.Thus, its performance meets our basic expect.

CONCLUSIONS AND FUTURE RESEARCH
This paper has introduced a real-time and precise algorithm based on color analysis and shape detection to find the traffic sign in the frame sequence without any restriction on prior experience.We have contributed to the state-of-the-art in two areas: 1)We combine color analysis and shape detection.In many other paper they only focus on just one field.But with help of the first feature-color, we can locate the potential area and decrease the computing quantity.Then in a smaller area, we use secondary feature-shape detection to find the ellipse or circle.Therefore its behaviour can be precise and swift.
2)Coarse prediction based on Kalman Filter and double-check statistically.Only simple color and shape conditions are not convinced enough to confirm the existence of traffic signs.So we put forward a rough Kalman filter prediction.Through the coarse prediction,we can set up the bond between the current frame and following frame, i.e we can use the center in this frame to predict its location in next frame and meanwhile it will accelerate the process.We can count the sum of success and failure frame and compute the confidence coefficient to double check the existence of traffic signs.
We have compared our method with other existing algorithm and observe its performance.A lot of experiments have been taken and its efficiency has been validated.

Figure 1 :
Figure 1: The Histogram of Standard Traffic Sign

Figure 3 :
Figure 3: The Diagram of the Procedure of Edge Drawing

Figure 5 :
Figure 5: Traffic Sign Processed by EDLine

Figure 6 :
Figure 6: Algorithm to extract line segments from a pixel chain.

Figure 7 :
Figure 7: Algorithm to cluster arcs from available line segments.

Figure 8 :
Figure 8: Arcs detected by our algorithm

Figure 9 :
Figure 9: Algorithm to fit circles from every circle cluster

Figure 11 :
Figure 11: Schematic diagram of correcting error

Figure 12 :
Figure 12: X coordinate of predicted point and measured value

Figure 16 :
Figure 16: Different RGB Threshold Results 02 are

Figure 17 :
Figure 17: Results Obtained By Different Color Space In Normal Light

Figure 19 :
Figure 19: Error Ellipse detection result by our algorithm in frame sequence

Figure 21 :
Figure 21: Several frames in second sequence

Table 1 :
Data to find the best threshold in HSV space