AUTOMATED VISIBILITY FIELD EVALUATION OF TRAFFIC SIGN BASED ON 3D LIDAR POINT CLOUDS

: Maintaining high visibility of trafﬁc signs is very important for trafﬁc safety. Manual inspection and removal of occlusion in front of trafﬁc signs is one of the daily tasks of the trafﬁc management department. This paper presents a method that can automatically detect the occlusion and continuously quantitative estimate the visibility of trafﬁc sign cover all the road surface based on Mobile Laser Scanning (MLS) systems. The concept of trafﬁc sign’s visibility ﬁeld is proposed in this paper. One of important innovation of this paper is that we use retinal imaging area to evaluate the visibility of a trafﬁc sign. And this makes our method is in line with human vision. To validate the reasonable and accuracy of our method, we use the 2D and 3D registration technology to observe the consistence of the occlusion ratio in point clouds with it in photo. Experiment of implementation on large scale trafﬁc environments show that our method is feasible and efﬁcient.


INTRODUCTION
Traffic signs are often occluded by growing trees, road facilities, and buildings etc.This leads to a great potentially dangerous for traffic safety, and increases the probability of traffic accidents.Until now, the method of checking and removing the occlusions in front of traffic signs by transportation department mainly rely on by manpower.As far as our best acknowledgment, there are not exist an automatic method to detect the occlusion and estimate the visibility of traffic sign cover all the road surface within a sight distance.How to detect the occlusion and evaluate the visibility of traffic signs accurately and efficiently in a large scale traffic environment is a challenging problem.
The visibility of traffic signs is affected by (1) traffic sign itself geometric factors, such as placement of the sign, mounting height, tilt, aiming direction, depression angle, shape damage, etc. (2) Traffic sign surroundings geometric factors, such as road facilities, buildings, etc. (3) Human factors, such as vision, attention, the age of the drivers, cognitive burden of traffic density (Costa et al., 2014), vehicle speed, Geometric Field Of View (Mourant et al., 2007), line of sight, etc. (4) other factors, such as weather conditions (Belaroussi , Gruyer, 2014), lighting (Rogé et al., 2004), etc.This research focuses on traffic sign itself geometric factors and traffic sign surroundings geometric factors.
Existing research on the evaluation of traffic signs' visibility is based mainly on simulator and image methods.Simulator based methods (Lyu et al., 2017, Motamedi et al., 2017, Li , Zhang, 2017) evaluate the visibility or recognizability by observing the simulated traffic environment in the screen.Those methods * Corresponding author can cover all of factors affected visibility.However, they cannot evaluate the visibility of a traffic sign in a real road.Image based methods (González et al., 2011, Doman et al., 2014, Khalilikhah , Heaslip, 2016) uses images to instead of human vision.Those method considered the two geometric factors affected visibility.However, they are limited by fixed viewpoints, thus cannot evaluate the visibility over the whole road surface.So far, there is no effective solution applied on real road to evaluate the visibility of traffic signs.
The appearance of Mobile Laser Scanning (MLS) systems that can proved accurate 3D point clouds on a large-scale traffic environment makes it possible to solve this problem.In this paper, we evaluate visibility of a traffic sign at a viewpoint in 3D point clouds according to human retinal imaging principle.We presented a traffic sign visibility evaluation model, proposed and automatic algorithm to detect occlusion and estimate visibility of a traffic sign, introduced a conception of visibility field.Experiment shows our method is reasonable and efficient.

RELATED WORK
Simulation-based methods.Sun et al. (Sun et al., 2011) recruited volunteers for visual cognition time in a driving simulator.The UTDrive platform used in (Li , Busso, 2015) investigates driver behavior associated with visual and cognitive distractions.Nengchao et al. (Lyu et al., 2017) evaluated traffic safety by analyzing the driving behavior and performance of recruited drivers under a cognitive workload.Motamedi et al. (Motamedi et al., 2017) used BIM enabled Virtual Reality (VR) environments to analyze traffic sign visibility.Some researchers (Bohua et al., 2011) (Li , Zhang, 2017) used eye tracker equipment to determine the visual cognition of the traffic signs under simulated driving conditions.These methods do not focus on the quantitative value of visibility.Most importantly, these methods cannot be applied on real roads to estimate the visibility of traffic sign.
Image-based methods.As part of nuisance-free driving safety support systems, Doman et al. (Doman et al., 2010) proposed a visibility estimation method for traffic signs by using image information.To compute visibility of a traffic sign, they used different contrast ratios and different numbers of pixels in the occluded area of an image.They improved their method by considering temporal environmental changes (Doman et al., 2011) and integrated both local and global features in a traffic environment (Doman et al., 2014).Belaroussi et al. (Belaroussi , Gruyer, 2014) investigated the effects of reduced visibility from fog in traffic sign detection.These methods limited by viewpoint position, cannot continuously evaluate visibility over an entire road surface.
Point clouds-based methods.Katz et al. (Katz et al., 2007) proposed a Hidden Point Removal (HPR) operator to obtain visible points from a given viewpoint, applied it to improve the visual comprehension of point sets (Katz , Tal, 2013), and studied the transformation function of the HPR operator (Katz , Tal, 2015).Huang et al.(Huang et al., 2017), based on the HPR operator, studied traffic sign occlusion detection from a point cloud.They considered occluded distribution and an occlusion gradient.However, other important factors in the visibility of traffic signs, including proportion of the occluded area, influence of vehicle speed, road curvature, number of lanes, etc., have not been considered.Besides, the HPR operator cannot detect all the occluding points when the occluding point clouds are composed of multiple objects.

METHOD
In this section, we will present the detail content of definition of visibility evaluation model in section 3.1, definition of visibility field in section 3.2, and automatic traffic sign occlusion detection and visibility evaluation algorithm in section 3.3, respectively.

Definition of Visibility Evaluation Model
In this model, we only consider the traffic sign itself geometric factors and the traffic sign surroundings geometric factors.In order to make the visibility is consistent with human visual recognition theory, we use the principle of retinal imaging to consider the impact of the geometric factor.The retinal imaging area of objects can be calculated in point clouds.
For the visibility of traffic sign from the j-th viewpoint in i-th col, we use geometric factors of traffic sign itself E self i,j and surrounding factors E sur i,j to construct the visibility evaluation model.The viewpoint visibility of a traffic sign, E sign i,j , is defined as follows: 3.1.1Traffic sign itself geometric factors evaluation For a traffic sign at a fixed viewpoint, various changes in its geometrical factors affected visibility can be comprehensively represented by its retinal imaging area at this viewpoint.For example, the wrong orientation of the traffic sign can cause the its imaging area to become smaller, defective traffic sign can make its retinal imaging area smaller, the difference in the installation height of the traffic sign can cause a corresponding change in the imaging area of the retina.Therefore, calculating retinal imaging area of a traffic sign in point clouds is a reasonable way to evaluate traffic sign visibility affected by itself geometric factors The evaluation of traffic sign itself geometric factors, E self i,j , is given as follows: where the A unit type is the retinal imaging area of a standard traffic sign viewed from the "unit viewpoint".The standard traffic sign has the same type with the traffic sign being evaluated.The "unit viewpoint" is a viewpoint having a unit distance d unit to the panel, and in the normal line passing through the center of the panel.To ensure E geo i,j ≤ 1, we set the unit distance, d unit , at less than 3 meters.It is unnecessary to compute the visibility of a traffic sign when the observation distance is less than three meters.Due to the vehicle almost passing through the traffic sign, it is highly impractical (almost impossible) for the driver to turn around 90 • to observe the traffic sign.

Traffic sign surroundings geometric factors evaluation
A traffic sign may occluded by the road facilities, plants, and near buildings, thus affected its visibility.The occlusion ratio has a crucial impact on the visibility of traffic signs.Apart from considering the ratio of the occluded area, we introduce an occlusion distribution factor as P. Huang et al. (Huang et al., 2017) did.The evaluation of the occlusion degree E occ i,j is given as follows: where the first term is the occlusion ratio, and the last term is the occlusion distribution under a fixed occlusion ratio.
The occlusion ratio in the last item is multiplied because the occlusion ratio restricts the occlusion distribution.If the occlusion ratio equals zero, the occlusion distribution must equal zero.
After adding the penalization weight, λ, the visibility evaluation of the occlusion factor, E sur i,j , is given as follows: when the degree of occlusion, E occ i,j , is constant, E sur i,j decreases as the penalty parameter λ increases; when the penalty parameter, λ, is constant, E sur i,j decreases as the degree of occlusion, E occ i,j , increases.To ensure E sur i,j is nearly zero when E occ i,j is nearly one, and the traffic sign cannot be recognized under the situation of half occlusion, λ must satisfy the condition: λ ≥ 6.

Definition of Visibility Field
Visibility Field: for a given surrounding around a target object, the visibility distribution of viewpoints in a 3D space constitutes a visibility field.The visibility field reflects the visible distribution around a target object in 3D space.Take a traffic sign as an example.The visibility field of its hemispherical space is shown in Fig. 1.The traffic sign (yellow) and distribution of its viewpoints (white points) are shown in Fig. 1

Automatic Traffic Sign Occlusion Detection and Visibility Evaluation Algorithm
In this part, we first introduce how to prepare the input data.
And then, we introduce how to compute the retinal imaging area of a given point clouds.Finally, we give the automatic traffic sign occlusion detection and visibility field evaluation algorithm.

Input data preparation
Input data includes the detected traffic sign in point clouds, viewpoints, traffic sign surrounding point clouds.We detect and classify the traffic signs by the method of Wen et al. (Wen et al., 2016).We get viewpoints from the trajectory of MLS system.According to the location of detected traffic sign, a section of trajectory within its sight distance is segmented.The points in the segmented trajectory are sampled and shifted to left and right to get multiple trajectories.The height of points in the shifted trajectories is modified to 1.2 m above the road.Finally we get the viewpoints covered road surface.The traffic sign surrounding point clouds indicate the point clouds which along and above the road, and at the same side with traffic signs.The traffic sign surrounding point clouds can be segmented along trajectory by octree segmentation method.

Retinal imaging area calculation Traffic sign
Retinal imaging area calculation.The first step in computing the retinal imaging area is to rotate the coordinates of the input data into the human view.For a group of input data: traffic sign panel point cloud, traffic sign surrounding point clouds, viewpoints, we first translate the origin of coordinate system to traffic sign panel center, by subtracting the coordinate of the panel center; and then rotating their z-axis coordinates to the translated line from the traffic sign panel point cloud center to the viewpoint using the quaternions rotation method (Kuipers et al., 1999).The rotary axis is the vector of the cross product result between the z-axis and the translated line from the traffic sign panel point cloud center to the viewpoint.The rotary angle is the angle between the z-axis and translated the translated line from the traffic sign panel point cloud center to the viewpoint.The following operating about the a group input data in the paper are based on the coordinate transformed point clouds.Traffic sign panel point cloud is projected onto XOY plane, then, we get the projected traffic sign point cloud.
The outer boundary of the projected traffic sign point cloud is computed by the alpha shape algorithm (Edelsbrunner et al., 1983).The alpha parameter in (Edelsbrunner et al., 1983) is set to about twice the interval between points.We use the polygon area formula to compute the area of the projected traffic sign point cloud.Finally, the area is mapped to the retinal imaging area using the human retinal imaging principle (Kaiser, 2009).
The distance from the center of the entrance pupil of an eye to the retina is set at seventeen millimeters.The coordinates of the surrounding point clouds and traffic sign point clouds are rotated into vertical to the line from viewpoints to sign center.The rotated traffic sign point clouds area is computed by alpha shape algorithm (Edelsbrunner et al., 1983) and mapped into retinal imaging area.The occlusion points and occlusion area are obtained by judge whether the intersect of the ray from viewpoint to the point from surrounding point clouds to the plane of rotated traffic sign is inside or outside of the polygon of the rotated traffic sign.The area where intersects inside into the polygon is the occlusion area.The occlusion area is also computed by alpha shape algorithm and mapped into retinal imaging area.
Occlusion Point Clouds Retina Imaging Area Computing.
From the vertexes of boundary of the projected traffic sign panel point cloud, we select a vertex which has the maximum distance from its center (origin) to the vertexes.After rotating the line segment from viewpoint to the selected vertex around the axis form the viewpoint to origin, we get a vertebral body.The points which possible occlude the traffic sign are included in the vertebral body.We segment points in the vertebral body from traffic sign surrounding point clouds by two conditions: (1) the angle between the vector from viewpoint to the point and the vector form viewpoint to origin, is less than, the angle between the vector from viewpoint to the selected vertex and the vector from viewpoint to origin; (2) the distance from viewpoint to the point, is less than, the distance from viewpoint to the selected vertex.For each point inside in the vertebral body, there is a ray from viewpoint to the point.We compute the intersection of ray to the XOY plane.If the intersection is inside the boundary of the projected traffic sign panel point cloud, then, the point is the occlusion point.All the occlusion points form occlusion point clouds.All intersections of the occluding points form block point clouds.The occlusion point clouds and block point clouds have the same retina imaging area.The retina imaging area of block point clouds is computed by the alpha shape algorithm and human retinal imaging principle.We call the method which obtain occlusion point clouds described above as "Ray Method".

Implementation algorithm
The automatic traffic sign occlusion detection and visibility field evaluation algorithm is summarized as follows: Input: MLS point clouds, trajectory Output: visibility field, occlusion points 1: Detect traffic signs (Wen et al., 2016) 2: Obtain viewpoints from trajectory 3: Segment point clouds in front of traffic sign 4: for each viewpoint 5: Translate and rotate (Kuipers et al., 1999) 6: Compute traffic sign retina imaging area 7: Compute occlusion retina imaging area 8: Compute visibility Eq.( 1) 9: end for

EXPERIMENT
We designed a verification experiment to verify that the occlusion ratio computed by our algorithm is in accordance with a real street scene.Using devices (including Leica Viva GNSS CS15 receiver, Huawei Honor V8 phone, bluetooth camera remote controller and a car), we created another dataset about photos and their GPS positions in street scenes.To improve the accuracy of the GPS positions, before taking a photo and recording its GPS position by CS15, we stopped the car and waited for the GPS signal to stabilize.2. The sign area and occlusion area are the retina imaging area computed by our algorithm.Their units are 10 4 square meters.
From this table, we can see that our algorithm can achieve about 95% accuracy in computing occlusion ratio.For all 144 photos in the verification experiment, the occlusion ratio of our algorithm is 94.89%.This experiment proves that our algorithm is in line with the actual situation.
The implementation of our algorithm on a mountain road is shown in Fig. 3.We can intuitively observe the visibility fields of traffic signs from the graphic interface.The means of color and symbols in the figure are as follows: the detected traffic sign (yellow) viewpoint visibility results (mesh planes with color from red to green), occlusion point clouds (red).The color of the mesh planes, changing from red to green, means that the values of viewpoint visibility change from small to big.The box marked with cross ("X") means that the traffic sign is occluded in that area.The cascaded pictures in the figure have the same position as the corresponding mesh on the road.

CONCLUSION
This paper, based on human retinal imaging theory and point clouds collected by an MLS system, presented a new way to detect occlusion and evaluate traffic sign visibility field.The propose method can automatically provide the traffic department with occlusion detection services in WGS84 coordinate system and can visually check the severity of occlusion through the visibility field.Therefore, this paper has important application value in the maintenance of traffic sign visibility.

Viewpoint
(a).In Fig. 1(b) and Fig. 1(c), the color of the pixels changing from black to white, means that, the value of viewpoint visibility change from small to big.Fig. 1(b) shows the visibility field of the traffic sign with occlusion.Fig. 1(c) shows the visibility field of the traffic sign without occlusion.

Figure 1 .
Figure 1.Hemispherical visibility field of a traffic sign.

Figure 2 .
Figure 2. The example of evaluating the occlusion ratio accuracy in a real environment.
GPS position accuracy, based on Realtime kinematic (RTK) technology, reached the centimeter level in open areas.Finally, 144 photos, with each photo (3968 × 2976 pixels) containing a GPS position, were obtained along the Longhu Mountain Road, Xiamen, China.Based on image verification data, we manually segment the traffic signs and occluded areas in the photo and count their number of pixels first.Then use the number of pixels to calculate the occlusion ratio as the ground truth to evaluate the accuracy of the corresponding occlusion ratio computed in point clouds.An example of evaluating the occlusion ratio accuracy is shown in Fig.2.From this figure, we can see the computed boundary of the sign and occlusion area are basically consistent with the original image.The middle image is the side view of a point clouds environment.The box in the upper left includes its front and top views.The other three corners of the box are the traffic sign images corresponding to the three viewpoints.These images include the original image, the segmented image, the image generated by the point clouds, and the boundary image calculated in point clouds.The computed result of the three viewpoints in the figure are shown in Table

Table 1 .
The visibility evaluation algorithm of a traffic sign.

Table 2 .
The occlusion ratio accuracy calculation result of viewpoints in Fig.2.
Figure 3.An application example of traffic signs' occlusion detection and visibility fields calculation algorithm on large scale data.