APPLICATION OF MLS DATA TO THE ASSESSMENT OF SAFETY-RELATED FEATURES IN THE SURROUNDING AREA OF AUTOMATICALLY DETECTED PEDESTRIAN CROSSINGS

During the last few years, there has been a huge methodological development regarding the automatic processing of 3D point cloud data acquired by both terrestrial and aerial mobile mapping systems, motivated by the improvement of surveying technologies and hardware performance. This paper presents a methodology that, in a first place, extracts geometric and semantic information regarding the road markings within the surveyed area from Mobile Laser Scanning (MLS) data, and then employs it to isolate street areas where pedestrian crossings are found and, therefore, pedestrians are more likely to cross the road. Then, different safety-related features can be extracted in order to offer information about the adequacy of the pedestrian crossing regarding its safety, which can be displayed in a Geographical Information System (GIS) layer. These features are defined in four different processing modules: Accessibility analysis, traffic lights classification, traffic signs classification, and visibility analysis. The validation of the proposed methodology has been carried out in two different cities in the northwest of Spain, obtaining both quantitative and qualitative results for pedestrian crossing classification and for each processing module of the safety assessment on pedestrian crossing environments.


INTRODUCTION
Nowadays, it is a fact that safety assessment is one of the key elements to take in consideration when infrastructures are designed or renovated.In urban environments, the most severe consequences of accidents are suffered by pedestrians and cyclists, who do not have protection against other vehicles.According to data from the European Road Safety Observatory (ERSO), pedestrian fatalities represent a 21% of all road fatalities (ERSO, 2016), and the number of fatalities has been reduced a 35% during the last decade.However, this positive trend has been stalled for the last couple of years.Pedestrian fatalities affect mainly to elder people, and according to a study about run overs carried out in Spain (AXA, 2014), a 55% of the fatalities involving elder people occur when they cross the road correctly, typically in pedestrian crossing areas.These crossing environments play an important role in urban areas as they should be accessible for wheelchairs and are related with the walkability (Kelly et al., 2007) of the environment.A proper location (Gitelman et al., 2017) and quality of different factors that influence the safety of the crossing areas (Basile et al., 2010) are essential for the development of humanized cities.
Another important factor that has an impact on the safety of a crossing area is the appropriate indication of the crossing areas for pedestrians and drivers, with visual cues such as road signs (both vertical and horizontal) or traffic lights.These elements should be assessed regularly, performing regular maintenance tasks.However, studies from the European Road Federation (ERF) state that the decline of expenditures for road maintenance is alarming, having this a clear impact on the road safety levels (European Union Road Federation (ERF), 2015a, 2015b) and on the quality of the road network assets: According to the Spanish Road Association, the maintenance deficit for vertical signage and road markings in Spanish roads amounts to 94 million € and 77 million € respectively (Asociación española de la carretera (AEC), 2016).
It is necessary, therefore, to invest in research and innovation to develop new methods of road inspection that take advantage of the newest technologies and artificial intelligence techniques to tackle the road network preservation necessities while being economically efficient.Specifically, road inspection methods based on laser technology have caught the attention of researchers and authorities in the last decade.Mobile mapping systems (MMS) consist mainly of mapping sensors (laser scanners and RGB cameras) together with navigation and time referencing units deployed on a mobile platform (Puente et al., 2013a).These systems are able to collect dense, accurate and three-dimensional representations of the surveyed environment called 3D point clouds.The research regarding the automatic interpretation of this kind of data has been a topic of interest for some years, and has proven to be useful for the inspection of road infrastructures and their assets, such as vertical traffic signs (Wen et al., 2015;Yu et al., 2016) and road markings (Cheng et al., 2017;Yu et al., 2015).Besides them, different supervised classification approaches can be applied to extract different objects such as cars, poles, façades, etc., from 3D point cloud data (Serna and Marcotegui, 2014;Yang et al., 2017).This work presents a methodology comprised of two main blocks, as depicted in Figure 1.The first one focuses on road marking detection and classification, and it is presented in Section 2.Then, the classified pedestrian crossings are employed to isolate crossing areas and perform an assessment of safety-related features, as shown in Section 3. The study case used for the validation of the methodology is presented in Section 4, and the results are depicted in Section 5. Finally, conclusions are outlined in Section 6.

ROAD MARKING EXTRACTION
The first part of the presented methodology focuses on road marking detection and classification.First, a raw 3D point cloud is processed in order to obtain the position of a number of road markings.Then, a hierarchical classification in two levels of hierarchy classifies road markings in seven classes: Pedestrian crossings, stop lines and five types of arrow.

Road marking detection
Let  = (, , , , ,   ) be a raw 3D point cloud where (, , ) are a set of 3D coordinates which are referenced spatially with the positioning system of the mapping vehicle, and (, ,   ) are intensity, angle of the laser beam and time stamp respectively, all of them properties associated to each point.The trajectory of the vehicle, as recorded by its navigation system, is defined as a matrix  = (, , ,   ), that is, a set of trajectory coordinates with associated time stamps.
First, there are some processing steps that need to be carried out within the point cloud in order to obtain the position of the road markings.Points whose distance with respect to the trajectory is more than 10 meters are filtered out from the original point cloud, reducing its size.Then, a pavement segmentation algorithm is applied using a curb-based approach to filter out points of the ground that do not belong to the pavement.This segmentation is performed locally, selecting several sections of the road which are defined along the trajectory.Curbs and similar obstacles are detected using an approach based on (Wang et al., 2015) normal saliency analysis, that separates points that belong to horizontal planes from points that belong to non-horizontal planes (salient points).This way, a region growing algorithm can be applied, considering that the pavement is a group of points that belong to a horizontal plane and are delimited by salient points at both sides of the trajectory.
Finally, an intensity filter is applied to the points in the pavement segment.Intensity is a key parameter for detecting road markings and traffic signs (González-Jorge et al., 2013) because they are reflective surfaces whose intensity parameter is higher than for points of the pavement or building façades (Figure 2a) For the pavement segment, two classes of points can be defined: Asphalt points (with low intensities) and road marking points (with high intensities).To distinguish them from each other, the distribution of intensities is fit to a Gaussian Mixture Model (GMM) (McLachlan and Peel, 2000) with two components.The component with lower intensity will correspond to asphalt points, that are filtered out from the point cloud.
At this stage, only points on the pavement with high intensity are available for further processing (Figure 2b).It is important to note that all the relevant information about road markings is approximately contained in the same plane, so it is straightforward to rasterize the point cloud, that is, to project the point cloud on the horizontal plane and to define a square grid within it, annotating to which cell -or pixel -of the grid belongs each point (Soilán et al., 2016).Then, it is possible to visualize grayscale images based on any feature of the points within each pixel of the raster grid.Using the point intensity as feature, a grayscale image that highlights road markings can be obtained.
Finally, the position and geometric parameters of a number of road markings can be extracted.The grayscale image is first binarized, defining an adaptive threshold that has into account the distance between each point and the trajectory (as the intensity parameter of a point is inversely proportional with respect to the distance between the laser beam origin and the position of the point, as stated in Höfle and Pfeifer ( 2007)) (Figure 2c).A connected components algorithm is applied over the resultant binary image, performing a clustering operation over each group of connected pixels.This allows to isolate each binary element for further filtering (Figure 2d).As the geometric properties of road markings are standardized by national regulations, the area and the ratio between area and perimeter of each element are computed and used for filtering only a number of road markings, including mainly arrows and rectangular markings.Note that markings that delimit road lanes are not considered here, being the main goal of this process to obtain the location of pedestrian crossings.
The position and geometric features of the binary elements that remain after the application of these filters will be stored, being the collection of semantic information the next step.

Road marking classification
Once each individual road marking is isolated, a hierarchical classification workflow in two levels of hierarchy is proposed.
First of all, it is necessary to define a feature vector that defines each binary image in order to train a classification model.Here, a Geometry-Based Feature (GBF) is defined for each binary image as: Where A is the area of the road marking, ( 1 ,  2 , ) are the length of major and minor axis, and the radius of the bounding ellipse, (, , ) are the ellipticity, triangularity and rectangularity (Rosin, 2003), and   represents the distribution of pixels in the binary image across rows and columns gathered in two 20-bin histograms.That is, a 47element feature vector is obtained for every binary image.A training set of images was gathered using a portion of the available data (as stated in Section 4) and their correspondent feature vectors were fed to a two-layer neural network whose output layer distinguished three different classes, namely rectangular markings, arrows, and other markings / negatives.This model defines the first hierarchy level of the classification framework, and allows to classify, in a second level of hierarchy, different rectangular markings and arrows using separate approaches.
Regarding rectangular markings, they are classified in stop lines or pedestrian crossings by analysing their contextual relationship.A pedestrian crossing is defined as a group of parallel rectangular markings whose centroids are close to each other (a threshold was set to 2.5 meters), and they are also parallel with respect to the trajectory of the vehicle.Furthermore, isolated rectangular markings which are perpendicular with respect to the trajectory can be defined as stop lines.
Road markings defined as arrows in the first level of hierarchy can be classified in five different types: Left, Right, Straight, Straight-Left and Straight-Right.They are classified by comparing the binary image representing the arrow with templates which are defined for each arrow class.In order to standardize the comparison, the binary images are rotated according to the principal axis of the arrow, defining two possible orientations for each one (pointing upwards or downwards).The comparison is carried out using two parameters: Correlation (Equation 2) and Structural Similarity Index (SSIM), as defined in Wang et al. (2004).
where ,  are the number of rows and columns of images A, B; and  ̅ ,  � are the mean of the images.
The class of each arrow will be defined by the largest result among all the comparisons, if it is over an empirically defined threshold (0.4 for correlation, 0.987 for SSIM).

SAFETY ASSESSMENT
Once the road markings have been extracted, it is possible to employ the information about the location of pedestrian crossings to study their surrounding area, focusing on parameters related with pedestrian safety on the crossing environment.Here, four modules related with different features are considered: (1) Accessibility analysis, (2) traffic light classification, (3) traffic sign classification, and (4) Visibility analysis.
The input data for this methodological block consists of a raw point cloud  and a trajectory  as defined in Section 2.1, together with a set of point cloud objects  = {  , … ,   , …   },  = 1 …  representing the location of the pedestrian crossings within .Before entering the first processing module, the point cloud is preprocessed.First, the surrounding area of each pedestrian crossing   is isolated, defining a road section 20 meters long and centered in the pedestrian crossing,   ⊂ .Then, the segments of points that belong to the ground   ⊂   and points that do not belong to it   ⊂   are separated in order to simplify the application of the different algorithms applied during the safety assessment.

Accessibility analysis
This processing module aims to get information about the accessibility of the crossing environment.On one side, it is important to ensure that the access to the crossing area does not have any obstacle that could impede a person in a wheelchair from crossing the road.On the other side, there exist regulations regarding the maximum slopes in an accessible crossing.In Spain, these regulations are given by the Ministry of Public Works and Transport (Ministerio de Fomento, 2010).
To get this information for a given pedestrian crossing   , the ground point cloud   is processed.First, the coordinates of the curbs in   are obtained using the same approach than in Section 2.1, that is, applying a saliency analysis.To define a curb, salient points within   are grouped using a Euclidean Clustering, and those groups of points that comply with certain elevation, horizontal length and distance to trajectory thresholds as defined in (Wang et al., 2015) are defined as curbs (Figure 3a).Then, in order to check whether or not the road is accessible in the crossing area, the ground segment is divided in slices perpendicularly with respect to the trajectory direction.For each slide, a region growing algorithm is applied at both sides of the trajectory: Starting from the closest point to the trajectory at each side, neighboring ground points are iteratively clustered together until a point previously defined as curb or obstacle is found (Figure 3b).This way, the length of each slide can be computed.Accessible road entrances can be finally defined comparing the length of neighboring slides, as a peak on the derivative of the slide length plot represents a road entrance as it is shown in Figure 3c.Finally, if the coordinates of the studied pedestrian crossing   overlap with an accessible road entrance, then it will defined as an accessible pedestrian crossing, as long as the length of that overlap complies with the official regulations (in this case, the road entrance should be between 1.2 and 1.8 meters long).
Furthermore, it is possible to calculate the transversal and longitudinal slopes of the crossing area.Given the coordinates of the points within the pedestrian crossing   , these slopes are obtained as: where (  ), (  ) are the height of two points defining a line which approximates the trajectory of a pedestrian who is crossing the road, and (  ), (  ) are the height of two points defining a line which approximates the trajectory of the driver.
This processing module outputs two Boolean variables (defining whether or not the pedestrian crossing is accessible at both sides) and two numerical values representing transversal and longitudinal slopes on the crossing environment.

Traffic light classification
An important safety feature on pedestrian environments is the presence of street lights which control flows of pedestrians and vehicle traffic.This module aims to define if a crossing environment is regulated by traffic lights.For that purpose, an automatic classification workflow is proposed.
For a given road marking   , the correspondent non-ground point cloud   is processed.First, a pole-like object segmentation has to be carried out in order to filter out both noise and other objects such as façades or parked vehicles.It consist of three main steps: (1) Object clustering: Points in   , which are originally unorganized, are grouped in a set of clusters following an Euclidean Clustering approach.
(2) Object parametrization: A group of horizontal profiles are selected from each object and projected to a horizontal plane.A circle K is fit to the projected points within each profile (Figure 4a), and different parameters are stored for each circle, namely its radius,   , the coordinates of its center   and the mean-squared error of the fitting process   .(3) Object segmentation: A cluster of points will be defined as a pole-like object if the selected parameters comply with a series of empirically defined thresholds based on previous knowledge about the geometry of pole-like objects: ������ < 0.001 ℎ  < 10 (5 where  is the angle of the principal direction of the object with respect to the vertical, max (  ) is the maximum radius allowed for the circles that have been fit,  ������ is the mean squared error averaged across the circles, and ℎ  is the height of the object.
After the application of this segmentation algorithm, there are mainly four classes of pole-like objects that appear in the surrounding area of pedestrian crossings: Column traffic lights, mast arm traffic lights, street lights, and trees.
The features selected for the classification of these objects are extracted from a binary image which is computed as follows: First, Principal Component Analysis (PCA) is applied to each object, and the coordinates of their points are transformed to the reference system defined by their three principal directions.
Then, the points are rasterized, being projected on the plane of the two principal directions (correspondent to the two largest eigenvalues) and computing a binary image where raster cells containing at least one point are set to '1', and empty cells are set to '0' (Figure 4b).Finally, two different feature vectors are defined depending on the aspect ratio of the resulting image.
For images whose aspect ratio is smaller than 3, they are resized to a [96 128] size and the pixel values are directly used as classification feature.For images whose aspect ratio is larger than 3, they are resized to a [42 128] size, and a pixel distribution feature (appending the proportion of '1' pixels in each row and column) is defined.
Note that doing this distinction separates column traffic lights (whose images have large aspect ratios) from mast arm traffic lights (with smaller aspect ratio).Street lights and trees, though, can be present in both groups.Images with small aspect ratio are classified using a simple Neural Network model with two layers, with sigmoid activations in the hidden layer, and images with large aspect ratio are classified using a Cubic Support Vector Machine.
This processing module outputs the objects classified as traffic lights, therefore defining if the vehicle flow in the pedestrian crossing environment is regulated.The objects are projected into a plane defined by their two principal components, and a binary image is defined from that projection.

Traffic sign classification
Another important element related with the safety on the surrounding area of a pedestrian crossing is the vertical signage.For a given marking   , traffic signs can be detected on the point cloud   following the methodology from the previous work in Soilán et al. (2016), where traffic signs are detected using the intensity property of the point cloud together with previous knowledge about the geometry of the traffic sign panels.Subsequently, semantic information is obtained using a deep convolutional neural network model as developed in Á. Arcos-García et al. (2017) on RGB images that are part of the data collected by the MMS during the survey.
The traffic sign classification module outputs the position of the vertical traffic signs in the surroundings of   together with the semantic description of each sign.Typically, traffic signs related with speed limit, and warning about the presence of pedestrian crossings or traffic lights will contribute to the safety of the crossing area.

Visibility analysis
The last module of the safety assessment focuses on the visibility between drivers and pedestrians in crossing areas.First, it is important to define the concept of visibility.Here, an object will be considered in the visible area of a driver when it is within its vision field, not occluded by any other object that may impede the visual contact, and located at a distance such that the vehicle can stop in safe conditions.This stopping distance (  ) is defined as: where V and   are the speed of the vehicle and the reaction time of the driver (which are set to 50 km/h and 2 seconds respectively),   is a friction coefficient which is function of the speed, and  is the longitudinal slope of the road as obtained in Section 3.1.
This processing module aims to obtain a qualitative measure of the visibility of a pedestrian at each side of a pedestrian crossing.For that purpose, it is necessary to define, on one side, the stopping distance of the vehicle (Equation 6) and, on the other side, an object that will define the visibility.For this study, a point cloud of a pedestrian will be used as a template and placed at both sides of each road marking   as shown in Figure 5a.
The model that defines the visibility is based on the approach by González-Jorge et al. ( 2016).First, the vehicle trajectory is employed to approximate a number of points of view (PoV) that represent the position of the driver.Then, a set of lines of sight (LoS) is defined joining the PoV with each point of the pedestrian template (Figure 5b).
The visibility analysis is carried out for each individual LoS.In the first place, it is necessary to check if the defined LoS is within the horizontal field of vision of the driver, which, for speeds in urban environments, can be approximated to 90 degrees.Then, a sphere of diameter  ℎ = 0.25 whose center slides through the LoS is defined.A total of 100 positions, equally spaced between the position of the driver and the considered pedestrian point, are defined for the sphere in order to look for occlusions (Figure 5c).An occlusion is defined whenever the point density within the sphere exceeds a predefined threshold based on the average density of the point cloud.For the study in this work, this threshold has been defined as 100   2 ⁄ , which implies that a LoS is occluded when more than approximately 50 points are found within any of the defined spheres.
Finally, the visibility of the pedestrian template with respect to the driver position is qualitatively defined as (1) Good visibility if more than the 75% of the template is visible, (2) Bad visibility if the percentage of visible points is between 25% and 75%, and (3) No visibility if more than the 75% of the points are occluded.This visibility analysis can be employed not only for pedestrians but for other elements such as traffic lights which were classified in Section 3.3, being the output of this module a set of qualitative descriptions of the visibility of the considered objects with respect to a series of points of view of the driver.

CASE STUDY
The methodology presented in this work has been validated using MLS data acquired by a LYNX Mobile Mapper by Optech.This system consists of a set of mapping sensors, specifically two LiDAR heads (FoV of 360 degrees, with 90 degrees between their rotational axes and 45 degrees with respect to the vehicle trajectory) and four 5-Mpix JAI cameras, together with a navigation system that comprises a two-antenna heading measuring system and an Inertial Measurement Unit (IMU).More insight about this system can be found in Puente et al. (2013b).
MLS data from two different cities located in the northwest of Spain were employed for the evaluation of the presented methods: A centric and cluttered avenue in Lugo, and narrow streets and an avenue close to the port and coast respectively in Vigo (Figure 6).Data from Lugo and from the port area in Vigo were used for the validation of both road marking extraction and safety assessment, while data from the coast area in Vigo was exclusively employed for training the classification models described in Section 2.2.

Road marking extraction
This methodological block aims to detect and classify different types of road markings from MLS data.First, evaluating the road detection algorithm, two representative binary images resulting for the process in Section 2.1 were compared pixel by pixel with reference images which were manually collected.The evaluation metrics have been Precision, Recall and F-score, as defined in Equations ( 7)-(9) (were TP, FP and FN are true positives, false positives and false negatives respectively), and the obtained results, compared with (Guan et al., 2014) and (Yu et al., 2015) can be seen in Table 1.For the first level of hierarchy, binary images representing individual road markings were classified using a Geometry Based Feature (GBF) as feature vector and a two-layer Neural Network as classification model, as explained in Section 2.2.The trained model was evaluated in a test set and the results, in the form of a confusion matrix, are shown in Table 2.
In the second level of hierarchy, rectangular markings are classified as pedestrian crossings relying on the contextual relationship among them, and arrows are classified by comparing their binary images with previously defined templates.Using SSIM as a metric for comparison, the final confusion matrix obtained can be seen in Table 3.For these results, it can be seen that the classification system is quite robust for the proposed classes.Although the test set is limited, the totality of the pedestrian crossings has been correctly classified, motivating the safety assessment on their surroundings.Obviously, this classification framework is capable to distinguish only a small number of road markings.However, information about pedestrian crossings and arrows can be useful for developing applications related with the safety (Section 3) and retrieving information about the road network.
Of course, all the information extracted can be gathered in objects whose properties should be easily exportable to geographic databases where this information would assist to inventory and maintenance processes.

Safety assessment
Information regarding the position of the pedestrian crossings is employed to obtain a set of safety-related features in the crossing environment.This processing block is divided in four modules, where quantitative or qualitative results are obtained.
Here, results are shown for all modules but for the traffic sign detection module, where the reader is referred to previous work in Soilán et al. (2016) and Arcos-García et al. (2017).
For the analysis of the accessibility as described in Section 3.1, a total of 54 road entrances were analysed.After a visual inspection it was found that all of them were accessible.The automatic accessibility analysis, though, got that 3 of them had accessibility issues.The main source of error was the presence of vehicles parked in the crossing area, creating occlusions that made the algorithm fail.
Regarding the classification of traffic lights, there were a total of 36 of them on the test set (28 column traffic lights and 8 mast arm traffic lights) which were classified with remarkably good results.Specifically, for column traffic lights the classification model was able to classify them with a precision of 90%, recall of 96.4% and F-score of 93.1%.For mast arm traffic lights, the classification model classified all of them correctly without any false positive, being all three classification metrics 100%.Of course, the number of objects in the test set is too small to ensure generalization, but the classification feature as defined in Section 3.2 seems to highly discriminate this type of object from other pole-like objects.The average F-score for the classification of traffic lights is, therefore, 94.6%.
Finally, qualitative results have been obtained for the visibility analysis module, where a set of points of view is defined for the driver along the stopping distance of the vehicle with respect to the pedestrian crossing, and the visibility of a pedestrian at both sides of the crossing is analysed for each point of view.For the 54 road entrances that were considered, the vast majority of the points of view was labelled as good visibility points.Only for 2 of the road entrances it was found that an object was causing an occlusion for the correct visibility of a pedestrian.

CONCLUSIONS
This work presents a processing framework for MLS data divided in two main blocks: Road marking extraction and safety assessment.First road markings can be detected and classified using only point cloud data, obtaining remarkable results for the classification of pedestrian crossings.This motivates the second processing block, which isolates the surrounding area of pedestrian crossings and extracts safety-related features regarding accessibility, visibility, presence of traffic lights and presence of traffic signs.This parameterization can be exported to Geographic Information Systems (GIS) were they can be not only visualized but also weighted in order to obtain a safety index to define the safety (globally or individually) of the crossing environment in the surveyed area.The definition of that safety index is out of the scope of this work, as there are a number of safety features that cannot be assessed with a Mobile Mapping System (as, for example, traffic light frequencies, or pedestrian / vehicle densities).
As future work, the classification models may be optimized and improved.Although the classification results are globally good, the defined features and models may be too simple, so deeper neural networks could help to ensure a better performance and generalization.Furthermore, there is room for improvement on the visibility module, developing a vision model which approximates better the actual field of view of a driver.

Figure 2 .
Figure 2. Road marking detection.(a) The intensity feature discriminates pavement points and road markings.(b) Point cloud after the application of an intensity filter.(c) Binary image representing road markings.(d) Each individual road marking in the binary image can be isolated for further analysis.

Figure 3 .
Figure 3. Accessibility analysis.(a) Curb definition within the ground point cloud.(b) A region growing algorithm is applied to divide the ground segment in slides, using the information of trajectory and curbs.(c) The length of the slides is analysed in order to find road entrances as peaks on the slice length plot.

Figure 4 .
Figure 4. Traffic light classification.(a) Pole-like object segmentation.A number of sections are studied for each object, fitting circles to each one of them.(b) The objects are projected into a plane defined by their two principal components, and a binary image is defined from that projection.

Figure 5 .
Figure 5. Visibility analysis.(a) Pedestrian templates are located at both sides of the pedestrian crossing.(b) For a PoV of the driver, diferent LoS are defined to analyse the visibility of each point of the pedestrian point cloud.(c) A sphere that slides through each LoS is defined in order to search for occlusions.

Figure 6 .
Figure 6.Case study data was acquired from three areas in two different cities located in the northwest of Spain.

Table 2 .
Confusion matrix for the first level of hierarchy

Table 3 .
Confussion Matrix in the second level of hierarchy.The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, VolumeXLII-2, 2018ISPRS TC II Mid-term Symposium "Towards Photogrammetry 2020", 4-7 June 2018, Riva del Garda, Italy