MOBILE LASER SCAN DATA FOR ROAD SURFACE DAMAGE DETECTION

: Road surface anomalies affect driving conditions, such as driving comfort and safety. Examples for such anomalies are potholes, cracks and ravelling. Automatic detection and localisation of these anomalies can be used for targeted road maintenance. Currently road damage is detected by road inspectors who drive slowly on the road to look out for surface anomalies, which can be dangerous. For improving the safety road inspectors can evaluate road images. However, results may be different as this evaluation is subjective. In this research a method is created for detecting road damage by using mobile proﬁle laser scan data. First features are created, based on a sliding window. Then K-means clustering is used to create training data for a Random Forest algorithm. Finally, mathematical morphological operations are used to clean the data and connect the damage points. The result is an objective and detailed damage classiﬁcation. The method is tested on a 120 meters long road data set that includes different types of damage. Validation is done by comparing the results to a classiﬁcation of a human road inspector. However, the damage classiﬁcation of the proposed method contains more details which makes validation difﬁcult. Nevertheless does this method result in 79% overlap with the validation data. Although the results are already promising, developments such as pre-processing the data could lead to improvements.


INTRODUCTION
Road damage detection is important to determine road safety and road maintenance planning.The damage of the road surface, like potholes, cracks and ravelling, affects driving conditions such as driving comfort and safety and increases fuel consumption, traffic circulation and noise emission.Localisation of these damage can be used for targeted road management and maintenance, which contributes to an improvement of driver safety and comfort (Vittorio et al., 2014).
The traditional method for road condition surveying is that inspectors drive slowly on the road looking out for road surface damages and stop the vehicle when damage is found, do measurements on the damage and mark visually.This is dangerous, time-consuming and costly (Cheng & Miyojim, 1998;Yu et al., 2007).For improving the safety road inspectors can evaluate road images.The results are, however, susceptible to human subjectivity.
Iv-Infra has a mobile mapping car, shown in Figure 2, including 3 laser scanners, 10 cameras for 360°photos, a GPS and an Inertial Measurement Unit.This system has been implemented successfully for lamp post identification.This paper is an attempt to study the feasibility of using such a system for road damage detection.In this research a method for road damage detection is developed using laser scan data of one of the three laser scanners of the car, a Z+F PROFILER 9012A.This laser scanner is mounted at the rear of the vehicle such that its profile lines are perpendicular to the driving direction.It measures the range and the intensity, along the profile.
There are several advantages of such a system, for example no road closure is needed for manual road inspection, which increases safety and decreases costs.When the damage detec-tion can be done automatically no differences due to subjective judgement are obtained.This paper is structured as follows.In the following section advantages and disadvantages of some of the popular alternatives to manual road condition survey will be discussed.Some details about the measurement car and research area will be given in section 3. The methodology will be explained in section 4. In section 5, results of this method will be given and finally the conclusion is presented in the last section.

BACKGROUND
Several methods have been developed to collect data of a road surface and determine damage from such data.The methods can be classified based on how the road surface information is acquired.This can be vibration, image and laser scanningbased methods.As this study focuses on investigating the feasibility of laser scanning in detecting potholes, ravelling, crack and craquelure (Fig. 1), these definitions of these damage are first presented, and then existing methods for damage detection are investigated.
Potholes are bowl-shaped holes with various sizes involving one or more layers of the asphalt pavement structure.Size and depth can increased whenever water accumulates in the hole (Tedeschi & Benedetto, 2017).They arise due to freezing of water in a soil, which results in expanding of the space.Thawing of the soil can weaken the road surface while traffic can break the pavement resulting in potholes.
Ravelling is dislodging of aggregate particles due to influences of traffic, weather and obsolescence of the binder (Kneepens & Heesbeen, 2017;Tedeschi & Benedetto, 2017) Due to traffic load, freezing and expanding of water in asphalt, cracks can be formed.Two types of cracks (longitudinal and transverse cracks) were considered in this study.Longitudinal cracks are cracks parallel to the road, while transverse cracks are perpendicular to the road.
Craquelure are cracks, which develop into many-sided, sharp angled pieces.This damage develops at the end of the structural life of an asphalt pavement, (Bouwend Nederland and emulsie asfaltbeton, n.d.).Craquelure at the outer 0.25 m of the pavement is named as boundary damage.Next, a survey of techniques for data capture and methods for processing data to determine road surface damage are present.

Vibration based methods
Accelerometers, microphones and tire pressure sensors are used to measure vibrations caused by pavement elevation differences and roughness.Accelerometers in mobile phones can measure the relative movement of the car in three dimensions.Examples are the Pothole Patrol by Eriksson et al. (2008) and Wolfrine by Bhoraskar et al. (2012).Filters and machine-learning approaches are used to detect road damages.A disadvantage of this data acquisition method is that the relative movement of the car is only influenced by the small contact area between the road surface and the four tires.So only a small parts of the road surface along the wheel paths can be analysed.

Image based methods
There are also methods collecting images from scanning, linescan and video cameras of the road surface, which can be used for detecting the damage.An example is the automated detection system RoadCrack, created by the Australian Commonwealth Scientific and Industrial Research Organization (CSIRO, n.d.).This system is based on high speed cameras mounted underneath the vehicle.These cameras collect high resolution images of small patches of the pavement surface and they are consolidated into bigger images of half-metre intervals.CSIRO (n.d.) stated that the system can detect cracks in a millimetre order, while driving up to 105 kilometres per hour.This is done fully-automated with a combination of machine vision and artificial intelligence (CSIRO, n.d.).Another system based on laser based imaging is the Digital Highway Data Vehicle (DHDV) from Waylink (n.d.).They use their Automated Distress Analyzer (ADA) which produces crack maps in real time.
RoadCrack and DHDV are two commercial systems, which use cameras as one of their acquisition methods.There are several more commercial systems, most of which have not published details on their algorithm.

Laser scan based methods
One of the advantages of using laser scanning sensors is that 3D topographic of the road surface can be captured highly accurately and quickly.Guan et al. (2014) used mobile laser scanning (MSL) data to detect road markings.From MSL data, they create intensity images, which they used in a point-densitydependent multi-threshold segmentation method to recognise road markings.
Pavemetric inc.developed the Laser Crack Measurement System (LCMS), which consists of two high performance 3D laser profilers and a camera as detector, in cooperation with government and research partners (Laurent et al., 2014).This system measured range and intensities, and produced 2D and 3D data.Yu et al. (2007) developed a system using a SICK LMS 200 laser scanner for reconstructing the 3D surface model, cracks in smaller regions can be identified from a variation of the 3D depth measurement.Mertz (2011) used a low cost "laser line striper" to evaluate the unevenness of the road with a step-operator to detect road damage.Based on the number of the data points in one line, significant road damage is found.However, noise data can trigger the larger number of the points in the line, which lead to incorrect damage to be detected.

Data acquisition system
As mentioned in section 1, Iv-Infra has a measurement car with 3 laser scanners, 10 cameras for 360°photos, 3 HR cameras in the bumper, a GPS and a IMU system (Fig. 2).

Figure 2. Measurement car
In this research, the data from one scanner, the Z+F PROFILER 9012A is used.This is a profile scanner using the the phaseshift method for measuring the range.An outgoing laser beam is intensity-modulated by a sine-wave signal.This signal is reflected back from an object and the received intensity pattern is compared with the original transmitted signal.A phase-shift in the modulated signal is caused by the travelling time of light forth and back to the measured object.The phase measurement can be transformed directly into a distance/range: d = c 2 * f , with c the speed of light (with atmospheric corrections) in m/s, f the modulated frequency in Hz.
The profile scanner produces measurement points with x, y, z coordinates.Each measurement point (x,y,z) is geo-referenced by the QINSy (Quality Integrated Navigation System) software (Quality Positioning Services B.V, 2018) such that the IMU, the GNSS locations, the vehicle odometer, the intensity and range are taken into account.This is done in the Dutch coordinate system, RD-coordinates.The z component is given in Normaal Amsterdams Peil (NAP), the Dutch height reference.Each measurement point contains the following data fields: intensity, range, profile number and beam number.The intensity is the amount of reflected light, which has no clear unit.The range is the distance between the scanner and a hit point on the object surface and it is given in meters.When the laser beam hits multiple "targets" of different heights, for example when the laser beam partly hits the road surface and partly falling into a crack, the laser scanner will detect a combination of multiple reflections, one for each target.Unfortunately phase-based ranging devices can never discern all the single vectors but only measure the resultant vector; the geometrical sum of all vectors.So the resultant range is a mixture of the distances to the surface and into the crack (Mettenleiter, M. (Zoller + Fröhlich GmbH), 2019).
A profile number is given to each new line which the profiler measures.A new profile starts nadir and the laser beam turns anticlockwise, see Figure 3.The beam numbers are given to each consecutive point in each profile.In this project, the laser scanner is configured such that each profile (360°) contains 5100 points (beams), with a spindle speed of 200 rotations per second (profiles).When the car is driving, a spiral pattern is formed, illustrated in Figure 3.The distance between each profile depends on the car velocity and the spindle speed of the laser scanner.In this case, this results in a distance of 4 cm between the profiles while driving 30 km/h and 14 cm at 100 km/h.The point spacing along the profile is approximately 3 mm on the road in nadir direction and does not depend on driving velocity, but on range.

Research area
A road section of the R106 near Haarlem city, the Netherlands, is selected for a pilot survey.This is a touristic and quiet road where the driving speed is between 30 and 50 km/h.On this road, 36 road damages are found by a road inspector from a third party and are categorised as 2 ravelling, 7 craquelure, 2 potholes, 8 longitudinal and 11 transverse cracks and 6 boundary damage, (van den Assem, 2019).Figure 4 shows the damage of the road as classified by a road inspector.For this paper a subset of around 120 meters of road is used, which includes 6 million points, given in Figure 4. Road sections 1 and 2 are evaluated extensively in Section 5.5.

Data selection
The laser beam width defines which sizes of damages can be measured.A large beam is more likely to hits multiple "targets" which results in a resultant vector.Therefore, it was decided to use only beam widths smaller than 5 mm for this research.To avoid that the beam widths are larger than 5 mm, the theoretic intersection of the laser beam with a horizontal was calculated based on trigonometric properties.For this laser scanner the beam divergence is 0.5 mrad and it has a beam diameter of 1.9 mm (at 0.1 m distance) (Zoller + Fröhlich GmbH, n.d.).In Figure 5 it can be seen that at 42 degrees the beam width is below 5 mm, so this is taken as the boundary angle.This results in around 600 beam numbers on each side of the nadir, when there are 5100 in one profile.

METHODOLOGY
To identify damage of the road surface from MLS data, the proposed workflow includes (I) feature creation, (II) K-means clustering to create training data set, (III) Random Forest classification and (IV) Mathematical morphological operations to reduce small damage points and connect larger damage points.

Deviation from the mean
The first feature is to calculate the absolute elevation deviation of the centre point from the mean of a window of length L. This can also be written as: where Z = the point data and L = the number of the points within the window.This can be interpreted as the surface roughness, which can be defined as the irregularities in the surface texture which are inherent in the production process and wear (Taylor Hobson Limited, 2011).This feature can also be calculated for height values as well as with the intensity values.

Difference to the surrounding points
Another method is to take the difference with the centre point of a window and the neighbouring points along the profile.This can written as: Zi. (2) This feature can be used with the height values as well with the intensity values.

Standard deviation of range
In this feature the range is calculated for each beam number when the road would be horizontal and flat.This is done by calculating the angle which each beam number should have, by taking the fraction of the beam number by the maximum beam number times 360°.The range is then calculated by the height of the scanner divided by the cosine of the above calculated angle.This calculated range is subtracted from the measured range.This is done because the range may vary with the angle.Over a window with length 20 points the standard deviation is taken over the range difference.In this case a radius of 0.02 metre is taken.Here the standard deviation is also taken over a window of length 20 points.

Sum of different windows
For the deviation from the mean and difference with surrounding points different window sizes can be used to calculate the feature.The results of different window lengths are added as a separate feature a new feature is created.

Step II: K-means clustering
In this step K-means clustering is used to create a training data set for the Random Forest classification.K-means clustering, (Hartigan & Wong, 1979), divides M points in N dimensions into K clusters so that each point belongs to the cluster with the closest centroid.
In this study, K-means clustering is used to classify a small selection of the data with known damage into two clusters ("no damage" and "damage").
But before the clustering is done, each feature is scaled.This is done by first subtracting the mean value, and scale it by the standard deviation of the feature.
Both scaling and clustering are done with the python scikitlearn module (Pedregosa et al., 2011).

Step III: Random Forest classification
After clustering, the small training data set can be used for training the Random Forest algorithm.Random Forest Classification is a supervised classification method, based on classification trees (Liaw et al., 2002).A classification tree is a multistage approach which breaks up a complex decision into a union of several simpler decisions (Safavian & Landgrebe, 1991).Each node in a tree makes a binary decision, and multiple decisions in a tree lead to a class label.This is done by dividing the small training data set in 3 parts, and use 1 part for training the algorithm.
In this research, the RandomForestClassifier from the scikitlearn module is used (Pedregosa et al., 2011).After training, the whole data set is classified by using this random forest classifier.

Step IV: Mathematical morphological operations
With the Python scikit-image (van der Walt et al., 2014) morphology module, objects smaller than 3 points are removed as a first step.This is done by projecting the data as a matrix with the number of profiles as rows and the number of beams as columns.After the small objects are removed, morphological closing is used.Mathematical morphological operations assign pixels in an image based on the values of neighbouring pixels.Mathematical morphological closing is a combination of dilation followed by an erosion operation (Smith, 1997).Dilation is an operation, which changes a "no damage" pixel into a "damage" pixel when a neighbouring pixel is classified as "damage".Erosion is the opposite operation of dilation.Erosion gives "damage" pixels a "no damage" value when the neighbouring pixels are classified as "no damage".Erosion decreases objects, while dilation increases objects, and can merge multiple objects into one (Smith, 1997).Mathematical morphological closing removes gaps in connected damage pixels.As neighbouring pixels a + shape of 1 centre point is used.

RESULTS
In this section, results from the proposed method are presented.Furthermore were the results validated based on the classification of a road inspector from a third party with no connection to this project.

Features
For this research 22 features are calculated as described above.
For the deviation from the mean and difference to the surrounding points four different window sizes (5, 21, 41, 101) are used.
Figure 6 illustrates the correlation between the features.It is clear that there is a large correlation between different window sizes of the same feature.
Examples of different features for road section 1, can be found in Figure 8.

K-means clustering
K-means clustering is done on road section 1. Results for this road section are given in Figure 9. From this figure it can be seen that large longitudinal cracks are classified as damage, while the transverse cracks are not detected.

Random Forest
The centre figure of Figure 9 gives the results of the Random Forest classification and morphological operations for road section 1, and Figure 10 gives the road damage classification for the whole research area.Figure 9 shows that less small objects are present in compare with the K-means clustering result.The transverse crack is as well not detected as damage.

Validation
The validation of the above described method is done with the help of damage shapefiles of a road inspector from a third party.
The shapefiles are three files with point, line and polygon data.These data files are converted to raster data with the GDAL (GDAL/OGR contributors, 2018) tool gdal rasterize.This tool rasterizes the shapefile (vector geometries) with a pixel size of 0.05 × 0.05 meter.Then the three raster files are combined to one large raster file.
This validation data is projected to the point data, such that each point gets a damage value.The areas of connected damage points are calculated, such that orthogonal and diagonal point neighbours are included.This is also done for the method data.
Through rasterising the shapefiles, some pixels are no longer connected to each other, such that the number of damage areas are increased.This results in 153 connected road damages instead of 20 damages.The results of the method contains 3512 damage areas, most of them are smaller than 30 points.The distribution of the damage areas (below 30 points) for the validation data and the method are given in Figure 12.Here it can be seen that there is a large amount smaller damages detected for the described method, and less for the road inspector.When the larger areas (>30 points) are compared, there are 139 damage areas for the method and 62 validation damage areas.
When the intersection of union should be calculated, this would result in a low number.The intersection of union can be calculated by the area of overlap divided by the area of union.This can explained by the large and rough damage areas of the road inspector.The area of union is large, while the classified damage of the method are detailed and relative small.So for calculating the intersection of union another more detailed validation data is needed.This can be done by taking orthogonal photos of the road and take the road inspector's classification as guide.
When only method damage points are compared with validation method points, 79% of the method points are right classified as damage.However, the question is whether the false positives really are false positives.

Cases
In this section, two cases are discussed in detail, road sections 1 and 2 in Figure 4.
In road section 1 (Fig. 10, between profile numbers 19360 and 19600), the proposed method classified only parts as damage, while the road inspector (Fig. 11) the whole area classified as damage.
Road section 2 (Fig. 10, between profile numbers 17500 and 17800) contains road markings classified as damage by the proposed method.This can be explained by the higher elevation of road markings and the higher intensity compared to the surrounding points.Therefore it is important to pre-process the point data, for example filter out high intensities.Also transverse cracks are difficult to detect with this method.The change that they are not detected is large, because the spacing between profiles is relative large, especially when the driving speed is high.

Conclusion
This paper presented a possible technique for detecting road damages with a Z+F PROFILER 9012A laser scanner mounted on a mobile mapping system.This is done by making features with a moving window method.Then K-means clustering is applied to create training data for a Random Forest algorithm.After that classification mathematical morphological operations are used to remove small objects and connect points which are close to each other.Validation is done with the help of a road inspector's classification.Although this validation data is too rough to calculate the intersection of union on areas, when points are compared to each other 79% of the points are correct classified as damage.However, more research is needed for analysing the false positives.
Also road markings are classified as damages, probably due to the high reflectivity and due the fact that road markings are elevated with respect to the road which results in a deviation when compared to the surrounding road surface.
Due to the spacing between profile lines, the probability is large that transverse cracks are not detected.

Recommendations
To create a higher level of accuracy for this method pre-processing of the point data is needed in order to remove road markings for example.To remove the K-means classification for extracting training data for the Random Forest algorithm self made training data can be used.Also a new and more detailed road classification can be used for validation.This can be done by making a detailed road damage classification by help of orthogonal road photos.Further, a look at the false positives is needed and a confusion matrix can be used to distinguish which damages types can and can not be recognised well.A next step in this research could be to identify different types of damages.

Figure 1 .
Figure 1.Examples of road surface damage.

LFigure 3 .
Figure 3. Scanning pattern of the profile.Each new colour represents a new profile number.The driving direction is marked with an arrow.The zoomed in section gives how the sliding window algorithm is used.

Figure 4 .
Figure 4. Part of the research area with locations of damages found by a road inspector.The black squares show the locations which are discussed in more detail.

Figure 5 .
Figure 5.Effect of the angle of the beam on the beam width.A larger angle causes a larger beam width.
deviation of number of points With Cloud-Compare (Girardeau-Montaut et al., 2017) the number of neighbours inside a sphere of radius R are calculated for each point.

Figure 6 .
Figure 6.Correlation matrix of the features.L gives the window size which is used for calculating the feature.