SIDEWALK DETECTION AND PAVEMENT CHARACTERISATION IN HISTORIC URBAN ENVIRONMENTS FROM POINT CLOUDS: PRELIMINARY RESULTS

Abstract. The definition of physical accessibility in urban environments is a topic of recognized importance by policy makers and by international organizations. A first step to address the accessibility topic is the definition and characterisation of urban elements, like sidewalks, roads, and ramps. Sidewalk inventory plays a crucial role in this phase. In literature there are several ways to extract sidewalks from a point cloud, but they are all tailored on modern and standardized situations. For example the presence of a curb is assumed as the normality and the roads are supposed to have the same width along the path. When dealing with an Urban Heritage, some difficulties arise. In fact, in an historic urban environment ground irregularities should be taken in consideration: the paving is composed by different materials, curbs are not always present, and a Z difference between road and sidewalks is not so sure. In such cases existing methodologies cannot be applied. This paper present a method to semantically segment a point cloud, labelling sidewalks and roads. Sidewalks are also characterized by detecting their pavings. The method is tested on an Urban Heritage: the Unesco site of Sabbioneta, in northern Italy. The results are promising, sidewalks are detected with a precision of 80%, main errors are in corner areas. Paving characterisation is based on thresholds derived from some samples, and the method shows an high precision (more than 90%) in all the pavings considered.



INTRODUCTION
Physical accessibility in urban centres is a topic of general interest and primary importance, addressed on several levels by both local and national authorities, but also by the European Union, and by international organisations such as the United Nations. Coherent and comprehensive management of accessibility is not simply a matter of providing ramps next to sidewalks, or design solutions to overcome differences in level without steps. Coherent management must necessarily take into account a series of parameters that allow any user to fully enjoy the urban space. Therefore, the presence or absence of lighting elements, of resting areas (e.g., benches), of differences in level, but also the use of correct materials and their state of preservation should be considered useful for accessibility. An assessment based on the elements actually present in an urban area can then lead to the definition of a set of routes that can be defined as accessible.
Sidewalk inventory is a basic step for the further definition of walkable routes inside the city and accessibility management. The sidewalk data could be related to several attributes, for example, their geographical position, their geometric characteristics, and the material of the paving surface.
Automatic recognition and classification of sidewalks and other urban elements from Mobile Laser Scanning (MLS) point clouds is a topic already treated by several authors, and even if with different methods (Serna and Marcotegui, 2013, Ishikawa et al., 2018, Balado et al., 2018, Díaz-Vilariño et al., 2016 and Ai, 2020), the main idea is to identify curbs and then separate the sidewalk area from the road by using the detected curb or basing on different elevation levels of elements. However, these methods do not apply to historic cities where there is a non-standardized disposition of elements, like sidewalks and streets at the same Z level (no curbs). Moreover, sometimes there are uncommon solutions and peculiar spatial organization to cope with physical constraints due to the surrounding historic environment. In such a case, sidewalk detection becomes a challenging task that should be solved employing different strategies.
This paper presents a method for the automated classification of ground elements in historic urban environments. The method relies on the fact that different paving materials are used to separate sidewalks from other road objects. The method is based on geometric features clustering, topological relations, and paving material identification. This method is tested in a real casestudy: the historic city of Sabbioneta, a Unesco site located in northern Italy. The paper is organized as follows: section 2 describes existing method that deals with accessibility data extraction from point clouds; the method proposed in this paper is presented in section 3, and the results of the tests made on the city of Sabbioneta are presented and analysed in section 4. Section 5 presents a discussion and the conclusion of the paper.

RELATED WORKS
In literature there are several approaches to sidewalk detection, and they mainly refer to road boundaries detection and curbs detection methods to separate road pavement from roadside (Ma et al., 2018). Curb detection can be based on basic rules related to elevation values, topological relation, and on machine learning and deep learning techniques. The purposes are various, from urban inventory to path planning. The typical instruments used to collect point clouds are mobile mapping systems and terrestrial laser scanning.
Accessibility diagnosis and itinerary planning, starting from a MLS point cloud was investigated by (Serna and Marcotegui, 2013). Basing on range images, height, and geodesic features they segmented urban objects and they detected curbs. Accessibility analysis was later defined using geometrical features and accessibility standards. Curbs detection and classification was also performed by (Ishikawa et al., 2018), in fact, from MLS they extracted curbstones and classified weather they allowed or not access to off-road facilities. Classifying the curb types they assessed also the accessibility. The method was based on analysis of the angles of adjacent points on a scan line.
Curbs detection is not the only way to perform urban classification. A method to automatically classify urban ground elements from MLS data was proposed by (Balado et al., 2018). The method is based on a combination of topological and geometrical analysis. Element classification is based on graph comparison. Road, tread, riser, curb and sidewalks were detected. Another example is the work of (Yadav et al., 2018), they extracted road surface in rural roads, where the curb detection cannot be performed cause there is no curb in rural roads. They base their method on parameters coming from Principal component Analysis (PCA), point density, intensity and height differences. A similar approach is presented by (Díaz-Vilariño et al., 2016), they propose a classification of urban pavements (stone and asphalt) relying on roughness descriptors of MLS, applying clustering on scan profiles.
A deep neural network approach to extract and characterize sidewalks was presented by (Hou and Ai, 2020). This stripebased sidewalk extraction is also able to measure sidewalks width.
Regarding urban paving material classification, (Degol et al., 2016) used photogrammetry and photos combined with machine learning to classify different materials in a construction site. They also introduced a new dataset. Other authors (Yuan et al., 2020) classified different building materials from a Terrestrial Laser Scanner (TLS), using a machine learning approach basing on three features: material reflectance, Hue Saturation Value (HSV) colour, surface roughness.
The presented works were meant to extract road elements mainly referring to standardized conditions, where for example curbs are always present. The methods are various and exploit point cloud features which are then processed, also using machine learning techniques. However, the difficulties of an historic site (irregular ground, different materials, not Z difference between road and sidewalks) are not yet investigated, making this paper more valuable.

METHOD
The idea at the basis of this work is that in many historic sites different materials are used for different elements of the urban ground. Consequently, there are different paving materials for roads and sidewalks. However, the same material can be used in one case for sidewalks, and in other for roads (making it even more challenging). Starting from this idea, a method that  takes into account those differences was developed to cluster the point clouds. In this work, the term "paving material" is intended as the layout and disposition of elements that make the surface of the flooring. For example, the paving material referred as "bricks" is intended as the external surface of a floor composed by closely aligned bricks with mortar interlayers. The proposed method is summarized in Figure 1, and is developed for a dataset coming from a Mobile Laser Scanning system, but with some minor changes, the idea can be applied also to other dataset, from different instruments.

Extraction on the Region of Interest
To easily analyze the entire point cloud, it is subdivided in several Regions Of Interest (ROI). Each ROI is firstly defined by cropping the point cloud with an oriented Bounding Box, based on two consecutive points extracted from the trajectory of the instrument. Then the crop is refined by selecting only points pertaining to almost horizontal surfaces.
The trajectory is subdivided in segments at a distance of 5 metres in straight portions and at a lower distance, 1 metre, in road curves position. From each segment the two vertices are projected onto the point cloud, in such a way that their Z coordinate corresponds to the ground surface. The oriented bounding box is defined by eight points and, for definition, must be a rectangular cuboid. In practice, the Bounding Box is defined starting from the two projected vertices (points A, B in Figure 2) from the trajectory. Using the line between points A,B as a symmetry axis, a rectangle (CDEF in Figure 2) is established. From the four vertices of the rectangle, 8 new points are formed by moving the vertices upward and downward of 1 meter. The bounding box is then defined based on those eight points. Actually the eight points does not define a rectangular cuboid, but a parallelepiped, because the angle between faces is not a right-angle. The Bounding Box is created simply by computing the oriented Bounding Box that enclose all those 8 points. The final shape will be very similar to the one desired.
In the case of the figure, only points E,C bottom and D,F top exactly correspond to the Bounding Box vertexes. Each Bounding Box is then used to crop the point cloud and to obtain a sub-cloud, from which extract the ROI.
The ROI of the presented method is the ground surface of the urban environment. To extract the ROI for each sub-cloud, each selection was filtered depending on z component of points normal (Nz). Points with Nz < 0.9 were excluded from the selection. The method then proceed by processing one ROI of one sub-cloud at a time.

Sidewalk segmentation
This method takes advantage from the fact that different paving materials are used for the different urban elements. Therefore, the points of the point cloud pertaining to each paving area have a specific spatial arrangement which result in different values of some geometric features.
In order to apply a semantic label to points, each ROI is firstly subdivided in macro-clusters according to specific points features; on each macro-cluster a split operation is performed, then the clusters are labelled as "sidewalk", "road", or "other". In this work the main interest are sidewalk and road, for this reason the class "other" simply collect all points which do not pertain neither to road nor to sidewalk. Semantic labels are applied basing on topological relations like position respect existing buildings, trajectory line, average Z values, and other information about geometry (such as symmetry and shape).
The clustering algorithm implemented is K-means (Jin and Han, 2010), basing on specific features it is able to subdivided the dataset in k clusters. Ideally, on a general ROI there is one road portion, two sidewalks and some points pertaining to other elements, like doorsteps, or porches floors, or zebra crossings. Basing on this fact, and after some empirical tests the "k" number of cluster chosen is 5.
The feature selected to be used by the algorithm are 5: Omnivariance, Sphericity, Roughness, Intensity of the returning signal emitted by the laser scanner, and Z coordinate of points. The first three are related to the distribution of points in 3D space and were computed relying on a spherical neighbour computed in a radius of r = 0.05 metres. For each point a spherical neighbour is extracted and the Omnivariance and Sphericity can be computed relying on the eigenvalues from the local covariance matrix (Blomley et al., 2014). The Roughness describes the surface variation of the neighbour and can be computed as the mean distance of points in the neighbour respect an interpolated First the ROI of a sub-cloud is selected and clustered, then the split operation is implemented and semantic labels are applied.
plane (Díaz-Vilariño et al., 2016). The Intensity is a function of several variables, including the distance from the laser, the angle of incidence of the laser beam on the surface and the specific material reflectance (Yuan et al., 2020). Since the method considers different paving materials, the intensity is an appropriate features. Even if the method is specifically tailored for non-standard urban area, it may happen that different urban elements have different elevation, so the Z coordinate is used as feature because it could be helpful in those areas.
After clustering, it can happen that some clusters are not homogeneous and contains some sparse points aggregated in subclusters. To split those elements, a split operation is performed implementing a density-based algorithm: DBSCAN (Sander, 2010). This algorithm is able to find clusters according to the density of points, the features provided to find clusters are the XYZ coordinates of points, and a minimum distance to consider two points in the same clusters. The selected distance is in accordance with the point density on ground and it was set to 5 cm. The result of this step is a set of sub-clusters which better subdivide the ROI. Figure 3 is meant to show the process from a single ROI, after the application of K-means clustering (the result is composed by k = 5 clusters), and to the application of DBSCAN. It can be noticed how the red cluster after the k-means is composed by two sub-clusters, after the subsequent step, those points are pertaining to 2 isolated sub-clusters.

Labelling
Semantic labels are applied to points according to the relative position of clusters respect the road centerline, or respect buildings footprints. The road centerline almost corresponds to the trajectory of the MLS system, but it can also be simulated by the road polyline that represent roads in Open Street Map (OSM). Open Street Map is a free, editable map of the whole world that is being built by volunteers largely from scratch and released with an open-content license(www.wiki.openstreetmap.org). OSM dataset can be downloaded for free directly from the main website or using GIS software. Buildings footprints are represented in OSM by polygons, which can be easily downloaded, as for the road centerline.
To easily analyse the clusters, their center point is used as representative. First, clusters with higher number of points and furhter away from the road centerline, are considered as sidewalk candidates. Then, to confirm a candidate as a sidewalk, a first assumption is made: every candidate which is in front of a building is a sidewalk. To do so, the proximity of a cluster to a building footrpint is performed: if the outcome is positive, the candidate is confirmed as sidewalk. The following check is for areas where there is not a building polygon (e.g. due to a missing data in OSM dataset or simply because there is a square near the sidewalk). To address this case, firstly an assumption on the typical road configuration is made: if there is a sidewalk on one side of the road, there should be also another one on the other side. After this first check, the shape of the cluster is evaluated, if the shape is elongated and almost parallel to the centerline of the road, the candidate is confirmed as sidewalk. A last consideration is for the corners, in those areas the central point of clusters with points that follow a curve can be a source of error, so only for those areas instead of considering the central point, the whole cluster points should be considered. The assumption in corners is simply that the bigger cluster, further away from the road centerline, is the sidewalk.
The lowest image in Figure 3 shows the ROI labelled, colours are used to visualize semantic clusters: green for the "road" class, blue for the "sidewalk" class and orange for the "other" class.

Pavement characterisation
Each labelled cluster is assumed, in first hypothesis, as made by a single paving material. By analysing the geometrical features of points of the clusters, they are classified into one type of pavements, according to the paving material actually present in the historic city: "sampietrini", "cobblestone", "stone" and "bricks" (see Fig. 6). Several samples of paving materials, each of 1 square meter of paving surface, were extracted and analysed. Among the features analysed, the two considered most interesting are reported in Figure 4. The presented graphs show the probability density functions of Omnivariance and Intensity values, for the 4 different paving materials. It is noticeable that the Intensity values were exported using ".pts" format from Leica Cyclone software, which export the Intensity in a range of values from -2048 to 2048. The Omnivariance was computed relying on a spherical neighbour of 0.05 metres.
Basing on the resulting graphs (Fig. 4) it is possible to develop a strategy which identify the paving material basing on some thresholds. The proposed method, presented in Figure  5, firstly check the Omnivariance mean value of the analysed  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B4-2021XXIV ISPRS Congress (2021 sample, if it is lower than a specific threshold (THR-1) the sample is considered as "sampietrini" or "cobblestone". Then, to define which is of the two pavings, the shape of the Intensity distribution curve is analysed, a narrow curve is for "sampietrini" and a wider curve is for "cobblestone". The shape of the curve is retrieved by looking to the standard deviation of Intensity value of the sample and checked with a second threshold (THR-2). If a sample has the mean Omnivariance higher than the first threshold, the sample could be "stone" or "bricks", to identify the correct paving material the mean value of Intensity is checked and compared with a last threshold (THR-3).
To determine the three thresholds, their value is computed starting from the graphs in Figure 4. The threshold on mean value of Omnivariance (THR-1) is determined as the intersection between the "bricks" and "sampietrini" curves, its value is 0.00009 metres. The threshold on the standard deviation of the Intensity distribution curve (THR-2) is determined as the average between the average of standard deviation of "sampietrini" and "stone", and the average of standard deviation of "bricks" and "cobblestone". The threshold value is set as 125.
The last threshold is for the mean value of Intensity (THR-3), it is defined as the intersection between the "stone" and "bricks" curves. Its value is -1300.

The dataset
The case study chosen to test the methodology proposed in this paper is the historic city of Sabbioneta: founded between 1556 and 1591, it was designed according to the Renaissance principles of the Ideal City. Since 2008 it is a UNESCO site.
Almost the entire city was surveyed by a MLS system: Leica Pegasus Two. The survey is composed by 10 acquisition missions (or tracks), which combined correspond to a total length of 7700 metres and 1,200,000,000 points. The points on ground have an average spacing of 3 centimetres.
Sabbioneta represents a good example of the peculiarities of an historic site; in fact it has fortified walls, narrow roads, pedestrian areas, and it has mainly 4 paving materials: "cobblestone", "sampietrini", "bricks", and "stone" (see Figure 6).
In order to test the presented method, two acquisition tracks out of the 10 of the survey on Sabbioneta were selected. The two selected areas (track A and track B) are interesting and representative under several points of view: the co-presence of various paving materials, the presence of strict curves, and crossing with other tracks. Track A is 650 metres long and track B is 170 metres long. The two point clouds are respectively composed by 121 Million of points and 35 Million of points. Track A presents 5 straight paths connected by four 90 degrees angle curves. The first half of track A is composed by narrow roads and the second half is composed by a wider road (almost doubled in width). The paving materials are various, in some portions of track A there are "cobblestone" (for road) and "bricks" (for sidewalks), in other areas there are "sampietrini" (for road) and "stone" (for sidewalks). Ground elements are at the same Z level in some portions and with height difference in other portions of track A. Track B is a straight road with 3 crossing with other roads, the materials are "cobblestone" and bricks, and all the elements are at the same Z level. Track A was subdivided in 119 ROIs and track B was subdivided in 33 ROIs. Table 1 summarise the two track main characteristics. Figure 6. Paving materials in the case-study. a: combination of "cobblestone" and "bricks", b: combination of "sampietrini" and "stone", c: combination of "sampietrini", "stone", and "cobblestone", d:"cobblestone", e:"bricks", f:"stone", g:"sampietrini".

Sidewalk segmentation and labelling
The purpose of the sidewalk segmentation is to apply a semantic label to points of each ROI. The method is tested on an Intel(R) Core(TM) i7-3820 CPU, 3.60GHz, 4 core, 8 logic processors, with 64 GB RAM. On average the processing time is 2 minutes each ROI.
The clustering-based approach (implementing the K-means and DBSCAN algorithms) is able to find clusters pertaining to different paving materials sometimes the clustering is excessive, for example one single element can be subdivided into more than one single cluster. The labelling method implemented is able to cope with this issue and produce a final satisfactory result. In fact, even if one ground element does not correspond to a unique cluster, the job of merging several clusters associated to the same ground element is assigned to the topological classification. The topological classification approach works well but it is slightly dependent on the correct presence of building polygons from OSM.
To evaluate the method performance, the results were compared with a ground truth, the confusion matrix is computed and Precision, Recall and F1 score are calculated. Results are provided in Table 2; Figure 7 shows the two tracks with points coloured according to their final label. The class with better score is the "road", a lower performance on sidewalk class can be related to some errors that still to be solved in the topological step, in fact some sidewalks in corners, or in areas where a parked car create occlusion in the dataset, the sidewalk is labelled as road or as  Table 3. Performance values of the paving material characterisation method, in terms of Precision, Recall and F1-score. Please note that dataset of track B does not have "sampietrini" and "stone" classes, so their performance values are not reported. Figure 7. The two tracks of the case-study selected, after the Semantic Labels computation. In green the "road" class, in blue the "sidewalk" class, and in orange the "other" class (which collect all the remaining urban elements).
"other". Track B has a low performance on class "other" and that is mainly due to errors due to errors in topological step, as previously described. Track B, in fact, has lots of parked cars and has 3 crossings with other roads.
By visually analysing the results, it is possible to find some repetitive errors. Possible classification errors occur when a polygon is absent, in such a case it is necessary to implement other ways to label clusters as sidewalks. Corners and crossing between roads still represent a possible source of errors. Errors in classification are also presents near squares, or where there are ramps on sidewalks (the ramp can be classified as "road"). Considering the small Z thickness of the sub-clouds (2 meters, centered on the road) and considering the ROI, which considers only points mainly pertaining to horizontal surfaces, it can be observed that the class "other" mainly collect the doorsteps and the floors under porches. Figure 8. The two tracks of the case-study selected, after the paving material characterisation computation. In purple the "cobbelstone" class, in orange the "sampietrini" class, in blue the "bricks" class, and in green the "stone" class.

Pavement characterisation
Different paving materials have different features, so the material classification rely on them. The purpose of this tests is to validate the proposed method, so (at this stage) it has been performed on the ground truth from previous step, such that the results from paving characterization tests are independent from errors in semantic labels. The final method will use the semantic labels coming from previous computations.
For each ROI, the semantic labels are used to create separate clusters: road, sidewalk left, sidewalk right. Then, a sample of points in 1 square meter is extracted from each semantic cluster, then the process implemented is the one described in previous section. The processing time (on the same machine as 4.2) is, on average, 7 seconds for each ROI. The results are compared with a ground truth and presented in Table 3 in terms of Precision, Recall and F1 score. The two point clouds with points coloured according to the computed paving material are shown in Figure 8.
The road in track B is simpler respect the one in track A, in fact, the paving materials in track B are only "cobblestone" (for the road) and "bricks" (for the sidewalks); track A presents roads with a combination of all the four materials. The performances are very positive, few times "cobblestone" are interpreted as "stone", and this is a result connected to the choice of threshold THR-1. The choice of a method based on thresholds can be improved, even if this first results are encouraging.

CONCLUSIONS
This work presents a method to perform a semantic segmentation of a point cloud of an historic urban environment. This work is challenging because in urban heritage other segmentation methods, mainly based on curbs detection, can fail. The proposed method is based on the fact that in historic areas different urban elements are identified by different pavings. The proposed method firstly subdivides the point cloud in ROIs, then each ROI is clustered according to specific features, and a semantic label is applied to points, according to topological relations between elements. To each semantic cluster (road or sidewalk) is then calculated an attribute: the paving material.
The sidewalk segmentation method has been proved to be robust in straight road portions, independently on the width, while some errors occur in corners and areas where there are some occlusions, for example where there are parked cars. This problems can be solved by applying some mathematical morphology methods on the continuity of elements, to close gaps (Balado et al., 2020). The labelling step relies on topological relations considering also the building footprint. In the presented work the building footprint is retrieved from OSM, which sometimes presents some missing data. The method can work also using building data coming from other cartographic sources which can be more complete respect OSM, for example cadastre data or from a topographic dataset.
The pavings characterisation method is dependent on some parameters, which were defined basing on a small sample of the point cloud. The usefulness of the defined thresholds was proved by applying them on the entire dataset, which correspond to two roads with a total length of 1156 metres. Even if the results are promising, in future works a better definition of this method will be studied, mainly relating it to statistical reasoning on data distribution instead of using thresholds. Possibilities given by supervised machine learning methods will be evaluated.
Future work will extend this method to the entire city of Sabbioneta, proving its efficiency and usefulness for the further definition of maps, to plan routes, and to better manage the urban environment of the city. The method will also be tested on dataset of different cities and coming form different instruments.