A REVIEW OF POINT CLOUDS SEGMENTATION AND CLASSIFICATION ALGORITHMS

: Today 3D models and point clouds are very popular being currently used in several fields, shared through the internet and even accessed on mobile phones. Despite their broad availability, there is still a relevant need of methods, preferably automatic, to provide 3D data with meaningful attributes that characterize and provide significance to the objects represented in 3D. Segmentation is the process of grouping point clouds into multiple homogeneous regions with similar properties whereas classification is the step that labels these regions. The main goal of this paper is to analyse the most popular methodologies and algorithms to segment and classify 3D point clouds. Strong and weak points of the different solutions presented in literature or implemented in commercial software will be listed and shortly explained. For some algorithms, the results of the segmentation and classification is shown using real examples at different scale in the Cultural Heritage field. Finally, open issues and research topics will be discussed.


INTRODUCTION
We are recently witnessing an increasing availability of notinterpreted point clouds and 3D models, often shared online using point-based rendering solutions (e.g.PoTree) of mesh-based portals (e.g.Sketchfab).If we focus on point clouds, there is a growing need of innovative methods for the treatment and analysis of these data and for their classification, aimed ultimately to exploit in-depth the informative value of these surveys and representations.3D point clouds are the simplest but at the same time powerful collection of elementary geometrical primitives able to represent shape, size, position and orientation of objects in space.This information may be augmented with additional contents obtained from other sensors or sources, such as colours, multispectral or thermal information, etc.For a successful exploitation of point clouds and to better understand them, we must first proceed with segmentation and classification procedures.The former refers to group points in subsets (normally called segments) characterized by having one or more characteristics in common (geometric, radiometric, etc.) whereas classification means the definition and assignment of points to specific classes ("labels") according to different criteria.Due to the complexity and variety of point clouds caused by irregular sampling, varying density, different types of objects, etc., point cloud classification and segmentation are very active research topics.There are multiple research studies related to these two topics, many driven by specific needs provided by the field of application (building modelling, Heritage documentation and preservation, robotics, etc.).Most of the segmentation algorithms are tailored to work with a 2.5D surface model assumption, coming for example from a LiDAR-based survey.Many algorithms require a fine-tuning of different parameters depending upon the nature of data and applications.Supervised methods are the majority with a training phase mandatory and fundamental to guide the successive machine learning classification solution.Some of the techniques developed for segmenting point clouds generated from airborne laser scanning can be applied or easily adapted to terrestrial point clouds.The results are generally affected by noise and density of the cloud as well as by the quality of the training data.Different benchmarks were proposed in the research community, the most comprehensive study being carried out by the previous ISPRS Working Group III/3 "3D Reconstruction from Airborne Laser Scanner and InSAR Data"' with the aim to segment and classify points in bare earth and object classes (Sithole & Vosselman, 2003;Sithole & Vosselman, 2004).The study was initiated to compare the performance of various automatic filters with the purpose of (i) determining the comparative performance of existing filters, (ii) understanding the influence of point density on the filter performance and (iii) identifying directions for future research on point clouds filtering algorithms.A more recent benchmark is the "Large-Scale Point Cloud Classification Benchmark" (www.semantic3d.net)that provides labelled terrestrial 3D point cloud data on which people can test and validate their algorithms (Fig. 1).

SEGMENTATION
A first attempt to group segmentation methods follows the works of Sapkota (2008) and Nguyen (2013) and a schematic representation is shown in Figure 2.

Edge-based segmentation
As described by Rabbani et al. (2006), edge-based segmentation algorithms have two main stages: (i) edge detection to outlines the borders of different regions and (2) grouping of points inside the boundaries to deliver the final segments.Edges in a given depth map are defined by the points where changes in the local surface properties exceed a given threshold.The mostly used local surface properties are normals, gradients, principal curvatures or higher order derivatives.Methods based on edge-based segmentation techniques are reported by Bhanu et al. (1986), Sappa and Devy (2001), Wani and Arabnia (2003).Although such methods allow a fast segmentation, they may produce not accurate results in case of noise and uneven density of point clouds, situations that commonly occur in point cloud data.In 3D space, such methods often detect disconnected edges making the identification of closed segments difficult without a filling or interpretation procedure (Castillo et al., 2013).

Region growing segmentation
These methods start from one or more points (seed points) featuring specific characteristics and then grow around neighbouring points with similar characteristics, such as surface orientation, curvature, etc. (Rabbani et al., 2006;Jagannathan and Miller, 2007).Region-based methods can be divided into:  Bottom-up approaches: they start from some seed points and grow the segments on the basis of given similarity criteria.
Seeded region approaches are highly dependent on selected seed points.Inaccurate selection of seed points will affect the segmentation process and can cause under-or oversegmentation results. Top-down approaches: they start by assigning all points to one group and then fit a single surface to it.Where and how to subdivide unseeded-region remain the main difficulty of these methods.Region-based algorithms includes two steps: identification of the seed points based on the curvature of each point and growing them based on predefined criteria such as proximity of points and planarity of surfaces.The initial algorithm was introduced by Besl et al. (1988) and then several variations were presented in the literature.The region growing method proposed by Vosselman et al. (2004) has introduced the use of colour properties beside geometrical criteria.Surface normal and curvatures constraints were widely used to find the smoothly connected areas (Klasing et al., 2009;Belton and Lichti, 2006) whereas Xiao et al. (2013) proposed to use sub window as the growth unit.Ackermann and Troisi (2010) used a region growing approach to segment planar pitched roofs in 3D point clouds for automatic 3D modelling of buildings.Anh-Vu Vo et al. (2015) presented an octree-based region growing approach for a fast surface patch segmentation of urban environment 3D point clouds.
A collection of region growing algorithms is available in the Point Cloud Library (http://pointclouds.org).Figure 2 shows the results of a segmentation done by a region growing algorithm implemented in the pcl::RegionGrowing class.The purpose of the algorithm is to merge/join similar points and deliver a set of clusters with points belonging to the same smooth surface.In general, the region growing methods are more robust to noise than the edge-based ones because of the using of global information (Liu and Xiong, 2008).However, these methods are sensitive to (i) the location of initial seed regions and (ii) inaccurate estimations of the normals and curvatures of points near region boundaries.

Segmentation by model fitting
This approach is based on the observation that many man-made objects can be decomposed into geometric primitives like planes, cylinders and spheres (Fig. 3).Therefore, primitive shapes are fitted onto point cloud data and the points that conform to the mathematical representation of the primitive shape are labelled as one segment.As part of the model fitting-based category, two widely employed algorithms are the Hough Transform (HT) (Ballard, 1981) and the Random Sample Consensus (RANSAC) approach (Fischer and Bolles, 1981).In case the primitives have some semantic meaning, then such approach is also performing a classification.The HT is used to detect planes (Vosselman et al., 2004), cylinders and spheres (Rabbani et al., 2006).The RANSAC method is used to extract shapes by randomly drawing minimal data points to construct candidate shape primitives.The candidate shapes are checked against all points in the dataset to determine a value for the number of the points that represents the best fit.A popular strategy in reverse engineering involves locally fitting primitives like planes, cylinders, cones using RANSAC based methods (Schnabel et al. 2009).
Chen et al (2014) has proposed a modified RANSAC segmentation algorithm that is less sensitive to noise, maintains topological consistency, and avoids over and under-segmentation of building primitives, through the localized sampling to segment the polyhedral rooftop primitives and then through the application of a region growing based triangulated irregular network (TIN) to separate the coplanar primitives.Several extensions are available within the Point Cloud Library:  MLESAC (Maximum Likelihood Estimation SAmple and Consensus)  MSAC (M-estimator SAmple and Consensus)  PROSAC (Progressive Sample and Consensus) Model fitting methods are fast and robust with outliers.Their efficiency for the 3D detection of geometrically simple parameterized shape such as cylinders, spheres, cones, torus, planes and cubes has been proven, providing an efficient shape descriptor with insight over the geometrical properties of a point cloud sample.As it falls short for complex shapes or fully automated implementations, the use of the richness of surface geometry through local descriptors provide a better solution (Poux et al., 2016).In the architectural field, details cannot always be modelled into easily recognisable geometrical shapes.Thus if some entities can be characterized by geometric properties, others are more readily distinguished by their colour content (Barnea and Filin, 2013).

Hybrid segmentation technique
In kind of approach, more than one method is combined, e.g. to detect planar segments, in order to exploit the strength a method and bypass the weakness of other methods (Vieira and Shimada, 2005;Lavoue ́ et al., 2005;Benko and Varady, 2004).The success of these hybrid methods depends on the success of the underlying methods.

Machine learning segmentation
Some segmentation algorithms are based on machine learning methods, such as hierarchical clustering, K-means or mean shift.Machine learning (including deep learning, neural network, etc.) is a scientific discipline concerned with the design and development of Artificial Intelligence algorithms that allow computers to take decisions based on empirical and training data.A learner can take advantage of examples (data) to capture and infer characteristics of interest of their unknown underlying probability distribution.Data can be seen as examples that illustrate relations between observed variables.In machine learning, unsupervised learning is a class of problems in which one seeks to determine how the data are organized.It is distinguished from supervised learning (and reinforcement learning) as they rely on a set of provided training examples (features) to learn how to correctly perform a task.Features play a fairly important role in these problems and their definition is one of the bottleneck of machine learning methods.Features with high quality can simplify learning models to interpret the models more easily and enhance algorithm performance with respect to both the speed and the accuracy.Weinmann et al. (2015) discusses the suitability of features that should privilege quality over quantity.This shows a need to prioritize and find robust and relevant features to address the heterogeneity in a point cloud.Machine learning methods are quite robust and flexible, however they rely on the point cloud density and normally feature very long computational time.

Segmentation based on K-means clustering
It is a method based on an algorithm able to classify or to group set of (3D) points into K groups using attributes/features.The grouping is done by minimizing the sum of squares of distances between point and the corresponding cluster centroid.The original K-means algorithm presented by MacQueen et al. (1967) was then exploited for point clouds by various researchers (Comaniciu and Meer, 2002;Lavoue ́ et al., 2005;Yamauchi et al., 2005, Zhang et al., 2008;LeCun et al., 2015).

Segmentation based on hierarchical clustering
These methods compute representative measures (features) for each (3D) point, based e.g. on geometrical and radiometric characteristics: point position, locally estimated surface normals, residuals of best fitting surface procedures, points reflectance, etc.They usually create a hierarchical decomposition of a dataset by iteratively splitting the dataset into smaller subsets until each subset consists of only one object (Ng and Han, 1994).Xiaohu Lu et al. (2016) recently presented a novel hierarchical clustering algorithm which clusters any dimensional data and can be applied to mobile mapping, aerial and terrestrial point clouds.

CLASSIFICATION
Once a point cloud has been segmented, each segment (group) of points can be labelled with a class thus to give some semantic to the segment (hence point cloud classification is often called semantic segmentation or point labelling).Point cloud (or mesh) classification is gaining interest and becoming a very active field of research (Weinmann et al., 2013;Guo et al., 2014;Niemeyer et al., 2014;Schmidt et al., 2014;Weinmann et al., 2014;Xu et al., 2014;Hackel et al., 2016).The class labelling procedure is normally achieved following three different approaches:  a supervised approach, where semantic categories are learned from a dataset of annotated data and the trained model is used to provide a semantic classification of the entire dataset.A large amount of annotated data is normally mandatory to train the model. an unsupervised approach, where the data is automatically partitioned into segments based on a user-provided parameterization of the algorithm.No annotations are requested but the outcome might not be aligned with the user's intention. an interactive approach, where the user is actively involved in the segmentation/classification loop by guiding the extraction of segments via feedback signals.This requires a large effort from the user side but it could adapt and improve the segmentation result based on the user's feedback.As suggested in Weinmann et al. (2015) most of the approaches for point cloud classification consider the different components of the classification process (i.e.neighbourhood selection, feature extraction and classification) independently from each other.However, it would be desirable to connect these components by sharing the results of crucial tasks across all of them.Such a connection would not only be relevant for the interrelated problems of neighbourhood selection and feature extraction, but also for the question of how to involve spatial context in the classification task.

APPLICATIONS IN THE CULTURAL HERITAGE FIELD
Processes like segmentation and classification can be apply at different scales in the field of Cultural Heritage, from an entire archaeological sites to a small artefact.In the following sections some examples are reported and commented.Figure 6: The digitized and textured Neptune temple in Paestum and the classification of its archaeological elements (Fiorillo et al., 2013).

Segmentation with a Prismatic Buffered TIN method
The Prismatic Buffered Triangular Irregular Network (PBTIN) method presented in Menna & Troisi (2007) was applied to separate man-made structures from bare earth in the UAV-based dense point cloud of the entire archaeological site of Paestum (Italy).The method is based on an iterative densification of a TIN built on some initial seed points belonging to the class bare earth.As shown in Figure 5 the algorithm was used to separate the terrain from any man made structure and vegetation standing off the bare earth.This segmentation method is better suited for structured point clouds while for unstructured point clouds it suffers of noise, outliers and inhomogeneous point densities.

Segmentation with primitive fitting
Digital copies of large and complex monuments could be hardly accessible and manageable by non-experts and policy makers.Segmentation and classification procedures could facilitate understanding, interpretation, management and access to complex information.In Fiorillo et al. (2013) semantic knowledge was manually added to a 3D model of a Greek-Roman temple to allow a better understanding and management of the heritage object (Fig. 6).

Machine learning segmentation
The Canupo segmentation algorithm (Brodu and Lague, 2012) implemented in CloudCompare was used to separate vegetation and stones on the Seiano's gatethe imperial entrance to the archaeological site of Pausilypon (Naples, Italy) (Fig. 7).The Canupo plug-in allows to create own classes as well as to use existing classifiers for segmenting point clouds into subsets (e.g.vegetation, ground, etc.).This supervised method is based on 3D geometrical properties of the point cloud across multiple scales and, employing a probabilistic approach, the points with high uncertainty can be removed from the wrong class.
Farella (2016) used the same classification algorithm for the separating man-made structures from natural elements (trees, bushes, etc.) in point clouds of military fortifications built during the First World War (WWI) on the hills around Trento, Italy (Fig. 8).

CONCLUSIONS
The paper presented a review of point clouds segmentation and classification algorithms.The review is probably not fully exhaustive but it reports many approaches suitable for the geospatial and heritage communities.The key difference among most segmentation approaches is the method or criterion used to measure the similarity between a given set of points and hence for making the grouping decisions.All methods were grouped in five categories based on their core approach.
According to what the user's needs and data type, the correct algorithm must be chosen.Many segmentation algorithms, for example, are tailored to work with structured or LiDAR data and thus are simply not applicable to other types of point clouds.
Other methods work with RGB information, but not every point cloud is always coloured.Many solutions rely on a large set of parameters which need to be fine-tuning for every processed dataset.Therefore it's impossible to make a generalization and decide which approach is the best as each dataset is distinct and needs to be treated as such.Although advances are being made in the field of segmentation and classification, particularly with machine learning methods, most of the processing steps still requires human intervention to achieve satisfactory results.

Figure 2 :
Figure 2: Synthetic representation of the segmentation methods.

Figure 3 :
Figure 3: Point cloud segmented with a region growing algorithm available in the Point Cloud Library (PCL).

Figure 4 :
Figure 4: Segmentation of 3D point cloud by geometric primitive fitting.
Tarsha-Kurdi et al. (2007) compared RANSAC and 3D Hough transform for automatically detect roof planes from LiDARbased point clouds.Despite the limitation encountered in both methods, RANSAC resulted the more efficient considering segmented results and running time.It can process a large amount of input data in negligible time.On the other hand, 3D HT resulted slower and more sensitive to the segmentation parameters values.

Figure 5 :
Figure 5: 3D point cloud of the archaeological site of Paestum (left) and segmentation to separate bare earth and man-made structures (right).