BUILDING FEATURES RECOGNITION FROM UNCERTAIN 3D LIDAR POINT CLOUDS: A SEMANTIC APPROACH

LiDAR technology allows rapid observation of high-resolution and precise 3D point clouds for diverse applications in urban and natural areas. However, uneven density and incomplete point clouds make LiDAR data processing more challenging for the extraction of semantic information on objects and their components. In this paper, we propose a knowledge based semantic reasoning solution for the recognition of building components (e.g. roofs) from segmentation results in the presence of uncertainties in LiDAR point clouds. The proposed solution uses a semantic reasoning approach as well as a similarity evaluation process for object recognition. We apply the proposed method to recognize buildings’ roof styles from a point cloud with uncertainty as a case study.


INTRODUCTION
LiDAR technology is increasingly used for the rapid acquisition of 3D high-resolution and precise data for diverse applications in urban and natural areas. However, in spite of the advancement of technology in recent years, different problems such as occlusions still may lead to an uneven point density or incomplete point clouds. This may complicate the process of segmentation and detection of objects from LiDAR point clouds. This is because the segmentation process uses geometric properties of objects to decompose it into more primitive features and provides information on their shape and their geometric and topological relations. Missing data on these properties in LiDAR point clouds may further complicate the extraction of semantic information on objects and their components that are fundamental to their modeling and identification.
Uncertainty in the information extracted from a LiDAR point cloud may include uncertainty in geometric properties of segmented objects, uncertainty in geometric relations between object components, as well as uncertainty on the topological relations between object components (Parkan, 2018). Segmentation provides segments with richer information for further processing than the processing of each isolated point, such as identifying geometric relation and topological relation, detecting geometric shapes. The analysis of segmentation results based on the constraints and rules for defining prior knowledge (Che, 2019) supports the extraction of higher-level semantic information of objects from point clouds. Knowledge-based approaches (Boochs, 2011;Hmida, 2012;Pu, 2011;Pu, 2009;Truong, 2013;Xing, 2018) provide interesting alternative methods to overcome some of these problems for object detection and recognition. However, in the presence of uncertainty and incompleteness in the data, not all the knowledge-based methods are suitable to assess such data. Due to the presence of uncertainty and incompleteness in the information, we do not have all the necessary information for the inference process to obtain higher-level semantic information of objects. * Corresponding author This paper proposes a knowledge based semantic reasoning solution for the recognition of building components from segmentation results in the presence of uncertainties in LiDAR point clouds. The proposed solution uses a semantic reasoning approach combined with a similarity evaluation process for object recognition. At the first stage, the knowledge of a specific object is formalized as semantic rules. Then the uncertain information extracted from point clouds is compared to the predefined knowledge for the recognition of the object. Then the similarity between geometric properties, geometric relations, and topological relations defined in rules and those extracted from point clouds is evaluated. Based on the similarity evaluation, the most appropriate semantic rules are chosen from the knowledge base to conduct semantic reasoning for object recognition purposes. We apply this method to recognize buildings' roofs from a point cloud with uncertainty as a case study. The following section presents a brief literature review on the topic. Then the proposed methodology and examples of results from a case study are presented and discussed.

RELATED WORK
Recent developments suggest that knowledge-based solutions offer interesting alternative solutions to tackle object recognition problems and help to extract semantic information from point clouds. For this purpose, formalized representation of knowledge on complex objects of interest (e.g. buildings) is critical for the recognition of their components from point clouds. For this purpose, ontologies are commonly used to specify and formally describe concepts and their relations in a given domain with a defined level of granularity. An ontology consists of concepts, relations, axioms, and instances (individuals in OWL) (Stuckenschmidt, 2009). Several languages are developed for formal representation of ontologies. Web Ontology Language (OWL) (Peter F. Patel-Schneider, 2004) can be used to explicitly represent meanings of concepts and their relationships (W3C). Semantic Web Rule Language (SWRL) (W3C, 2004) offers the capacity for representing a high-level abstract syntax for Hornlike rules that are a human-readable format for representing knowledge. However, the limitation of the formal knowledge representation is that the classic binary semantic (true or false) cannot represent imprecise knowledge and reason new knowledge from uncertain information (Bobillo, 2016). Although fuzzy ontology (Bobillo, 2011;Bobillo, 2016) has been explored to represent vague concepts and properties, the reasoning process is designed based on the fuzzy set theory and the new language needs to be developed to implement fuzzy ontology and reasoning from fuzzy information.
Several approaches were developed for dealing with uncertainties in different steps of LiDAR point cloud processing. Geometric reasoning between entities (e.g., point, line, and plane) is used in the 3D modeling from point clouds. Projective geometry (Heuel, 2004) is capable of representing geometric entities and reasoning geometric relations from point clouds. This method was explored to determine the boundaries of man-made structure (Loch-Dehbi, 2011).
For determining topological relations from uncertain observation for qualitative reasoning, the morphological distance (Winter, 1996;Winter, 2000) and the topological distance by comparing the intersection matrix represented by the 9-Intersection model (Egenhofer, 1992;Kang, 2004) were developed to compare the similarities between topological relations. The distance between qualitative topological relationships evaluated by the number of direct transitions was used to detect the relations of objects in 3D space in the case of movement and occlusion (Sabharwal, 2013). Topological distance can be used to evaluate the degree of similarities between the topological relations extracted from a point cloud with uncertainties and the ones which are used as reference topological relations defined in a knowledge base.
Comparing similarities in the presence of uncertainty for the extraction of higher-level information has proven its potential in different domains. Cosine similarity (Han, 2012) was widely used in data mining, recommendation systems (Kotu, 2019), and search ranking (Long, 2014). Hence, for dealing with uncertainties in our context, we propose to integrate the similarity evaluation of properties and relations into the reasoning step in the knowledge-based solution for extracting higher-level semantic information of objects.

METHODOLOGY
Our proposed knowledge based solution combines a semantic reasoning method and a similarity evaluation process for the identification and recognition of an object's components from point clouds. The solution includes the following main steps: (1) Formalize geometric and topological information, and geometric and topological relations extracted from point clouds as properties and relations related to individuals that represent objects and their components.
(2) Compare the similarities between properties and relations. The similarities between the properties and relations of individuals and those defined in rules representing knowledge of objects are estimated at the semantic level.
(3) After estimating similarities, the similarities corresponding to each atom in the rules are used to construct a similarity vector. The cosine distance between the similarity vector and the reference vector that has the same dimension as the similarity vector is used to choose the most probable rule to reason the semantic information for a given individual.

Translate Segments into Individuals
First, a given point cloud is segmented and information such as geometric properties, geometric relations, and topological relations are extracted and formalized before adding to the knowledge base. For example, a building is segmented into geometric segments and the geometric and topological properties and relations between segments are identified. Then, for each segment, the geometric and topological information is formalized in the ontology. This information is considered as facts in the reasoning step describing a specific instance of a building and its components. This process is done for all the objects detected in the point cloud.

Build a Knowledge Base:
A knowledge base composed of concepts and their relations and semantic rules. It formally represents concepts and their properties and relations between concepts described in an ontology. The knowledge instances of the concepts (objects in reality) can be extracted and formalized in the knowledge base.
For instance, for the concept in the knowledge base mainly include two groups of concepts: building-related concepts (e.g., building components (walls, roofs, ceilings, columns, and doors), and building roof styles (gable, hip, pyramid hip, and hip and valley roofs)), and geometry-related concepts (e.g. "PlanarRegion_3D" represents a planar segment in 3D space segmented from point clouds, Cylinder_3D and Sphere_3D represent the cylinder and sphere extracted from point clouds).
It also includes geometric properties that describe the above concepts. The properties, such as "hasLength", "hasWidth", "hasShape", "hasHeight", "hasBoundary", describe the concept "PlanarRegion_3D". Other properties, such as color and texture can also be added to describe concepts.
The geometric relations and topological relations between instances of concepts are crucial to describe the spatial relations between concepts and to compose other concepts. The geometric relations "isParallelTo", "isVerticalTo" can describe relations between a building's components. For instance, the geometric relations between walls could be vertical or parallel. The relation between a wall and a floor is generally vertical, and a column is vertical to a floor as well. The topological relations describe the spatial relations of geometric objects, independent of their spatial transformations such as translation, scaling, and rotation (Egenhofer, 1990). Topological relations between object's components are abstracted as the topological relations between planar regions in 3D space. Then, these relations are formalized as Tp1-Tp2-Tp3-Tp4, where Tpi is a semantic description of topological relation (see section 3.1.3 for more details) and it could be one of the topological relations (e.g., disjoint, meet, overlap, contain, cover and equal) as defined in a 2D space. Based on the formalized topological relations, the possible topological relations between object components are formalized as a four-word semantic description.
In addition, the knowledge base includes semantic rules. Based on the properties, concepts, and relations, a set of atoms (e.g., OWL class), properties, built-in relations, instances or data value, are used to construct a rule with a "Human Readable Syntax" form ( 1 ∧ 2 ∧ ••• ∧ ⇒ ) when both antecedent and consequent are conjunctions of atoms. Here an atom may represent a geometric property, geometric relation, topological relation, etc. The consequent represents the conclusion after reasoning. For example, building roofs with different styles (gable, hip, and pyramid hip) can be represented as rules based on the building components (planar regions) and their properties and the relations between them.
In the following paragraphs, more details on the formalization of properties, geometric relations and topological relations, and translating the information extracted from point clouds into individuals are presented.

Translate Geometric Information of Segments to Properties of Individuals:
In the proposed knowledge base, a building component represented by a planar segment is regarded as an individual of the concept "PlanarPolygon_3D". The geometric information of planar segments first needs to be extracted from point clouds. Point Cloud Library (Rusu, 2011) provides basic functions to calculate the geometric dimension of a planar segment from points, to identify the boundaries from points, and to calculate the intersection between lines and planes. In this paper, we directly use these functions to calculate the geometric information of a planar segment from points, such as its length, width, height, area, and boundary. These geometric properties are added into the properties of individuals through the predefined properties "hasLength", "hasWidth", "hasArea", "hasBoundary", "hasShape".

Translate Geometric Relations between Segments to
Relations between Individuals: Geometric relations between segments can be determined based on the parameters of the geometric equations of segments detected from point clouds. A plane equation a + b + c + d = 0 detected from point clouds can represent the plane that a planar segment locates. The numeric computation of the parameters of the equation decided the geometric relations between planar segments by calculating the relations between normal vectors of planes. A threshold is defined to decide the geometric relation parallel and vertical. The geometric relations are formalized as relations between individuals in the ontology, such as "isParallelTo" and "isVerticalTo".

Translate Topological Relations between Segments into Relations between Individuals:
The topological relations between components are abstracted as the topological relations between two planar regions in 3D space. A planar region in 3D space is defined as a planar surface area with a non-empty, connected interior similar to the definition of a region in 2D space for identifying topological relations. A planar region in 3D space is determined by the plane equation in which the planar region is located and its boundaries together (Xing, 2016). For example, two planar regions can be on the same plane in 3D space, but their boundaries separate them as two independent regions. Therefore, the topological relations between two planar regions are divided into three cases: (1) When two planar regions share the same plane equation in 3D space, it is same as the case of topological relation defined in 2D space; (2) When two planar regions locate on two parallel plane equations in 3D space, they have relation "disjoint"; (3) When two planar regions locate on two intersected plane equations in 3D space, the cases of topological relations between these two planar regions are more complex than the cases in 2D space.
In this paper, an extended DE-9IM is used to identify the topological relations between planar regions for obtaining the topological relation between 3D object components in the third case (Xing, 2016).
In the DE-9IM matrix, the interior, the boundary of planar regions, and the intersection line of two plane equations containing planar regions constitute a 3*3 matrix. The dimension operation on the elements in the matrix is the same as in 2D space. The DE-9IM 3*3 matrix is defined as follows: Apart from the element  , the dimension operation on other elements of the matrix is obtained as DE-9IM cases in 2D space. The last element  describes the topological relations between intersection parts on the intersection line. The intersection parts created by two planar regions and the intersection line could be points and lines. Thus, the topological relation between the intersection parts could be the relations of "point-point", "pointline segment" and "line segment -line segment".
Based on the matrix, the topological relations between the two planar regions are primarily determined by the relation between the intersection line and two planar regions and the relations between intersection parts on the intersection line. The formalized representation of topological relations between two planar regions can be represented as a four-word semantic description: Tp1-Tp2-Tp3 -Tp4. The detail of each part is presented as follows:  Tp1 is the overall description of the topological relation between two planar regions, including disjoint, meet, intersect.  Tp2 is the relation between planar region A and the intersection line, including disjoint, meet, overlap.  Tp3 is the relation between planar region B and the intersection line, including disjoint, meet, overlap.  Tp4 is the topological relation of intersection parts on the intersection line (Xing, 2016). The possible relations between two points on the intersection line are disjoint or equal. The topological relations between a line segment and a point are disjoint, meet, and contain. Similarly, the topological relations between two line segments are disjoint, meet, overlap, cover, coveredBy, contain, containedBy, and equal.
The proposed formalized representation of topological relations is used to describe the relations between individuals in the ontology. The topological relations are also used to define semantic rules to formalize the knowledge defined through the relations between object components.

Evaluation of the Similarities of Properties and Relations at the Semantic Representation Level
To reason with semantic information with uncertainty, a similarity evaluation approach for comparing formalized representations of properties and relations is required to evaluate how similar are the properties defined in the rule and those in the The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B4-2020, 2020 XXIV ISPRS Congress (2020 edition) This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLIII-B4-2020-89-2020 | © Authors 2020. CC BY 4.0 License. individuals translated from point cloud segments. The similarities between geometric relations and topological relations need to be compared as well.

Evaluation of the Similarities between Properties:
For quantitative properties (e.g., height, length, area, width), the similarity can be evaluated by: where is the given value extracted from point clouds. ℎ is the defined threshold for properties in the rules.

Evaluation of the Similarities between Formalized
Geometric Relations: Formalized geometric relations evaluated by numeric computation (e.g., parallel, vertical) are represented as "isParallelTo" and "isVerticalTo". The similarities between geometric relations are compared by the angle between plane normal vectors. We define the equations to calculate the similarities of the geometric relations ( ∥ for the similarity of the parallel relation and ⊥ for the similarity of the vertical relation).
Following the above definitions, when the normal vectors of two planes have an angle 0 degree, the similarity of their relationship compared to the parallel relation is 1. Similarly, the similarity compared to the vertical relation is 0. Thus, the proposed definition is capable of comparing the similarities of geometric relations with the help of a numeric computation of the angle between normal vectors.

Evaluation of the Similarity between Formalized
Topological Relations: For the formalized topological relations (Tp1 -Tp2 -Tp3 -Tp4) between object components identified from point clouds (Xing, 2018), the similarity is obtained by calculating the distance between semantic descriptions of topological relations. In this paper, the distance between formalized topological relations (e.g., Tpi) is defined as the steps of the topological transition (Randell, 1992). As shown in Figure  1, the topological transition step is 1 when the topological relation between a region A and a line is changed from "disjoint" to "meet". Similarly, the topological transition step is 1 from "meet" to "overlap" and the topological transition step is 2 from "disjoint" to "Overlap". Based on the definition of the topological transition step, the distance between formalized topological relations 1 1 − 2 1 − 3 1 − 4 1 and 1 2 − 2 2 − 3 2 − 4 2 is defined as: Where 1 = ( 2 1 , 2 2 ), 2 = ( 3 1 , 3 2 ), 3 = ( 4 1 , 4 2 ).
For calculating the distance 1 and 2 , the topological transition steps are presented in Table 1 For calculating the distance 3 , the topological relations between intersection parts on the intersection line could be disjoint, meet, overlap, cover, contain, and equal. The topological relations are grouped by the step of topological relations into five sets: (disjoint), (meet), (overlap), (cover, equal, coveredBy), and (contain, containedBy). The same way is used to calculate 3 as the definition of the step of the topological transitions. Every two adjacent sets have one step and their distance is 1. Consequently, the distance between "disjoint" and "contain" is 4. Following this way, all the possible distances of 3 can be known.
After calculating , the similarity of topological relations is defined based on the distance between formalized topological relations as follows.
Where indicate the minimum of and its value is 0. is 8 because the maximum values of 1 and 2 all are 2 and the maximum value of 3 is 4. Finally, the similarity between topological relation ranges from 0 to 1. The smaller similarities demonstrate the topological relations are closer.

Selection of Semantic Rules Based on the Similarities
The knowledge of the object is formalized as semantic rules based on the OWL class, relations, and individuals. The rules are represented as "Human Readable Syntax" form ( 1 ∧ 2 ∧ ••• ∧ ⇒ ). When an antecedent composed of the conjunctions of atoms hold, the consequent hold during the reasoning step. In the knowledge base, the individuals translated from the segments identified from point clouds are considered as facts for reasoning purposes. In our proposed solution, when an individual corresponding to a building component is given, all its related properties or relations are compared with the antecedent defined in the rules. The similarity between properties and relations related to an individual and those contained in the semantic rules are calculated. The similarities corresponding to properties and relations constitute a similarity vector. This similarity vector is produced referring to a selected rule. Then, the cosine distance between the similarity vector S = ( 1 , 2 , … , ) and the reference vector ref = (1, 1, … , 1) whose dimension is n is used to decide which rule is the most probable one for reasoning building features for a given individual. The rule corresponding to the minimum cosine distance is selected as the most probable one. Finally, the consequent in the selected rule is considered as the approximated conclusion of the given individual.

CASE STUDY AND RESULTS
The proposed method was applied to a point cloud with uncertainty obtained from a building. Here we aim at the recognition of the roof component with a specific style. For this purpose, we used an airborne LiDAR point cloud of buildings with uncertainties. We have defined a few semantic rules based on the qualitative information on buildings and their roofs with different styles (e.g. gable roof, hip roof). We then applied the segmentation process to extract geometric properties and relations as well as topological relations from the uncertain dataset.
According to the common knowledge of buildings, a roof is a structure of the upper covering of a building, and a wall is any opaque part of the external envelope of a building that is at an angle of 70° or more to the horizontal (DesigningBuildings, 2020). In the experiment, two basic types of building roof styles are chosen and the rules defined for recognizing building roof styles are presented in Table 2. First, we define rules to distinguish roof components from planar segments extracted from point clouds. According to the definition of a wall, the planar segments have an angle of less than 70° compared to the horizon are defined as a roof. Then, based on the segmented roof components, the rules for defining a hip roof and a gable roof are formalized based on the topological relations between roof components. The definitions of rules refer to the knowledge of building and formalized semantic rules presented in (Xing, 2018).
Following the step of the proposed solution, first, the input point cloud is segmented into planar segments as shown in the segmentation results of point clouds in Table 3. Then, the planar segments are translated as individuals of the concept "PlanarRegion_3D". The geometric information of planar segments and the topological relations between planar segments are formalized as properties and relations of individuals in the knowledge base. As shown in Table 3, the intersection lines between adjacent planar segments are extracted. The topological relations between each planar segment and the intersection line is translated as the Tp2 and Tp3 in the formalized topological relations. The identification of topological relations between planar segments is conducted and formalized as a four-word semantic description after identifying the topological relations between intersection parts on the intersection line (Tp4).

Hip roof
Gable roof Roof styles that are served to define semantic rules for reasoning purpose (https://www.roofcostestimator.com/top-15-rooftypes-and-their-pros-cons/roof-types-diagram/) The hip roof is a roof that slopes upward from all four sides of a building (Goodier, 2006). (https://www.roofcostestimator.com/top-15-rooftypes-and-their-pros-cons/roof-types-diagram/) Gable roof is a roof having two sides of continuous slope meeting at a ridge point (Goodier, 2006).  Table 2 Building roof styles and the defined rules for recognizing building roof styles from point clouds For example, in the hip roof example, the intersection line between Pr2 and Pr4 detected from point clouds is presented in Figure 2. The intersection parts between the intersection line and planar segments are shown in Figure 3. The endpoints representing the line segments are detected from points as shown in the bottom of the figure and the large size points represent the detected endpoints and the endpoints with the same color constitute a line segment. After identifying the topological relations between two line segments, the topological relation between Pr2 and Pr4 is formalized as "Meet-Meet-Meet-Overlap". Similarly, the topological relations among all planar segments are identified and formalized as shown in Table 3    In the step of semantic reasoning, the individuals of Pr1, Pr2, Pr3, and Pr4 meet completely the conditions of the rule (1) defined in Table 2. They are reasoned as the roof components. However, the set containing Pr1, Pr2, Pr3, and Pr4 cannot be reasoned as a hip roof because the topological relations between them do not conform to the conditions related to topological relations in the defined rule (2).
Following the step of evaluating the similarities of properties, relations related to a given individual and those defined in the rules, the similarities vector compared to the rule (2) is calculated as S = (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 (3) is calculated as 1 because the similarity between the identified topological relation "Meet-Meet-Meet-Cover" and "Meet-Meet-Meet-Equal" defined in the rule is 1. In the same way, the subset containing Pr3 and Pr4 has a similarity value of 0.999 compared to the rule (3). In conclusion, the roof composed of planar segments Pr1 and Pr2 belongs to a gable roof with similarity value 1. The roof composed of Pr3 and Pr4 also has a gable roof with a similarity value of 0.999.
It is important to note that according to the definitions provided in Table 2, roof styles are defined by the connection of roof components without being constrained by their dimension. Hence, topological relations between roof components play a vital role in identifying roof style from point clouds. For example, in the case of a hip roof with several hip-and-valley forms, we can construct several sets to identify the hip structures using the rule for reasoning a standard hip roof. In the experiment, the point density has an impact on identifying topological relations and estimating the similarities as well. In practice, a high-density point cloud provides more details on the intersection line between roof components. This will also result in higher similarity value for identifying a roof style from point clouds compared to a rule defining a specific roof style.
In summary, the information extracted from point clouds could be uncertain in the steps of segmentation, the identification of topological relations and geometric relations as well as properties.
Based on the formalized knowledge of a building, the similarities evaluation for properties and relations build the connection between individuals translated from segments and the predefined rules. After comparing the similarities, the cosine distance between vectors helps to select the most probable rule for reasoning the semantic information of a given individual. From the above two examples in the experiment, it showed that the proposed solution is effective to reason most likely building roof styles from point clouds with uncertainties based on the similarity evaluation of properties and relations.

CONCLUSION AND PERSPECTIVE
In this paper, we have proposed a method that leverages from both geometric and semantic information for the extraction of knowledge from LiDAR point clouds. The method deals with different types of uncertainties in data including uncertainties in the geometric and topological relations. This information is combined with complementary knowledge to extract semantic information on building components. The solution is effective in extracting semantic information of buildings, for example, roof styles, and it has potential for the recognition of other components such as walls, windows, and doors from point clouds as well. Integrating deep learning methods to make semantic reasoning robust in the recognition of varying objects will be interesting to explore in the future.