STREET TREE INFORMATION EXTRACTION AND DYNAMICS ANALYSIS FROM MOBILE LIDAR POINT CLOUD

Street trees are common features and important assets in urban scenes. They are huge in numbers and are constantly changing, thus are difficult to monitor on a regular basis. A method of automatic extraction and dynamic analysis of street trees based on mobile LiDAR data is proposed. First, ground and low objects are filtered from the point clouds. Then, based on a geometric tree model and semantic information, each tree point cloud is extracted, and geometrical parameters such as location, trunk diameter, trunk structure line, tree height, crown width, and crown volume of each tree is obtained. A dynamic analysis combined with the growing characteristics of trees is conducted to compare and analyse the street trees from different epochs, in order to understand whether the trees have grown or been pruned, replanted, or displaced. The proposed algorithm was tested on three epochs of mobile LiDAR data, obtained in 2010, 2016 and 2018, respectively. Experimental results showed that the proposed method was able to accurately detect trees and extract tree parameters for detailed dynamics analysis. * Corresponding author


INTRODUCTION
Most cities have a large number of street trees, which play an important role in urban ecosystems. They have to be regularly monitored to understand their dynamics and to maintain their functions. However, it is a difficult task to monitor street trees at a city scale, which could be labour intensive and time consuming. Efficient and effective acquisition and information extraction of street trees could significantly benefit dynamics analysis, and subsequent city environment management and maintenance.
Street tree information can be obtained through different data acquisition techniques. Remote sensing images can obtain the spatial distribution of street trees from a bird's eye perspective at a large scale but in 2D. Aerial remote sensing and airborne LiDAR technology could further refine the street tree information by obtaining relatively accurate parameters in 3D (tree height, crown width, etc.) (Galvincio et al., 2016), but it is difficult to obtain lower layer parameters (such as breast diameter). The traditional method of using total station or RTK for manual measurement is reliable, but is inefficient and costly (Hauglin et al, 2017). It is difficult to deal with large numbers of street trees. Mobile LiDAR systems have obvious advantages over other data acquisition techniques as they can collect accurate and detailed 3D point cloud of trees on both sides of the streets in a whole city, covering relatively complete trunk and crown information from a side-view.
The research on street tree information extraction based on mobile LiDAR technology mostly consists of ground feature filtering/classification and tree localisation and parameter extraction. At present, it is an inevitable trend to detect street tree changes for long-term monitoring. Using mobile LiDAR point cloud, this paper proposes a method of automatic extraction of street trees by combining geometric parameters and semantic information, and a dynamic analysis framework for detailed change analysis. The contributions of this paper are: (1) a street tree parameter model is constructed according to the characteristics of trees in the mobile LiDAR point cloud; (2) based on the tree parameter model, a detailed parameter extraction algorithm for the street tree is proposed; (3) based on multi-temporal data, an analysis framework for street tree dynamics understanding is provided.

RELATED WORK
Point cloud voxelization: When the mobile LiDAR system is traveling in the street, massive amount of points was recorded, containing many types of objects connected by the ground points. Many researchers have adopted similar methods (Yue et al, 2015;Guan et al, 2015;Guan et al, 2016;Guan et al, 2019;Kang et al, 2018b;Liu et al, 2017) when extracting street trees from mobile LiDAR point clouds. The point clouds are assigned to three-dimensional voxels to simplify and regularize discretely distributed point clouds. Then voxel-based algorithms are used to filter out ground points, and different types of objects become scattered point cloud clusters. Euclidean distance-based clustering algorithm is then adopted to cluster non-ground points. A bottom-up searching method has been used to extract tree clusters according to the characteristics of street trees. There are two main search modes. One is to use a cylinder model to fit with trunks and crowns (Kang et al, 2018b) to filter point clouds. The other is a hierarchical grid projection method, which searches and extracts the street tree point cloud based on the projection grids (Liu et al, 2017). However, the voxel-based method is relatively simple. Each voxel represents all the points inside, with the density and elevation of the points as the attributes (Yue et al, 2015). On the basis of voxels, deep learning methods can be further adopted to realize the segmentation of trees (Guan et al, 2019). Super-voxel is an improved point cloud representation mode, which can derive more attributes. Kang et al, (2018a) presented a probabilistic graphical model to automatically classify various objects in street scenes. Super-voxels are first used to obtain geometric and spectral attributes, and then point-based multi-scale descriptor is constructed for various objects. A theme model is used to analyse the semantic correlation between the points in the super-voxel to establish a semantic expression, based on which each point is assigned to a probabilistic graphical model. Yang et al, (2015) adopts multi-scale super-voxels, taking the colour, intensity and distance of points as attributes, and segments the super-voxels by graph cut to realize the classification of various objects in urban scenes.
Street tree extraction and segmentation: In urban scenes, the street trees normally do not exist in isolation, and are often connected with each other or other objects such as street lamps. It is a key area of research to identify and extract the street tree from the point cloud as an independent individual. There are two main methods of street tree extraction and segmentation. The first one is direct segmentation based on the spatial characteristics of street trees. Zhong et al, (2013) adopted a grid-based method to separate the street trees and fits the crown with ellipse to separate the point cloud of adjacent street trees after filtering ground points and low ground objects. Guan et al, (2015) removed ground points and extracts tree point clouds based on voxel region growing, and segments the point cloud into individual trees by Euclidean distance-based clustering and voxel normalization. Li et al, (2019) based on the observation that the street trees and other objects have different 3D density characteristics, used vertical region growing algorithm to segment the street tree and adjacent ground objects. Guan et al, (2016) extracted clusters by Euclidean distance-based clustering, and realizes the segmentation of adjacent street trees based on voxel normalized cutting. Yadav et al, (2020) determined the trunk based on the linearity and the uniformity of the data distribution, and uses the compactness index of the circular or near-circular cross section of the crown and its symmetry around the trunk to extract the trunk point cloud. The other method is to use machine learning to train samples to match and segment trees from the point cloud. Wang et al, (2019) trained and tested samples based on deep learning regional convolutional neural networks, Faster R-CNN, to automatically recognize the street tree. Yu et al, (2015) used paired 3D shape contexts to match and retrieve street trees in urban scenes. Kumar et al, (2019a) improved the deep convolutional neural network (CNN) structure by preserving the geometry of the input point cloud to distinguish between street trees and nonstreet trees. Yu et al, (2015) used 3D object matching templates to extract street trees from 3D points. The direct segmentation algorithm is relatively intuitive, simple, and efficient, but less tolerant of extreme cases. The machine learning segmentation algorithm is relatively complicated, and needs a large number of samples in advance. Such algorithm can be time-consuming, but highly stable.

Tree parameter extraction:
The point cloud can intuitively express the spatial morphological characteristics of street trees, but in order to simplify the data and store key information in a database (Yadav et al, 2018), tree parameters, besides the DBH, need to be extracted. Researches have demonstrated a variety of parameter extraction methods. Zhong et al, (2013) used the horizontal span of the point cloud in cross-sections to estimate the fitted diameter of the trunk and crown. Herrero-Huerta et al, (2018) used principal component analysis to obtain the coordinates of trees, then the RANSAC algorithm to perform circular fitting on points of different heights, and a volumetric technique to calculate the crown volume. Rutzinger et al, (2010) used the alpha shape method to reduce the street tree data and extracted the parameters from the simplified point cloud. More research (Wu et al, 2013a;Chen et al,2019;Wu et al, 2013b, Zhong et al, 2017 carried out on parameter extraction such as tree height, breast diameter, crown width diameter, crown height. Husain et al, (2019) used three-dimensional slicing to process the external points of trees, and calculated the tree parameters. The above studies can extract the street tree parameters from the mobile LiDAR points. However, the change detection of street trees needs a more comprehensive and detailed parameter description to better understand street tree dynamics at different times. The geometric and semantic information of street trees, such as the position, height, breast diameter, crown width and volume, are extracted to dynamically analyse the street trees to determine the damage, replanting, displacement and growth of street trees. Multi-temporal street tree data acquisition can have a time span of several years or even longer, the operating conditions of the test area and the equipment (including the settings) may all be greatly different. This requires the parameter extraction algorithm to be consistent, robust, and efficient.

STREET TREE INFORMATION EXTRACTION
When the mobile LiDAR system travels along the street, the laser scanner mounted on the vehicle emits a laser beam toward the street tree. The trunk facing the scanner will reflect the laser signal back if is not occluded so that points can be captured. However, there is no signal back from the trunk side that is facing off the laser scanner, causing a certain area of data missing (the ratio is 30% -50%, depending the operating mode of the system). Due to the crown's relatively greater height and penetrability, the part facing the laser scanner has a large density of point cloud and relatively complete data, while the part facing off the laser scanner is relatively sparse and data are missing to a certain degree.
The acquisition of street tree point cloud by the mobile LiDAR system is affected by a variety of factors, e.g., LiDAR system settings and specification, vehicle travelling speed, distance to trees, and tree types/species. The street scene is often complex. In addition to street trees, it also includes various objects such as shrubs, grass, streetlights, street-signs and other public facilities, roadside buildings, as well as pedestrians, vehicles and others. To realize the dynamics detection of the street tree, the tree information must be accurately extracted from the complex point cloud. The general process is as follows: (1) to establish a fine geometric parameter model of the street tree; (2) to pre-process and filter the other objects such as ground points and low ground objects in the point cloud; (3) to extract the tree point clouds from the data; (4) to segment the tree point clouds into individual trees; and (5) to extract parameters for each tree. The detailed steps are described in the following sections.

Street tree parameter model
A geometric model of a tree, which mainly includes the trunk and the crown, is shown in Figure 1. The trunk is composed of four sets of parameters: bottom centre position P, trunk diameters Di, trunk structure line L, and trunk height Ht. The thickness of different parts of the trunk varies, so the diameter of the trunk is expressed by the fitting diameters of a sequence of small sections of the trunk. On the one hand, it can avoid the bias expressed by a single breast diameter (e.g., DHB). More importantly, the overall shape of the tree trunk can be better described together with the structural line, showing the vertical orientation of the tree trunk. The canopy is also composed of four sets of parameters: segmented sequence of outer contour lines OLi, the canopy height Hc, the canopy volume V, and the horizontal projection area A.

Ground point filtering
In street scenes, low ground-level objects such as grass, shrubs, vehicles, pedestrians, garbage bins, account for a large proportion of the data, causing certain interference to the extraction of street trees. The ground points are connecting all kinds of objects together, which cause difficulties in the extraction of street trees. By filtering out ground points, the connection between objects can be removed, and interferences can be eliminated, simplifying the extraction of street trees. Among the ground point filtering algorithms, TIN-based ground point filtering algorithms are relatively mature (Chen et al, 2014). However, the density of ground points from mobile LiDAR is very high, so the computation of such algorithms is relatively complicated and time-consuming.
Considering the complexity of the street scene and the efficiency and accuracy, the grid algorithm (Cai et al, 2017) is adopted from the mathematical morphology filtering algorithms to filter the ground points. One issue of mobile LiDAR is that data can be missing on the ground due to occlusions, which will cause erroneous ground filtering. The filtering algorithm is therefore improved by a new secondary filter ( Figure 2). The original point cloud is first filtered with a smaller grid size (Figure 2(a)), and it can be seen that some point cloud of trees and street lights will be misjudged (Figure 2(b). Then a larger grid size, as shown in Figure 2(c), is applied to restore the misjudged points, as shown in Figure 2(d).

Street tree extraction
Feature analysis: After filtering the ground points, the remaining points are independent point clusters. The geometric and morphological characteristics, horizontal plane projections, and related semantics have become the most effective means of distinguishing between various types of objects. Table 1 shows the projections of several typical street objects. It can be seen that the differences between the projections of the upper and lower parts of different objects are obvious. For independent street trees, the upper and lower layers are horizontally projected and gridded respectively. The upper-layer projected area is much larger than that of the lower-layer, and the upperlayer is approximately distributed in a circle, whereas the lowerlayer is more concentrated. The upper-layer projections of adjacently connected street trees are distributed in a wide area, but the lower-layer projections are in the pattern of a number of isolated blobs, and the area is significantly smaller. The other objects have different shapes, so their projection characteristics are obviously different from that of the street trees. Extraction details: A horizontal 2D distance-based clustering method is used first to cluster each point, and those clusters with too few points are removed. At this stage, some clusters contain only one object, but others have a mixture of multiple objects. Therefore, a 3D Euclidean distance-based clustering is adopted to further segment large clusters into smaller units with a smaller distance threshold for the points that lower than a threshold (for example, 2 m) from the ground. In this way, the remaining small ground objects, e.g., bicycles, pedestrians, warning signs, can be filtered by checking their cluster features.  occupied by the cluster. The cluster is considered to be a nontree object if the ratio of the upper projected area to the lower projected area is less than a threshold SS (SUpi / SDowni < SS (i = 1 ... n), SS can be set between 5 and 10), or street trees otherwise. The upper and lower projected spans of each cluster are simplified as the projected grid numbers. If the projected area (calculated above) differs greatly from the span area (similar to bounding box), the object is considered as a non-tree object. If the object has a small span in the X or Y direction, it is also considered as a non-tree object. The above criteria can eliminate street lamps, low-rise buildings, road signs and billboards .

Extraction of street tree parameters
In real-world street scenes, it is common that street trees are closely adjacent or connected. Each tree cluster may contain one or more trees. Therefore, the number of trees is firstly determined in each cluster, and then the adjacent or connected tree points can be segmented. Then the trunk and crown of each tree are separated. Finally, the corresponding parameters of the trunk and crown are obtained. The specific processing is as follows: (1) Determining locations and numbers of trees. For each tree cluster, a section of point cloud at a height of around 1.3 m from the ground is projected onto the horizontal plane and a 2D clustering is performed. The projection of the center of gravity of each sub-cluster on the DTM is taken as the location of the trunk, which is labelled as Pi (xi, yi) (i = 1, 2, 3 ... n). The number of trunks is considered the number of trees (n).
(2) Segmentation of adjacent trees. The distance weighting algorithm is adopted based on the tree growth model to accurately divide the adjacent tree point clouds into individual trees. Firstly, the distance-based segmentation method is used to roughly divide the adjacent street trees, and the parameters such as the height and breast diameter of each segment is calculated. The point cloud is horizontally projected and fitted to obtain the crown diameter as in (Yang et al, 2014). The theoretical crown width of the tree is calculated by the estimated parameters of the crown (tree height, breast diameter, and fitted crown width) with a theoretical model. Then the weight of crown distance was obtained by the weighted average of the theoretical and fitted crown widths. Finally, the weights of the crown distance are used to process the adjacent street trees in pairs iteratively, to obtain accurate segmentation of the street trees. The calculations of tree height, breast diameter, crown width, and the distance weighted segmentation algorithm based on tree growth model are described in detail in (Zhang et al, 2015).
(3) Separation of trunk and crown. There are huge differences in, for example, shape, composition, and laser reflectivity, between the trunk and the crown. As a result, the trunk and the crown have different manifestations when they are represented in a point cloud. Therefore, the trunk and crown can be separated when extracting their parameters separately. A grid growing algorithm is adopted in this paper to separate the trunk and crown. The grid (side length/size set as 0.2 m) at the bottom point of the trunk is used as a seed grid to start the upward search, and one layer of grids is added each time. If there is a point in the eight neighbour grids in the XY plane of the seed grid, they will be marked as new seed grids to continue the search upward. When the grid search reaches the boundary between the crown and the trunk, the number of newly marked seed grids will increase sharply, and the search will stop. The points in the marked seed grids will be recorded as trunk points, and the remaining as crown points. Figure 3 illustrates the separation process.
(4) Tree trunk fitting. The trunk is mainly expressed by the trunk structure line, which reflects the orientation and shape changes of the trunk during the growth of the tree. By crosssectioning the trunk points and using the least squares fitting method, the radius and center of each fitted circle are obtained at each certain height interval. The trunk structure line is constructed by connecting the centers of the sections. However, there are noise points in the point cloud that need to be filtered before the sectioning and fitting. The Radius Outlier Removal algorithm in the PCL (point cloud library) is used to de-noise the points before circle fitting is performed. The trunk structure line is then smoothed by using a smoothing algorithm. Figure 4 shows the trunk parameters before and after de-noising and smoothing.
(5) Tree crown parameter extraction. The volume of crown is an important measurement for the growth detection and biomass calculation of street trees. Therefore, the crown volume is calculated for change detection. After tree trunk separation, the crown volume is calculated based on the 3D grids. The crown grids that have points are taken as effective voxels. The total number of voxels is counted layer by layer. The crown volume is then computed by multiply the number of voxels and the volume of a voxel.

DYNAMICS ANALYSIS
The dynamics of street trees are determined by comparing and analysing the corresponding parameters and their correlation with adjacent trees from two or more epochs. The dynamics mainly include death of tree, newly planted or replanted, grown and pruned.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition)

Multi-temporal data comparison
The street tree parameters are extracted and stored as an independent unit in a database to facilitate dynamics analysis and indexing. The extraction of tree parameters in the later epoch(s) is not carried out full independently, but is based the previous parameters, in order to accelerate the extraction. Even both epochs of data use the same coordinate system, due to factors such as geo-referencing error, the positions of the same tree in two mobile LiDAR point clouds do not completely overlap (usually 5-50 cm offset). So the location of the tree in the first-epoch is used as a reference to search and determine whether there is a tree within a radius of 0.8 m in the secondepoch. If the tree exists, the relevant parameters of the tree are extracted. If the tree does not exist, it means that the original tree no longer exists. Similarly, based on the information of the trees from the later epoch, we can effectively check the missed extraction and difference in the earlier epoch.

Tree planting and replanting
New planting and replanting of trees often occur in road scenes for new road construction or greening management and improvement. There are two main scenarios. If there is no tree in the early epoch but there is in the later epoch, the tree is considered newly planted. This type of detection mainly uses position parameters. A further assessment is made to avoid possible miss detections (omissions) because of small trees or partial occlusions in the tree detection stage. The point cloud within a certain range (equivalent to the size of the canopy in the later epoch) is extracted in the earlier epoch. If the point cloud in the earlier epoch meets the crown characteristics of a street tree, it is considered that there are omissions in the previous detection. If not, it is considered that there is no street tree in the earlier epoch, and the tree is newly planted. Second scenario is that the tree is replanted in its original location due to death of the original tree or other reasons. This can be judged by the parameters of the trunk and the crown. In general, the newly planted street trees are relatively small. If the street trees in the earlier epoch are significantly larger in parameters than the later such as trunk diameter, tree height, and crown size, the tree is considered as newly planted.

Tree growing and pruning
The height, trunk diameter, crown width, crown volume and other parameters of the street tree should normally increase over time. If these parameters are all increasing, it means that the trees are growing normally, and the difference is taken as the growth value. Due to safety and aesthetics reasons, the street trees are pruned from time to time, mainly removing or shortening lower branches and sometimes the top of the crown, which will affect the extraction of tree height. Among all the street tree parameters, if the tree is not newly planted, the trunk diameter will have a relatively stable increase, and the trunk structure line is also relatively stable. The main result of pruning is the decrease of the crown circumference and volume. The scope and level of pruning can be estimated by the differences of the crown parameters.

EXPERIMENTAL ANALYSIS
Three epochs of experimental data on the campus of Henan Polytechnic University, China, were acquired to test the proposed method. The three datasets were obtained using two mobile LiDAR systems, SSW-1 and SSW-2, in 2010 (SSW-1), 2016 (SSW-2), and 2018 (SSW-2), respectively. The SSW-1 system was mounted on a van, and the center of the scanner was about 3 meters above the ground. The SSW-2 system was mounted on a flatbed cart, and the scanner center was about 1 meter above the ground. The study site was divided into four areas: A, B, C, and D ( Figure 5). Area A is a newly planted grove in 2010. Area B has a large ginkgo that was replanted before 2010, as it was growing slowly. The street trees in area C were transplanted due to road widening between 2010 and 2016. The rest trees in Figure 5 are considered in area D. The trees in areas A, C and D are evergreen large-leaved privet trees. Only the trees along the street (areas B, C, and D) are quantitatively analysed in this paper. The trees in area A are small and occluded by the street trees, so are only qualitatively analysed.
Using the proposed method, 100% of the street trees are detected and extracted ( Figure 6). It can be seen that in 2010, the street trees were small, and each tree was independent. By 2016, the crown of each tree had increased, and most trees were touching each other. The trees in area A had a certain degree of growth. By 2018, the connectivity between street trees was strengthened, and the number of trees in area A was further increased. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) After extracting the street trees, detailed tree parameters, as described in Section 3, were estimated for each street tree. Figure 7 shows an example of tree trunk separation and modelling by fitting a series of circles.
Based on the extracted street tree parameters, a comparative analysis was carried out among the three different epochs. Figure 8 shows the growth of the Ginkgo tree in area B. Since the tree is a transplanted mature tree, the growth of the tree has been slow. The difference in DBH is no more than 2 cm, height no more than 0.6 m, and crown volume no more than 2 m³, showing that the growth is insignificant.
It was found that between 2010 and 2016, 11 street trees in area C disappeared, and 11 new street trees appeared at the same time, with a shift of about 4 meters in position. The reason was that in 2012, due to the widening of the road, these 11 street trees were transplanted (Figure 9). During 2016-2018, the street trees did not change in position, and 19 trees were detected in area A. The reason was that the data collected in 2018 used a high point density mode. Although the trees in area A are far away, there are still more points, so that more trees could be detected.
According to the statistics analysis, the average DBH of detected trees in areas C and D increased by 5.3 cm, the average tree height increased by 0.91 m, the average crown increased by 1.58 m, and the average crown volume increased by 8.6 m³ between 2010 and 2016. From 2016 to 2018, the average DBH increased by 1.4 cm, the average tree height increased by 0.11 m, the average crown width increased by 0.38 m, and the average crown volume increased by 2.5 m³ .
It can be seen from Figure 10 that the growths of trees from 2010 to 2016 are more obvious, as the height and crown of each tree have increased significantly. In the 2016-2018 comparison, due to the shorter time interval, the growth is less obvious.

CONLUSION
This paper proposed a method for automatic extraction and dynamic analysis of street trees based on mobile LiDAR point cloud data. The improved mathematical morphological filtering algorithm can effectively remove ground points and low ground-level object points, facilitating the clustering and identification of street trees. According to the morphological characteristics of street trees in the mobile LiDAR point cloud, the tree geometric model and related semantic information are constructed to accurately detect tree points and segment them into independent individuals, and then to extract parameters of the trees, such as trunk diameter, trunk orientation line, crown volume, which are used as basis for the dynamics analysis. The comparative analysis of the parameters of the multi-epoch street trees can effectively reveal the dynamics of street trees, including death, transplantation, new plantation and growth, which has great contribution for street tree maintenance and management.
The tree parameter extraction accuracy is highly related to the quality of the point cloud. The position of the street tree is the center of the fitted circle at the bottom of the trunk. The absolute accuracy of the position is depending on the real-time positioning accuracy of the mobile LiDAR. Since the data acquisition time between adjacent street trees is very close, the relative position accuracy is high, so it can be used as supplementary evidence for assessing the position change of street trees. The trunk diameter is an important parameter of the street tree, and its fitting accuracy is greatly affected by the point cloud density. When the point cloud is sparse, the fitting accuracy will be low. In order to accurately assess the overall condition of the trunk, the trunk direction line becomes an important supplementary. In particular, if they are damaged by natural disasters such as typhoons, and become inclined, the direction line can be used for street tree displacement detection. However, mobile LiDAR systems can only scan from one side, thus cannot obtain complete data on street trees. Therefore, using the obtained point cloud for dynamics detection has inherent defects, and the modelling of the tree crown can be inaccurate. It will be the focus of future work on using the symmetry of the crown to tackle data incompleteness and further improve the crown model, and then to accurately detect changes in the crown.