A PROJECTION-BASED RECONSTRUCTION ALGORITHM FOR 3D MODELING OF BRIDGE STRUCTURES FROM DRONE-BASED POINT CLOUD

This paper presents a projection-based method for 3D bridge modeling using dense point clouds generated from drone-based images. The proposed workflow consists of hierarchical steps including point cloud segmentation, modeling of individual elements, and merging of individual models to generate the final 3D model. First, a fuzzy clustering algorithm including the height values and geometrical-spectral features is employed to segment the input point cloud into the main bridge elements. In the next step, a 2D projection-based reconstruction technique is developed to generate a 2D model for each element. Next, the 3D models are reconstructed by extruding the 2D models orthogonally to the projection plane. Finally, the reconstruction process is completed by merging individual 3D models and forming an integrated 3D model of the bridge structure in a CAD format. The results demonstrate the effectiveness of the proposed method to generate 3D models automatically with a median error of about 0.025 m between the elements’ dimensions in the reference and reconstructed models for two different bridge datasets. * Corresponding author


INTRODUCTION
Today, overpopulation and its concentration in the central nucleus of cities have led to more complex urban problems. Communication and transportation networks are vital urban spatial infrastructures which disruption them can lead to irreparable damages to the safety, health, and economy. The bridge structure as one of the most important urban structures has a particular role in the field of communication and transportation. Understanding the vulnerability of bridge structure toward ignoring hydraulic standards in design as well as natural and human disasters (Deng et al., 2016) has led to the use of efficient methods for technical and quality inspection. 3D CAD models are an accurate data source to measure and update geometric and spatial information (Cabaleiro et al., 2016). In this regard, 3D models of the bridge structures, in addition to purposeful and comprehensive coverage of urban problems, are widely used in many applications such as highspeed monitoring and inspections (Zhang et al., 2014;Cheng et al., 2019;León et al., 2019), safety, crisis management, and transportation system. The main contribution of this paper is to provide a projectionbased approach for automatic 3D reconstruction of the major elements of the bridge including the railing, body and abutment elements from photogrammetric point clouds. The main idea is to employ the geometric relations between the elements in different bridges as knowledge to reduce the geometrical and structural complexities during the modeling of bridges.

RELATED WORK
Bridge structures can be damaged for different reasons during construction and operation phases. 3D CAD models can show the real changes of bridge elements compared to the original design and plans (Barazzetti, 2016;Brilakis et al., 2010). Due to the complexity and diversity of the geometric shapes of the bridge elements, as well as the low-quality point cloud data including noise and gap, 3D reconstruction of bridges has become a challenging topic (Rashidi et al., 2020). In most studies, 3D models of bridges presented as finite element meshes and Industry Foundation Classes (IFCs) models (Yan et al., 2017;Sánchez Rodríguez et al., 2020) which are not ideal for geospatial CAD-based software. During the last decades, the use of airborne technologies such as aerial laser scanners (e.g. LiDARs) and drones has been increased for 3D modeling of bridge structures (Lee et al., 2019;Cabaleiro et al., 2016;Yoon et al., 2018). Accuracy, time, and cost are the three key factors in choosing any of these technologies. Although the LiDAR makes it possible to obtain more accurate and high-quality information of the structure, the use of drones is required to collect data from upper and lateral parts of the bridge (Popescu et al., 2019;Pepe et al., 2019). On the other hand, automatic 3D modeling of a bridge requires a precise and efficient segmentation algorithm such as k-means or region growing (Chen et al., 2018) to separate the bridge elements before modeling. (Yue et al., 2019) developed a segmentation method based on geometric features, the super voxel structures and global graph optimization for grouping the points of bridge components. In addition, (Lu et al., 2019) proposed a slicing algorithm based on the surface normal, oriented bounding boxes and density histograms for automatic identification of structural elements of concrete bridges. Their results showed an accuracy rate of 100% for automatic segmentation of 10 concrete bridges. The geometry of a bridge is an important factor in assessing the load-bearing capacity (Gilbert, 2007;Heyman, 1969;Livesley, 1978). Therefore, it is necessary to develop a 3D reconstruction method to accurately model different elements of a bridge. The current methods include calculating the surface features of elements such as Gaussian and mean curvatures (Teza et al., 2009), and examining the differences between the fitted geometric shapes and the original points (Ye et al., 2018). In this study, we developed a novel hierarchical framework to generate 3D models of different bridges, automatically. The final models are in CAD formats, which are suitable to use in safety and inspection investigations.

METHODOLOGY
As shown in Figure 1, the main steps of the proposed method include point cloud segmentation, projection of 3D points into a 2D space, modeling of individual elements, and merging them into an integrated 3D model in order to form the final CAD model. The details of steps are given in the following sections.

Point Cloud Segmentation
Due to hierarchical and vertical positions of the bridge elements in the 3D space as well as the high performance, high accuracy and adaptability of the fuzzy clustering algorithm for segmenting of noisy data (Zhuang et al., 2008;Zhao et al., 2008), we used the fuzzy c-means clustering (Dunn, 1973;Bezdek, 1981) to segment the bridge point cloud into the main elements. For this purpose, the point features including the height and spectral values, 3D density, normal vectors (Amini Amirkolaee, Arefi, 2017) and planarity conditions (Blomley et al., 2014) are employed to group the points into three clusters. The clusters contains the railing, body and other elements of the bridge (e.g. ground and embankment segments). Due to connections of abutment elements to the ground, body, and embankment elements, the separation of abutment points are more difficult and often accompanied by errors. Therefore, the points of abutment elements will be extracted from the raw point cloud after modeling the body element to increase the segmentation accuracy.

Projection-based Modeling
In this paper, the proposed 3D modeling workflow is a projection-based method (Arefi and Reinartz, 2013;Sahebdivani et al., 2020) which transfers the 3D point cloud to an orthogonal plane and generates 2D models using primitive points. The final model is formed by merging individual models based on horizontal and vertical curvatures of the bridge.

Railing Element
Modelling: Similar to our previous study (Sahebdivani et al., 2020), we model the linear railing elements on both sides of the bridge, separately. First, the points of each railing element are identified and extracted using the RANSAC-based line-fitting algorithm. Next, the point segments are divided into smaller patches and for each patch, the angle between the fitted RANSAC lines and the Y-axis is calculated (e.g. αz) (Figure 2, a). As shown in Figure 2, b, the points of each patch are then rotated and projected to the X-Z plane using the corresponding angle. Finally, a 2D plane with the dimensions of W and H is fitted to the points in each patch on the X-Z plane, as a 2D model for the railing segment. Each patch is partially rotated in two directions (e.g. around the Z and X axes) at the predefined intervals in order to model the curvatures of the railing elements in the bridge. The best partial rotation angle (e.g. Δαz, Δαx) is selected based on the maximum number of inlier points on the fitted model. Next, the 3D model of each patch is reconstructed by adding the Y values to the coordinates of the 2D model's vertexes and then converted to the original coordinate system by inverse rotating of points around the X and Z axes (Figure 3). There are some distortions in joints because of patching points and merging models. To solve the distortions and improve the quality, five parameters are calculated for each 3D model using a Fourier interpolation technique, given by Equation 1 (Sahebdivani et al., 2020).
where : j = [1… 5] are the five parameters of X, Y, Z, Δαx and Δαz. The parameter i is the number of patches, and w are the unknown coefficients of the interpolation function, and n is the degree of the Fourier curve.

Body Element Modelling:
The body element of the bridge consists of various structural elements such as decks, slabs, beams, girders, sidewalks, and plates that connect the beams to the supports. Due to the physical connections between the railing and body elements through the sidewalks, we use the information extracted from the 3D model of the railing element to divide the points of the body segment into patches and then project them to a 2D space. This knowledge includes the endpoint coordinates of patch lines and the total (αz) and partial rotation (Δαz, Δαx) angles. After patching the points, each patch is divided again into smaller sections with the length of K along the Y-axis and then projected to the X-Z space. Next, a noise removal technique based on the 2D neighbourhood point density is applied to the points to eliminate the outliers.
To form a 2D model of each section, the coordinates of the primitive points should be determined on the X-Z plane. Based on the existing knowledge in designing the cross-sections of the body element in different bridges, the number of primitive points is 12 for a bridge with a one-way transverse slope and a box girder structural element, as shown in Figure 4. As a rule: If the deck slope is two-way, an additional point where the transverse slope changes, is added to primitive points. If there are two or more elements of the box girder structure in the design of the bridge, for each box girder structure, four extra points are added to primitive points. For 2D modeling of the cross-section, the parametric equations and geometric constraints are developed according to Table 1. As shown in Figure 5, a, 2D models of sections are fitted to the points on the X-Z plane by employing the Nelder-Mead optimization technique (Nelder, Mead, 1965). The cost function is the distance of points from the optimized 2D model. To generate the final 3D model of the body element, all optimized models are re-projected to the 3D space by adding the Y coordinates and applying the inverse total and partial rotation angles. Finally, all 3D models are merged using mutual points ( Figure 5, b). Similar to the railing element, a curve fitting method is applied to the final 3D model to smooth and improve the distortions caused by point cloud patching. a b Figure 5. a: Fitting 2D primitive models to all sections of each patch, and b: 3D model of the body element.

Abutment Element Modelling:
The abutment element is a perpendicular or oblique structure at the ends of a bridge, which consists of three parts such as a retaining wall and two lateral walls (i.e. abutment wings), as shown in Figure 6. Since the 3D model of the bridge body is connected to the retaining walls of the abutment element, the abutment points can be separated from the main point cloud using the patch information of the body points and the final 3D model of the body element. First, the points of the retaining walls are separated from the abutment points by fitting a plane using the RANSAC algorithm. Next, the inlier points of the fitted plane are projected to the X-Z plane and an optimal bounding rectangle is fitted to them (Figure 7). Accordingly, the 2D positions of four primitive points are defined and the 2D model of the retaining wall is generated. Lastly, the 2D model can be projected to the 3D space using the Y coordinates of inlier points.   As illustrated in Figure 6, the geometrical shape of the abutment wings can be considered as a 2D triangle and therefore, can be modeled by three primitive points. Two required primitive points are the points in the left and right sides of the retaining wall. The third point is also the endpoint of the bridge which is already calculated in the 3D model of the railing element.

Merging of 3D Models
In this study, we proposed two merging methods including the plane-line and plane-plane intersections to connect the elements to generate an integrated model of the bridge structure. For this purpose, the coordinates of the primitive points in the 3D models of the elements are used to calculate the parameters of corresponding lines and planes. Therefore, the accuracy of primitive points has a significant impact on determining the intersecting points of the elements. Finally, the 3D model of the bridge structure is prepared in a CAD format (e.g. dxf).

Case Study
To evaluate the proposed method, multi view images are captured by a Mavic 2 Pro drone and a 20 MP camera over two bridges with different cross-section designs (e.g. beam and arch bridges) in Germany. All images are processed by Pix4Dmapper software with a spatial resolution of 0.36 cm and the colored point cloud of each bridge is generated (Figure 8). Table 2 contains the main characteristics of the data.  Table 2. Data used in this study.

Experimental Results of Point Cloud Segmentation
To segment the point cloud into the main bridge elements, required features including the height and spectral values, 3D density, normal vectors and planarity conditions are calculated for points in a one-meter neighborhood. Next, the fuzzy cmeans clustering is applied to the points. In Figure 9, the results of the point segmentation are illustrated in three classes. a b Figure 9. The results of point cloud segmentation: a: bridge 1; and b: bridge 2.
To evaluate the segmentation step, manually-annotated reference data and the segmented points are compared and the results are reported in Table 3 Table 3. Accuracy evaluation of segmentation.

Experimental Results of 3D Modeling
After extracting the railing element, a line-based RANSAC algorithm is employed to separate all railing elements on both sides of the bridge. The fitted RANSAC lines are then patched and the total rotation angle of each patch is calculated using the corresponding endpoints. Next, the points of each patch are rotated using the total and partial rotation angles, and a 2D primitive model with the dimensions of 0.05 × 1.10 m 2 is fitted to the points in the X-Z plane ( Figure 10). The 2D primitive models are then re-projected to the 3D space by adding Y values and inverse rotating of points around the X and Z axes. Finally, to reconstruct the bridge curvature precisely, parameters including X, Y, Z, Δαx and Δαz are interpolated separately using a second degree Fourier function. The final 3D models of the railing element are presented in Figure 11. a b Figure 10. Fitting 2D primitive models of the railing elements: a: bridge 1; and b: bridge 2. a b Figure 11. 3D models of the railing elements for: a: bridge 1; and b: bridge 2.
To model the body element, the points are patched and then rotated using the railing model information. Next, each patch is divided into sections with the length of one meter along the Yaxis. Next, the body points are projected to the X-Z plane, and the noises are removed based on the point density ( Figure 12). a b Figure 12. The noise removal for: a: bridge 1; and b: bridge 2.
According to the equations mentioned in section 3.2.2, the locations of 12 primitive points are calculated to reconstruct the 2D primitive model of each section in each patch ( Figure 13). Next, the 2D models are re-projected to the 3D space using the Y coordinates of endpoints. In order to generate a unique 3D model for the body element, the points of individual 3D models of each patch are rotated using the corresponding total and partial inverse angles and then connected to each other. Finally, a second-degree curve is fitted to the 3D coordinates of primitive points to improve the smoothness of the 3D model and reconstruct the horizontal and vertical curvatures. Figure 14 shows the final 3D models of the body element.
Initial model 25th iteration 3000th iteration a Initial model 25th iteration 3000th iteration b To model the abutments, the related points are first extracted from the original point cloud using the 3D model of the body element. Next, a plane with a maximum distance of 0.02 meters is fitted to the segmented points using the RANSAC algorithm, and the corresponding points of retaining walls are extracted. A bounding rectangle is fitted to points on the X-Z plane and the 3D model of the retaining wall is reconstructed by adding Y The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLVI-4/W1-2021 6th International Conference on Smart Data and Smart Cities, 15-17 September 2021, Stuttgart, Germany coordinates. Finally, the locations of the primitive points in the 3D model of the retaining wall and the points at the beginning and end locations of the railing model are employed to generate the 3D model of lateral walls ( Figure 15). a b Figure 15. 3D models of abutment elements for: a: bridge 1; and b: bridge 2.
As a final step to generate the integrated 3D model of the bridge, the 3D models of elements should be connected, respectively. For this purpose, the perpendicular planes in the railing 3D model should be connected to the planes that are passing through the primitive points of the sidewalk parts of the body element. Besides, the retaining wall planes at the beginning and endpoints of the bridge are intersected with the lines crossing the primitive points of 3D models of the body element. The final models are illustrated in Figure 16. The results of the 3D modeling are evaluated based on the distances and lengths of the bridge elements in the design plans (e.g. reference models) and the generated models. For each dataset, the standard metrics are presented in Table 4. The overall measures are 0.032 m, 0.04 m, and 0.025 m for the mean errors, standard deviations, and median errors, respectively. The results indicate that the proposed method can provide highquality 3D models for bridge structures.

Dataset
The final and reference models differences ( Table 4. The evaluation of final 3D models.

CONCLUSION
In this study, a novel hierarchical framework is presented to generate 3D models of different bridges using point clouds. The main idea is to use the knowledge in geometric relations between the bridge elements and developing a projection-based algorithm in order to decrease the challenges in 3D modeling by converting the data into a 2D space. The proposed method is an automatic framework for modeling different shapes of bridges with an overall median error of about 0.025 m. Unlike recent approaches, the results of the proposed method are CAD models (not mesh models) which can provide geometric measurements of different bridge elements in geospatial software.