POWER TRANSMISSION LINE RECONSTRUCTION FROM SEQUENTIAL OBLIQUE UAV IMAGES

With the accelerated development of the global economy, the scale of UHV (Ultra High Voltage) large-capacity transmission lines has increased dramatically. Nevertheless, existing transmission line inspection methods rely on human inspectors detecting abnormal facilities from ground level or helicopters. Those methods not only require extremely high labor and equipment costs, but also are timeconsuming. In this paper, a thin object 3D reconstruction method is proposed for the extraction of 3D structures of power lines from sequential oblique UAV images. The proposed method recovers the 3D structure of the power lines from three sequential images that depict the structure of power lines in 2D. Proposed method was tested in a typical transmission line corridor and the reconstructed results show the effectiveness of the proposed method.


INTRODUCTION
Modern society has particular dependence on electrical energy. The importance of building an electrical power system capable of an uninterrupted, safe and sustainable power supply cannot be overstated. Therefore, it is urgent to develop an automatic and flexible ROW(Right-Of-Ways) anomaly detection method for monitoring transmission lines. In recent year, it is common to utilize aerial vehicles equipped with the artificial vision system and multi-sensory, such as ALS, ultraviolet sensors, infrared and visible-light cameras, for the execution of faults detection (Matikainen et al., 2016). State Key Laboratory in Wuhan university proposed a method to diagnose the safety distance of transmission line with UAV loaded airborne laser scanning (ALS) (Chen et al., 2018). The researchers at the Australian Research Centre for Aerospace Automation (ARCAA) have been working on the generation of 3D Occupancy Map of the environment using stereo vision and laser scanning (Hrabar et al., 2010). Although there are many related work in recovering 3D structures form 2D images (Fabbri and Kimia, 2010;Teney and Piater, 2012;Hofer et al., 2015;Hofer et al., 2017;Li et al., 2018;Langlois et al., 2019;Tian et al., 2019), the wire-like thin object reconstruction remain unsolved. The power lines are typical thin and linear structure. Considering the unique characteristics of the reconstruction wiry objects in this study, such as lack of distinctive image features, uniform color, thin elements, and severe self-occlusion (Liu et al., 2017), its extremely hard to digitize the wiry lines, by either using commodity depth sensors or traditional multi-view stereo (MVS) approaches (Seitz et al., n.d.). Both of the methods suffer from obvious noise, loss of data and failed to detect global curve topology, resulting in plausible reconstruction outputs. The # Equal contribution * Corresponding author curve structure, on the construct, is more invariable compared with interest points to the transformation of illumination of image. In this paper, a 3D thin structure reconstruction method was proposed for power line by utilizing sequential oblique images collected by low-cost camera loaded on min-UAVs. In Section 1 the related study has been reviewed briefly. And the detection of 2D pylon structures as well as the reconstruction of 3D transmission lines are detailed in Section 2. Section 3 discusses the details of our implementation and analyses the result of the experiment. In Section 5 we concluded the paper and introduce the future work of our project.

PPOWER LINE 3D STRUCTURE RECONSTRUCTION FROM SEQUENTIAL ONLIQUE IMAGES
The proposed method can be briefly divided into two parts: the first part is 2D power line extraction from UAV oblique images. The second one is 3D power line reconstruction from detected 2D line segments. The specific details are stated in the following sections.

Extraction of Power Lines form 2D UAV Oblique Images
The proposed approach of transmission line extraction from UAV imagery consists of Data preprocessing and power line extraction stage. An overview of this pipeline is provided in Figure 1. Before the curve extraction process, a pre-processing stage is implemented firstly. The accuracy of the pure curve geometry-based reconstruction relies highly on the precision of the calibrated camera parameters. Thus, the sequences required to be imaged under controlled conditions and a large number of precise calibrated camera parameters are calculated via bundle adjustment. Then a sequence of undistorted images is obtained and as input to the next extraction stage.
Considering the characteristics of the problem addressed in this study, Richer Convolutional Feature (RCF) network (Liu et al., 2018) is used for the edge detection on undistorted key frame images to generate high-quality 2D lines.  Figure 2), which consists of 13 convolutional layers and 3 fully connected layers, is designed by modifying VGG16 network. The input is an image with arbitrary size and the output is an edge probability map with the same size. By utilizing this network, we can take advantage of multiscale information, such as low-level and object-level which is helpful to edge detection.
Considering the characteristics of the problem addressed in this study, Richer Convolutional Feature (RCF) network is used for the edge detection on undistorted key frame images to generate high-quality 2D lines. RCF network (shown in Figure 2), which consists of 13 convolutional layers and 3 fully connected layers, is designed by modifying VGG16 network. The input is an image with arbitrary size and the output is an edge probability map with the same size. By utilizing this network, we can take advantage of multiscale information, such as low-level and object-level which is helpful to edge detection.
The Otsu thresholding method (Abutaleb, 1989) that capable of adaptively finding the optimal threshold to distinguish twoclass data is applied on the output of the RCF network to get the final binary edge map. To make sure the algorithm performs well in terms of the line segments (i.e. the gaps between the curve of power lines are less and the noises decrease simultaneously in the binarized image), the line structures are selected out from the image manually and other background issues such as electric tower are deleted. Thus, a final accurate edge map with clear background is generated.

Image-based 3D Reconstruction
During the procedure of 3D power line reconstruction, two principle and unique features of wire structure are exploited in this method, which is simplicity and smoothness. In the light of these considerations, we address this challenging problem about featureless lines by obtaining candidate 3D curve segments and selecting out the correct 3D curves and then recovering a decomposition of such wires into a set of continuous and different wires by utilizing a multiple traveling salesman problem(mTSP) (Worboys et al., 1986). The methodology of 3D wire structure reconstruction includes many steps and the workflow is shown in Figure 3.

2D Curve Segments Detection:
The input are three oblique RGB images, which meets the smallest input requirement and related calibrated camera parameters.The output is a set of 2D curve fragments detected separately from each view. The approach is pixelwise based and all adjected pixels with similar color are divided into the same cluster according to flooding method. After that, a mathematical morphology approach is applied to thin and capture one-pixel wide wires. Finally, the crossed curves are cut off at the intersection point to generate a set of 2D individual curves from each image.

3D Curve Selection:
Next, the generation of 3D candidate curves is operated by pairing curves from two arbitrary views, namely a reference view and a neighboring view at a time via epipolar-geometric constraints. A 2D curve pair reconstructs to a 3D candidate curve, whose reprojection onto one another view named projection view and calculated the re-projected errors taking angle and distance deviation between their tangential directions into consideration. The matching cost is calculated as the weighted sum of the aggregated distance and tangential deviation. In this experiment the weight is assigned to 1, i.e. it is of equal importance for these two parts. Minimizing the matching cost, then a series of true 3D curve segments for each 2D curve segments are obtained. Figure 3. Workflow of 3D wire structure reconstruction procedure

Global Topology Recovery and Smoothness:
Having obtained the optimal 3D curve fragments in the previous step, the next goal is to recovery the global geometry of the wires. The set of 3D curves can be formulated as a multiple traveling salesman problem (mTSP), which belongs to a NP-hard problem. In this case, a set of 3D curve segments is defined as intermediate nodes, and optical paths which start and end at a new start node are calculated by minimizing the total cost of the selected edges and the number of paths. Considering the small gaps along the final path, a cubic B-spline is chosen to smooth and true object in image better. As a result, there is no gap in the resulting B-spline representation of each wire and the 3D pylon is smooth and continuous.   The flight to collect data for this experiment was executed with a low-cost drone, the Inspire 1(DJI). The specification of the drone and camera sensor are shown in Table 1 ("Inspire 1 -Aircraft -DJI," n.d.).

Experiment data and overall result
Because the reconstruction accuracy of this project depends highly on the bundle adjustment and the accuracy of the calibrated camera parameters, obtained multiple images are firstly preprocessed on the professional pixel4Dmapper software ("Pix4Dmapper," n.d.). In this experiment, we select about 20 unmanned airborne images satisfying the 3D reconstruction constraints mentioned above and segment the initial images The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) (4000*3000) into 100*100 pixel scale, which is an acceptable input size for the RCF architecture, shown in Figure 6. Next, we load the pre-trained model BSDS500 and execute the program to obtain the probability edge maps. In addition, we do the standard non-maximum suppression (NMS) (Neubeck and Van Gool, 2006) and edge thinning. The results are shown in Figure 7. As we can see, it is noticeable that the edge is thinner compared with previous results.  In order to further optimize the experimental results, we use the iterative cubic B-spline curve model and back-project the 3D lines onto three input images for the consistency checking. Figure 10 indicates the reprojection results before the optimization process, and Figure   The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 11. Reconstructed power lines The performance can be quantitatively evaluated by projecting points on the reconstructed 3D wires to each input view. As can be seen in Figure 9 and Figure 10, the maximal error often occurs at each end of the model, which located between the insulator of the tower and the wire itself, resulting from the reason that the area of target pylons is selected out manually and human eyes have subtle differences in pylon discrimination. Besides, the high-resolution images are vitally important in the reconstruction process. The image with lower resolution will lead to blurred or even broken power lines, which often occurs at the junction of the electric tower or the pylon far from the shooting point, resulting in lack of pylon structure and the loss of precision of the 3D reconstruction stage. Comparing the reprojection errors in Table 6, both of the average and maximum errors of the line without literation are larger, indicating the necessity of the final B-Spline optimization stage.

CONCLUSIONS
Aiming to reconstruct 3D structure of power lines from 2D UAV oblique images, we proposed a two-state thin object reconstruction algorithm. First, a CNNs based Richer Convolutional Features (RCF) is used to extract the keyframe of the power lines. Then, considering the smoothness and simplicity features of power lines, an image-based method is used to reconstruct continuous and high-quality 3D lines. The experiment demonstrates that the proposed method can effectively reconstruct 3D structure of the power lines form sequential images collected at different angles and positions. The future work will be aimed at reconstructing 3D multiple power lines from a set of input images (only three in this study) simultaneously. Besides, how to successfully reconstruct the pylon which is approximately tangent to the epipolar line is also a problem worthy of study.