LINE SEGMENT MATCHING ALGORITHM BASED ON FEATURE GROUPING AND LBD DESCRIPTOR

This paper proposes a line-matching algorithm based on feature grouping and a line band descriptor (LBD) to address the insufficient reliability of individual line descriptors for line matching. First, the algorithm generates line-pairs according to geometrical relationships such as the distances and angles between line segments extracted from a single image. Subsequently, the algorithm employs the epipolar line of intersection between two lines in a reference line-pair to constrain candidate pairs corresponding to the reference line-pair. Thereafter, each line in the reference line-pair is considered individually, and its support region and the corresponding support region of each candidate line in the candidate pairs are established, following which an affine transformation is used for unifying the sizes of the reference support region and the candidate support region. Moreover, the LBD descriptor is then used for describing the reference and candidate lines. The Euclidean distances between the reference line and each candidate line descriptors are calculated, and the nearest neighbor distance ratio (NNDR) is used as a criterion for determining the final matching. Finally, the one-to-many and many-to-one line correspondences in matching results are transformed into one-to-one line correspondences by fitting multiple lines to the new line; simultaneously, incorrect matches are eliminated. The experimental results show that the proposed algorithm yields reliable line-matching performance for close-range images. * Corresponding author


INTRODUCTION
Feature matching is at the core of computer vision research and has been widely used in three-dimensional (3D) reconstruction, image registration and retrieval, image merging, and other such fields (Taylor et al., 1992;David et al., 2005;Chandraker et al., 2009). In feature matching, features are first extracted, and the feature descriptors are constructed from stereo pairs. Thereafter, feature similarity is computed according to some similarity measure. Finally, feature correspondence is established. This process focuses on establishing highly distinctive feature descriptors, with good feature representation ability and significant discrimination. Because line features contain more semantic and structural information than point features and are also easier to extract and describe than region features, they are widely used for matching man-made scenes, which contain an abundance of lines.
The existing line matching algorithms can be divided into two categories, according to the number of matching primitives: The methods in the first category match individual line segments and often utilize the properties of these segments (such as the length, gradient, and neighborhood information) (Schmid et al., 2010;Elaksher, 2011;Sun et al., 2015). In addition, some geometrical constraints are also often used for matching (Fan et al., 2010;Fan et al., 2012), such as the epipolar constraint (Hartley, 2003), triangulation constraint , and homography matrix constraint (Schmid et al., 2010;Lourakis et al., 2000). However, these constraints require some prior knowledge, such as the corresponding point and orientation parameters. Consequently, matching with these constraints strongly depends on the availability and quality of this additional information. Owing to the development and application of the scale-invariant feature transform (SIFT) (Lowe, 2004), a few researchers have focused on the construction of line feature descriptors, which employ the local texture information regarding lines to perform real-time matching. Wang et al. (2009) proposed the mean standard deviation line descriptor (MSLD), which is constructed by computing the mean and standard deviation vectors of the gradients of pixels in the vicinity of a line segment. The MSLD is highly distinctive and robust; however, it is sensitive to scale change. Similar to the MSLD, Zhang et al. (2013) proposed a line band descriptor (LBD) based on line appearance similarity. During the construction of this descriptor, the line support region is divided into several equal sub-regions, and the global and local Gaussian weight coefficients are applied to each row of the line support region, respectively. The description matrix is formed by calculating the gradient vectors of different directions in the sub-regions, which further improves the distinctiveness and robustness of existing descriptors. However, as the relationship between the features of adjacent lines was not considered during the matching process, mismatches still exist in different scenes with similar structures or similar texture; this affected the matching performance of line descriptors to some extent. geometrical constraint between the line segments. The most commonly used group is the line-pair constructed from two line segments, which can be matched as a whole. Compared with single line segments, additional geometrical information is available for line-pair matching. The key information for a linepair is the virtual intersection point between both line segments in line-pair; this information is widely used in line-pair matching methods. For example, Ok et al. (2012) utilized the epipolar line of the intersection for limiting matching candidate pairs. Kim et al. (2012) utilize the region with the center at the intersection to find corresponding pair. Li and co-workers (Li et al., 2016) proposed the line-junction-line (LJL) descriptor, which avoided the weak reliability of individual line descriptors and yielded good matching results for a variety of stereo images. In addition, geometrical properties between both line segments are also used for reducing the search space and filtering out ambiguous matches. Another advantage of pair matching is that a single line can be matched repeatedly because it is allowed to appear in different line-pairs, which yields several redundant matches. Although this significantly increases the computational complexity, it provides redundant information for the check of matching results. In view of the abovementioned advantages, group line matching is likely to be significantly useful in future applications.
To optimize the matching constraint in single line matching, a line-matching algorithm based on feature grouping and the LBD descriptor is proposed in this paper. This algorithm makes use of the advantage of line-pairs and the stability of the LBD. In this method, line-pairs are generated according to the geometrical relationships between constituent lines; for each line-pair, the epipolar line of the intersection point is used for reducing the search space from two-dimensional to onedimensional. This improves the reliability and the computational efficiency of single line matching and also provides a reliable initial value for subsequent matching. On the other hand, support regions are established for the candidate and reference lines, and their sizes are equalized by performing an affine transformation, which increases the robustness of the LBD toward different types of images. Moreover, we introduce the checking and integration algorithm for one-to-many and many-to-one line correspondences in matching results, which effectively eliminates incorrect matches and improves the reliability of the matching results. In summary, we demonstrate that combining the virtual intersection of line-pairs and the improved LBD descriptor can effectively improve the quality and the performance of the straight line matching.

METHODOLOGY
In this work, we employ global and local information for matching line segments. Based on the line segments detected using the line segment detector (LSD) algorithm (Grompone Von Gioi et al., 2010) and on the corresponding points obtained using the SIFT algorithm, which utilizes random sample consensus (RANSAC), our algorithm consists of the following five steps: (1) Generate line-pairs in each image using the permutation and combination function model and remove meaningless pairs according to the geometrical relationships, such as the distance and angle between two lines.
(2) For any reference line-pair, limit candidate line-pairs using the epipolar constraint of the intersection point between the two lines in the reference pair.
(3) Construct the corresponding support region for the reference line and each candidate line. For this process, epipolar geometry is used for building corresponding endpoints of the two line segments. Thereafter, the corresponding support regions based on the overlap of the two lines are constructed, and an affine transformation is used for unifying the sizes of both support regions. Finally, the LBD descriptor of the line is computed using the line support regions.
(4) Apply the nearest neighbor distance ratio (NNDR) criterion for calculating the similarity measure between the LBD descriptors for determining initial matching results.
(5) One-to-many and many-to-one line correspondences in the matching results are checked to obtain reliable one-to-one correspondences.

Generation of Line-pairs
A set of line segments is extracted from the reference image using the LSD algorithm, this set is denoted as Similarly, a set of line segments is extracted from the search image, using the LSD algorithm, this set is denoted as Here, n and m are the number of line segments extracted from the reference and search image, respectively. For a set of line segments extracted from a given image, the line-pairs are generated using the permutation and combination function model, as shown in Eq. (1). Grouping line-pairs can reduce the requirements for line continuity and also provide more abundant geometrical information.
Where, G is the number of generated line-pairs, and NS is the number of extracted line segments. For the line-pairs generated using the above mentioned random combination, which need to be pruned by geometric constraints between two line segments for reducing the matching complexity. The line-pairs should satisfy the following three conditions for subsequent matching: (1) The angle between the two lines in the line-pair should be within a certain range. As the two line segments that maintain a parallel relationship are not strongly related, the geometrical characteristics are not evident. The intersection relationship between the two lines can directly reflect the geometrical attribute information of the line-pair. As shown in Fig. 1(a ), the line-pair will be moved to the next analysis stage.
(2) The maximum distance between the four endpoints of the two line segments should satisfy a certain condition. To avoid generating meaningless line-pairs which increases the amount of calculation, it is necessary to ensure that the line segments that are far from the target line segment are not joined with the target line segment to form a pair. As shown in Fig. 1(a), the distances from endpoints a and b of i l to endpoints c and d of ' i l are ac, ad, bc, and bd, respectively. If the maximum distance between these endpoints is less than the threshold Td (Td = 100 in this study), the line-pair will be registered as a candidate, and the subsequent analysis will be performed.
(3) The distance from the intersection point of the line-pair to the starting point of each line segment in the line-pair should be below a certain threshold value. The endpoint of the line segment close to the intersection point o is selected as the starting point. As shown in Fig. 1(a), a and d are the starting points of the line segments i l and ' i l , respectively. The distance Doa represents the distance from point o to point a, and Dod is the distance from point o to point d. If max(Doa, Dod) < TD (here, TD = 50) holds, the line-pair will be reserved for the following matching.
After the above-described processing steps, a set of the linepairs in the reference image is obtained as ' , whereas a set of the line-pairs in the search image is obtained as

Epipolar constraint:
For the reference line-pair, the search space of candidate pairs has to be pruned to limit the matching complexity. For individual line matching, an epipolar quadrilateral region constraint can be employed to limit the search space, and for the line-pair, we utilize the epipolar of the intersection point of line-pair to find candidate pairs that the intersection points are close to the epipolar line.
During the process of matching, the corresponding points obtained via the SIFT algorithm are used for real-time calculations of the epipolar of the intersection of the reference line-pair. A determination is made to deduce whether the vertical distance from the intersection of each line-pair in the search image to the epipolar is smaller than a given threshold Th. According to the results of the experimental analysis, the value of Th is set to 1, and the line-pairs in the search image satisfying the distance threshold are regarded as candidate pairs. In Fig. 3 2), where x and y represent the coordinates of the pixels in the support region after the affine transformation, x′ and y′ represent the coordinates of the pixels in the support region in the initial image, and a0, a1, a2, b0, b1, and b2 are the transformation parameters.
After the affine transformation, we construct the LBD descriptor of the line using the line support regions with uniform directions and sizes. As shown in Fig. 4, the support region is divided into five equal sub-regions, each of which is called a band, and w indicates the width of the band (w = 3 in this study). The gradient of the pixels within each row of the line support region is accumulated. Along the direction perpendicular to the line, two Gaussian functions fg and fl are assigned to each row of the band. To avoid the influence of the length of the line segment on the dimension of the descriptor, the mean and standard deviation of each band are calculated. Moreover, to reduce the influence of changes in illumination, the mean and standard deviation vectors are normalized. Finally, we obtain a unit LBD descriptor, as follows:

Post-processing of matching results
According to the method of generating line-pairs in this paper, a single line can be matched repeatedly based on the fact that it is allowed to be partially present in different pair combinations. This increases the probability that the same line will generate different matching results. For example, as shown in Fig. 5, the line indexed by 1 in the reference image corresponds to the lines indexed by 9, 7, and 2 in the search image. Thus, it is necessary to check for one-to-many line correspondences based on the geometrical relationships between the lines. If multiple lines meet the condition of zero overlaps between multiple line segments, and if multiple line segments are adjacent and collinear, then the case is regarded as a correct matches, and multiple line segments are fitted into one line segment. Otherwise, the matching result for this group should be removed if multiple lines in the result do not satisfy the abovementioned conditions. After checking the one-to-many line correspondences, we need to further determine whether there are many-to-one correspondences. In Fig. 5, the line indexed by 7 in the search image corresponds to the lines indexed by 1 and 4 in the reference image. Again, a check is performed according to the above principle, and finally the matching results in the two views are all transformed into one-to-one correspondences with line checking and fitting.

EXPERIMENT
To evaluate the performance of our algorithm, we selected four representative close-range image pairs for matching experiments, as shown in Figs. 6(a)-(d), which contains common image transformation, namely, rotation change, viewpoint change, scale change, and illumination change. The images are 640 480 pixels, 800 600 pixels, 640 480 pixels, 900 600 pixels, respectively. The matching performance results of our algorithm are shown in Figs. 6(a)-(d), in which correct matches are shown in red and wrong matches are shown in blue. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) The statistics of the line matching results for the four image pairs are listed in Table 1. These capture the number of lines extracted for the two views, the number of line-pairs in the reference and search images (denoted by N1 and N2, respectively), the number of line-to-line correspondences, the number of lines from the two images included in the correspondences (denoted by M1 and M2, respectively), the number of final corresponding lines after result checking, the number of correct matches, the correct matching rate (CMR), and the computation time. Among them, the correct matches are determined by artificial judgment. Table 1 shows that the correct matching rate of our algorithm is above 97% for the images with rotation, viewpoint, scale, and illumination change, demonstrating good robustness and universality. However, with the correspondence of one-to-many and many-to-one in the matching results, most of the mismatches are deleted by post-processing with the collinearity constraint and line segment fitting.

CONCLUSIONS
This paper proposed an algorithm for matching line segments, based on the pairwise constraint and the LBD descriptor. First, the proposed algorithm generated line-pairs based on the geometrical relationships between adjacent lines, which compensated for the effects of incompleteness and fragility of extracted line segments. Thereafter, the LBD descriptor combined with an affine transformation was constructed based on the local neighborhood of the line segment. Finally, the correspondence relationships were established based on the similarity measure. This descriptor improved the performance of our algorithm for the stereo images with complex transformations, such as scale and illumination changes. Four groups of representative close-range image pairs were selected for completing the matching, and the experimental results demonstrated the reliability and stability of our algorithm. However, the algorithm proposed herein has a few shortcomings. A large number of line-pairs are generated because the same line is allowed to be present in different pair combinations; this increases the computational complexity of the proposed approach. On the other hand, the thresholds involved in the algorithm were all set to empirical values, and a sensitivity analysis of the effect of threshold variations on matching was not performed. Hence, future studies should aim to address both these issues.