THE USE OF COMPUTER VISION ALGORITHMS FOR AUTOMATIC ORIENTATION OF TERRESTRIAL LASER SCANNING DATA

The paper presents analysis of the orientation of terrestrial laser scanning (TLS) data. In the proposed data processing methodology, point clouds are considered as panoramic images enriched by the depth map. Computer vision (CV) algorithms are used for orientation, which are applied for testing the correctness of the detection of tie points and time of computations, and for assessing difficulties in their implementation. The BRISK, FASRT, MSER, SIFT, SURF, ASIFT and CenSurE algorithms are used to search for key-points. The source data are point clouds acquired using a Z+F 5006h terrestrial laser scanner on the ruins of Iłża Castle, Poland. Algorithms allowing combination of the photogrammetric and CV approaches are also presented.


INTRODUCTION
Point clouds are acquired from terrestrial laser scanning (TLS) in a local reference system of the instrument.For a large number of data sets (where big and complex objects are processed),the point clouds need to be transformed for a global reference system.This process consists of determination of orientation parameters, i.e.,three angles of rotation and three elements of linear transformation.For this purpose,affine 3D transformation is performed; it is also possible to achieve 3D transformation by similarity, when a scale difference exists between the exterior and the scanner co-ordinate systems.This transformation is performed based on control points, which should be evenly distributed across the entire area of the whole point clouds .At least three points are required; when their number is increased, redundant observations are generated, which allows gross errors to be eliminated and increases the orientation accuracy.The point-based registration method is the one most commonly applied for TLS data orientation (Vosselman, Mass, 2010).There are two types of this method: target-based and ICP.The target-based method is the conventional approach to pointbased orientation, and is usually performed with the use of measuring marks, which are previously distributed on the object.The iterative method, ICP, involves the detection and comparison of nearest pairs of points in clouds (McKay, 1992).The ICP algorithm requires a good first approximation in order to converge to a global minimum.However, even if there is considerable overlap, convergence to a global minimum is not guaranteed.The ICP algorithm can also be computationally intensive and time-consuming in its search for conjugate points in overlapping scans (Kang et. al., 2009;Sequeira, 1999).At present, research works aiming at elimination of measuring marks from the orientation process are being carried out (Kang et. al., 2009;Sequeira, 1999).Recent works are focused on orientation of TLS point clouds for estimation of transformation parameters by means of recognition of shapes on scans .The existing approaches are based on detection of corresponding characteristic features in a point cloud, such as lines or planes.Methods which utilize detectors and 3D descriptors of key-points are also used (Theiler et al., 2013;Theiler et al., 2014.).Another approach which is applied for point cloud orientation uses raster images, generated on the basis of TLS data, and successive image processing algorithms, applied using the computer vision (CV) method for identification of tie points.In this paper, target-based analysis is compared with the transformed feature-based registration method.

RELATED WORKS
The performed research used different detectors for orientation of TLS data.Algorithms searching for characteristic points (so-called key-point detectors) acquired based on spherical 2D intensity images were applied.A general diagram of scan matching using CV algorithms is presented in Fig. 1.

Conversion of point clouds to raster forms
In order to utilize the method of searching for key-points on digital images, the point cloud should be transformed into Figure .2 Relations between spherical co-ordinates and co-ordinates on spherical photographs (Fangi, 2007).the raster form.Two basic methods have been described in the technical literature.The first method consists of transformation of the point cloud into the central projection form (Melierhold, 2010;Moussa, et al. 2012), where pixels are interpolated.The second method consists of conversion of the point cloud into the form of a spherical image (Shahara, Sagi 2010;Alba et al., 2011;Wang, Brenner 2008;Kang et al., 2009) this utilizes raw data and does not require interpolation of new values when the photograph is generated with maximum resolution.The point cloud acquired from TLS is an arranged set of data.Two angles are measured for the distance between the instrument position and the analysed object; the laser beam reflectance intensity is also recorded.The mathematical method of data processing from the polar form to the spherical image form was described by Fangi (2007) (Rys.2).The proposed methodology of using spherical images for TLS data orientation utilizes two values as the grey level value: the laser beam reflectance intensity, and the distance from the analysed object.Since the point co-ordinates in the spherical photograph are known, it is possible to determine the XYZ co-ordinates of the analysed points.

Key-point extraction
Generally, different types of visual feature may be extracted from images in order to detect corresponding image contents (Weinmann, 2013).However, local features such as corners, blobs, or small image regions offer significant advantages.Since such local features (i) may be extracted very efficiently, (ii) are accurately localized, (iii) remain stable across reasonable variations in viewpoint, and (iv) allow individual identification, they are well-suited for a variety of applications such as object recognition, autonomous navigation and exploration, image and video retrieval, image registration or reconstruction, and interpretation and understanding of scenes (Tuytelaars, Mikolajczyk, 2008;Weinmann, 2013).Generally, the extraction of local features consists of two steps: feature detection and feature description (Urban, Weinmann, 2015).Usually, three types of detector are used to detect characteristic points in photographs (e.g., features, corners).The first group includes detectors that aim to detect corners, such as the Harris (Harris, Stephens, 1988), BRISK (Leutenegeer, 2011), andFAST (Rostem, 2006) algorithms.Another method of detection of characteristic points is based on detection of regionsso-called BLOB detection.For this, one of the following detectors is usually used: the Difference-of-Gaussian detector, implemented in the SIFT algorithm (Lowe, 2004), or the Determinant-of-Hessian (DoH) detector, utilized in the SURF algorithm (Bay et al., 2006).During the process of data orientation with the use of features detected in the image, it is important to detect stable key-points and to describe their features by means of descriptors.Detected points should be free from any noise and independent of the scale or rotation of images, the influence of illumination, and changes in the position of the projection centre (Urban and Weinmann, 2015).The SURF and SIFT algorithms are those most commonly applied for detection of features in photographs (Urban and Weinmann, 2015).Unfortunately, all algorithms which are based on detection of features in photographs have been designed for central projections.This approach assumes that standard image deformations occur as expected for a series of successive photographs.Where spherical photographs are applied, acquired based on TLS data, image deformations are big enough to generate problems concerning explicit identification and matching key-points (Markiewicz et al., 2015).The problem may be solvable through transformation of a point cloud into the form of "a virtual photograph in the central projection" (Moussa, et al 2012), scan transformation into an orthoimage (Markiewicz et al., 2015), or use of descriptors with consideration of affinity, such as with the ASIFT algorithm (Yu and Morel, 2011).

Key-point matching
Detection and description of features for each characteristic point are an important component in the process of detection of conjugate points in digital images.The next stage of classification of points as tie points in the process of image data orientation is mutual point matching.Usually, two approaches are applied: approximate nearest-neighbour-based point matching (Tran and Marchand, 2007) and brute-force matching (OpenCV, 2016).In the nearest-neighbourhood problem, a set of data points coded by the descriptor in the K-dimensional space is given.These points are pre-processed into an appropriate structure, so that given any query point Ωb, the nearest points to Ωb can be reported as quickly as possible.Although nearest-neighbourhood searching can be performed efficiently in low-dimension spaces, the search time grows exponentially as a function of dimension.To efficiently match two sets of points, we use the approximate nearest-neighbour technique proposed by Mount.The idea is to organize feature points into a kd-tree structure and compute the nearest neighbours approximately (Tran, Mrchand, 2007).The brute-force matcher is simple.It takes the descriptor of one feature in the first set and matches with all other features in the second set, using a set of distance calculations.The closest feature is returned.

Point-cloud registration
Detection of tie points allows their further utilization in the process of mutual orientation of TLS data.For this,estimated rigid-body transformation in the least-square sense is applied − rotation around x-axis,  -rotation around y-axis, ℵ -rotation around z-axis, x,y,z -coordinates in local system, X 0 , Y 0 , Z 0 -coordinates of TLS station in global system. (1) Since the point position in the spherical image is known, it is possible to determine XYZ co-ordinates for each point.
In this case, the formula 1 is applied: Unfortunately, when spherical images are utilized for orientation of TLS data, there is a problem concerning correct identification of the same tie points (Markiewicz et. al, 2015).For this, filtration of detected points is performed with the use of the well-known RANSAC algorithm, which increases the accuracy and correctness of the data orientation process.

ANALYSIS OF ALGORITHMS FOR AUTOMATIC ORIENTATION OF RASTER SCANNED IMAGES
Automatic TLS data orientation was performed with the use of the processed target-based registration method, supplemented by detection of points using the feature-based method.Typical CV algorithms were used in experimental works.A new, original approach to the adjustment of measurements has been presented by Markiewicz and Zawieska (2015) it considers the accuracy of measurements, previously determined by means of the Z+F 5006h scanner.
In the process based on feature-based registration, raster images were first acquired and then used for automatic searching for corresponding natural points,using algorithms based on detection of corners: The co-ordinates of detected points were processed from the 2D into the 3D form.X and Y co-ordinates determined in the spherical image were used to read the values of horizontal and vertical angles.Additionally, distance values between the scanner station and the analysed object were also recorded (from the depth map), and then re-calculated based on XYZ co-ordinates.Data acquired in this way allowed analysis to be performed using automatically identified natural points for TLS data orientation.In order to objectively assess and select the best algorithm:


the number of points and their distribution in spherical images were tested,  the total processing time of searching (by means of different algorithms), describing (with the use of one descriptor -SIFT) and matching key-points (using the brute-force matcher) was checked,  the percentage of correctly detected and matched key-points was analysed,  points were divided into two groups: control and check points,  the accuracy of natural control and check points was checked,  the accuracy of signalled checkpoints was checked,  deviations between point clouds were analysed,  the influence of particular factors was assessed and a method for overall evaluation proposed.

Data characteristics
Point clouds obtained from Iłża Castle, Poland, were used for the analysis; they were obtained using a Z+F 5006h terrestrial laser scanner.The resolution of the acquired data was 6 mm/10m.The maximum range of the point clouds was 360/310.The maximum distance between the scanner station and the object was 30 m, and the minimum distance was 3 m.Acquired data were initially filtered by reflectance intensity.
Additionally, marks were distributed on the analysed object in the form of Maltese crosses; these served as check points (Fig. 3).
Figure .3Distribution of marked check points.

Searching for tie points
During the first stage of the analyses of algorithms searching for tie points, the distribution of points was analysed for a fragment of the utilized spherical image.Parameters proposed by the authors of the above algorithms were used as default parameters.Fig. 4 presents the distribution of points for the following detectors, respectively: A) BRISK, B) FAST, C) MSER, D) SIFT, E) SURF, F) ASIFT, and G) CenSurE (STAR).It is noticeable in Fig. 4 that for all the applied detectors, points were evenly distributed in the analysed fragment of the scan.Moreover, it may be clearly seen that the best tie point detection results among algorithms based on edge detectors are obtained with the FAST algorithm.For the group of BLOB algorithms, the best results were obtained for the ASIFT algorithm.When the entire scan area is analysed, only the ASIFT algorithm allows the detection of points which are evenly distributed.For the FAST, SURF, and MSER algorithms, tie points were detected for approximately 75% of the scan area.The worst distribution of tie points was achieved when the CenSurE algorithm was used.
Unfortunately, none of the algorithmswith the exception of ASIFTdetect points which are located within high "distortion" areas.
Operator  1 presents the number of points detected by different algorithms.It may be seen that most points were detected by means of the ASIFT algorithm, while the BRISK algorithm detected the smallest number of points.

Description of detected key-points and their mutual matching
The SIFT descriptor was used to unify descriptions of points for all algorithms (with the exception of ASIFT).This assumption allowed unification of the time of searching, describing, and matching of points, in order to correctly analyse the calculations.The brute-force matcher algorithm was used for matching of points, which was implemented in the OpenCV library.Table 2 presents the time of searching and matching of features for tested detectors.As shown in the presented data, the shortest processing time was with the CenSurE (STAR) algorithm, and the longest with the SURF algorithm (excluding the ASIFT algorithm, for which the processing time was 10 times longer).

Analysis of accuracy of results
In order to check the correctness of the detected points, as well as to evaluate the accuracy of the orientation of point clouds, an accuracy analysis was performed on check points and control points; and deviations between point clouds were also checked.
At the first stage of elimination of incorrect points from the set of conjugate points (detected by means of the above algorithms), the iterative method, based on the least-square method, was applied.
Operator Table2 Time of computations of algorithms for searching and matching conjugate points.
As a criterion of the determination of correct points, the difference between the value calculated in the least-square adjustment process and the nominal value was assumed to be smaller than the scanning resolution (equal to 6 mm).When historical objects are analysed it is not possible to distribute any points marked on the object's surface.The presented solution proposes to utilize certain automatically detected points as check points.For this, an algorithm was developed to allow appropriate selection of points from a set of earlier detected points, evenly distributed over the entire object.Operations may be presented in four steps: 1.A raster (spherical) image is divided into four equal parts.2. For each analysed quarter of the image, the number of points is checked.If it is greater than six, every sixth point is selected.3. XYZ co-ordinates are determined for each point, based on pixel co-ordinates.4. Other points are considered as control points.An important factor which influences the accuracy of TLS data orientation is the intensity of the laser beam reflectance.In the performed works, this feature was used to weigh observations in the process of adjustment.

𝑝𝑣𝑣 = 𝑚𝑖𝑛 𝑝 =
1  2 Since pixel co-ordinates are known, it is possible to read intensity values within the area of 4x4 pixels, close to the point.In order to determine the accuracy of point measurements, the formula proposed in "Quality assessment of the TLS data in conservation of monuments" (Markiewicz, Zawieska, 2015) was applied (Fig. 5).(Markiewicz, Zawieska, 2015).

Figure. 5 Diagram of relations between reflectance intensity and accuracy of matching the reference plane
In order to better represent deviation values on control points, the values of errors are presented in the form of histograms (Fig. 5).Analysis of Fig. 6 reveals similar RMSE values for control and check points.Following the distribution of errors on control points, it may be stated that the MSER, STAR, BRISK, SURF, SIFT and ASIFT algorithms are characterized by similar values of errors.Only for the FAST algorithm is the RMSE error bigger, by an average of approx.0.5 mm.Similar analysis was performed for check points.Comparing the accuracy achieved for control and check points, differences in measurements performed with different detectors may be noticed.
 X co-ordinates: The smallest error was achieved for the STAR algorithm.The BLOB algorithms, i.e., MSER, SIFT and ASIFT, gave similar point-cloud matching values on check points.Only the SURF algorithm varied, by approx.0.5 mm.
 Y co-ordinates: The mean error values were much higher than for the X co-ordinates.The worst results were achieved for the BRISK algorithm, while the STAR algorithm turned out to be the best.All the algorithms (except BRISK) were characterized byaccuracy smaller than 3 mm.
 Z co-ordinates: The smallest error was achieved for the BRISK algorithm.The worst results were obtained for the STAR algorithm.The error of the FAST algorithm was 3.5 mm, while the remaining algorithms (except BRISK) were characterized by accuracy lower than 2.5 mm.For the BRISK algorithm, the percentage of points smaller than 2RMSEwas not determined.This was a result of the fact that only three points were assigned by the algorithm to the group of check points; the BRISK algorithm was therefore not utilized for further testing.Table 3 shows the best results were obtained for the SURF, MSER, FAST and SIFT algorithms.Additionally, in order to test the orientation accuracy, signalled marks were usedwhich are usually utilized for the targetbased orientation method.All marks were evenly distributed on the analysed object, at different heights.In order to analyse the accuracy of matching point clouds, the theoretical accuracy (based on the formula presented in Fig. 4), the theoretical resolution (based on the scanning distance), the obtained resolution, and the obtained error of matching (based on elements of orientation determined by the least-square method) were tested.The final results are presented in Table 5.
No The obtained results confirm the expected accuracy of the projection of a marknot lower than 4 mm.Based on the presented diagram (Fig. 7), it may be stated that the obtained accuracy corresponds to the mean theoretical value of the error, obtained on the basis of the intensity and the theoretical scanning resolution.As seen in the diagram, the largest distribution of errors was achieved for the STAR algorithm; the lowest distribution was achieved for the ASIFT algorithm.
Other algorithms are characterized by similar data orientation accuracy.Additionally, in order to independently check the point cloud matching, the CloudCompare software was used to analyse differences between fragments of point clouds (so-called octrees).This allowed the accuracy of point cloud orientation to be checked not only for selected points, but also for larger parts of point clouds.Where products are generated based on the processed point cloud, i.e., orthoimages or 3D models, the accuracy on check points is important, as well as the accuracy of entire parts of the point cloud.Fig. 8 present maps of deviations between point clouds.The results of point cloud registration are presented in Table 6.
The analysis of deviations between point clouds proves that higher deviations occur at the edges of the analysed fragmentthis may be caused by the structure of the analysed object (ruins of a castle constructed of bricks).The smallest distribution of errors is presented by point clouds oriented with the use of SURF and STAR detectors.
When the results presented in Table 5 are analysed, it may be noticed that the highest deviations between point clouds occur in the case of orientation on points detected by the STAR detector.The best results were obtained with the use of the ASIFT and SIFT algorithms.
Additionally, the obtained results were related to the theoretical resolution of a point cloud (Fig. 9).
Based on the analysis of the RMSE error value related to the mean resolution of point clouds for the specified sample (Fig. 9) the high difference for the STAR algorithm may be noticed.The respective results obtained are as follows:  Sample 1: for the FAST, MSER, STAR algorithmshalf of the resolution of a point cloud; for the SIFT, SURF and ASIFT algorithmsa little more than 0.4. Sample 2: the mean RMSE value for all algorithms with the exception of STAR-0.25 of the point cloud resolution of the analysed sample.The value for the STAR algorithm is four times higher.

DISCUSSION
In order to perform an objective analysis of the suitability of CV algorithms for detection of tie points on spherical images, a series of criteria are proposed (Table 7).Scores from 0 to 6 are assigned to each criterion, where 0 means the weakest and 6 the best result.It should be stressed that, for the majority of the tested algorithms, small differences between the final values were obtained.However, deviations for the BRISK and STAR algorithms were clearly visible; it is therefore not recommended to use these algorithms in the process of tie point detection for TLS data orientation.The results of the performed tests are presented in Table 6.As shown, the best results are obtained for the ASIFT algorithm.It has a considerable advantage over the remaining algorithms.

CONCLUSIONS
This paper has presented an analysis of CV algorithms for orientation of TLS data.The proposed method -processing into raster form, searching for and connecting tie points considerably improved the accuracy of the orientation of TLS data.Additionally, when a new parameter is added into the process of orientationnamely the influence of the reflectance intensity of a laser beam (expressed as the function of the point projection error) -standard procedures of adjustment can be considered, such as weighting of observations with the leastsquare method.This allows a consideration of the influence of the quality of local control points on the final values of the parameters of the exterior orientation of the point cloud.
Selection of appropriate detectors of tie points on point clouds, processed into spherical images, allows the time of orientation of large data sets to be reduced, without the need to manually measure tie points.When complex objects are being processed, such as cultural heritage objects, where it is not possible to distribute signalled tie points, the applied technique eliminates the problem of low accuracy, caused by detection of natural points in point clouds.

Fig. 1
Fig. 1 Diagram of orientation of TLS data with the use of CV algorithms.

Figure. 6
Figure.6 Diagram of RMSE values for control and check points.Descriptors applied in the data orientation process are marked by different colours.

Figure. 7
Figure.7 Distribution of values of linear errors on signalled control points, X axispoint number, Y axisdeviations in mm.

Figure. 8
Figure. 8 Deviations between point clouds oriented with the use of point detectors: A) FAST, B) MSER, C) SIFT, D) SURF, E) ASIFT and F) CenSurE (STAR).

Figure. 9
Figure. 9 Diagram of RMSE error values related to the average resolution of point clouds for a specified sample.X axissample number; Y axiserror value/average resolution of a point cloud for a given sample. Sample 3: for the FAST, MSER, SURF and SIFT algorithms -0.9 of the point cloud resolution of the sample;for the ASIFT algorithm -0.95;for the STAR algorithmequal to the resolution of the analysed sample. Sample 4: similar values are visible for all analysed algorithms, with the exception of the STAR algorithm.

Table 1 .
Number of detected key-points.Table

Table 4
Table 4 presents the percentage of points for which error values did not exceed 2RMSE.
Percentage of points with error not exceeding 2RMSE values (control and check points).

Table 6
RMSE errors of deviations between point clouds and values of deviations for 95% points.

Table 7
Test results for analysed algorithms.The performed test proved that the BRISK and STAR algorithms are not worthy of recommendation in the TLS data orientation process.On the other hand, the efficiency of the ASIFT and SIFT algorithms means that they can be considered.It is still an open question whether the ASIFT algorithm should be applied, because although it achieves better results than the SIFT algorithm, it takes 10 times longer