THE EXAMPLE OF USING INTENSITY ORTHOIMAGES IN TLS DATA REGISTRATION – A CASE STUDY

The paper presents the orientation analysis of terrestrial laser scanning (TLS) data. In the proposed data processing methodology, point clouds are considered as panoramic images and orthoimages enriched by depth maps. Computer vision (CV) algorithms were used for the orientation; they were applied to test the correctness of the detection of tie points, as well as the accuracy, number and point distribution. For the source data, point clouds acquired from the terrestrial laser scanner Z+F 50063h for the two chambers in the Museum of King John III’s Palace in Wilanów were utilized.


INTRODUCTION
Terrestrial laser scanning (TLS) point clouds are recorded in a local reference system for the instrument.For a large number of data sets (as is the case when big and complex objects are being processed), the point clouds must be transformed to the global reference system.This process consists of the determination of orientation parameters: three rotation angels and three elements of the linear transformation.For this purpose, the 3D affine transformation is performed.It is also possible to apply the similarity transformation in 3D, when a scale difference exists between the exterior and the scanner co-ordinate systems.This transformation is performed based on control points, which should be evenly distributed across the entire area of the common cover of point clouds.At least three points are required for this purpose; when their number is increased, redundant observations are generated, allowing gross errors to be eliminated and increasing orientation accuracy.The Point-Based Registration method is mostly applied for the registration of TLS data (Vosselman, Mass, 2010).This method has two different types: the Target-Based and ICP methods.The Target-Based method is the conventional approach to the Point-Based orientation, which is usually performed using measuring points, which are earlier distributed ion the object.Another commonly applied method of TLS data orientation, is the iterative method to detect the closest pairs of points in the clouds being compared (ICP) (McKay, 1992).The ICP algorithm requires a good first approximation in order to converge to a global minimum.However, even if there is considerable overlap, convergence to a global minimum is not guaranteed.The ICP algorithm can also be computationally intensive and time-consuming in its search for conjugate points in overlapping scans (Kang et al., 2009;Sequeira, 1999).At present, research that aims to eliminate measuring marks from the orientation process is being performed, and recent works are focused on the registration of TLS point clouds, based on their estimated shapes on scans.Existing approaches are based on the detection of corresponding characteristic features in a point cloud, such as lines or planes.Methods utilizing detectors 3D keypoints descriptors are also used (Theiler et al., 2013;Theiler et al., 2014.).A different approach applied for point cloud orientation is the use of raster images, generated on the basis of TLS data and the successive use of image processing algorithms, applied in the CV to identify conjugate points.

RELATED WORKS
Modern software packages dedicated to processing terrestrial and aerial photographs utilize algorithms based on a combination of methods commonly applied in ComputerVision (CV), as well as algorithms applied in conventional photogrammetric works.Algorithms applied for those purposes allow for the complete automation of orientation and data processing.At present, attempts to use such solutions for the automatic orientation of TLS data and integration with other data sources are being carried out.Generally, different types of visual features may be extracted from images in order to detect corresponding image contents (Weinmann, 2013).However, local features such as corners, blobs or small image regions offer significant advantages.Since such local features (i) may be extracted very efficiently, (ii) are accurately localized, (iii) remain stable over reasonably varying viewpoints and (iv) allow an individual identification, they are well-suited to a variety of applications, such as object recognition, autonomous navigation and exploration, image and video retrieval, image registration or the reconstruction, interpretation and understanding of scenes (Tuytelaars and Mikolajczyk, 2008;Weinmann, 2013).Generally, the extraction of local features consists of two steps, which are feature detection and feature description (Urban and Weinmann, 2015).Usually, three types of detectors are used to detect characteristic points on images.The first of these includes detectors based on corner detection, such as the Harris (Harris and Stephens, 1988), BRISK (Leutenegeer, 2011) and FAST (Rostem, 2006) algorithms.Another is based on region detection, or the so-called blob detection, such as the Difference of Gaussian detector (DoG;Lindberg, 1993) implemented in the SIFT algorithm (Lowe, 1992) or the Determinant of Hessian (DoH) detector, utilized in the SURF algorithm (Bay et al., 2006).During the data orientation process using features detected in the image, it is important to detect stable keypoints and to describe their features by means of descriptors.The points detected should be free from any noise and independent of the influence of illumination and position changes of the centre of projection on the scale or rotation of the images (Urban and Weinmann, 2015).The SURF and SIFT algorithms are the most frequently applied to detect features in photographs (Urban and Weinmann, 2015).
Unfortunately, all of the algorithms have been designed for images projected in the central projection.Such an approach assumes that standard image deformations might be expected.In the case of spherical photographs, acquired based on terrestrial laser scanning data, image deformations are big enough to generate problems concerning explicit identification and matching keypoints (Markiewicz et al., 2015;Markiewicz, 2015).It seems that the problem could be solved by transforming a point cloud into the form of "a virtual photograph in the central projection" (Moussa, et al 2012) by transforming the scan into an orthoimage (Markiewicz et al., 2015) or the use of descriptors which consider affinity, such as the ASIFT algorithm (Yu and Morel, 2011).

ORTHOIMAGE GENERATION
Orthoimages are attractive for archaeological and architectural documentation, as they offer a combination of geometric accuracy and visual quality, and can also be applied to different measuring techniques.Two main approaches to orthoimage generation exist.The first one is based on TLS data and using terrestrial photographs.Despite its many advantages, the TLS technique also has a lot of limitations, such as problems with acquiring sharp edges, poor quality or reflective surfaces, etc.The second approach to orthoimage generation consists of only using terrestrial photographs, processed using software based on Computer Vision (CV) algorithms.The conventional approach to orthoimage generation is based on processing a terrestrial photograph via the orthorectification process.This process consists of processing a photograph based on a digital surface model (DSM); as a result, deformations caused by distortion and the influence of the central projection can be eliminated.Photographs in orthogonal projection are obtained as a result of such processing.The most important stage in the generation of orthoimages is the interpolation of the digital surface model.The hierarchic way of recording data acquired with the use of terrestrial laser scanning allows the DSM to be recorded in the form of a regular square grid (GRID).Additionally, as a result of using this data recording structure, it is possible to fill blind spots resulting from the filtration of a point cloud.The reflectance intensity value and the distance between the measuring station and the object are assigned to each point of the GRID structure in the form of a depth map.Each generated orthoimage is georeferenced; XZ co-ordinates are determined for each pixel of the image, and the Y co-ordinate is determined from the depth map.When the point cloud is represented by means of an orthoimage, it does not have geometric deformations, as is the case for spherical images.
As a result, data are recorded in an ordered way, the generation of the DSM in GRID form is not a problem and there is no need to interpolate new pixel values.Disadvantages of this approach include the issue of the detection and interpolation of a reference plane, as this is the basis of DSM generation.A description and comparison of different algorithms for detecting reference planes was presented in the paper entitled A New Approach to the Generation of Orthoimages of Cultural Heritage Objects-Integrating TLS and Image Data (Markiewicz et al., 2015)

TLS DATA ORIENTATION -PERFORMED EXPERIMENTS
The author of this paper has closely co-operated with the Museum of King John III's Palace in Wilanów in conservation works, as well as in other projects.The development of metric documentation for complicated historical objects requires an automated technological process to be implemented.One of the most important products used at the museum is the orthoimage.Based on this experience, the orthoimage generation process has been automated.In order to increase the accuracy of point cloud orientation, orthoimages acquired from non-oriented point clouds were utilized.Data orientation using spherical images from processing TLS point clouds was also selected.Thanks to this, it was possible to compare the results obtained from two independent methods.

Characteristics of the source data
Point clouds from the two chambers in the Museum of King John III's Palace in Wilanów were used for the analysis.Data were acquired using two different terrestrial laser scanners: Z+F 5006 and 5003.The resolution of the acquired data was equal to 6 mm/10 m.The maximum point cloud range was equal to 360/310 and the distance between the scanner station and the object was 2 to 10 m.In the first step, the acquired data were initially filtered by the reflectance intensity.Additionally, control points commonly applied in TLS data orientation were located on the second test site.The first test site is characterized by a number of adornments such as bas-reliefs, facets or fabrics hanging on the walls.The walls are painted with spatial patterns (Fig. 1a).The second test site is more geometrically complicated.It has rich adornments, such as bas-reliefs, facets or fabrics on the walls.Additionally, mirrors in golden frames and decorative fireplace are located in some sections of the walls (Fig. 1b).

The orthoimage generation process
The first stage consisted of detecting planes in the point cloud.For that purpose, the transformed Hough algorithm was applied as described in the paper A New Approach to the Generation of Orthoimages of Cultural Heritage Objects-Integrating TLS and Image Data (Markiewicz et al., 2015).Using this data it was possible to automatically generate an orthoimage using LupoScan software.A point cloud of 3 mm resolution was used.Thus, it was not possible to generate an orthoimage with a GSD smaller than 3 mm.The following input parameters were assumed: • NN interpolation; • assumed deviation of the automatically generated plane = 5 cm; • buffer of projected points = ± 3 * 5 cm; • resolution of generated object = 3 mm.Additionally, depth information was recorded for each orthoimage, as well as information concerning the laser-beam reflectance intensity.Fig. 2 presents examples of orthoimages applied for the orientation of terrestrial laser scanning data.

Keypoint detection and description
In order to search for tie points, the SIFT detector and descriptor were applied.Resulting from the use of orthoimages of walls, the point cloud was divided into fragments.This allowed for the detection of evenly distributed points within the entire scan area.Fig. 2 demonstrates examples of orthoimages with tie points (colour dots).In order to check the correctness of the detected points, as well as to evaluate the accuracy of the point cloud orientation, the accuracy analysis was performed on check points and control points, and the deviations between point clouds were checked.At the first stage, where incorrect points were eliminated from the set of conjugate points (detected by means of the above algorithms) the iterative method, based on the least square method, was applied.As a criterion for the correct point determination, the value of the difference between the value calculated in the least square adjustment process and the nominal value, smaller than the scanning resolution (equal to 5 mm), was assumed.When analysing historical objects, it is not possible to distribute any points marked on the object's surface.In the presented solution, it is proposed to utilize certain automatically detected points as check points.For that purpose, an algorithm was developed which allows for the appropriate selection of points from a set of earlier detected points, evenly distributed across the entire object.The diagram of operations may be presented in four steps:

Test site I
The first stage consisted of testing the number of correctly detected keypoints.Tab. 1 presents the results for particular orthoimages and for the spherical image.

Tab. 1 The number and percentage of correctly detected keypoints
It may be noticed from Tab. 1 that the number of points detected in the orthoimages in these two cases is higher than in the case where spherical images are applied.The smaller number of points in spherical images results is due to the scanner being positioned closed to the wall, resulting in the effect of image "distortion".This geometric deformation may be eliminated by using orthoimages.Position four of the scanner was located close to position three.Only the scanning height was changed; as a result, more points were detected in the spherical images than when orthoimages were used.
When the keypoints distribution is analysed, it may be noticed that when spherical images were utilized, the SIFT algorithm incorrectly detected points on the floor and ceiling.Fig. 3 presents diagrams of the numbers of detected control and check points.Tab. 2 shows that, in the case of the point cloud representation in the form of orthoimages being applied, the number of correctly detected and matched points is considerably higher than when applying spherical images.Terrestrial laser scanning data were not acquired with the full angular resolution.As a result, the SIFT algorithm for spherical images with "distortion" was incorrectly matching features detected in pairs of orthoimages.By representing the point cloud in the form of orthoimages, it was possible to eliminate the impact of that error.This resulted in the increased number and matching of detected points.Similar to Test site 1, the use of orthoimages influenced the correct detection of keypoints on facets, the floor and the ceiling.Fig. 4 presents the diagrams of the numbers of detected control and check points.

Accuracy analysis of results
In order to analyse the accuracy of terrestrial laser scanning data, the orientation values of deviations were tested for the control and check points.

Test site I
Tab. 3 presents the results of the orientation on the control points.When Tab. 3 is analysed, it may be noticed that the accuracy of the orientation of point cloud fragments (walls) are similar.When spherical images are used, the accuracy of the orientation is insignificantly lower.It results from the uncomplicated geometry of the objects (straight, unadorned walls).
Tab. 3 RMSE of the orientation of point clouds on the control points.
The distribution of errors, based on histograms of the deviations of the control and check points, was analysed.An example is presented in Fig. 5. Fig. 6 presents the spatial distribution of points detected using both methods.Intervals of between -3 and 3 mm for the X coordinate, -3 and 3 mm for the Y coordinate and -3 and 3 for Z coordinate were taken as measures of accuracy; however, the majority of points oscillated between -5 and 5 pixels, corresponding to a range of ±4 mm.For the check points, deviations are around ± 4 mm.When spherical images are used, the errors achieved on the check and control points are higher.Intervals of between -4 and 4 mm for the X coordinate, -4 and 4 mm for the Y coordinate and -4 and 4 for the Z coordinate were taken as measures of accuracy; however, the majority of the points oscillated between -6 and 6 mm, corresponding to a range of ±6 mm.The deviations for the check points are around ± 5 mm.Additionally, the percentage of points for which the error value on the check points was greater than 2 RMSE is also shown.
In the case of Test site 1, a slight improvement in the percentage of such points may be noticed.It may be concluded that using orthoimages for TLS data orientation for objects of uncomplicated geometry only has an influence on improving the distribution of keypoints, and not on the accuracy of the data orientation process.

Test site II
Tab. 4 presents the results of orientation on the control points.When Tab. 4 is analysed, it may be noticed that the accuracy of the orientation of point cloud fragments (the walls) are similar.
Tab. 4 RMSE of point clouds orientation on control points.
When spherical images are used, the orientation accuracy is insignificantly lower.Significant improvements in data orientation may only be noticed when using orthoimages when fragments are scanned from a wide angle.This is caused by the "pseudo-distortion" effect on spherical images.Additionally, smaller deviations on the control points for the Z component may be noticed.Additionally, in the upper fragments of a scan (on facets) it was not possible to detect the correct tie points on spherical images.In order to prove the advantages of orthoimages as compared to spherical images, Fig. 8 presents the spatial distribution of points detected using both methods.The distribution of errors, based on histograms of the deviations of the control and check points, was analysed.An example is presented in Figs. 9 and 10.Intervals of between -5 and 5 mm for the X coordinate, -5 and 5 mm for the Y coordinate and -4 and 4 for Z coordinate were taken as measures of accuracy.Deviations for the check points are around ± 5 mm.When spherical images are used, errors achieved on check and control points are similar, but the number of points is lower.When analysing the histograms, it may be noticed that the error values for orthoimages are close to 0. The fact that a greater number of points is utilized for the orientation process (of check points) and for testing the results may be also noticed.(f) Z direction in pixelscheck.

FINAL REMARKS AND CONCLUSIONS
This paper presents the analysis of computer vision algorithms for the orientation of terrestrial laser scanning (TLS) data.The proposed method of processing data to the raster format (orthoimage and spherical image), as well as searching for and connecting tie points, considerably influenced the improved accuracy of TLS data.When processing complex objects, such as cultural heritage objects, where it is not possible to distribute the tie points indicated, the applied measurement technique allows the problem of achieving low accuracy values to be eliminated due to the detection of natural points in point clouds.
The following conclusions may be drawn based on the performed experiments:  The use of intensity orthoimages increases the accuracy of TLS data registration.


As a result of the use of orthoimages, image deformations which occur as a result of the transformation of a point cloud to a spherical image can be eliminated.


The use of orthoimages for the orientation of terrestrial laser scanning data reduces the time it takes to detect keypoints.


When orthoimages are used, more correctly detected tie points are obtained as compared to the use of spherical images.


The detected points are evenly distributed across the entire area of works.


Thanks to the use of orthoimages, it is possible to increase the accuracy of orientation in selected fragments of the scan.This is important in terms of results generation, high resolution orthoimages or precise 3D modelling.


It is planned to perform successive tests on the facades of historical buildings.


Using orthoimages for images with less complicated textures may contribute to increasing the number of number of keypoints in the terrestrial laser scanning data orientation process.


The analysis of data acquired by close-range scanners, which considers the accuracy of the proposed method, allows the resulting elements of exterior orientation to be considered as final data, without it being necessary to conduct a final orientation of the point clouds using the ICP method.

Figure 1 :
Figure 1: Point clouds of the test sites 1a and 2b in spherical projection, with fragments of the utilized orthoimages marked

Figure 2 :
Figure 2: An example of the distribution of control points in images: a) the first test site, b) the second test site

Figure 3 :
Figure 3: The numbers of control and check points detected in spherical images and orthoimages (Test site I)

Figure 4 :
Figure 4: The numbers of control and check points detected in spherical images and orthoimages (Test site II)

Figure 6 :
Figure 6: Distribution of the control and check points detected in a) orthoimages and b) spherical images

Figure 7 :
Figure 7: Histograms of the deviations of the control and check points for spherical images: (a) X direction in pixels -control; (b) Y direction in pixels -control; (c) Z direction in pixels -control; (d) X direction in pixelscheck; (e) Y direction in pixels check; (f) Z direction in pixelscheck.

Figure 8 :
Figure 8: Distribution of the control and check points detected in a) orthoimages and b) spherical images

Figure 10 :
Figure 10: Histograms of the deviations of the control and check points for spherical images: (a) X direction in pixels -control; (b) Y direction in pixels -control; (c) Z direction in pixels -control; (d) X direction in pixels -check; (e) Y direction in pixels -check;(f) Z direction in pixelscheck.