MIXED REALITY VISUALIZATION OF POINT CLOUDS FOR SUPPORTING TERRESTRIAL LASER SCANNING

3D point clouds from terrestrial laser scanners (TLS) are used in a variety of fields and applications. To acquire high-quality point clouds that have enough point density, small scanning errors, and no lack of points in important regions, appropriate scan planning, including determination of scanner positions and scan conditions, is required. Currently, planning is supported by knowledge and experience of skilled workers, and it is difficult to ensure the quality of acquired point clouds. In this study, we propose a system for visualization of point clouds to support the acquisition of high-quality point clouds using TLS. The system allows the user to see and check the quality of scanned TLS point clouds and unscanned regions intuitively by superimposing the point clouds onto the real world using a mixed reality (MR) device. In addition, the system supports finding the next best scanner position for additional laser scans based on predicted scan quality visualization to acquire higher-quality points or fill the unscanned regions.


Background and Objective
3D point clouds acquired by terrestrial laser scanners (TLS) are used in a variety of fields and applications. In many applications, accurate 3D models from scanned point clouds are often required, and high-quality point clouds that have enough point density and small scanning errors without a lack of points in important regions are essential for reconstruction of accurate 3D models. To acquire high-quality point clouds, appropriate scan planning, including determination of scanner positions and scan conditions, is required. Currently, planning is supported by the knowledge and experience of skilled workers, and some studies on supporting scan planning have been conducted. However, it is difficult to guarantee the quality of acquired point clouds.
In this study, we propose a system of mixed reality (MR) visualization of point clouds to acquire high-quality point clouds using TLS. Figure 1 shows a scenario of the use of our system in laser scanning using TLS. First, point clouds of the environment or objects are acquired by laser scanning using TLS (Step 1). Then, the user checks the point clouds and unscanned regions while seeing the superimposed quality of point clouds using the MR device (Step 2). If low-quality points or a critical lack of points in an important region exist, the user finds the next best scanner position according to the MR visualization of predicted scan quality of points (Step 3) and marks the position using a virtual TLS (Step 4). Finally, additional laser scanning is performed by the TLS set at the marked position (Step 5).
Step 2. Checking point clouds (to check the quality and the lack of points (on site)) Step 4. Additional scan planning 2 (to mark the NBT in real space) Step 3. Additional scan planning 1 (to find next best TLS position (NBT)) TLS MR device

TLS
Step 5. Additional scan (to set TLS at the NBT and perform scanning) Step1. Laser scanning using TLS Mixed reality visualization in each step Step 2.
In this study, we introduce the system overview and some technologies for achieving MR visualization of point clouds in support of laser scanning. A new automatic global localization method based on point cloud registration, current and predicted scan quality estimation methods of point clouds for visualization, and a method to realize efficient and real-time visualization of point clouds are proposed. The system is evaluated for two different environments: a plant and laboratory.

Related Works
Research on scan planning support by automatically determining scanner positions has been conducted to efficiently acquire TLS point clouds , Kitada et al., 2015, Wakisaka et al., 2019. These methods can estimate the best scanner positions for acquiring TLS point clouds efficiently. However, to estimate the positions, pre-created 2D maps or 3D models of the environment are often required. Furthermore, it is not possible to guarantee the quality of acquired point clouds (e.g., a lack of points caused by surface reflectance property often exist). Our system's objective is to efficiently guarantee the quality of point clouds, and these existing methods can be used for obtaining initial TLS point clouds (Step 1 in Figure 1).
The system we developed is based on a recent MR device, localization, point cloud quality estimation, and efficient point cloud management technology. The studies that are related to each technology are described below.
MR device: We use the Microsoft HoloLens2 (Microsoft, 2022) as the MR device. The HoloLens has many effective functions for mixed reality visualization (i.e., optical see-through display, laser-scanning by time-of-flight sensors for environment mapping and users' hand scanning, head tracking using inertial measurement units and images, eye-tracking, and wireless communication). Recently, MR visualization technology and devices are widely used in the fields of industrial engineering, medical and healthcare, architecture, and civil engineering (Park et al., 2021, Yamanaka et al., 2019. The device is also used in 3D scene reconstructions (Hübner et al., 2020, Weinmann et al., 2021, and its performance is evaluated. Superimposing various types of virtual objects onto the real world is used in many applications. However, superimposing TLS point clouds using an MR device for supporting laser scanning operations has not been developed to the best of our knowledge. One related technology is the augmented reality system for superimposing point clouds onto digital images (Ohno et al., 2021). This study describes an application method of MR visualization of point clouds and proposes a set of related technologies using HoloLens2.
Localization: In our MR visualization, point clouds are superimposed onto real environments using an optical seethrough display. Therefore, accurate and efficient localization and tracking of the MR device position and orientation are required. The MR device we used has a function for short-range (~3.5m) 3D scanning, so we can use point clouds acquired by the device for the global localization in TLS point clouds. Many global localization methods have been developed based on graph structures, Monte-Carlo simulation (particle filters), and machine learning (neural network) (Park et al., 2009, Yongjin et al., 2013, Zang et al., 2021. Markers in the environment can also be used (Hübner et al., 2018). Depending on the scale or size of the environment, these methods often require lengthy computation times, pre-computation, and preparations. In our situation, global localization should be performed quickly after laser scanning by TLS on site. Therefore, pre-computation should be avoided as much as possible, and short computation time should be desirable. To meet these demands, in our system, a global localization method using simple and efficient rough registration for TLS point clouds (Sumi et al., 2018) has been developed and evaluated in a real-world scene.
Point cloud quality estimation: The quality of laser-scanned point clouds is important for the stable and accurate application of the point clouds. The quality includes the point density, the lack of points caused by the occlusions, and measurement errors. These depend on the scan conditions (i.e., the setting positions of the TLS in the environment) and the scan setting (e.g., scanning pitch). The density can be easily estimated using the nearest neighbor search and adopted in our system. A lack of points can be identified using a superimposed visualization of point clouds. Research on measurement of random and systematic errors of TLS point clouds has been investigated. The random errors mainly depend on the scan range, angle of incidence of the laser, and surface reflectance properties , Ozendi et al., 2017). In our system, scan distance and laser incident angles are used as the scan quality related to measurement errors, and we do not consider systematic errors.

Point clouds for visualization (PCV)
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France Point cloud management: Large-scale point clouds can have over hundreds of millions of points, and an efficient data management data structure should be implemented based on the hardware used. Space partitioning such as octree and voxels are simple but powerful structures to handle large-scale point clouds (Wand et al., 2007, Rusu, 2013). In our system, uniform space subdivision is used for efficient frustum culling implemented in the software we used. In TLS point clouds, overly dense points are often acquired near the scanner or in overlapped regions of several scans. The redundant points can be caused for low performance of point cloud applications, including rendering. The point clouds with appropriate density in the application can be obtained by down-sampling. In our study, a new method for real-time point insertions with voxel sampling into the point cloud for visualization has been developed. The method operates in real time on the MR device, and it is possible to use the points from each scan frame from the MR device in the visualization. Figure 2 shows the overview of our system. We use an MR device (Microsoft Hololens2), which is an optical see-through head-mounted display with 3D scanning and head tracking functions. Each function of our system is explained below.

Overview of Our System
To realize MR visualization of point clouds, first the coordinate systems of the MR device and TLS point clouds should be aligned. We call this process global localization ( Figure 2. A, Section 2.2). In our system, global localization is performed by the registration of TLS point clouds and MR device point clouds. After global localization, tracking of the device's position and orientation is always required for superimposed visualization of point clouds. This tracking is performed by the device's function. In our pre-experiments, position and orientation tracking errors for dozens of meters movements are few to ten centimeter in a laboratory and a corridor environment, and we consider that the tracking accuracy is enough in our applications.
Point qualities and predicted qualities are estimated using point clouds from TLS and MR device ( Figure 2. B, Section 2.3). Quality of TLS point clouds is calculated on the server (note PC or desktop PC), and the predicted quality is computed on the MR device in real-time using point clouds from TLS and the MR device.
In the MR visualization of point clouds, the quality of points is represented by colors. The virtual TLS is used to mark the scanner position for the best additional scan found by predicted quality (Figure 2. C, Section 2.4). For the efficient visualization of point clouds, point clouds for visualization (PCV) are created from point clouds from TLS and MR device efficiently using a visualization point generator (Figure 2. D, Section 2.5). PCV is generated by space subdivision and down-sampling for given point clouds.

Overview:
To achieve superimposed visualization of point clouds on the real world using an MR device, the device must first be global localized in the point cloud space. In our system, global localization is automatically performed by registration between each point cloud from TLS and point clouds from MR device. An efficient rough registration method (Sumi et al., 2018) followed by the iterative closest point (ICP) algorithm (Besl andMcKay, 1992, Rusinkiewicz andLevoy, 2001) is used in the process. To make the localization more robust, a new simple correctness measure of the registration results based on the point cloud overlap and inconsistency of the space has been developed. In this section, the rough registration method that we used and a measure of registration correctness are first described in Sections 2.2.2 and 2.2.3. Then, in Section 2.2.4, the global localization method is discussed in detail.

Rough Registration Method:
Many methods for rough and precise registration of point clouds have been proposed , and we have options for choosing the method. In our system, a pair-wise registration method using 2D images of sliced points (Sumi et al., 2018) is implemented. The method first creates the point projection images using horizontally sliced points of point clouds, and feature points in the image are extracted by polygonization of the figures in the image (Figure 3). Extracted feature points in the images are efficiently and robustly matched using the hash tables of pairs of feature points and the RANSAC framework. The method requires a planar floor or ground plane to make horizontally sliced points. The used MR device can acquire the points of the floor or ground in natural situations. We consider that the method's efficiency and robustness are sufficient in our application, so we implemented it in our system.

Measure of Registration Correctness:
To find the best registration result from the rough registration between multiple TLS and MR device point clouds, we use a registration correctness measure. Proper registration results of two-point clouds of static scenes have some properties related to points and space, as shown in Figure 4. First, the space that the laser passes in a scan does not include points from another point cloud. Second, the overlap of points becomes larger compared with incorrect registration results. Third, the space that the laser 3. the space that the laser passes in two scans has a certain amount of overlap.
1. the space that the laser passes in a scan does not include points from another point cloud. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France passes in two scans has a certain amount of overlap with each other. These properties can be evaluated using certain criteria. However, the computation cost will increase for large numbers of point clouds and large scenes. To evaluate these properties efficiently, in our study, space occupancy images were used to find the best registration results from multiple ones. The space occupancy image is generated by lay casting on the point projection image from the scanner position. The pixels that the lay passes in lay casting are classified as FREE, the cells including projected points are classified as OCCUPIED, and the others are classified as UNKNOWN, as shown in Figure 3(c).
Correct registration results have less overlap between FREE and OCCUPIED pixels, enough overlap between FREE pixels, and a larger overlap between OCCUPIED pixels. From these properties, we define a measure for evaluating the registration results of a pair of point clouds, as expressed in Eq. (1).

( )
ocpy ocpy q P P   X ocpy P and X free P are the sets of pixels of OCCUPIED and FREE pixels of point cloud X, and w is a user-defined parameter ( 2 w  is used in our experiments). Figure 5 shows the examples of the measures applied to eleven TLS point clouds of a heat source plant and an MR device point cloud acquired at different initial device positions. In each graph, the vertical axis shows the measures of AB V , and the horizontal axis shows the number of scans of the MR device point cloud. From the experiments, it was observed that the higher values can be obtained from correct registration results (red lines in the figure).

Global Localization Method:
Point cloud registration is performed for each point cloud from TLS and point clouds from the MR device when the moving distance of the MR device, the changes of the device orientation and the number of points from the MR device exceed the given thresholds. The registration results are evaluated using a measure defined by Eq. (1). As shown in Figure 6, registration and evaluation are performed by using the point clouds from the MR device successively. If the TLS point cloud, which gives a measure over than a given threshold and continuously maximum in several times compared with the other point clouds, the registration result (a coordinate transformation) to the TLS is adopted as the transformation matrix for initial global localization. Thereafter, the ICP algorithm (Besl andMcKay, 1992, Rusinkiewicz andLevoy, 2001) is applied to the TLS point cloud and MR device point cloud to refine the device position and orientation.
In our implementation, the point clouds from the MR device are transferred to the server using TCP/IP communication, and the registration is performed on the server. The resulting coordinate transformation matrix is transferred to the MR device and applied to the TLS point clouds in the MR device.
Global localization is performed by registration between TLS point clouds and local point clouds near the user's initial positions. Therefore, non-negligible superimposition errors often occur at a position far from the initial position. In our system, the user can modify the error when they recognize nonnegligible superimposition errors. This modification is performed by using the ICP algorithm between point clouds from TLS and the point cloud from the MR device near the current position.

Point Quality Estimation and Prediction
In Steps 2 and 3 in Figure 1, the scan quality of point clouds is superimposed and visualized using the MR device. In our system, scan distance, laser incident angle, and point density, which are related to the measurement errors , Ozendi et al., 2017 and the stability of data processing, are used as the scan quality of points. In Step 2, these quality values are computed for each point in the TLS point clouds. For Step 3, the predicted quality of the point cloud acquired by additional scanning using the TLS set at the current user (device) position is computed in real time on the MR device using the point clouds for visualization, which consists of points both from the TLS and the MR device.
The quality for Step 2 can be pre-computed for the TLS point clouds. The scan distance dist i q , laser incident angle angle i q , and the approximated density density i q of a point i can be computed as follows:  In our implementation, the four closest neighbors on the structured point clouds (Masuda and Tanaka, 2010) are used as Xi, and the normal of each point is estimated using principal component analysis of local points (Rusu, 2013).

The predicted quality for
Step 3 can also be computed by similar equations with small changes. In the predicted quality computation, D o is the MR device position, and X i is a set of virtual neighbors of point i. The positions of the virtual neighbors are defined using the local virtual plane at the point i and lays that is defined by a scan pitch in laser scanning, as shown in Figure 7. They can be calculated using Eq. (5).

MR Visualization
In Steps 2 and 3 in Figure 1, the current quality of TLS point clouds and predicted quality are displayed by colored points. For efficiency, the point clouds for visualization (PCV) described in the next section is used in the rendering. In our implementation, points within 10 m of the device are used in rendering under the assumption that the user will not check far away point clouds. When the user finds the next best TLS position for additional scanning in Step 3, the position is marked in the 3D space using the virtual TLS, and used to set the TLS in additional scanning, as shown in Figure 8.

Visualization Point Generator
The PCV using mixed reality should be created according to the requirements of the system and device. We considered the following three requirements. First, the rendering framerate should be kept high enough in MR visualization. Second, point clouds from TLS and MR device can be individually visualized because only TLS point clouds are used for checking the scan data (Step 2 in Figure 1) and both point clouds from TLS and MR device are used for visualizing predicted quality (Step 3 in Figure 1). Third, point clouds from the MR device are successively added to PCV in real time on the MR device for visualizing the predicted quality.
To keep the rendering framerate high enough in MR visualization, we use spatially subdivided down-sampled points as PCV. In our implementation, we use Unity (Unity Technologies, 2022) to create rendering objects of point clouds in MR visualization. Unity adopts the view frustum culling of rendering objects for efficient rendering. To make the effectiveness of the frustum culling higher, we use small-sized rendering objects consisting of a set of local points by uniformly subdividing the space of the point clouds. We call each subdivided cubic space as a subdivision cell. In addition, to keep the rendering framerate, point clouds are down-sampled using voxel sampling and stored as PCV. The PCV from TLS and MR device is individually stored in the corresponding rendering object to satisfy the second requirement described above. The cell sizes of subdivision cells and down-sampling voxels are determined to keep the allowable maximum number of points in a rendering object (1.04m and 13mm are used as sizes of the subdivision cell and down-sampling voxel respectively in our implementation).
In our system, PCV is created using an integrated occupancy grid for efficiently creating PCV and to avoid overlaps of TLS and MR device points. The global size of the grid equals the one of the subdivision cells, and the size of the cell of the grid is same as the one of the voxel for down-sampling. Each cell has the indices of the occupied subdivision cells, and the indices are stored using a binary search tree for efficient addition of MR device points to PCV. Figure 9 shows the PCV generation using an integrated occupied grid. First, the initial PCV is created from TLS point clouds. In each subdivision cell, TLS points are sampled using a voxel. If the cell of the voxel includes the point, the index of the subdivision cell is stored in the corresponding cell in the integrated occupancy grid. This process is applied to all subdivision cells, and the integrated occupancy grid of TLS points is obtained, as shown in Figure 9(a). When we add the MR device point cloud to the PCV, first, a sampling of points is applied. Then, if the cell of the integrated occupancy grid corresponding to the sampled point i does not include the index of the subdivision cell of i, point i is added to the PCV, and the index of the subdivision cell is stored in the cell of the integrated occupancy grid. Otherwise, point i does not get added to the PCV. In this process, we have to efficiently find the index of subdivision cell, so a binary search tree is used in each cell of the integrated occupancy grid. Figure 9(b) shows the results of the addition of MR device point cloud to the PCV. Addition of the MR device point cloud is performed on the MR device in real time in the predicted quality visualization.

EXPERIMENTAL RESULTS
Our system was implemented using Microsoft Hololens2 and tested in a heat source plant using 11 TLS point clouds and a laboratory using 4 TLS point clouds. Photos and TLS point clouds of each environment are shown in Figure 10. The results of quality visualization of TLS point clouds and predicted quality visualization are shown in Figures 11 and 12. Figure 13 shows a result of MR visualization before and after global localization. It was confirmed that all functions of our system worked appropriately. An example of supporting additional scanning is shown in Figure 14. Checking the quality of points and finding and marking the next scanner position was realized. Although small visualization gaps between the points and real objects were observed, the user could check the point clouds of the scene intuitively.
The performance of the global localization was evaluated using 12 and 5 different initial user positions in the plant and laboratory, respectively. As a result, global localization of 16 initial positions were succeeded. Figure 15 shows a point cloud registration result in the global localization. In the plant, passages with similar structures between large tanks existed. Global localization in failure was resulted in the position in the next passage of the correct one. Global localization took about 20~30 s in total (In an example, 7.6 s for data transmission, 10.4 s for rough registration and 3.0 s for the ICP algorithm, CPU: Core-i7 7820HQ). The accuracy was evaluated using a marker The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France set a few meters away from the initial position. In our experiments, positional errors were ranging from few millimeters to about 200 mm, and angle errors were a few degrees in maximum. In our experiments, these errors are acceptable in the MR visualization of point clouds for checking the scanned regions and point quality. Figure 16 shows visualized points from TLS in the PCV ( Figure  16 (a)) and points from both TLS and MR device (Figure 16 (b)) generated by the integrated occupancy grid. We observed that no redundant (overlap) sampled points can be appropriately created as the PCV using our method. Rendering framerates were kept at about 30 FPS. This is also enough in our applications.
In the visualization of point cloud, hidden point removal is performed automatically on the MR device. The process uses the mesh created from the points acquired by the MR device's scanner, and the mesh is updated about every 3 s. The scan range of the MR device scanner is 3.5 m (in specification). Therefore, it is not possible to remove the hidden points away from the device position. To improve the visibility in largescale scenes, accurate hidden point removal will be required, and this may be performed by using the mesh from TLS point clouds.

CONCLUSIONS
In this paper, a mixed reality visualization system for supporting laser scanning of environments using TLS is described. The system allows the user to check the scanned point cloud by superimposing the points colored by the quality onto the real environment using see-through head mount display. In addition, the predicted quality visualization mode of our system supports in finding the next scanner positions of additional scanning, and the position can be marked using a virtual object. To realize these functions, a new global localization method based on point cloud registration and registration correctness measure, a new method for creating uniformly subdivided down-sampled point clouds for visualization, and a new method for predicting point cloud quality were introduced.
Our system was tested in two different environments. Using our system, the quality of point clouds was superimposed onto the real world, and it was possible to check the scan points, find the The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France next scanner position to obtain high-quality points, and mark the position using a virtual object. The proposed global localization works well for two environments (with a success rate of 94% in our experiments), and enough rendering performance (about 30 FPS) was observed. The superimposing errors were about 200 mm at the maximum, and the error was acceptable in our applications.
Future work includes more accurate hidden point removal to improve the visibility and application to outdoor scenes.