EVALUATION OF MOBILE MAPPING SYSTEMS FOR INDOOR SURVEYS

With their high recording rate of hundreds of thousands of points acquired per second, speed of execution and a remote acquisition mode, SLAM based mobile mapping systems (MMS) are a very powerful solution for capturing 3D point clouds in real time, simply by walking in the area of interest. Regarding indoor surveys, these MMS have been integrated in handheld or backpack solutions and become fast scanning sensors. Despite their advantages, the geometric accuracy of 3D point clouds guaranteed with these sensors is lower than the one reachable with static TLS. In this paper the effectiveness of two recent mobile mapping systems namely the GeoSLAM ZEB-REVO RT and the more recent GreenValley LiBackPack C50 is investigated for indoor surveys. In order to perform a reliable assessment study, several datasets produced with each sensor are compared to the high-cost georeferenced point cloud obtained with static laser scanning target-based technique.


INTRODUCTION
3D digital models are essential in diverse applications especially since the extension of building information modelling (BIM), or more generally the documentation of cultural heritage. In the last couple of decades, static terrestrial laser scanners (TLS) have proven to be the most appropriate tool for acquiring in a fast way and simultaneously with high level of details, high quality point clouds. Their extensive use and the fastidious georeferencing process led, in the last decade, to the development of new 3D mapping technologies, enabling dynamic acquisition outdoor and indoor. With the purpose of indoor surveys, these mobile mapping systems (MMS) have been integrated in handheld or backpack solutions and become fast scanning sensors. In outdoor environment GNSS signal enables satisfying positioning accuracy. In indoor environments, accurate positioning of the scanning stations is certified with conventional approaches such as the use of total stations and target based terrestrial laser scanning. Obviously, this solution is not effective in large and rapidly changing indoor environments. Currently, the most promising approach for resolving this issue is the use of Simultaneous Localization And Mapping (SLAM).
With their high recording rate, speed of execution and remote acquisition mode, SLAM based MMS are a very powerful solution for capturing 3D point clouds in real time, simply by walking in the area of interest. Thanks to their portability, handheld scanners are interesting for acquiring both small objects and larger ones, since these scanners enable easy access to occluded areas. In the field of cultural heritage, it is common that local areas need to be completed by detailed and dense data. Thanks to the very simple interfaces offered by the manufacturer, these sensors are easy to use even without surveyor's knowledge. Nevertheless, the geometric accuracy of 3D point clouds guaranteed with these sensors is lower than the one reachable with static TLS. Thus, over the last few years, a lot of research has been carried out to evaluate the performance of these new systems, which are constantly evolving (Tucci et al., 2018, Nocerino et al., 2017, Lachat et al., 2017. * Corresponding author This paper investigates the effectiveness of two recent mobile mapping systems namely the GeoSLAM ZEB-REVO RT and the more recent GreenValley LiBackPack C50 for indoor surveys.
After presenting some works about the evaluation of MMS data in a related work section, the two devices under study will be presented as well as the three study areas used for the evaluation of these systems. Then, the comparison methodology will be explained, and several comparison results will be presented. Next, the measurement noise will be evaluated. Finally, conclusion and future works will be proposed.

RELATED WORK
In the last few years, several studies provide information on the performance that can be expected from the GeoSLAM ZEB-REVO depending on the type of survey carried out. A synthesis of the approaches used for the evaluation of this device is proposed in this section. Since the GreenValley LiBackPack C50 is new in the market, no study about its performance is available yet. However, the two devices use the same technology and the approaches used to evaluate the ZEB-REVO can also be applied for the LiBackPack C50.
The most common approach applied to evaluate mobile acquisition systems is a cloud-to-cloud comparison with a reference point cloud. After transforming the two clouds into the same coordinate system or aligning them, a comparison is performed (Maboudi et al., 2017). Metrics for point cloud to point cloud comparison for indoor point clouds are mentioned and discussed in Lehtola et al. (2017). The reference point cloud is usually a point cloud acquired by a TLS which provides a higher accuracy. A reference model can be used instead of the reference point cloud provided that it was generated with more accurate data. Maboudi et al. (2017) and Tucci et al. (2018) propose to evaluate the quality of the acquisition of the GeoSLAM ZEB-REVO by using targets for the comparison (checkerboards or spheres). The results of these studies show that all the targets can't be detected due to the low density of points in the MMS point cloud. Tucci et al. (2018) also highlights that because of the low resolution, small architectural details cannot be reconstructed.
An evaluation of MMS data without a reference point cloud is proposed by Karam et al. (2019). They use architectural constraints to analyze the point cloud namely the parallelism between wall faces and the perpendicularity between walls. They also exploit an existing 2D plan to make comparisons. These methods can be use only if the geometry follows some rules and if up-to-date deliverables are available for the studied area.
In order to evaluate the measurement noise, geometric primitives can be fitted to a point cloud and deviations analyzed (Nocerino et al., 2017). This method is applied to the MMS point cloud and the reference point cloud to compare their measurement noises.

Mobile mapping systems under study
The performance of two real time MMS (Mobile Mapping Systems) based on SLAM technology are evaluated in this paper: the GeoSLAM ZEB-REVO RT and the more recent GreenValley LiBackPack C50 launched on the market in 2019. Whereas the first one is handheld, the second one is a backpack system. Table  1 lists some features of both systems notably the accuracies announced by manufacturers which are checked in this study.

LiBackPack C50 GreenValley
Weight (  The field of view is limited to 270° in the horizontal and vertical plane, in front of the displacement of the user. The system includes a 2D infrared laser scanning profilometer coupled to an IMU sensor, without GNSS receiver. The 2D laser profilometer is a compact scanning laser rangefinder that consumes less power, is more compact and lighter than a conventional static 3D laser scanning system. No camera is integrated that means that the sensor does not provide colorized point clouds.

The LiBackPack C50 system, developed by GreenValley
International, is equipped with a laser profilometer, which is mounted in a rotating housing with a 360° horizontal and 30° vertical field of view. The device integrates also a panoramic camera providing co-registered images and therefore RGB colorized point clouds. Its outdoor positioning is provided by GNSS, while indoor positioning and positioning in environments where GNSS is denied is based on IMU and a SLAM algorithm.

Various datasets
The study presents a variety of datasets that have been acquired, processed and evaluated to define the survey accuracies that may be achieved using both laser scanners. Moreover, the diversity of the data in terms of geometry, access complexity and cluttering will make it possible to define the performances of MMS for a given type of survey. Figure 1 illustrates the three datasets chosen for our study. Given its higher accuracy compared to dynamic systems, static scanner FARO Focus3D X330 has been used for producing reference datasets for every study area. The first dataset has been acquired in a 20 meters high tower of the "Ponts Couverts" site, in the historical district of Strasbourg, France. It is a very narrow and vertical survey, made in a dark environment. The survey was carried out with the ZEB-REVO RT in a closed loop from the ground floor of the tower to the top floor and back to the ground floor.
The second dataset has been acquired in the Strasbourg's Zoological Museum. The square shape building is composed of five floors of about 1900 square meters each and has an internal courtyard. It presents various rooms sizes and a lot of objects due to its multiple functions (offices, classrooms, labs) and the huge collection of the museum. Several closed loops have been carried out with the ZEB-REVO RT scanner mainly in the basement floor and also in several floors at the same time.
The third study area is the topographic measurements laboratory of the graduate engineering school of INSA (National Institute of Applied Sciences) in Strasbourg. This large room characterized by large glass fronts covers about 850 square meters. It is located in the ground floor and enables a direct access to the outdoor. Its survey was carried out with the LiBackPack C50 in a single closed loop and with the ZEB-REVO RT in a single closed loop as well. It is interesting in this survey to study the measurement noises produced by the various devices in this environment composed of reflective or transparent surfaces: walls composed of bay windows and tiled floors.

Pre-processing and alignment of point clouds
In order to perform comparisons between point clouds acquired by the different acquisition systems, a spatial resampling of the point clouds of 1 cm is first performed. Indeed, a high density of points involves a long calculation of the deviations between two point clouds. The point clouds are then segmented to remove objects that are not the same in the scene (moving persons of objects move between two acquisitions) as well as measurement noise that corresponds to points far from the scene. These removed points could disturb the alignment process as well as the comparison results by introducing high distances between the two point clouds.
Finally, the alignment of point clouds is carried out so that the two point clouds to compare are in the same coordinate system to allow calculating distances. The CloudCompare software (EDF R&D) is used for alignments and comparisons. This is a wellknown software designed for the comparisons between point clouds. A two steps alignment is proposed: A rough alignment is first achieved by clicking manually common points between the two clouds. As soon as at least 3 points are selected in both point clouds, the resulting RMS is displayed. This step is illustrated Figure 2 for the tower dataset. The alignment is secondly refined thanks to the Iterative Closest Point (ICP) algorithm.

Deviation analysis
In order to evaluate the accuracy of the mobile systems, distances between the MMS clouds and the cloud obtained by the static scanner are analysed for several surveys. The point cloud acquired by the static scanner is the reference point cloud since this system has the highest measurement accuracy. In addition, distances between the point clouds obtained with the two MMS are also analysed for one of the datasets.
For the tower, the point cloud captured on the whole edifice is considered in the deviation analysis, therefore the impact of the edifice's geometry will also be studied. For the other test sites, only samples of common parts are mentioned in this paper.
To analyse deviations between point clouds, the M3C2 plug-in available in CloudCompare was used. The M3C2 plug-in computes signed and robust distances between two aligned point clouds (Lague et al., 2013). It is less sensitive to noise than the cloud-to-cloud function included in CloudCompare and has the advantage of calculating positive and negative distances.
At the end of distances calculation, a point cloud with the calculated deviations as a scalar field is created. Thanks to a colour map, deviations can be easily visualized in the software. One can also straightforward select and display points for which the deviations belonging to a specific range. In our study, the range -5 cm to +5 cm was considered for the analyses of the results. Additionally, a statistical analysis based on deviations is proposed by CloudCompare. It provides notably the accuracy associated to the point cloud compared to the reference.

Tower of "Ponts Couverts"
The acquisition of the tower was carried out with the FARO Focus3D X330 static scanner and the GeoSLAM ZEB-REVO RT. About 3 million signed distances divided into 256 classes were calculated with the M3C2 plug-in between the two point clouds. The created classes represent intervals of values. The software fills in the number of calculated distances belonging to each interval. These classes can be exported in a file (CSV format) for further analysis.  The mean deviation is 5 mm and the maximum deviation value is 1.79 m. 87% of the distances between the two point clouds are less than 5 cm and 92% are in the range -10 cm to +10 cm. One should note that there are high deviations due probably to remaining outliers which are difficult to isolate properly from the The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-4/W1-2020, 2020 3rd BIM/GIS Integration Workshop and 15th 3D GeoInfo Conference, 7-11 September 2020, London, UK point clouds. After removing those outliers, 91% of the deviations are less than 5 cm and 95% are less than 10 cm. This represents an accuracy of 5 cm without outliers.

Zoological Museum
The static scanner FARO Focus3D X330 and the GeoSLAM ZEB-REVO RT were used. Several loops were carried out with ZEB-REVO RT and 3 areas of interest were selected for further analyses.

Closed loop in basement floor:
The first area is a closed loop located in the basement floor of the museum. Figure 4 shows the reference and MMS point clouds aligned in CloudCompare.

Figure 4: Reference and MMS point clouds aligned
The results of the comparison are similar to the previous dataset. After removing outliers, 91% of the deviations are less than 5 cm and 97% are less than 10 cm. An accuracy of 5 cm is reached.

Closed loop in several floors:
The second area is a closed loop performed in two different floors consisting in a back and forth from one floor to the other. About 3.3 million signed distances were calculated with the M3C2 plug-in. Figure 5 shows the result of the comparison. After removing outliers, 55% of the deviations are less than 5 cm and 80% are less than 10 cm. An accuracy of 9 cm is reached without outliers. The SLAM technology obviously had a hard time consolidating the loop accurately. There is a discrepancy which may be due to the fact that the loop is located on 2 different floors and the narrow and long stairwell between them. The algorithm probably didn't find enough points of interest to align the clouds.

Closed loop indoor and outdoor:
The third area deals with a loop performed both indoor and outdoor. The accuracy of the outdoor part of the MMS point cloud was investigated.
Although the manufacturer announces an outdoor range of 15 to 20 meters, as showed in Figure 6, the ZEB-REVO RT acquired only a limited part of the façade compared to the static scanner. Again, a proper segmentation of the point clouds has to be carried out before comparing the two point clouds.

Figure 6: Reference point cloud (green) and MMS point cloud (red) before alignment and segmentation
For this area, 81% of the deviations are less than 5 cm and 92% are less than 10 cm. The accuracy obtained is 6 cm.

Topography Laboratory of INSA Strasbourg
The third study area was acquired with the FARO Focus3D X330 static scanner, the GeoSLAM ZEB-REVO RT and the Green Valley LiBackPack C50. The complexity of this survey is that this laboratory has a large number of mobile objects such as furniture, chairs and tables which are often moved according to the events that this room hosts. Moreover, it is composed of large windows, which may produce artefacts in the captured cloud. In addition, the acquisitions with the different devices were made at different times, so we will focus on specific homologous areas in order to compare point clouds.

FARO-GeoSLAM comparison:
In order to perform the comparison, two homologous areas were segmented in the reference and the MMS point clouds. Figure 7 presents the result of the comparison; about 4 million signed distances were calculated.

Figure 7: FARO-GeoSLAM comparison result
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-4/W1-2020, 2020 3rd BIM/GIS Integration Workshop and 15th 3D GeoInfo Conference, 7-11 September 2020, London, UK The above result is not satisfactory since there are a high number of points which don't belong to the range -5 cm to +5 cm (grey points in Figure 7). It is especially the case for the ground points. The low acquisition density of GeoSLAM compared to TLS combined to the large area of study doesn't allow a proper comparison of the ground. One should note that high deviations are not due to the floor material. As illustrated Figure 8, the measurement noise is limited on the floor. To perform the comparison, a portion of a pillar of about one cube meter acquired by both systems was considered. About 34 000 signed distances were calculated with the M3C2 plug-in. A mean value of 3 mm and a maximum value of 9.1 cm are observed. An analysis of these discrepancies shows that 97% of the distances between the two clouds are less than 5 cm and 93% less than 1 cm. An accuracy of 1 cm is reached.

FARO-LiBackPack comparison:
Considering the point cloud obtained with the LiBackPack, the density is also lower than the reference point cloud. A comparison in a large area also led to high deviations on the ground even if there are fewer than with the ZEB-REVO RT. A visual inspection of Figure 9 permits to conclude that the high deviations are only due to the point cloud density and not the measurement noise at ground level which is very limited. Figure 9: Vertical slice in the LiBackPack point cloud A comparison was also made thanks to a portion of pillar of about one cube meter. A mean value of 1 mm and a maximum value of 9.2 cm are obtained. An analysis of the discrepancies shows that 100% of the distances between the two clouds are less than 5 cm, 96% less than 2 cm and 81% less than 1 cm and an accuracy of 1 cm is reached. These results are close to those obtained with the ZEB-REVO RT.

GeoSLAM-LiBackPack comparison:
A comparison was also performed between the point clouds obtained with the two MMS. The portion of pillar used for the comparisons with the reference point cloud is considered. For this comparison, the point cloud obtained with the ZEB-REVO RT was used as a reference because of its higher quality. Figure 10 presents the result of the comparison. 100% of the distances between the two clouds are less than 5 cm and 95% less than 1 cm. An accuracy of 1 cm is reached.

Discussion about completeness and density of points of MMS data
One advantage of the MMS compared to the static scanner is that the environment is acquired more easily. One can move around objects of the scene. In order to acquire the whole scene with a static scanner, it is necessary to increase the number of scanning stations. This is generally not done because it's time consuming and there are therefore some missing parts in the point clouds because of occluding objects.
On the other hand, the MMS data has a lower density of points than the static scanner. This made the comparisons more difficult for the laboratory dataset which is quite large. Regarding the number of points per second and the resolution provided by Table  1, the LiBackPack C50 data will provide a higher point density than the ZEB-REVO RT. Besides a lower points density, the MMS also present a higher measurement noise; this noise will be evaluated in the next section.

Measurement noise assessment based on a plane
In order to quantify the measurement noise, the dispersion of points around a plane is analysed. A portion of a wall face smooth of about one per two meters was segmented in the three point clouds. Using the Fitting tool provided by RealWorks software (Trimble), a plane is fitted to each point cloud. Table 2 presents the results for each device namely the number of points used to fit the plane and the standard deviation around the plane.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-4/W1-2020, 2020 3rd BIM/GIS Integration Workshop and 15th 3D GeoInfo Conference, 7-11 September 2020, London, UK As expected, the lowest standard deviation of points around the fitted plane concerns the TLS cloud. The mobile devices deliver noisier data, however the noise observed in measurements performed with the ZEB-REVO RT is lower than the noise measured in the LiBackPack cloud.

Computation of regression lines
In order to validate the analysis of measurement noise based on a plane computed in RealWorks, a statistical analysis of the X and Y coordinates of point clouds was also performed. The aim is to determine an orthogonal regression line by the least square method. The principle of this method is to minimize the orthogonal distances between points on the segmented wall face and the calculated line. Figure 11 presents the computed orthogonal regression lines for the two MMS devices and the static laser scanner. The equations obtained are very close and therefore the lines appear similar. However, it can be seen that the two MMS lines are very close and far from the TLS reference line. It can also be seen that the line closest to the reference line (in red) is the one relative to the ZEB-REVO RT (in green), which supports the previous results.
Finally, the correlation rate of each point cloud was calculated. The more the data are correlated (the more the X coordinate is correlated with the Y coordinate), the less scattered they are and therefore the less measurement noise there is. The device with the least measurement noise is unsurprisingly the TLS with a correlation of 99.32%. The correlations are almost the same for the ZEB-REVO RT and the LiBackPack C50. The percentages are respectively 96.21% and 96.17%.

CONCLUSION AND FUTURE WORKS
In this paper the effectiveness of two recent mobile mapping systems namely the GeoSLAM ZEB-REVO RT and the GreenValley LiBackPack C50 was investigated for indoor surveys.
In order to perform a reliable assessment study, several datasets produced with each sensor were compared to the high-cost georeferenced point cloud obtained with static laser scanning target-based technique. Quantitative analyses of the results implied the analysis of differences between the point cloud produced with these SLAM based sensors and 3D reference point clouds.
An analysis of the distances between the registered point clouds produced automatically with the mobile devices and after processing with the static scanner shows that more than 90% of the deviations between the two clouds are less than 5 cm for the first and second dataset. This represents an accuracy of 5 cm. For the third study area, comparisons between the point clouds acquired with the two mobile systems and the reference point cloud show also very satisfying results since more than 97% of the distances are less than 5 cm. This means that the reachable accuracy of the mobile systems is 1 cm. It is therefore noticeable that the point clouds acquired via MMS are very close to those obtained with TLS. Based on this assessment, it can be confirmed that the accuracies obtained through these experiments correspond to those announced by the manufacturers (Table 1).
Regarding the noise, the lowest standard deviation of points around a reference plane concerns the TLS cloud. The mobile devices deliver noisier data, however the noise observed in measurements performed with the ZEB-REVO RT is lower that the noise observed in the LiBackPack cloud.
The results are very satisfying and promising since the MMS data, which are acquired much faster than TLS data, are close to the reference data. Both mobile systems provide similar results.
As expected, the registration quality of MMS point clouds remains lower than the quality obtained with static systems. Especially the operating mode plays a crucial role with SLAMbased systems. For instance, the SLAM technology still suffers from limits for the registration of acquired data when loops are too large and performed in several floors. However, many studies are aimed at improving the technology (Zhao et al., 2008). A combination of both systems for improving registration might be interesting, i.e. to perform static acquisitions in strategic areas of the building and register the mobile system point clouds between to static stations. Additionally, measurement noise of MMS is higher than TLS and the density of points is lower which made the comparisons more difficult for large spaces.
In the future work, an acquisition method combining both a mobile system and a static scanner will be developed. Indeed, it may be appropriate to help the SLAM technology by providing some fixe scanning stations at specific points. The use of MMS data for BIM reconstruction will also be investigated. Of course, the MMS data have a lower density of points, but it is possible to reconstruct some elements such as walls, ceilings and floors. The question is not about if the MMS data can answer BIM requirements but rather how far is it possible to reconstruct buildings with those data in term of level of details and which quality of the reconstruction can be reached.