FIRST ASSESSMENTS INTO THE USE OF COMMERCIAL-GRADE SOLID STATE LIDAR FOR LOW COST HERITAGE DOCUMENTATION

Heritage documentation has benefitted greatly from significant developments in sensor technology during the past two decades. Miniaturisation of sensors is also an important aspect in the development of low cost sensors, always interesting in heritage projects where budgetary constraints are often present. Among these sensors, the solid-state lidar has begun to attract attention, partly due to its integration in Apple Inc.’s latest version of the iPhone and iPad series. We hypothesise that this type of sensor will see a lot of use in the near future; however, the question remains whether they are sufficient for heritage documentation purposes. In this paper, results from the 2020 iPad Pro SSL point cloud will be assessed and compared to more traditional techniques for 3D scanning (photogrammetry and terrestrial laser scanning). While understandably the geometric quality of benchmark-level techniques such as these remain undeniably better, at least for the moment, the paper concludes that SSL sensors may nevertheless be sufficient for some lower-precision applications. * Corresponding author


INTRODUCTION
The need for 3D heritage documentation has increased significantly in recent decades due to technological advancements. Technologies such as lidar or laser scanning and dense matching in photogrammetry enables a democratisation of 3D sensing techniques particularly in the field of heritage documentation (Barsanti et al., 2014;Munumer and Lerma, 2015). More recently, miniaturisation and developments in both processing algorithm and computing power has also led to more low-cost 3D sensor alternatives. The low-cost aspect is an interesting topic of discussion since many heritage conservation communities try to strike a balance between budgetary constraints and geometric precision (Murtiyoso et al., 2019).
In 2020, Apple Inc. released two products fitted with 3D sensors marketed as lidar sensors: a new iteration of the iPad Pro and the iPhone 12 pro. While generating much hype, the use of 3D sensors in smartphones is not new as they have previously been used for minor applications such as facial recognition. The technology used in such non-professional and thus low cost 3D sensors most commonly involve a type of Time-of-Flight (ToF) depth camera or even structured light sensors (Lachat et al., 2015).
The aim of this paper is to perform experiments focused mainly on the use of Apple's solid state lidar (SSL) for heritage documentation purposes. Aspects related mainly to geometric precision, details/resolution of the result, acquisition protocols, cost, and ease of use will be discussed. The main final output would be an assessment on if and how this commercial-grade 3D sensor can be used for heritage documentation.

RELATED WORK
Although the type of 3D sensor used in the iPad and the iPhone 12 remains naturally a trade secret, it is most likely a type of solid-state lidar (Wang et al., 2020) which means that it eschewed the motorised optomechanical laser scanning setup usually encountered in geomatic applications (Granshaw, 2020). Further literature research has shown that Apple's lidar may be based on a single-photon avalanche diode (SPAD) coupled with a laser light source (Niclass et al., 2019;Tontini et al., 2020).
While Apple has declared that their 3D sensor is mainly aimed for AR purposes requiring a lower level of geometric precision, this setup presents nevertheless an interesting combination for heritage documentation purposes. This is because such systems provide compact 3D sensors capable of providing almost realtime results with a relative low-cost compared to traditional surveying methods. A similar precedent was a research on the use of Microsoft Hololens mixed reality glasses for indoor mapping (Khoshelham et al., 2019).
Other types of sensors performing similar tasks as the SSL exists. ToF depth cameras are often present in smartphones and mainly used for face recognition. A more professional version of the ToF camera exists, e.g. Topcon IS Imaging Station (Evgenikou and Georgopoulos, 2015). The ToF camera is generally considered more accurate than structured light sensors, although the latter are also considered as more low-cost (Lachat et al., 2016;Menna et al., 2017). Another sensor which has become more well-known to the geomatics community is the Wearable Laser Scanning (WLS) such as GeoSLAM (Cabo et al., 2018;Salgues et al., 2020). However, most of these professional sensors are specialist devices which require user training and generally cost more. The SSL fitted into Apple Inc.'s smartphone and tablet ( Figure  1) is an interesting attempt to democratise a real-time approach to 3D reconstruction. Previous attempts on this democratisation was shown, for example, by the Microsoft Kinect (Lachat et al., 2015) and the Google Tango (Froehlich et al., 2017). On the subject of real-time 3D reconstruction using commercial-grade smartphones, a noteworthy study is the REPLICATE project .

CASE STUDIES AND RESEARCH METHOD
The experiments aim to compare the point cloud generated by the 2020 iPad Pro to those produced with other techniques which are more commonly used in heritage documentation, i.e. photogrammetry and laser scanning. Two applications were tested in the iPad Pro: EveryPoint (https://everypoint.io/ accessed 1 February 2021) and SiteScape (https://www.sitescape.ai/ accessed 1 February 2021), both chosen due to their capability in generating point clouds instead of 3D mesh models.
Several case studies were performed to assess the SSL's capabilities: (1) small to medium sized objects, (2) building exterior (façades), and (3) indoor mapping. For the first type of case study, a statue located inside the St-Pierre-le-Jeune church in Strasbourg was used, while a tympanum of the same church's main portal was used for the exterior dataset. The case study on the indoor volumetric mapping uses data from a medieval cell located inside the Heinrichsturm, one of the 13 th century Ponts Couverts of the UNESCO World Heritage site Grande Île of Strasbourg, France. For each case study, a TLS scan was performed using a FARO Focus X330. As a benchmark comparison, the objects were also recorded using photogrammetry by using a Canon EOS 6D DSLR camera.
In each case study, the TLS data is used as reference. For case (1), the TLS point cloud is used. In this case, a Multiscale Model to Model Cloud Comparison (M3C2) analysis (Lague et al., 2013) available on the open source software CloudCompare (CC) (https://www.danielgm.net/cc/ accessed 14 April 2021) was performed. In case (2), the overall completeness of the TLS point cloud made it possible to compute rather the mesh-tocloud deviations. In case (3), both TLS point cloud and mesh were used for several analysis. Finally, overall comparison between SSL results and photogrammetry and TLS was performed in terms of geometric precision, detail in the point cloud (including presence of noise), influence of ambient lighting, overall cost, and ease of use. In the final section, several recommendations on the use of SSL sensor for heritage documentation are presented.

RESULTS AND DISCUSSIONS
In all case studies, a fine registration between the respective technique and TLS data was performed using an Iterative Closest Point (ICP) algorithm. For the photogrammetric data, the 3D model was previously scaled using available scale and/or coordinate data before registered using ICP. In the case of photogrammetry, scaling was not computed for the ICP. Furthermore, taking into account the arbitrarily chosen theoretical resolution of 1 cm, we determined a tolerance value for the point cloud deviation from the reference data of 2.6 cm. This tolerance value corresponds to 2.6 σ and therefore a confidence interval of 99%, a formula habitually used in geomatics applications.   Case (1) consists of an object located in the interior of a building. In this case, the one-meter tall statue is made of marble which is notoriously difficult to model using both TLS and photogrammetry. Taking into account the TLS point cloud as a reference point, the comparison results show that nevertheless the DSLR managed to correctly model the object albeit with some degree of error. This error is however mostly random, as evidenced by the Gaussian distribution of the deviations. The virtual absence of systematic error also shows that the data was correctly scaled. Slight systematic errors were however observed for SSL: 0.1 cm in the case of SiteScape and 0.6 cm for EveryPoint. Since SSL produces scaled data, this error is most likely caused by noisy point cloud which therefore influenced the ICP registration.
The presence of noise in case (1) is further demonstrated by the level of out-of-tolerance outliers in each technique. While DSLR produced almost 5% of its points out of the set tolerance, both SSL applications generated in average 35% of points which are classed as outliers. Visually this is also shown by the form of the point cloud in which the statue's details are not easily discernable. It is also worth noting that besides the problem with the material, case (1) also did not present good acquisition conditions as ambient lighting was not ideal.
The problem with ambient lighting was more or less expected with photogrammetry; however, this proved to also play an important role for SSL results. Indeed, the iPad's SSL seems to consist of a ranging device (the SSL sensor in question) and a position-tracking setup which relies fully on RGB video feed for its real-time positioning, most probably using a SLAM-like approach. The majority of problems with the SSL in case (1) was due to failure of real-time point cloud registration due to this reason. Using the versions tested in this study (February 2021), EveryPoint seems to suffer more from this problem than SiteScape.
The results for case (2) are shown in Figure 3. Case (2) was the tympanum of a church, made from red sandstone. The material of the object means that theoretically no textural problem should be present for any of the tested methods. For this case study, the TLS point cloud was deemed complete enough to warrant the creation of a 3D mesh reference surface. Meshing was performed using the Poisson algorithm with an octree level of 12 in order to keep as much as possible the details represented by the point cloud. DSLR data unsurprisingly fared well with virtually no systematic error and minor noises (0.3% of points are considered as outliers).
The virtually complete nature of the mesh reference also means that a completeness analysis was possible. In this regard, DSLR data managed to reach 94.25% completeness using data from 26 images. Both SiteScape and EveryPoint apps showed much better results than in case (1), possibly due to the absence of the problem with lighting. Indeed, the outdoor scene was taken during cloudy weather which is a near-ideal setup for natural lighting. Even so, the presence of noise in both SSL point cloud is still evident, with almost seven times more points classed as outliers than the DSLR data.
Completeness level for both SSL results are very satisfactory. This displays one of the advantages of SSL-based methods: ease of device handling and therefore minimal risk of holes in the 3D point cloud, terrestrial point of view notwithstanding. On the contrary, photogrammetric point cloud is often influenced by filtering algorithms which may result in holes especially on texture less objects (Murtiyoso et al., 2016). This problem is minimal for active ranging sensors such as TLS and SSL.
In Figure 4, results for case (3) are shown. Case (3) covers the interior of a medieval prison cell. For technical reasons, only results for SiteScape was used in this case study. Visually, the SSL point cloud succeeded in giving the correct form of the cell. A first numerical analysis was performed by comparing the volumetry of the tested methods. The TLS data was once again used to generate a reference volume for the cell, giving a value of 18.13 m 3 . Both DSLR photogrammetry and SSL gave an overestimation of the volume. However, DSLR gave a lower overestimation of 0.14 m 3 which represents 0.8% of the reference volume.
SSL on the other hand overestimated by 0.72 m 3 or 4% of the reference volume. A similar mesh-to-cloud analysis as the one used in case (2) was also performed on this case study, yielding very satisfactory results for DSLR (0.12% outlier) and more significant error for SSL (3.66% outlier). While ambient lighting may also play a role in this error, the results for SSL is much better than expected contrary to results from case (1).
A particular feature of case (3) is the presence of some old engravings, created by past prisoners. These markings and engravings are of great importance for archaeologists to understand the socio-cultural fabric of the 13 th to 16 th century. For this reason, a specific sub-case study was conducted for case (3) to showcase the level of detail users can achieve using each technique. We attempted to capture a specific engraving inside the cell using TLS, DSLR photogrammetry, and SSL.
The results for this test are shown in Figure 5. As can be seen from the figure, DSLR data did not manage to detect enough details geometrically as opposed to TLS, although further point cloud processing may rectify this problem. SSL point cloud was not at all suitable for such applications; in addition to the lack of level of detail, the presence of significant noise (standard deviation of to 5 mm and up to 3% outlier points) also plagues this data.

CONCLUSIONS AND RECOMMENDATIONS
Assessment based on available data showed that from the point of view of geometric quality, SSL managed to attain acceptable results. However, the problem mainly lies with the noise of the resulting point cloud from both SSL apps. Indeed, the dim lighting situation of the object posed great problem for the SSL's real-time registration algorithm (of which most likely to be a form of SLAM). The problem is most evident in the results from EveryPoint. It is therefore advisable to follow a proper acquisition protocol when performing a scan in such dim conditions. As far as the other aspects are concerned, SSL acquisition is very quick and easy even for untrained operators. Photogrammetry on the other hand, requires experience and additional post-processing. Additionally, we noticed that SSL point cloud spatial resolution is difficult to manage in the iPad apps. While the user has a certain control on the "density" of the point cloud in both EveryPoint and SiteScape, this parameter is vaguely determined and is not presented in a metric sense. The density seems to also play a role in the positiontracking algorithm, with lower density point clouds performing better real-time position tracking. Figure 6 displays a simplified synthesis of the comparison between the three tested techniques, namely TLS, DSLR photogrammetry, and SSL. While SSL quality still leaves a lot of room for improvement, it fared very well in terms of acquisition and processing. Indeed, the fact that the tested SSL is integrated with a tablet (and a smartphone in another version) means that, if anything, the presence of this sensor is a major step towards the democratisation of 3D data.
Based on these observations, we formulate several recommendations for the use of this type of SSL sensor for heritage documentation purposes, which are naturally valid at the moment of writing of this paper: 1. Applicability: the SSL sensor has shown to display promising results in term of geometric quality, but the noise may reduce its applicability. As with most projects, knowing the final application is crucial in choosing which sensor to use. At this state, the tested SSL sensor is deemed sufficient for applications such as: 3D visualisation, AR, VR, and to some extent 3D printing. It is not yet suitable for high precision application, such as: detailed 3D printing, digital twins, HBIM, orthophoto, texture analysis, mesh analysis. 2. Acquisition mode: ambient lighting proved to be an important aspect in SSL data acquisition. This is due to the dual nature of the sensors, consisting of the SSL active sensor itself and an RGB-based tracking system. In difficult scenarios, the best result may be obtained by not moving at all, i.e. single position scanning. Multiple pass of a same part of the object does not necessarily mean higher quality of registration and may even lead to the double layering phenomenon of the point cloud. 3. Object material: SSL sensor encountered problem when faced with reflective or light absorbent material such as marble, as is the case with traditional TLS. When encountering these types of material, caution and proper acquisition protocol is necessary. 4. Point cloud quality: as has been mentioned previously, the density of the SSL point cloud can be modified in both SiteScape and EveryPoint. We noticed that lower density point clouds seem to perform better position tracking even in bad lighting situations.
As the SSL technology is still young relative to other wellestablished technologies such as TLS and photogrammetry, these first assessments show the potential and limitations of this sensor at this stage of development. In the context of heritage documentation, further developments of this technology may prove to be very important in democratising 3D heritage data. Indeed, even at this stage we predict a significant increase in the applications and publications on the SSL sensor for heritage documentation in the near future. Much remains to be studied on the use of this novel technology for heritage documentation.
Further studies remain to be conducted to assess this method in a larger selection of sample objects. Comparison with other alternatives for low-cost solutions such as smartphone photogrammetry and videogrammetry is also interesting to test.