THE IPAD PRO BUILT-IN LIDAR SENSOR: 3D RAPID MAPPING TESTS AND QUALITY ASSESSMENT

The main goal of this ongoing research is the evaluation of the iPad Pro built-in LiDAR sensor for large scale 3D rapid mapping. Different aspects have been considered from the architectural surveying perspective and several analyses were carried out focusing on the acquisition phase and the definition of best practices for data collection, the quantitative analysis on the acquired data and their 3D positional accuracy assessment, and the qualitative analysis of the achievable metric products. Despite this paper is a preliminary analysis and deeper studies in various application environment are necessary, the availability of a LiDAR sensor embedded in a tablet or mobile phone, appears promising for rapid surveying purposes. According to test outcomes, the sensor is able to rapidly acquire reliable 3D point clouds suitable for 1:200 architectural rapid mapping; the iPad Pro could represent an interesting novelty also thanks to its price (compared to standard surveying instruments), portability and limited time required both for data acquisition and processing. * Corresponding author


INTRODUCTION
In 2020 Apple released the new iPad Pro and iPhone 12 Pro with built-in LiDAR (Light Detection and Ranging) sensor representing a considerable innovation in the market segment of tablet and smartphone. The LiDAR embedded in these two devices is mainly devoted to Augmented Reality (AR) applications and some tests on its performances in this field have been already carried out (e.g. https://www.vgis.io/2020/12/02/lidar-in-iphone-and-ipadspatial-tracking-capabilities-test-take-2/). The use of iPad Pro and iPhone 12 Pro for metric survey purposes is obviously of high interest for the research community, for both outdoor and indoor environments, and especially when speed, portability and cost-benefit optimisation are essential requirements, not always easy to meet using high-end surveying instruments. Also, the technical features, cost and manoeuvrability of both devices represent an interesting solution when compared to other more consolidated range-based techniques, already employed in various fields, such us Terrestrial Laser Scanning (TLS) or Time of Flight (TOF) cameras (Chiabrando et al., 2011). The new iPad Pro could be compared to recently studied sensors not specifically developed for 3D metric survey; cameras embedded in smartphones are tested for metric reconstruction as low cost and rapid solutions in case of complex environment as underground cave (Dabove et al., 2019) or for small objects (Vogt et., 2021). Other studies are focused on flexible and light weight mobile mapping systems built-in tablet, as in the case of an object survey with the Lenovo tablet (Donlic et al., 2017), or wearable instrument, as the Microsoft Hololens tested for metric survey of architectures (Khoshelham et al., 2019). Today devices as iPad Pro and iPhone 12 Pro are commonly used and can be easily stored in a pocket or a bag; the integration of a LiDAR sensor can speed up both acquisition and processing phase, so their uses in surveying could be widespread by many professionals in different fields.

SENSORS AND DATA
The tests presented in this manuscript are based on an iPad Pro, a small (28 x 21.5 cm 2 ) and lightweight (about 650 g) device, provided with 8-core GPU (Neural Engine, A12Z Bionic), and dual RGB cameras (Apple). The build-in LiDAR is composed by a matrix of a Vertical Cavity Surface-emitting Laser (VCSEL) and a direct Time of Flight (dToF) Near Infra-Red (NIR) CMOS Image Sensor (CIS), with a Single Photon Avalanche Diode (SPAD) array (https://www.systemplus.fr/reverse-costing-reports/apple-ipadpro-11s-lidar-module/). It means that iPad Pro lidar is "a solidstate LiDAR that creates a fine grid of points, with the distance to each point measured individually" (https://www.vgis.io/2020/12/02/lidar-in-iphone-and-ipadspatial-tracking-capabilities-test-take-2/). It must be highlighted that the rate of measurements enables the acquisition of multitemporal 3D point clouds of moving objects. Unfortunately, at least until the time of writing, no official technical specifications have been released by Apple. A limited number of iOS applications are today available to test the survey capabilities of the LiDAR sensor at the start of the research: nevertheless, the number is constantly growing. Among the existing applications, SiteScape (https://www.sitescape.ai/) -developed at the beginning of 2020 -seems to be the one more oriented in the use of 3D point cloud for documenting the surveyed area, while the approach of the other apps (Polycam, RoomScan LiDAR, 3D Scanner App, LiDAR Scanner 3D) is related to the use of mesh derived from the acquired point clouds. Moreover, SiteScape was selected for additional reasons: possibility to customize acquisition parameters (point cloud density and acquisition mode), realtime visualization of acquired 3D data, automatic storage of the acquired point cloud (in separate *.ply files), automatic coregistration of subsequent scans, free of charge.
In the performed tests, two different types of acquisition configurations were tested: static acquisitions mainly aimed at evaluating the sensor characteristics and dynamic acquisitions aimed at analysing the 3D point clouds. For the static acquisition, the iPad Pro was mounted on a photographic tripod to acquire static scans in a controlled indoor environment with diffuse light ( Figure 1); a white vertical and planar wall was scanned from different distances (1,2,3,4 meters). For each distance 6 scans were acquired combining two acquisition configuration (max area, max detail) and three types of quality (low, medium, high) available in SiteScape settings, for a total of 24 scans. The max area configuration is aimed at acquiring a longer scan with a lower density, the max detail increases eightfold the density of the acquired 3D points limiting the scan to a smaller area. The density of the point cloud is also influenced by the selected quality type: high (medium) quality corresponds to a quadruple (double) density with respect to low quality. In the dynamic configuration the iPad Pro was handled by an operator outdoor with natural illumination (cloudy day). In this second test, mainly devoted to the analyses of the sensor features and capabilities to reconstruct geometries, an existing 3D calibration test field (created ad hoc for digital and 360° cameras calibration as described in Teppati Losè et al., 2018) was employed as test area ( Figure 2). The test field is composed by an outdoor emergency stair, connected to a historical building, representing a diversified geometrical structure composed by various architectural features and materials (mainly plaster and concrete).  The test field has been surveyed by the operator trying to maintain a steady and slow pace while walking, with distances between the iPad Pro and the surveyed surfaces between 1 and 3 meters. The iPad was handled following horizontal or vertical movements with the main axis of the sensor perpendicular to the surfaces, controlling the point cloud in real time on the iPad screen during the acquisition. Two different resolution of the iPad Pro were considered to test the lowest and highest level of detail on the same portion of the test area. The same trajectory was followed performing a not closed loop. The dataset (a) is a single acquisition with the lowest resolution (max arealow) lasting 4 minutes for recording the test field (Figure 3, top). The dataset (b) has highest (max detail -high) resolution and required about 20 minutes to record 7 different scans (Figure 3, bottom). Each scan was captured with an adequate overlapping to the consecutive scan ( Figure 4). The number of scans required for mapping the area is connected to the employed application and to the iPad limit, for each scan is possible to store a maximum number of 11 million points that correspond to 160 MB file weight (.ply format).  In order to obtain a ground truth, the test field was also mapped with 4 scans ( Figure 5) by a TLS, the Faro Focus 3D X330, in about 30 minutes with 1/5 resolution (7.67 mm between adjacent points at 10 m distance) and 4x quality (each point is measured 4 times), representing the dataset (c). The TLS point clouds were selected as reference data for accuracy assessment. Finally, to have Ground Control Points (GCPs) and Check Points (CPs) on the test field nine artificial markers have also been positioned and measured with a total station (mean RMS on the markers < 1 cm).

METODOLOGY
As mentioned in the introduction, two different analyses have been carried out: the first one is aimed at examining the characteristics of the sensor and the application settings, the second one to estimate the point cloud quality, its geometrical reconstruction and its geometrical accuracy in comparison to the topographic Control Points measures and to the TLS dataset. The 24 static scans are considered to evaluate the different settings available in SiteScape as well as the sensors main characteristics exploiting CloudCompare software to compute relevant metrics. The composition of the sensor matrix can be analysed, enabling the counting of the number of rows and columns, and the total number of points for each of the 6 possible settings: max area low, max area medium, max area high, max detail low, max detail medium, max detail high. Then the average distance between two consecutive points of the matrix of the 6 possible settings are measured for each distance from the surface (1,2,3,4 m) to assess the resolution of each scan. The dynamic scans have been analysed to compute roughness and density (number of neighbours) of 1 m 2 point cloud subset of the concrete stair of the outdoor test field. The iPad Pro point clouds with lowest and highest resolution are compared to the TLS scans considered as ground truth. For the 3D positional accuracy assessment, the point clouds recorded in the complex outdoor environment are studied in Leica Cyclone 3DR software (https://leica-geosystems.com/).

Sensor analysis
It is important to highlight, as reported before, that all the consideration related to the iPad Pro sensor, are based on data acquired through the SiteScape application: other applications may lead to different results. First of all the effective maximum distance of acquisition was verified: the sensor reaches an orthogonal distance of 4.9 m from the plane of the sensor, which is in line with the declared maximum range of 5 meters (https://www.apple.com/ newsroom/2020/03/apple-unveils-new-ipad-pro-with-lidarscanner-and-trackpad-support-in-ipados/ ). The iPad Pro sensor measure points organized in a regular matrix following a diagonal pattern ( Figure 6). The number of columns and rows depends on the mode and the quality SiteScape settings (Table 2), while the field of view is constant. According to Table 2, the total number of acquired points increase exponentially depending on the settings: starting from the lowest resolution (max arealow) with 520 points, the other resolutions increase by a factor of 2 (max areamedium), 4 (max areahigh), 8 (max detaillow), 16 (max detailmedium), 32 (max detailhigh), as shown in Figure 7. The vertical distance between rows and the horizontal distance between columns (corresponding to the red and blue lines in Figure 6) of the acquired point matrix were measured for each acquisition distance from the surface and for each possible setting, combining both mode and quality to identify the spatial resolution (Table 3). Vertical and horizontal distances are equal, so this measure can be considered the sensor resolution in ground units. The max detail high quality is the highest spatial resolution setting, reporting a distance between two points of about 1 cm at 1 meter distance from the object, covering with more than 16000 points an area of about 97x127 cm 2 , while the lowest resolutioncorresponding to the max area low settings - Figure 6. Example of the matrix of point of the max area low resolution static acquisition. In red the vertical distance and in blue the horizontal distance between two adjacent points. The roughness of the iPad Pro point cloud was assessed in comparison to the one of the TLS point cloud considering the same portion of the vertical wall of the stair (Figure 8) that according to the results related to the TLS scans could be considered flat. Indeed, the roughness of a single TLS point cloud -dataset (c) -with a kernel radius of a sphere equal to 1 cm shows that 99.5% of the points are far less than 1 mm from the best fitting plane calculated considering the points in the sphere. The iPad Pro point cloud acquired with lowest resolutiondataset (a) -has 97.4% of points with a roughness lower than 1 cm in a sphere of radius 2 cm. The point cloud with highest resolutiondataset (b) -is similar to (a) with the 98.7% of points with roughness values lower than 1 cm ( Figure  9). Therefore, the iPad point cloud roughness does not change according to the application settings of resolution, being related to the accuracy of the sensor regardless the resolution settings.
In conclusion according to the analysis of the roughness, (TLS ~ 1 mm, iPad Pro ~ 1 cm) the TLS point cloud is characterised by a roughness of one magnitude order lower than the tested sensor data: dataset (c) can be therefore considered as ground truth for validating the iPad Pro scans. A further evaluation of the characteristics of the acquired point clouds has been performed considering the density, calculated as number of neighbours in a sphere of 2 cm radius in 1 m 2 area (the same portion employed for the roughness computation). In Figure 10 yellowish/reddish colour represents higher density, greenish/bluish colour lower density, while white is used for missing points. The results of this test show the homogeneous distribution in the TLS point cloud, noticeable by the red colour and low standard deviation (Table 4, c), while iPad Pro point clouds (Table 4, a,b) show not homogeneous distribution with high standard deviation, as expected due to the influence of the operator behaviour (movements and distances from the mapped surface).   Table 4. Mean and standard deviation of the density (number of neighbours) in 1 m 2 for a single TLS scan and the two resolutions of the single iPad scans.

Positional accuracy assessment
As reported before, each dataset has been oriented using 5 GCPs and 4 CPs to evaluate the point cloud 3D positional precision ( Figure 11).   The registration process using ICP algorithm between the iPad scans and the TLS scans as reference reports the same results (RMS ~ 2 cm ± 3 cm) for the point clouds (A), (B1) and (B2), as visible in Table 6. Furthermore, the cloud-to-cloud distance analysis has been performed comparing the TLS point cloud and the iPad Pro clouds in the different tested configuration. The results show slightly better results for (B2) point cloud than the (A) and (B1), confirming that the application of filter and ICP algorithm between iPad scans before the alignment to the TLS dataset improve the overall registration process ( Figure 12 and Table 7).
The main critical parts of the iPad Pro point clouds are located on the side wall of the stair and in the left wall (red colour in Figure 13), probably because they are at the endpoints of the iPad scans. The last performed evaluation was carried after the removal of these portions. As reported in Table 8, if these two portions are not considered in the cloud-to-cloud distance analysis, at least 90% of the points of all datasets are within 5 cm distance from the reference point cloud (97% for B2). Point cloud Cloud to cloud distance 0-2 cm 2-5 cm 5-10 cm (A) iPad -single scan 61% 29% 10% (B1) iPad -raw scans 70% 24% 6% (B2) iPad -edited scans 76% 21% 3% Table 8. Evaluation of the positioning through ICP algorithm to the TLS scans by means of cloud-to-cloud distance without considering portion where gross errors appeared in the previous analysis.

CONCLUSIONS
The main aim of the paper is a preliminary study about the new LiDAR sensor integrated in the iPad Pro, released by Apple in 2020 with the main purpose to apply it in Augmented Reality field. The performed analyses were focused on the metric characteristics evaluation of the point clouds acquired by the iPad Pro using the free SiteScape application. The sensor analysis allows to understand that is possible to acquire data with six different resolution, letting to the operator to establish the point cloud density (from 500 points in a single static acquisition up to 16000 points). As far as the 3D positional assessment is concerned, evaluated in comparison to reference points measured via topographic method and using TLS data, the iPad point clouds have a centimetric precision and accuracy, suitable for 1:200 map scale (precision 2 cm, accuracy 4 cm, according to commonly adopted Italian standards). The roughness analysis confirms the noise over a flat surface is comparable to the aforementioned centimetric 3D positional accuracy. The density analysis highlights the influence of the acquisition modality, leading to heterogenous density values.
In conclusion, according to the first tests the LiDAR sensor integrated in the iPad Pro seems to be very promising in the architectural surveying field as portable cost-efficient 3D scanner, considering its low cost, portability, speed, easy usability especially in articulated complex within a range of 4 meters. Further tests will be performed to explore different iOS applications available on the market.