ACCURACY ASSESSMENT AND CALIBRATION OF LOW-COST AUTONOMOUS LIDAR SENSORS

A number of low-cost, small form factor, high resolution lidar sensors have recently been commercialized in an effort to fill the growing needs for lidar sensors on autonomous vehicles. These lidar sensors often report performance as range precision and angular accuracy, which are insufficient to characterize the overall quality of the point clouds returned by these sensors. Herein, a detailed geometric accuracy analysis of two representative autonomous sensors, the Ouster OSI-64 and the Livox Mid-40, is presented. The scanners were analyzed through a rigorous least squares adjustment of data from the two sensors using planar surface constraints. The analysis attempts to elucidate the overall point cloud accuracy and presence of systematic errors for the sensors over medium (< 40 m) ranges. The Livox Mid-40 sensor performance appears to be in conformance with the product specifications, with a ranging accuracy of approximately 2 cm. No significant systematic geometric errors were found in the acquired Mid-40 point clouds. The Ouster OSI-64 did not perform to the manufacturer specifications, with a ranging accuracy of 5.6 cm, which is nearly twice that stated by the manufacturer. Several of the individual lasers within the OSI-64’s bank of 64 lasers exhibited higher range noise than their counterparts, and examination of the residuals indicate a possible systematic error correlated with the horizontal encoder angle. This suggests that the Ouster laser may benefit from additional geometric calibration. Finally, both sensors suffered from an inability to accurately resolve edges and smaller features such as posts due to their large laser beam divergences.


INTRODUCTION
There has been an explosion of small form factor, low-cost lidar units commercially available over the past several years. This growth has primarily been a result of the autonomous vehicle market and the need for small and cheap sensors suitable for providing real-time 3D situational awareness. A variety of these low-cost laser scanners have been integrated into unmanned aerial vehicles (UAV), indoor mapping platforms, and autonomous vehicle designs as the primary mapping device for providing obstacle detection and avoidance, e.g., (Wang et al., 2017) and (Asvadi et al., 2016). Beyond situational awareness, these devices are also being routinely employed as primary data acquisition sensors for high resolution surveying and mapping (Lin et al., 2019, Elaksher et al., 2017.
However, to date, for a majority of the sensors a systematic evaluation of their accuracy, repeatability and stability has not been presented. Most examination of accuracy for mapping products using these sensors have relied upon spot checks using GNSS check points, or static tests of ranging accuracy versus an external reference, e.g. (Ortiz Arteaga et al., 2019). While important for understanding overall mapping precision, it does not provide any understanding of the raw accuracy of the sensor observations, and the possibilities for improving this accuracy should systematic errors be present in the resultant point cloud measurements. A detailed analysis of the sensors in a wellcontrolled environment is required to determine base observational noise levels and the possible presence of systematic errors in the resultant 3D point cloud. This analysis is fundamental to understanding the capabilities of these sensors for 3D modelling and mapping as well as autonomous vehicle navigation applications. To our knowledge, currently, this type of detailed analysis has only been performed for Velodyne sensors, for example (Glennie, Lichti, 2010, Glennie et al., 2016.
While an evaluation of all low-cost lidar units currently being employed in 3D surveying and mapping is required, such an exhaustive examination is beyond current resources. Therefore, we have chosen to demonstrate an evaluation methodology using two representative sensors, the Livox Mid-40, and the Ouster OS1-64, with the hope that this framework will provide a basis for analysis and comparison of additional low cost lidar sensors.
Herein, a detailed analysis of the OS1-64 and Livox Mid-40 laser scanners is presented. A preliminary evaluation of the Mid-40, primarily focusing on ranging accuracy is presented in (Ortiz Arteaga et al., 2019). Previous work on similar autonomous scanners (i.e. Velodyne HDL-64E, HDL-32E and VLP16) have shown that the factory calibration of the instruments was not optimized, that the instruments exhibited temporal instability for their calibration values, and also required a significant warm-up period to reach steady-state (Glennie et al., 2013, Glennie, Lichti, 2010. With this prior experience in mind, each of the scanners was examined with the following goals: (1) characterization of precision with respect to range, angle of incidence and reflectivity of target surface, and, (2) presence of systematic errors in resultant point clouds. For the analysis we collected several static datasets from varying locations and orientation from both scanners in a scene with multiple hard target planar surfaces. The entire control scene was also scanned at high resolution with a survey grade terrestrial laser scanning system (Riegl VZ-2000) to provide an independent reference. Attempts to identify residual systematic errors using the least squares adjustment results constrained to planar surfaces sim-ilar to that reported in (Skaloud, Lichti, 2006) are also presented.

Mathmatical Formulation
Scanners built for operating on autonomous vehicle platforms are often difficult to analyze in a static environment because they rely on vehicle motion to build up a high-resolution 3D model of their surroundings. In static mode, their fixed laser positions and field of views make it difficult to calibrate in a traditional sense using signalized targets (see for example (Lichti, 2007)) because their static sampling density to too coarse to precisely determine target locations. Therefore, an approach using geometric primitives as targets is implemented. Herein, we use planar surfaces as solution constraints, similar to that detailed in (Glennie, Lichti, 2010, Glennie, 2012. The model used for constraining a lidar point to a planar surface is given as: where g = g1, g2, g3, g4 are the parameters of the kth planar surface r is the 3D lidar point in a global coordinate frame.
For a static analysis and calibration, the raw laser scanner data are normally collected from a number of different locations and/or orientations in order to collect data from differing view geometry. Therefore, any point, i, collected from any of the scanner setups, j, must be converted to a global coordinate frame via a rigid body transformation given as: where R(ω, φ, κ)j is the rotation matrix from scanner frame j to a global coordinate frame tj is the translation vector between scanner frame j and a global coordinate frame lij is the 3D lidar point i in scanner frame j.
The functional model described by the above equations is solved using a standard Gauss-Helmert adjustment model. A detailed discussion of this adjustment model is given in (Skaloud, Lichti, 2006), and is therefore not repeated here. The solution to the model can either be accomplished by treating the plane parameters as unknown and solving for them simultaneously with the rotation matrix and translation vector in the adjustment, or by treating them as known values from an external reference. For our purposes, the latter case is chosen, with the planar reference surfaces provided by the point cloud from a survey grade terrestrial laser scan collected simultaneously with the tested autonomous scanners.

Livox
The Livox Mid-40 sensor (see Figure 1) has a 38.4 • field of view, and employs a unique non-repetitive rosette scanning pattern that increases data density in a fixed direction over time as demonstrated in Figure 2. Detailed specifications for the Mid-40 sensor are given in Table 1. The sensor weighs less than a kilogram, has a volume of less than 10 cm 3 and is IP67 rated at a price point of $599 USD.

Ouster
The Ouster OS1-64 has a 360 • by 33.2 • (± 16.6 • ) field of view with 64 individual laser beams, and rotates at a rate of either 10 or 20 Hz. The OSI-64 acquires data in a similar configuration to the well known Velodyne laser scanners. The sensor weights less than 0.5 kilograms, and has a 8.5 cm diameter and 7.5 cm height with an IP68 rating. The OSI-64 also includes an integrated 3 axis gyro and accelerometer package, the InvenSense ICM-20948. Detailed geometric specifications of the OSI-64 are given in Table 2 and an image of the scanner is given in Figure 3. The price of the OS1-64 is listed as $12,000 USD. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition)

Datasets
The data capture requirements for a rigorous geometric analysis of the scanners using the planar methodology, described in Section 2.1, is a collection area with a number of planar surfaces at a variety of distances and orientations. An ideal location, located in the University of Houston student center and shown in

Data Processing
After data acquisition, the 64 laser scans (40 for Mid-40 and 24 for OSI-64) first needed to be converted into a format suitable for display, processing and analysis. The Livox Mid-40 data acquisition software has a module that allows the export of raw scan data into an LAS format output file. However, the OSI-64 acquisition software has no such functionality. Therefore, a custom script was written in C++ to convert the raw binary packets, saved in UDP (User Datagram Protocol) format, into an LAS file format using both the WinPCAP (www.winpcap.org) library and PDAL (point data abstraction library -www.pdal.io). The script can be obtained from at (github.com/pjhartzell/ousterextract).
The output LAS files were then approximately oriented to the Riegl VZ-2000 dataset using the Alignment tools provided in the software package CloudCompare. The approximate alignments (rotation and translation) were exported from CloudCompare for each scan, and then PDAL was used to apply the transformations to the raw LAS point clouds to roughly reference all scans to a common reference frame.
The roughly aligned datasets were then amalgamated and used to extract a number of planar surfaces in a variety of orientations. Overall, 127 and 133 planes were selected from the Mid-40 and OSI-64 datasets respectively. The extracted planes were used in a least squares adjustment, using Equations 1 and 2 to determine refined scanner positions and orientations. The residuals from these adjustments were then analysed to determine the precision of each scanner and to investigate the presence of any systematic errors in the acquired datasets.

Livox
The least squares adjustment of the Livox data contained 40 instrument set-ups in various locations with 127 observed planes. The final least squares adjustment considered 621,323 measurements on these planar surfaces. Statistics on the final residuals of the Livox points from the VZ-2000 reference planes are given in Table 3, and plots of these residuals w.r.t. various observables are given in Figure 6.
The 127 observed planes ranged in distance from 3 to 35 m from the sensor (top panel in Figure 6). The overall standard deviation of the adjusted point cloud is 1.8 cm, which is very near the Livox specification of 2 cm at 20 m (see Table 1). If 2σ outliers are removed (24,996 points or ∼4% of observations), then the overall standard deviation is 1.3 cm. Examination of the top panel in Figure 6 seems to imply that range residuals for the Mid-40 are larger at smaller ranges ( i.e. < 20 m). In fact, if the standard deviation of the planar residuals are divided into two groups, those from ranges less than or greater than 20 meters, their standard deviations are 2.1 cm and 0.8 cm respectively -the Livox sensor appears to provide more accurate ranges at longer distances. A larger test field is required to determine if this lower noise level is consistent over the dynamic range of the instrument. Unfortunately, given the paucity of information regarding the hardware configuration of the Livox sensor, we are unable to provide a possible explanation for this sharp change in range precision.
The second panel (from top) in Figure 6 plots residuals with respect to angle of incidence on the planar surface. Here, the scatter plot of the residuals has a fairly uniform distribution up until ∼65 • , where there is a significant increase in the dispersion of the residuals. This behavior is consistent with other examinations of both autonomous laser scanners (Glennie, Lichti, 2010) and high-accuracy tripod mounted scanners (Lichti, 2007), and is due primarily to laser beam divergence.
The middle panel in Figure 6 plots planar residuals versus intensity, where the intensity value is the raw reported value from the Mid-40, which is given as a unitless 8 bit value. Higher residuals are found below an intensity of ∼40. Again, this decrease in accuracy due to a lower SNR is common for laser scanners, see for example (Wujanz et al., 2017), and therefore not unexpected. There does not appear to be any systematic error correlated with intensity.
The final two panels in Figure 6 show planar residuals with respect to vertical angle and horizontal angle. The angles were calculated based on the raw cartesian coordinates in the scanners own coordinate system reported in the raw data files. The intent of these plots was to examine if there was any location dependent distortion within the instrument field of view. However, an examination of residuals versus both horizontal and vertical angle does not show any obvious systematic trends. This observation was further examined by computing the average RMSE of residuals for a 0.5 • square grid of horizontal and vertical angles (see Figure 7). Overall, no obvious systematic trends can be seen in the grid. It should be noted that (Ortiz Arteaga et al., 2019) observed a noise propagation visible in the point cloud, which they termed a "ripple effect", that appeared to be noise artefacts propagating outward when observing flat planes. We tested our Mid-40 instrument in a similar manner but were unable to duplicate their result.
Overall, the geometric performance of the Livox Mid-40 was satisfactory w.r.t. to the manufacturer specifications. Expected ranging precision appear to be met overall, and there does not appear to be any significant systematic distortions in the resultant point cloud that are correlated with the examined observables. Of course, a better understanding of the hardware would be required to say with confidence that no systematic errors remain in the scanner. Given our lack of knowledge of its operating principles there may be remaining systematic errors that we were simply unable to uncover given the lack of raw observations from the scanner.

Ouster
The Ouster OSI-64 data least squares adjustment contained 24 instrument set-ups and 133 observed planes. The final least squares adjustment considered 413,765 measurements on these planar surfaces. Statistics on the final residuals of the Livox points constrained to the VZ-2000 reference planes are given in Table 3, and plots of these residuals w.r.t. various observables are given in Figure 8. The top panel of Figure 8 shows that the observed ranges for the OSI-64 data varied between 3 and 37 m. Within this range, the overall standard deviation of the planar residuals (given in Table 3) is 5.6 cm. If we remove the 2σ outliers from the Ouster results (approximately 21,833 points, or 5.3% of the observations), then the resultant standard deviation is 5.0 cm. The specifications for the OSI-64 gives varied range precision over the dynamic range of the instrument (see Table 2), but the relevant specifications when comparing to our results are a standard deviation of 1.5 cm and 3.0 cm for ranges from 2 to 20 m and 20 to 60 m respectively. Therefore as a direct comparison, the planar residual standard deviation was computed separately for these two range envelopes, and 6.7 cm (2 to 20 m) and 2.6 cm (20 to 60 m) were obtained. With this breakdown, for ranges above 20 m the OSI-64 appears to meet specifications, but for shorter ranges the computed standard deviation is >4 times the specification.
The second panel (from top) in Figure 8 plots residuals versus angle of incidence. For the OSI-64, the increase in residual standard deviations at larger (>70 • ) angles of incidence is not as apparent as for the Livox sensor. This is likely because the effect is masked by the overall larger residuals for the Ouster scanner. As a result, the increase is only readily apparent above ∼80 • .
The middle panel of Figure 8 plots raw intensity (as reported by Ouster scanner) versus planar residuals. Note that the OSI-64 reports intensity using a 16 bit scale. As would be expected, the lower accuracy observations are observed when the reported intensity is low. However, the drop in accuracy w.r.t. intensity is significantly more pronounced for the Ouster scanner, when compared to the Livox residuals in the center plane of Figure 6.
The bottom two panels in Figure 8 show planar residuals plotted versus horizontal angle (bottom) and vertical angle (second from bottom). The vertical angle figure shows a striped pattern -this is because of the configuration of the OSI-64 sensor. The sensor has 64 individual lasers pointed at fixed angles between ±16.6 • , and therefore each vertical band corresponds to an individual laser in the sensor. The figure clearly shows that the lasers pointed between 0 and -10 • in the scanners own coordinate system have significantly higher noise levels than the other lasers. This could be an indication of pointing errors for those individual lasers. Finally, the plot versus horizontal encoder angle shows potential sinusoidal systematic error correlated with angle -although likely hard to detect on the small panel plot in Figure 8. These systematic effects suggest that there are also potential calibration pointing errors in azimuth for the individual OSI-64 laser/detector pairs. The systematic error could also be due to a misalignment between the horizontal encoder and the spin axis of the sensor (see (Glennie et al., 2013) for a description of this error). Overall, the presence of systematic er-rors correlated to encoder angle and individual laser suggest the presence of an improper calibration or other error sources. The OSI-64 provides raw measurements of range, encoder angle, and a calibration file that enables a detailed analysis of unit calibration, similar to that done for the Velodyne sensor in (Glennie, Lichti, 2010). However, this detailed analysis is beyond the scope of this research and is left as a possible future research direction.

General Remarks
When comparing the two sensors, it should first be noted that the scales of the graphs in Figure 6 and 8 are different. The yaxis limits for the Ouster datasets are double that of the Livox figure to account for the significantly higher noise level of the OSI-64. It is quite clear that overall the Livox sensor significantly outperforms the Ouster sensor. While the Livox sensor has a more limited field of view, it's price point which is currently 20x cheaper than the OSI-64.
While the Mid-40 clearly outperforms the OSI-64, there is one large error source common to both scanners which is a direct consequence of their rather large beam divergence (when compared to survey grade terrestrial laser scanning systems). This is the inability of the sensors to accurately depict surface edges, as the large beam divergence causes an extended range envelope at edges. Examples of this effect are shown in Figure 9 below. The green data is from the VZ-2000, while the red points are from the Livox scanner. The figure clearly shows how both the light pole and the edge of the staircase are stretched in the final point cloud. This problem may not be a significant concern for autonomous vehicles, where it is more important to detect the presence of an object, but it would be of significant concern if the sensors were used for a primarily mapping or modeling task.

CONCLUSIONS
A rigorous least squares adjustment constrained to planar surfaces and a high accuracy terrestrial laser scan were used to investigate the geometric accuracy and systematic error sources of the Ouster OSI-64 and Livox Mid-40 lidar sensors. The geometric accuracy of the Livox Mid-40 laser scanner matched the manufacturer specifications for ranging accuracy. The system behaved as expected, showing increased planar errors for decreased lidar intensity returns and increased angle of incidence on the target. No significant systematic errors were found in the resultant point cloud. However, the system does not provide access to raw measurements (i.e., mirror angles and ranges), and information on the internal operation of the system, including scanning and ranging methods, was not available. Therefore, systematic errors may still be present, but were not correlated with the point cloud derivatives against which they were compared (e.g., range, polar angle, intensity).
On the other hand, the Ouster OSI-64 significantly under-performed when compared to its stated manufacturer specifications, with a ranging error that was almost double the stated accuracy. An analysis of the residuals identified possible systematic errors correlated with horizontal encoder angle, and several individual lasers which appeared to have poorer accuracy than the system aggregate. These errors, similar to those discovered for the Velodyne HDL-64E sensor in (Glennie, Lichti, 2010), point to the need for a rigorous geometric calibration of the OSI sensor to improve overall point cloud accuracy and consistency. Fortunately, the math model for the OSI-64 is provided by the manufacturer, along with access to the raw measurements that would enable such a calibration. A detailed geometric calibration of the Ouster sensor is an area of future research.
Finally, owing to the large beam divergences from each of the scanners, they were unable to properly model sharp edges and small features such as poles. While this may not be a problem for obstacle detection and avoidance use cases, the application of the sensors to mapping and modelling scenarios may require special filtering of the final point clouds to remove beam divergence artifacts.