SYSTEMATIC EVALUATION AND CHARACTERIZATION OF 3D SOLID STATE LIDAR SENSORS FOR AUTONOMOUS GROUND VEHICLES

3D LiDAR sensors play an important part in several autonomous navigation and perception systems with the technology evolving rapidly over time. This work presents the preliminary evaluation results of a 3D solid state LiDAR sensor. Different aspects of this new type of sensor are studied and their data are characterized for their effective utilization for object detection for the application of Autonomous Ground Vehicles (AGV). The paper provides a set of evaluations to analyze the characterizations and performances of such LiDAR sensors. After characterization of the sensor, the performance is also evaluated in real environment with the sensors mounted on top of a vehicle and used to detect and classify different objects using a state-of-the-art Super-Voxel based method. The 3D point cloud obtained from the sensor is classified into three main object classes “Building”, “Ground” and “Obstacles”. The results evaluated on real data, clearly demonstrate the applicability and suitability of the sensor for such type of applications.


INTRODUCTION
Lately, 3D LiDAR (Light Detection And Ranging) sensors have proved to be an integral part different autonomous navigation and perception applications, ever since the first implementations of SLAM (Simultaneous Localization And Mapping) for robotics (Thrun, 2002) and practical demonstrations on vehicles (Thrun et al., 2006), LiDARs have had an important role in realizing high accuracy occupancy maps and 3D point clouds. However, the introduction of new solid state LiDAR technology is gaining great interest in the scientific community. Even though these sensors have not yet fully commercialized, they still promise higher operational life, low power, small sizes and lower manufacturing costs.
In this paper, we present the preliminary evaluation results of working with one such sensor (as to date the sensor has not yet been launched). Different aspects of this new type of sensors are studied and their data are characterized for their effective utilization for object detection for the application of autonomous ground vehicles (AGV). The paper provides a set of evaluations to analyze the characterizations and performances of such LiDAR sensors. After characterization of the sensor, the performance is also evaluated in real environment with the sensors mounted on top of a vehicle and used to detect and classify different objects using a state-of-the-art Super-Voxel based method.
The paper aims to provide a first-hand analysis of this new LiDAR technology which is fundamental in understanding the capabilities of these sensors for different autonomous vehicle applications. In the state-of-the-art, we find that many works have evaluated different 2D and 3D LiDAR sensors like Ye and Borenstein (Ye, Borenstein, 2002) studied the characterization of the Sick LMS 200 laser scanner while a newer version of smaller Sick lasers such as LMS-100 family were evaluated in (Rudan et al., 2010). Similarly, the Hokuyo series was also well studied with the characterization of the URG-In (Stone et al., 2004), the authors review in detail the different types of LiDAR technologies, highlighting the different problems associated with several "off-the-shelf" LiDAR systems. The general trends in LiDAR sensor technology as well as their likely impact on manufacturing and autonomous vehicle application are also discussed. Other LiDAR technologies based on rotating beam mechanism, like the Velodyne family, is also extensively studied. (Atanacio-Jiménez et al., 2011) calibrated the HDL-64E using large cuboid targets while (Muhammad, Lacroix, 2010) by extracted wall surfaces. An automatic RANSAC-based plane detection algorithm was used in (Chen, Chien, 2012) for the same purpose. A more recent version, VLP-16, was evaluated by Wang et al. (Wang et al., 2018) and compared to cheaper RS-LiDAR manufactured by Robosense. After studying different aspects such as drift, orientation, surface color, material, etc., performance of the two sensors was reported to be similar.
Time-of-flight cameras, such as SwissRanger from Mesa Imaging, were characterized in (Kahlmann et al., 2006) and (May et al., 2009) and corresponding calibration models were also developed. Similarly, the authors of (Khoshelham, Elberink, 2012) characterized the Kinect sensor, used in several robotics applications. Not much work on 3D solid state LiDARs can be found in the state-of-the-art as it is a rather new and evolving technology. Some very recent works (Schleuning, Droz, 2020) (Ruskowski et al., 2020) just provide an overview of the underlying technology especially in the context of autonomous driving applications. However, to the best of our knowledge no prior work on the systematic evaluation and characterization of such solid state LiDARs has been presented. In Section 2, we give a brief introduction of the sensor and its underlying technology and, in Section 3, we present the characterization results of the sensor. In Section 4, we present the experimental results in a real application while we conclude in Section 5.

3D SOLID STATE LIDAR
LiDARs measure by emitting a laser beam that is reflected back from different objects and focused into a receptor to determine the corresponding distances. Traditional LiDARs are electromechanical devices relying on moving parts, to scan the surrounding environment, that need to be precise and accurate in order to obtain suitable measurements over a large field of view. The moving parts involved in the system not only result in size restrictions and increased manufacturing costs but it also implies that the sensor would be more susceptible to perturbations and vibrations resulting in lower operational life. However, despite the high cost, restrictions, and difficulty to manufacture, LiDAR is still widely used for different perception tasks.
Solid state LiDAR sensors are based on silicon chip technology without requiring mechanically moving parts. Apart from lower manufacturing costs, they promise higher operational life, low power and small sizes. The solid state 3D LiDAR sensors mainly employ either Micro-Electro-Mechanical System (MEMS) technology to drive the mirror in order to redirect a single laser in different directions or a Flash array that illuminates the entire field with a single flash.
In this work, the 3D solid state sensor used is based on a flash array type mechanism employing 905 nm wavelength laser and a time-of-flight distance measurement method. As the sensor has not yet been released in the market, it is named F-01 for convenience. The main specifications of the sensor are summarized in Table 1.

CHARACTERIZATION OF 3D SOLID STATE SENSORS
In order to characterize the sensor data, different tests were conducted for the study of drift analysis, effects of surface color and material, incident angle, luminosity, target distance and the phenomena of pixel mixing. These are described in the following parts of this section.

Drift Analysis
The drift analysis sheds light on the stability of the LiDAR. To analyze the drift effect and the stability of the 3D Solid state sensor, measurements of a plane surface was performed over a long period of time. The LiDAR was placed at a fixed distance from a white wall and kept the y-axis of the sensor vertical to the plane of the wall. The drift effect is mainly due but the rise in temperature as the device keeps working. Repeated measurements were taken over a period of 2 hours. Figure 1 shows the variation in distance measurements with respect to the time. From the figures it could be seen that a large variation of about 2 cm is observed in the first 20 min of running.

Effect of Surface Colors
The laser is usually affected by the color of the target surface.
In order to evaluate the LiDAR's performances with respect to different colors, the three primary colors red, green, blue and two secondary colors black and white and also a shiny silver (high reflectivity) color were tested. Each of these 6 colored targets are all the same material of papers and were all fixed on exactly the same place during each test. The results, in the form of distance distributions, are presented in Figure 2.

Effect of Surface Material
Just like varying colors, the material of the target's surface also has an effect on the reflection of the laser. In order to evaluate this effect, tests were conducted on three different material target sheets. The three targets were all the same white color however the materials were different: concrete, metal and tissue cloth. The results are shown in Figure 3.
The variation in distance measurements observed due to different materials was not much. However, it could be seen that for the metallic surface the number of returns (reflected 3D points) were much higher.

Variation of Incident Angle
The effect of different incident angles on the range measurement accuracy was also studied. In all other experiments/tests this angle was kept constant at 0°. The target (a 0.5 m × 0.5 m reflecting screen) was mounted on a precise rotary table and the central axis of the sensor and the rotation axis of the target were The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 3. Variation of distance measurement with respect to different materials carefully aligned ensuring that the distance between the two remains constant. The target was rotated in both clockwise and anti-clockwise directions from −45°to +45°. We were not able to get reliable range measurements for angle superior to ±45°d ue to lack of reflected 3D points. The results are presented in Figure 4. It can be seen that the results are almost symmetric for the +ve and −ve angles and that the distance measurement is more accurate within ±30°incidence angle.

Influence of Luminosity (Ambient Light)
In order to evaluate the sensor's performance at different luminosity levels, distance measurements of a fixed white target were taken. The luminosity levels were modified using a large external lamp. The corresponding luminosity levels were measured using a LUX meter. The results presented in Figure 5 show that the sensor is quite robust to changes in luminosity as there is not much variation in distance measurements. Figure 5. Variation of distance measurement with respect to different luminosity levels

Problem of Pixel Mixing
When a laser spot is located at the very edge of an object, the measured range is that of a combination of the foreground object and the background object, i.e., the range falls in between the distances to the foreground and background objects. This condition is called "mixed pixels" (Cooper et al., 2018).
In order to evaluate this phenomenon, we placed a white target at about 3.1 m with a uniform background at about 5.0 m in front of the scanner as shown in Figure 6. Figure 6. Side view of the 3D points belonging to the target (shades of blue) and uniform background. Circled in white, we find few false points due to the problem of mixed pixels The number of false measurements (points) due to pixel mixing was found to be very few as shows in Figure 6 (encircled in white). These can be easily removed or filtered out in post processing.

Distance Analysis
In order to analyze the effect of target distance, we took the measurements of the same target surface at different distances (1 m to 20 m). The measured distances with respect to the ground truth are presented in Figure 7. The small measurement error with a standard deviation of 2.08 cm demonstrates the high precision of the sensor.

EXPERIMENTATION IN REAL APPLICATIONS
Once the sensor data was characterized, it was also tested in real application environment. The sensor was mounted on a forklift The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B1-2020, 2020XXIV ISPRS Congress (2020 as shown in Figure 8. The tests were conducted in a large factory shed with different types of objects placed/scattered on the factory floor. These included common objects such as cartons of different size, pallets, metallic poles, other forklifts, storage racks and pedestrian etc. The sensor mounted vehicle affected several known trajectory around these objects. The 3D point clouds obtained from the sensor were then processed to detect and classify these objects in each frame. In order to avoid the problem of ego motion the vehicle moved at slow speeds of 2 m/s, 1 m/s and 2.5 m/s. In order to process the 3D point clouds, a state-of-the-art Super-Voxel based method (Aijazi et al., 2016) was employed. In this method the 3D points are first grouped together to form tetrahedral voxels of different sizes based on an R − NN and then properties are assigned to each of these voxels based on the constituting points. Using these properties the voxels are linked together along the three dimensions to form objects. The ground, assumed to be a large plane is extracted and then the remaining objects are classified using geometrical features. Although, the original work classified 5 different classes including cars and trees which are of less relevance in our application, we modified it to classify the point cloud into three main classes, i.e., "Ground", "Building" and "Obstacles". The main geometrical features used include size of bounding box, direction of surface normal and planarity, etc. All objects that could be an obstacle for the moving vehicle are included into the class called "Obstacles". Some results are presented in Figure 9 and Table 2 shows the evaluation results using standard metrics like Accuracy (ACC) and F1 measure (F1) using (1) and (2) where Precision = TP TP+FP and Recall = TP TP+FN . In order to evaluate these metrics, analysis was done using 3D points of the different object classes. The high accuracy results clearly demonstrate the applicability and suitability of the sensor for such type of applications.

CONCLUSION
In this paper we present the preliminary evaluation results of a solid state LiDAR sensor. Different aspects of this new type of sensor are studied and their data are characterized for their effective utilization for object detection for the application of autonomous ground vehicles (AGV). The paper provides a set of evaluations to analyze the characterizations and performances of such LiDAR sensors. After characterization of the sensor, the performance is also evaluated in real environment with the sensors mounted on top of a vehicle and used to detect and classify different objects using a state-of-the-art Super-Voxel based method. The highly accurate results (with ACC about 90%) clearly demonstrate the applicability and suitability of the sensor for such type of applications.  (c) show the classification results. In blue are the 3D points classified as "Ground", in green are the 3D points classified as "Building" and in red are the 3D points classified as "Obstacles"