3 D MOBILE MAPPING WITH A LOW COST UAV SYSTEM

In order to accomplish the automatic mobile mapping task in a small area of interest, a low cost UAV system is proposed in this paper. Multiple sensors including a global shutter camera and an inertial measurement unit are calibrated and synchronized to collect data from the area of interest. First the images are matched by the chronological order and the SfM is utilized. Then the origin SfM result is integrated with the IMU data by adding the IMU constraints into the bundle adjustment. At last the photogrammetry point clouds are generated using PMVS according to the extrinsic parameters. Experiments are undertaken in a typical scene with photogrammetry point clouds generated. The trajectory estimated by the proposed integration method are compared with the method that relies on image only, showing that the proposed method has better performance.


INTRODUCTION
Using point clouds of a cultural domain or urban building has become the focus for different purposes, such as cultural relic protection, emergency & disaster relief and building of smart city (Adams and Friedland, 2011;Reiss et al., 2016;Yang et al., 2015).It is a common technique to capture images using an Unmanned Aerial Vehicles (UAV) system and generate point clouds for the region of interest.The price of a set of mobile mapping system based on a UAV is usually more than 500,000 dollars, which is expensive for the civilian users.In particular, for some private sectors and local government agencies, they have a strong demand for a mobile mapping system based on a UAV that has low cost, miniaturization and medium performance for collecting data from a small area of interest.Although the common commercial mobile mapping systems mounted on a UAV is powerful, but they can hardly meet these demands.As a result, the low-cost and small mobile mapping systems mounted on a UAV with medium performance attract a large number of researchers (Wallace et al., 2012;Yang and Chen, 2015).
Usually the mobile mapping systems are equipped with the integrated navigation system including a GNSS receivers (GPS) and an Inertial Measurement Unit (IMU), so their state (location, orientation, speed, etc.) can be estimated by fusing the GPS and the IMU data directly.However, a differential GPS system also increases the cost of the mobile mapping system and the difficulties in operation.Furthermore, in many scenarios, it would be difficult to ensure that the received GPS signals are reliable and accurate.For example, the GPS signals would be poor when the UAV system is operating between tall buildings.Therefore, it is essential to develop a solution that does not rely on the GPS information for the mobile mapping system, it would increase the application scope and reduce the cost efficiently.
In this paper, a strategy for the low-cost UAV based mobile mapping is developed by integration of imagery and IMU data.Image based matching and loop-closure detection can limit the error accumulation of the system effectively, while the accuracy of pose estimated by Structure from Motion (SfM) depends on the 3D feature points.Only when the baseline length is greater than a certain range, the 3D feature points can be obtained through triangulation accurately.Therefore, images are not suitable for the estimation of the extrinsic parameters in short time duration.Moreover, the absolute scale cannot be derived from the imagery data.In a short time duration, precise poses as well as absolute scale can be derived from IMU data efficiently, while the errors accumulate seriously after long time.So the imagery and IMU data are effectively complementary.
The remainder of this paper is organized as follows.The system description is detailed in Section 2. The proposed data integration method is elaborated in Section 3. In Section 4, the experimental studies are undertaken to evaluate the proposed method, after which conclusions are drawn at the end.

Sensors and UAV platform
Our system is consisting of two kind of measurement instruments including an IMU and a global shutter camera.The modularity of the system shows that new sensors can be easily added to the system.A typical configuration, including the synchronization signals is shown in Fig. 1 (a).The black lines are the ground lines.The red line represents the synchronization signal send from arm board to the global shutter camera to trigger the camera.The orange wire represents the synchronization signal send from IMU to the arm board so the arm board can record the accurate time stamp once the IMU message comes.Table .1 is sensor specifications.
The IMU used in the system is Xsens MTI-300, which provides the measurements of the acceleration and the angular velocity.
The noise density of the gyroscopes is 0.01 º/s/√Hz, and the inrun bias stability of the gyroscopes is 10 º/h.The noise density of the accelerometers is 60ug/√Hz, and the in-run bias stability of the accelerometers is 15ug.This IMU is a low-cost MEMS-IMU products can meet our needs.
The global shutter camera used in the system is Pointgrey Flea3.It has a 1280 × 1024 pixel sensor with the pixel size of 5.3 µm.
It can collect at most 40 images per second and has no rolling shutter effect so it can be used to estimate motion of system.The wider the view of the camera is, the more reliable features can be detected in the image in the SfM algorithm.So the lens used in the system is a wide-angle lens (3.5mm/F1.4) made by Kowa.
The arm-board control unit contains a Cortex-A9 multi-core processor and 2 GB memory, which is the processing instrument of the system.The weight and power consumption of the instruments are severely limited in a UAV system.The armboard is suitable for the UAV platform because it can work in the absence of a radiator and its power consumption is very low.
The UAV platform used in the system is the DJI m600 as shown in Fig. 1

Synchronization and calibration
The synchronization of the different sensors is fulfilled electronically.The two signals used for synchronization are the 200Hz signal from IMU and 40 Hz signal generated by armboard.These signals were selected because the IMU cannot be synchronized accurately with an external signal while the camera can be triggered by an external signal easily.Whenever the IMU collects a data, it sends a pulse signal to the arm board.
Then the arm board generate a timestamp when it receive the pulse signal.It can be assumed that there is no time delay during the pulse signal transmission.The arm-board will send a signal to the camera after receiving 5 signals from the IMU.When the system is collecting data, the arm board save all the data in the SATA disk.When post-processing, the data from different sensors is combined using the timestamp information recorded by the arm-board.
The intrinsic parameters of the global shutter camera are calibrated using the Matlab calibration toolbox (Bouguet, 2008).The calibration tool for Xsens IMU is offered by the MTmanager program.The bore-sight calibration of system is carried out using the method proposed by Furgale et al. (2013).Before collecting data, the intrinsic parameters of the global shutter camera, intrinsic parameters of the IMU and the bore-sight between IMU and camera are calibrated manually.

INTEGRATION OF IMAGERY AND INERTIAL DATA
We follow the notation introduced by (Barfoot, 2016) through this work.The definition of coordinate frames of the system is shown in Fig. 2. F represents the world frame; F represents the inertial frame; F represents the image frame.
The variables to be estimated are the states | 1,2 … at each time the k th image is captured and the feature points | 1,2 … .The state x is written as: where SO is the special orthogonal group.and are the position and the orientation at time of index k, which are represented in world frame.
is the velocity represented in world frame.and are the biases of accelerometers and gyroscopes.
All the variables will be estimated and optimized as one joint optimization through a cost function including backprojection error and IMU error term : (2) where is the visibility set of the k th image.
, and are the information matrices.This non-linear optimization problem is optimized by the open source graph based optimization library G2O (Grisetti et al., 2011).The information matrices and error definitions are detailed as follow.The adjacent images are matched by the chronological order and the key frames are selected according to the parallax of the adjacent images, thereby increasing efficiency of bundle adjustment.Loop closures are detected by utilizing Bag of Words (BoWs) (Philbin et al., 2007) for suppressing cumulative error.The incremental SfM proposed by Schonberger and Frahm (2016) are to estimate the initial value of the scale free coordinates of feature points, meanwhile the false-matches are removed too.
Back-projection error is defined as follow: , where is the measurement of feature point whose distortions are removed by using the calibrated camera distortion parameters. is the calibrated projection matrix of the camera.and are the calibrated relative position and orientation between camera and IMU.information matrix , can be obtained as follow: where and are the error of feature points in image coordinates.In our experience, the error of feature points in image can be set to 2 pixels in each direction.
The IMU kinematics model used is the simplified model for low-cost MEMS-IMUs proposed by Shin and El-Sheimy ( 2004): (5) ( 6) where and are the measurements from accelerometers and gyroscopes with White Gaussian Noise (WGN) and .
is the gravity vector.The biases are modeled as random walk process with WGN and .Therefore the error term of IMU can be find: where is the IMU measurement.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W8, 2017 5th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 28-29 November 2017, Hamburg, The error state kinematics can be obtained as follow: The information matrix of the IMU error term can be propagated according to the error state kinematics directly as follow: (15) The extrinsic parameters and sparse landmarks can be obtained after solving the above optimization problem, then we use the PMVS proposed by Furukawa and Ponce (2010) to generate dense photogrammetry point clouds.

EXPERIMENTS AND ANALYSIS
In order to verify the effectiveness of the platform, we conducted experiments in Wuhan University.We calibrate the intrinsic parameters of the global shutter camera, IMU and bore-sight between the IMU and camera first.Then it took about ten minutes to collect data for a teaching building using the integrated UAV system.As shown in Fig. 3, we are operating our low-cost UAV system.
The state variables and extrinsic parameters are estimated by utilizing the proposed method, then the dense photogrammetry point clouds are generated by utilizing the PMVS algorithm as shown in Fig. 4. In Fig. 4, the yellow line is the estimated trajectory of the UAV system and the color point clouds are the dense photogrammetry point clouds.
To compare the proposed integration with SfM only using imagery data, results of the proposed method and results of the SfM only using imagery data are shown in Fig. 5.The two trajectories are registered manually using the open source software CloudCompare (Girardeau-Montaut, 2015).The yellow line is the trajectory estimated by the proposed method and the red line is estimated by imagery data only.From the white box in Fig. 5, we can find that the estimated trajectory only using imagery data has more noise than the proposed method.Because the extrinsic parameters estimated only using images are not reliable in short time duration caused by short baseline.While the propose integration method has higher robustness.Integration of imagery and inertial data Imagery data only

CONCLUSION
For the purpose of mobile mapping using low-cost UAV system, a low-cost UAV system integrating two kind of sensors are investigated in this paper.To estimate accurate extrinsic parameters, an integration method fusing the imagery data and the IMU is proposed in this paper.First the images are matched by the chronological order and the SfM is utilized.Then the origin SfM result is integrated with the IMU data by adding the IMU constraints into the bundle adjustment.The experimental results show that the designed low-cost UAV system can be used for mapping small area of interest by integrating imagery and inertial data effectively.Moreover, imagery and IMU data are complementary, so compared with SfM only using imagery data, more reliable extrinsic parameters and the 3D landmarks can be estimated by integrating IMU and imagery data.

Figure 1 .
Figure 1.System description.(a)The green line represents the data stream.The black line is the ground wire.The red and orange wires represent the synchronization signals.(b) Our UAV platform.

Figure 2 .
Figure 2. Coordinate frames involved in the system.Image data is acquired in F , and IMU data is acquired in F .The poses and feature points need to be estimated are represented in F

Figure 3 .Figure 4 .
Figure 3.The proposed low-cost mobile mapping system is operating under control.

Figure 5 .
Figure 5.Comparison of the proposed method and SfM only using imagery data.Yellow line is the trajectory estimated by the proposed method and red line is estimated by imagery data only.The two trajectories are registered manually.
board.With no payload, the hovering time of DJI m600 is about 35 min.With 6 kg payload, the hovering time of DJI m600 is about 16 min.So the DJI m600 is suitable for mobile mapping in a small area of interest.