PERFORMANCE EVALUATION OF 3 D MODELING SOFTWARE FOR UAV PHOTOGRAMMETRY

UAV (Unmanned Aerial Vehicle) photogrammetry, which combines UAV and freely available internet-based 3D modeling software, is widely used as a low-cost and user-friendly photogrammetry technique in the fields such as remote sensing and geosciences. In UAV photogrammetry, only the platform used in conventional aerial photogrammetry is changed. Consequently, 3D modeling software contributes significantly to its expansion. However, the algorithms of the 3D modelling software are black box algorithms. As a result, only a few studies have been able to evaluate their accuracy using 3D coordinate check points. With this motive, Smart3DCapture and Pix4Dmapper were downloaded from the Internet and commercial software PhotoScan was also employed; investigations were performed in this paper using check points and images obtained from UAV. * Corresponding author


INTRODUCTION
Recently, UAV (Unmanned Aerial Vehicle) photogrammetry, which combines UAV and freely available internet-based 3D modeling software, has been receiving attention as a low-cost and user-friendly photogrammetry technique in various fields.The development of high performance and user-friendly 3D modeling software has contributed significantly to the expansion of UAV photogrammetry.
3D modeling software or SfM (Structure from Motion) is composed of a matching, scene reconstruction, and point cloud generation functions.In conventional photogrammetry, expensive equipment and extensive experience are required, and it is necessary to determine the position and orientation of the camera beforehand using GCPs (Ground Control Points), etc.However, 3D modeling software can obtain such camera information from multi-view image without using GCPs.Consequently, the effectiveness of UAV photogrammetry using 3D modelling software has been receiving attention for 3D measurement of very dangerous areas, such as landslide zones (Niethammer et al., 2010) and Greenland ice sheets (Ryan et. al., 2015).
In general, precision and accuracy are used to evaluate 3D measurement from the viewpoint of photogrammetry.Because algorithms for 3D modeling software is a black box and generated 3D model is similarity model, the accuracy of a 3D modeling software should be verified after transforming the similarity model to an absolute model using GCPs, which are known coordinates.Various 3D modeling software have been verified in this way (Rosnell and Honkavaara, 2012;Vallet et al., 2011;Neitzel and Klonowski, 2011), evaluation of 3D modeling software from the viewpoint of photogrammetry has been insufficient.With this motive, Smart3DCapture and Pix4Dmapper were downloaded from the Internet and commercial software PhotoScan was also employed; investigations were performed in this paper using CPs (Check points) and images obtained from UAV.

UAV and camera
A hexarotor UAV and camera in Table 1 were used in this study.A SLR (Single Lens Reflex) camera was utilised because its accuracy is influenced significantly by resolution (Kung et al., 2011).and 100 m (H = 100 m) above ground level, respectively, using SLR and UAV at one-second intervals, without using GPS, IMU, three acceleration sensor nor three gyroscope.
Thus, performance of each software are evaluated using the mean square error in the check points after generating the 3D model by automatic process function of software.In this study, the images that are used in all verification are not preprocessing to input image and post-processing to output results for keeping fair of verification.The 3D model (point cloud data) which is automatically generated by the 3D modeling software is a similarity model.In order to verify the accuracy of the CPs, the coordinate system of the 3D model has to be transformed from relative coordinates to real-world object coordinates by absolute orientation using GCPs.In this study, 3D Helmert transformation was used for coordinate transformation with more than three GCPs.However, PhotoScan cannot acquire the coordinates of CPs on a relative coordinate system.Thus, in scenario 1, the performance of Smart3DCapture and Pix4Dmapper after absolute orientation of a similarity model that was automatically generated from the software were verified.On the other hand, Pix4Dmapper and PhotoScan have to acquire the coordinates of CPs on the absolute coordinate system because the absolute model is generated by inputting the coordinates of GCPs in the software and clicking the GCPs points on each image.Consequently, these software were verified in scenario 2 using the directly obtained absolute coordinates.

PERFORMANCE EVALUATION
The type of 3D model and associated software used in each scenario is shown in Table 3. (i) H = 100m Table 4 shows the horizontal and vertical RMSE for CPs for the case where the altitude was 100 m.In Table 3, PA (Proportional Accuracy) is computed using equation (1) (Fraser, 1987): where In addition, the performance was verified using four GCP arrangement patterns when the number of GCPs that was selected was three and two arrangement patterns when the (1) Figure 1.Test site (white circles signify check points) number of GCPs that was selected was seven.The results for 3 GCPs and 7GCPs listed in Table 4 show the average values obtained.
From Table 4, it is clear that Pix4Dmapper is more accurate than Smart3DCapture in each number of GCPs.However, let take into account that the resolution for one pixel is 22 mm (H = 100 m), it should be understood that the practical accuracy obtained for these software was at the sub-pixels level without using any particular function such as GPS, and the number of GCPs necessary for absolute orientation is understood to be sufficient at three points.
Computation of the PA using the bundle adjustment method, which is a conventional photogrammetry technique using the three images, produced results of 2,453, 7,278, and 7,647 for respective number of 7, 9, and 13 GCPs in centre image.
Therefore, in order to obtain the same 3D modeling software accuracy using the bundle adjustment method, it is clear that GCPs need more than nine points.The bundle adjustment method improves the accuracy by increasing the number of GCPs; however, it is clear that the 3D modeling software is not significantly effect by the number of GCPs.
(ii) H = 50m Table 5 shows the horizontal and vertical RMSE for CPs for the case where altitude was 50 m.In addition, the number of CPs was 35 at this altitude because the imaging range narrowed.
From Table 5, it is clear that Pix4Dmapper is more accurate than Smart3DCapture in each number of GCPs, as was the case for an altitude of 100 m.However, let take into account that the resolution for one pixel was 11 mm (H = 50 m), it is understood that the practical accuracy obtained by these software was at the sub-pixels level, and the number of GCPs necessary for the absolute orientation is understood to be sufficient at three points.
On the other hand, computing the PA using the bundle adjustment method and the three images resulted in values of 8,637, 13,080, and 16,581 for 7, 9, and 13 GCPs in the centre image, respectively.Thus, for an altitude of 50 m, the results computed using the bundle adjustment method with nine GCPs is higher than the results that computed using the 3D modeling software.This implies that the respective free versions of Pix4Dmapper and Smart3DCapture, which were downloaded from the Internet, are sufficiently practical for UAV photogrammetry.
From these results, the results obtained from these software are not affected by the number of GCPs.The results show that the accuracy of altitude 100 m is smaller than that for the results at altitude 50 m.It can be seen that the accuracy decreases as altitude increases when the bundle adjustment method is used, the relationship between altitude and accuracy is discussed in the next section.In addition, considering that the results are not affected by the number of GCPs when using 3D modeling software, the performance is verified using five GCPs in following section.Rock et al.(2011)verified that accuracy decreases quadratic curve manner as altitude increases.This relation is defined in equation ( 2):

Altitude and accuracy:
where σ X0 , σ Y0 , and σ Z0 are the standard errors; H is the altitude; f is the focal length; B is the baseline; and σ P is the pointing accuracy Therefore, in this section, the relationship between altitude and accuracy is verified from the viewpoint of ground resolution.
Table 6 shows results of normalised accuracy, which is the averaged accuracy obtained with five GCPs in Tables 4 and 5 normalised by each ground resolution in terms of altitude.From  Rosnell and Honkavaara (2012) verified the relationship between overlap ratio and accuracy.They showed that a large overlap reduces the accuracy.However, in using multi-view images, 90% overlap results in higher accuracy than 80% overlap.Thus, an overlap ratio in the range 80% to 90% is recommended in using 3D modeling software.
The relationship between overlap ratio and accuracy using 17 continuous images (Case 1 in Figure 2), nine images (Case 2 in Figure 2, selecting one skip image from 17 continuous images), and six images (Case 3 in Figure 2, selecting two skip images from 17 continuous images) were verified.
Figure 3 shows the respective accuracy for each case using five Figure 2. Selection images GCPs.Incidentally, the dashed line in Figure 3 shows the PA corresponding to the ground resolution of one pixel, where at H = 100 m the value is 3,100 and for H = 50 m the value is 5,500.
From Figure 3, it is clear that in the case of Smart3DCapture accuracy decreases with decreasing overlap ratio.However, the ground resolution of the sub-pixels level is obtained in each case.
On the other hand, the results for Pix4Dmapper show that the accuracy decreases even in the case of 91.3% overlap ratio.
From these results, it is recommended that the overlap ratio be more than 80% for Smart3DCapture and more than 93% for Pix4Dmapper.Micheletti et al. (2014) verified that accuracy increases with increases in the number of images.This study speculated whether increases in the number of images result in large overlaps.The relationship between number of images and accuracy using nine images in Case 2 (Figure 2) and nine consecutive images in the vicinity of the centre were verified in this section.

Number of image and accuracy:
Figures 4 show the results of the investigation into the relationship between number of images and overlap rate.The same applies in Case 3.
From Figures 4, it can be seen that in the case of Smart3DCapture, the accuracy is improved using the nine consecutive images in the vicinity of the centre (Case 2), which remains the same as the number of images decreases.Conversely, in the six images (Case 3), the accuracy does not improve, regardless of altitude.
On the other hand, in the case of Pix4Dmapper, the accuracy significantly improves for both altitudes of 100 m and 50 m using nine images.In contrast, using six images, no improvement in accuracy is observed, regardless of altitude.It is surmised that the available image at both ends of the course in the case of a narrow imaging range is small if a large overlap ratio is adopted at low altitude, whereas in the case of a wide imaging range it is large at high altitude.The accuracy of Smart3DCapture decreases continuously.However, the accuracy of the Pix4Dmapper improves significantly.Therefore, Pix4Dmapper is estimated to be more robust than Smart3DCapture even when the number of available images is small.Incidentally, the number of images naturally depends on the size of the measurement object, thus it is important that the measurement target is covered with equal multi-view images.

Input order of images and accuracy:
In 3D modeling software, it is estimated that images are processed in order using position information recorded in the GPS data, time information, or image name that is recorded in the EXIF information.GPS data were not used in this study.The results obtained by investigating the process order of Pix4Dmapper and Smart3DCapture showed that Pix4Dmapper processes in order using EXIF information and Smart3DCapture processes in order using numbers and image filenames.
Consequently, the performance was evaluated in terms of the time described in the EXIF information of the 17 images randomly changed in Pix4Dmapper.The input order of the images was changed after changing the numbers in the image filenames in Smart3DCapture.
Table 7 shows the results obtained.It should be noted that the results were computed using five GCPs.Table 7. Relationship between input order of images and accuracy From Table 7, it is clear that accuracy is not influenced by input order of images in Smart3DCapture.However, it is estimated that accuracy is influenced by input order of images in Pix4Dmapper.

Changing of altitude and accuracy :
In this study, to investigate the relationship between changing altitude and accuracy, firstly, the effect of combining all 17 images obtained at altitudes 100 m and 50 m were investigated.Table 8 shows the results obtained.Table 8. Results obtained after combining 17 images From Table 8, it is clear that Smart3DCapture accuracy improved as a result of the combination, whereas Pix4Dmapper accuracy decreased.
On the other hand, considering that the accuracy improvement at altitude is low, the changing of accuracy was investigated by using the images obtained at altitude 50 m and adding them to the 17 images obtained at an altitude of 100 m.Table 9. Results obtained using added images Table 9 shows the resulting accuracies computed when one, three and, five images obtained at altitude 50 m are combined and added to 17 images (100 m).From Table 9, it is clear that adding the image obtained at low altitude improves the accuracy in Smart3DCapture, whereas no improvement occurs in Pix4Dmapper.Thus, whereas Smart3DCapture is robust to large vertical movements in altitude accuracy may decrease in Pix4Dmapper.
Therefore, when using Pix4Dmapper, it is recommended that only images obtained at specific altitude be used.

Scenario 2
Pix4Dmapper and PhotoScan can both obtain the coordinates of CPs on a real-world object coordinate system directly using a 3D model generated automatically after entering the coordinates of GCPs on a real-world object coordinate system and clicking the position of the GCPs on each image shown on the display.Therefore, although this feature both time-consuming and labour-intensive, it is possible to save the absolute orientation step in scenario 1.
Thus, in this section, the performance using the absolute coordinates of the CPs read directly from the absolute model automatically generated is evaluated.

Number of GCPs and accuracy:
In this section, as section 3.1.1,the relationship between number of GCPs and accuracy is investigated.
Table 10 shows the result for data obtained at altitudes 100 m and 50 m for Pix4Dmapper.From Table 10, considering that the resolutions for one pixel are 22 mm (H = 100 m) and 11 mm (H = 50 m), the result for Pix4Dmapper gives a much lower accuracy than that obtained in scenario 1.However, the accuracy of the sub-pixels level is obtained without absolute orientation.
On the other hand, the bundle adjustment method in SfM is based on confluence residual minimisation using the multi-view image, and it differs from the bundle adjustment method based on the collinearity condition in photogrammetry.In other words, the coordinates of GCPs have an error in the process of confluence residual minimisation in the case of SfM.Table 10.Horizontal and vertical accuracy Therefore, the absolute orientation is computed regarding the absolute coordinates of the GCPs obtained from the generated absolute model as the initial value-as in scenario 1.
Figure 5 shows the computed results.The results computed using PhotoScan in each case from the initial step of work procedure doesn't show cm order of accuracy.Thus, the results computed by PhotoScan are not included in Figure 5.
In Figure 5, scenario 2 is shown to have higher accuracy than scenario 1 for altitudes of 100 m and 50 m, and accuracy improves significantly especially at high altitudes.From this result, it can be surmised that the number of image points increases to use constraint condition as residuals minimisation for the image point in the case of high altitude in SfM.
From these results, although scenario 2 by Pix4Dmapper is time-consuming and labour-intensive, the functions of the software are an effective technique that can be performed easily in UAV photogrammetry.Consequently, it is recommended to recompute the absolute orientation using obtained absolute coordinates for improving accuracy.
In addition, the relationship between number of GCPs, altitude, overlap ratio, input order of images, number of images, and changing of altitude; function of software and accuracy were skipped because the direction of the results is same as scenario 1, and the result of using other than the five GCPs is same as five GCPs.

CONCLUSION
This paper investigated the performance of freely available, Internet-based 3D modeling software and commercial software from the viewpoint of UAV photogrammetry.The results obtained indicate that both Smart3DCapture and Pix4Dmapper are able to obtain practical accuracy at the sub-pixels level.Further, whereas Smart3DCapture is not affected by overlap ratio and altitude of vertical movement, Pix4Dmapper is significantly influenced.Therefore, it is recommended that Smart3DCapture be used in cases where there is significant vertical movement in altitude and small overlap ratio (however, more than 80%), and Pix4Dmapper be used in cases where a certain altitude is maintained, such as automatic flight, and there is a large overlap ratio (more than 93%).
In scenario 2 using Pix4Dmapper, the accuracy of the sub-pixel level is obtained as the RMSE of CPs without computing absolute orientation, and the accuracy is improved to compute the absolute orientation using the absolute coordinates of GCPs obtained as the initial value.Further, the accuracy improves significantly at high altitudes.
In this study, the performance of 3D modeling software in UAV photogrammetry was evaluated in a narrow range.In future work, the performance of 3D modeling software in UAV photogrammetry will be evaluated over a broad range, such as using hundreds of images.
of GCPs and accuracy: In this scenario, Smart3DCapture and Pix4Dmapper were verified using the RMSE (Root Mean Square Error) of the CPs after absolute orientation of the similarity model automatically generated by the software.The GCPs selected from the CPs on the test site were evenly distributed, and 3, 5, 7, 9, and 13 GCPs were verified in this study.The position and number of CPs selected were the same regardless of the number of GCPs (H = 100 m: 45 points, H = 50 m: 35 points).
Figure 3. Relationship between overlap and accuracy

Table 4
DX, DY, and DZ are the object field diameter; and σX, σY, and σZ are the RMSE of the CPs . Horizontal and vertical accuracies (H = 100 m)

Table 6 ,
it is clear that the normalised accuracy improves at high altitude.Therefore, it is clear that whereas accuracy decreases with increasing altitude, normalised accuracy increases with increasing altitude.Thus, it can be inferred that matching accuracy is improved because the imaging range is widens with high altitude and more interest points are taken from the image.Conversely, matching accuracy decreases because the images of CPs obtained are small with increasing altitude.