INVESTIGATIONS ON THE BUNDLE ADJUSTMENT RESULTS FROM SFM-BASED SOFTWARE FOR MAPPING PURPOSES

Since its first inception, aerial photography has been used for topographic mapping. Large-scale aerial photography contributed to the creation of many of the topographic maps around the world. In Indonesia, a 2013 government directive on spatial management has re-stressed the need for topographic maps, with aerial photogrammetry providing the main method of acquisition. However, the large need to generate such maps is often limited by budgetary reasons. Today, SfM (Structure-from-Motion) offers quicker and less expensive solutions to this problem. However, considering the required precision for topographic missions, these solutions need to be assessed to see if they provide enough level of accuracy. In this paper, a popular SfM-based software Agisoft PhotoScan is used to perform bundle adjustment on a set of large-scale aerial images. The aim of the paper is to compare its bundle adjustment results with those generated by more classical photogrammetric software, namely Trimble Inpho and ERDAS IMAGINE. Furthermore, in order to provide more bundle adjustment statistics to be compared, the Damped Bundle Adjustment Toolbox (DBAT) was also used to reprocess the PhotoScan project. Results show that PhotoScan results are less stable than those generated by the two photogrammetric software programmes. This translates to lower accuracy, which may impact the final photogrammetric product.


INTRODUCTION
The advent of UAV (Unmanned Aerial Vehicles) technology and developments in the domain of computer vision has largely facilitated the traditional photogrammetric workflow (Murtiyoso and Grussenmeyer, 2017).Several traditionally manual photogrammetric tasks such as tie point marking and orthophoto production have been automated to a degree where a simple button may be sufficient to perform these tasks (Remondino et al., 2011).This is a very important advantage for users, more so for those who are not in the photogrammetry community.Indeed, these automations have given an impetus to the democratisation of photogrammetry, where almost anyone can take digital pictures and turn them into 3D models.This has also influenced the mapping industry, since it provides potential users with fast, easy, and fairly low-cost solution to perform mapping activities (Chiabrando et al., 2015).However, this advantage is also a double-edged sword precisely due to its simplicity.The black-box nature of some of the SfM (Structure from Motion) based commercial software has hindered a proper statistical analysis of its results (Jaud et al., 2016).
The Indonesian Geospatial Information Agency (BIG, formerly BAKOSURTANAL) specifically uses aerial photogrammetry for mapping purposes; especially large scale mapping.Since 2013, the government policy on spatial planning requires the production of large-scale maps for effective spatial management.Consequently, as the only institution responsible for base geospatial information in the country, BIG is required to be able to provide large-scale maps in large quantities.Budgetary limitations mean that SfM software has become interesting solutions for this purpose.However, due to the black-box nature of most commercial SfM software, there is a need to test their quality in order to fulfil the required national standard.The objective of this paper is to determine the feasibility of Agisoft PhotoScan, a commercial SfM software programme, to be used in mapping works compared to the results of classical photogrammetric solutions.This will be done by comparing the bundle adjustment results on the same dataset processed in Trimble Inpho and ERDAS IMAGINE Photogrammetry, two photogrammetric programmes which have been used extensively for mapping purposes.Similar assessments to PhotoScan's quality can be found in the literature, for example in the case of close-range photogrammetry (Murtiyoso et al., 2018) or aerial UAV photogrammetry (James et al., 2017;Ouédraogo et al., 2014).

RELATED WORK AND MOTIVATIONS
Aerial photogrammetry has seen an extensive use in the generation of maps.For decades it was the main means for large-scale mapping (Wolf et al., 2014).Improvements in analytical and later on digital photogrammetry enabled several automations in traditional photogrammetric tasks e.g.tie point detection (Gruen, 1985) and dense matching (Hirschmüller, 2005).Analytically speaking, the photogrammetric problem of determining camera external parameters may be solved by using the block bundle adjustment computation.This process takes into account various ray observations from images and calculates camera exterior orientations, 3D object point coordinates, and eventually internal camera parameters in a least squares adjustment.
The introduction of Structure from Motion (SfM) solutions has also revolutionised the domain of photogrammetry, as it enables a rapid detection of tie points (Lowe, 2004) as well as pose estimation.These improvements to the photogrammetric workflow have given rise to various software solutions providing relatively accurate results with increasingly less and less user involvement.The growing trend of the use of SfMbased software in topographic mapping is also aided by the democratisation of UAVs (Murtiyoso and Grussenmeyer, 2017).That being said, in terms of topographic survey, mapping agencies such as BIG still prefers to employ photogrammetryfocused software (e.g.Trimble Inpho, ERDAS IMAGINE, etc.) over SfM-based ones (e.g.Agisoft PhotoScan, Pix4D, etc.).The main objective of this paper is to determine if one of those software solutions, namely PhotoScan, can be used for largescale topographic mapping.
Comparisons between PhotoScan and other solutions has been much addressed in the literature, due to the popularity of the software (Bedford, 2017).For example, comparisons has been done in terms of dense point cloud (Murtiyoso and Grussenmeyer, 2017;Sapirstein, 2016), Digital Surface Models (DSM) (Remondino et al., 2013;Runkel et al., 2017), bundle adjustment results (Murtiyoso et al., 2018), or orthophoto generation (Chiabrando et al., 2015).This paper will focus on bundle adjustment or aerotriangulation assessment, being one of the most important factors influencing the final precision of topographic mapping.
Due to the absence of some bundle adjustment statistics in PhotoScan's results, DBAT (Damped Bundle Adjustment Toolbox) (Börlin and Grussenmeyer, 2013) will be used to reprocess the PhotoScan projects.Results were compared to those generated by two photogrammetry-based software programmes, namely Trimble Inpho (Inpho) and ERDAS IMAGINE Photogrammetry (ERDAS).
Agisoft PhotoScan (PhotoScan) is a commercial software which has seen a lot of use in the market (Burns and Delparte, 2017;Verhoeven, 2011).Inpho on the contrary is a standard aerial photogrammetry software solution which has been employed by BIG for metric mapping projects.It is capable of performing a complete photogrammetric data processing, including bundle adjustment.Tie points extraction and matching are run automatically using two algorithms, feature based matching (FBM) for creating new points and least squares matching (LSM) for refining existing points (Cramer, 2013).ERDAS has a similar algorithm with Inpho to process the data.Meanwhile, the bundle adjustment process in PhotoScan is hidden so the quality control is difficult to perform due to the scarcity of project diagnostic.To address this problem, the open source Damped Bundle Adjustment Toolbox (DBAT) (Börlin andGrussenmeyer, 2016, 2013)

Study site
The main data was acquired on several islands in the area of North Lombok Regency in West Nusa Tenggara Province, more precisely the Mandalika, Gili Air, Gili Laba, and Gili Trawangan islands.In this research, a subset of this data, namely images over Gili Air island was taken as a sample to be studied.The total area of this island is about 175 hectares.The topography is relatively flat with a combination of open land and urban areas.The urban area is mostly concentrated in the southern part and along the coast of the island.An aerial view of Gili Air island is shown in Figure 1.

Data
The study used an aerial photogrammetry and Lidar data set acquired by the BIG.The camera and Lidar sensors were installed in the same aircraft and were operated simultaneously.Data acquisition was conducted in 2016.  1.The flying altitude was set to approximately 1000 m, giving an average Ground Sampling Distance (GSD) of 11 cm.The aerial photography used the block-strip configuration with a 70% overlap rate and 60% sidelap rate.The total number of photos used in this study was 37 images, comprising of eight photos for each of the first four strips and five photos in the fifth strip (Figure 2).

Camera
Considering the limited budget in this project, there was only five ground control points (GCPs) measured on the ground.These points were measured using the static GNSS method, with a precision of 1.5 cm for the planimetry and 2.5 cm for the altitude.As illustrated in both Figure 1 and 2, two points were located to the south of the island, two at the centre, and one at its north eastern part.Theoretically, the total number of GCP met the minimum requirement for bundle adjustment, but there are no extra observations for adjustment or validation.As such, additional points were measured from the Lidar data in order to act as checkpoints (CP) for the bundle adjustment result.

Method
This paper used three photogrammetric software programmes for data processing.The SfM-based Agisoft PhotoScan was used, with comparisons to be performed with Trimble Inpho and ERDAS IMAGINE Photogrammetry.The last two programmes have been used extensively for mapping purposes.As shown in Figure 3, data processing in the three software solutions was performed using the same data set.Furthermore, identical a priori weighting for the bundle adjustment was used for all solutions.In this regard, the measurement standard deviations were used as weightings.For the GCPs, the a priori standard deviation was set to 15 mm in planimetry and 25 mm in altitude.The manual GCP and CP marking standard deviation was set to 0.1 pixels, while automatic tie point marking used 1 pixels as a priori value.In this study, all available GCPs were selected as control points due to the limited number of measured premarks.Since as of version 0.7.0.0 DBAT cannot take into account of onboard GNSS and IMU data as approximate exterior parameters, these values were not used in the aerotriangulation process in all solutions in order to homogenise the processing conditions.
All images were processed to generate automatic tie points and calculate the bundle adjustment.The bundle adjustment is an important part of the photogrammetric process which is used in resolving the exterior orientation problem, often using initial values calculated from other methods (relative orientation, space resection, direct linear transformation, etc.) (Grussenmeyer and Al Khalil, 2002;Luhmann et al., 2014).This process determines the overall precision of the project and will also have an influence on the dense matching results further down the pipeline.The Inpho and ERDAS results served as a reference data to compare the feasibility of PhotoScan projects for mapping purposes.Metrics generated by DBAT was compared to the metrics from Inpho and ERDAS, in order to assess its quality.
In order to analyse the results, the residual values of GCP from each solution were compared.Due to the fact that all GCPs were selected as control points, additional points were measured on the Lidar data acquired in the same time as the photo acquisition.These check points (CPs) were determined manually by choosing nine objects of interest from the Lidar data.However, this entrains a different standard deviation value for the CP measurements.Indeed, due to the Lidar data's resolution of 50 cm, the CP's precision cannot be higher than this value.An error on the posterior CP residuals of this order is therefore to be expected.The validation was performed by comparing the CP coordinates acquired at the end of the aerotriangulation process and the ones measured from the ground truth Lidar data.
Another criterion of quality which will be discussed in this paper is the standard deviation of the exterior orientation parameters.Since the onboard GNSS and IMU data was not used in the computation, no residuals could be computed.The standard deviation of each exterior parameter (X, Y, Z, ω, φ, κ) for each camera was therefore compared between the three solutions in order to see the quality of each camera pose.The calculated GCP 3D residuals showed that Inpho and ERDAS produced similar results in terms of precision (Figure 4) where the residual values range from 0.3 to 1.2 cm.However, PhotoScan produced markedly worse results; with residuals ranging from 0.7 to 6.4 cm.The root mean square (RMS) value of GCP residuals was 0.6 cm for Inpho, 0.8 cm for ERDAS, 3.7 cm for PhotoScan, and 4.4 cm for DBAT.However, it should be noted that in general the difference between measured and calculated GCP coordinates is below the theoretical average GSD (approximately 11 cm).Admittedly, the small number of available GCPs presents a weak source of interpretation.However, this gives an idea on the overall internal precision of the aerotriangulation process.The comparison of CP residuals as represented in Figure 5 shows the quality of the result.A total of nine objects, such as the edge of buildings or swimming pools, were chosen as points of interest.Due to the lack of GNSS-measured points, these CPs were measured on the Lidar data with lower resolution (50 cm).The residuals ranged from 12.4 cm to 134.3 cm between all software solutions, with RMS values of 40 cm in Inpho, 48.4 cm in ERDAS, 58.1 cm in DBAT, and 72.7 cm in PhotoScan.

GCP and CP Assessment
Note that the higher order of residuals is to be expected due to the circumstances in their measurement.Some significant differences can be seen from points C3, C6, and C8 where the residuals difference between Inpho/ERDAS and PhotoScan reach up to 104.1 cm.Some errors might reflect the difficulty in pinpointing a particular object in the Lidar data.
The GCP and CP residual values indicate, respectively, the internal precision and external accuracy of the bundle adjustment process in each software solution.Although PhotoScan can produce more tie points than Inpho/ERDAS, these tie points are generally not of high precision.This produces noises on its bundle adjustment process, and may well be an important factor in explaining its results.Also, it is likely that the tie points are connected to only two to three photos, weakening its geometric strength.This, however, remains to be studied.Meanwhile, tie points in Inpho or ERDAS are fewer but more precise due to their use of the LSM method.This corresponds to a better GCP and CP residual values, as can also be seen in Figures 4 and 5.

Diagnostics Assessment
A further assessment on the quality of the bundle adjustment process can be performed by looking at the computed exterior orientation parameters.Figures 6(a), (b), and (c) show the standard deviation values of the exterior orientation parameters of each photo, calculated by Inpho, ERDAS, and DBAT.The averages of standard deviation of Inpho, ERDAS, and DBAT were respectively 4.9 cm, 7.4 cm, and 36.6 cm for horizontal (x, y) and 1.9 cm, 3.0 cm, and 245.7 cm for vertical (z).In this calculation, DBAT represents the PhotoScan quality since the input in DBAT was the PhotoScan project.
The Inpho and ERDAS values are similar within 3 cm; however DBAT presents a much larger error.In the horizontal component, it presents in average an error of almost 5 times higher from those generated by Inpho and ERDAS, while in the vertical component it is almost 10 times higher.It is also worth noting that both ERDAS and Inpho displays a more stable standard deviation variations, while those from DBAT is more volatile.The average standard deviation of κ was however similar in all software, that is 2.2 mdeg, 1.8 mdeg, and 1.6 mdeg each for Inpho, ERDAS, and DBAT respectively.In this rotational aspect, similarly DBAT generated worse values except for κ.This may be due to the amount of tie points scattered on a relatively flat terrain, which means that rotations around the z-axis is less likely.On the other hand, the quality of tie points may once again contribute to the error on ω and φ.
These results also show the stability of the bundle adjustment results from the three tested algorithms.As illustrated in Figure 7, in the case of standard deviations of the planimetric exterior parameters, results from Inpho and ERDAS are relatively stable throughout the whole dataset.The same cannot be said of the values generated by PhotoScan-DBAT.Here a systematic trend on the planimetric error can be observed; for example errors towards the X axis seem to increase as the image comes nearer to the edge of the strip.Similarly, errors toward the Y axis are more visible towards the centre of the flight strips.This may be due to fact that no cross-flights were performed on the project, rendering errors on the centre of the strip higher towards the Y axis (perpendicular to the flight direction), and the errors on the strip edges higher towards the X axis (parallel to the flight direction).
This systematic trend is actually present in all three solutions, albeit much more remarked in the case of PhotoScan-DBAT.This further shows that the source of this systematic error comes from the acquisition part, rather than from the bundle adjustment process.As regards to the rotational aspect, a case in point can be seen for the φ value (Figure 6e) for images 1, 2, 3, and 8 which are bigger than images 4, 5, 6, and 7.Those images are located in the first strip where images 1 and 8 are located in both extremes of the strip.Another noticeable point from Figure 6 is that the trend for the errors in X and φ are similar.
Likewise, errors in Y and ω share a similar trend.This is due to the fact that errors in the φ direction (rotation on the Y axis) entrain errors to the X axis, and vice versa for Y and ω.This pattern may be potentially caused by the non-ideal GCPs distribution and the unavailability of perpendicular cross-line on each side of block-strip.

CONCLUSION AND FURTHER WORK
In this paper, a comparison of the bundle adjustment results in the case of large scale aerial mapping was addressed.Results generated by the SfM-based software PhotoScan and two photogrammetric software solutions (Inpho and ERDAS) were discussed and compared.The selected criteria of quality to compare them were GCP and CP residuals, as well as exterior orientation standard deviations.In the latter case, DBAT was used to reprocess the PhotoScan project in order to extract useful additional bundle adjustment diagnostics.Table 2. Comparison of the bundle adjustment quality Based on the analysis, PhotoScan still falls short in terms of the generated RMS.Analysis on the exterior orientation also shows that PhotoScan's solution is less stable than the other two photogrammetric algorithms.While it may be enough for fast visualisation purposes, its use in topographic large scale mapping still requires further improvements.A caveat to this conclusion is that in this study, only a minimal amount of GCP was used.However, while the project is not perfect, it represents real world conditions, in which not only budgetary but often times also topographical limitations must be taken into consideration in performing mapping projects.Further work still needs to be done in order to perform tests using more control points, preferably also incorporating GNSS/IMU data.Some of our observations regarding the use of the software solutions include the fact that photogrammetric programmes are much more complicated to understand and therefore to master compared to SfM solutions.This complexity often translates to further costs, such as training of users.However, projects requiring a high level of precision and dependability may benefit from this complexity, where the user is given much more control over the process.For example, PhotoScan still lacks a tool to edit bad tie points; instead relying more on the high number of tie points.For a specialised user such as a photogrammetrist, this presents a problem rather than a convenience.That being said, PhotoScan has in recent years adopted a more metric approach (e.g. standard deviation reports, support for GNSS/IMU).If this may be an indication of the future, than it is not impossible that SfM-based solution may someday take the place of traditional photogrammetry solutions.

Figure 1 .
Figure 1.The area of study on Gili Air island, with the distribution of GCPs.

Figure 2 .
Figure 2. Configuration of the 37 images used in the project, including image numbers, flight direction, and the distribution of GCPs and check points over Gili Air.

Figure 3 .
Figure 3. Flowchart of the experiments conducted in this paper.

Figure 6 .
Figure 6.Plotted standard deviation values for the exterior orientation parameters.

Table 1 .
Camera specificationsThe specifications for the camera used are shown in Table Table 2 shows a recapitulative of the obtained values.