EVALUATION OF THE QUALITY OF ACTION CAMERAS WITH WIDE-ANGLE LENSES IN UAV PHOTOGRAMMETRY

The application of light-weight cameras in UAV photogrammetry is required due to restrictions in payload. In general, consumer cameras with normal lens type are applied to a UAV system. The availability of action cameras, like the GoPro Hero4 Black, including a wide-angle lens (fish-eye lens) offers new perspectives in UAV projects. With these investigations, different calibration procedures for fish-eye lenses are evaluated in order to quantify their accuracy potential in UAV photogrammetry. Herewith the GoPro Hero4 is evaluated using different acquisition modes. It is investigated to which extent the standard calibration approaches in OpenCV or Agisoft PhotoScan/Lens can be applied to the evaluation processes in UAV photogrammetry. Therefore different calibration setups and processing procedures are assessed and discussed. Additionally a pre-correction of the initial distortion by GoPro Studio and its application to the photogrammetric purposes will be evaluated. An experimental setup with a set of control points and a prospective flight scenario is chosen to evaluate the processing results using Agisoft PhotoScan. Herewith it is analysed to which extent a pre-calibration and pre-correction of a GoPro Hero4 will reinforce the reliability and accuracy of a flight scenario.


INTRODUCTION
The availability of action cameras and their easy-to-use capabilities are of high interest for UAV photogrammetry. Due to their light weight, their robustness, their high resolution in video and still-image capture and their wide field of view, action cameras provide different advantages for photogrammetric purposes, especially in UAV photogrammetry. However, the wide field of view caused by a fish-eye lens yields some difficulties in camera calibration. Typical fish-eye lenses are characterized by a short principal distance and a high radialsymmetric lens distortion. As on-the-job camera calibration in UAV photogrammetry using standard consumer cameras is of low quality due to the flight characteristics, the estimation of the interior orientation parameters of action cameras with wideangle lenses is expected to be even more unreliable. A precalibration is therefore recommended in order to provide best interior orientation parameters and to allow for precise photogrammetric reconstructions.
Camera calibration and the evaluation of high-quality interior orientation parameters is a major topic in research and development of photogrammetry since decades. Remondino & Fraser (2006) and Luhmann et al. (2015) summarize the requirements on reliable and stable self-calibration procedures. Calibration procedures using planar testfields have to be handled carefully. The authors strongly recommend to use rolled images around the optical axis and varying object distances. These conditions have to be considered throughout all projects dealing with the estimation of interior orientation parameters. UAV flights usually do not fulfil these requirements for self-calibration. Nevertheless, in practice this is often neglected.
With respect to nowadays UAV missions a detailed summary on UAV systems, techniques, software packages and applications is given by Colomina & Molina (2014). According to the camera's interior orientation parameters the relevance of their estimation and its impact in object space is rarely analysed or documented. Douterloigne et al. (2009) present a test scenario for camera calibration of UAV cameras based on a chess-board pattern. The repeatability of the interior orientation parameters is evaluated by using different image blocks and error propagation systems. Hastedt & Luhmann (2015) discuss investigations on the quality of the estimation of interior orientation parameters and its impact in object space for UAV flights. The influence of the calibration structure for consumer cameras is evaluated. Different flight scenarios and camera settings in UAV flights are evaluated within a simulation process in order to estimate the impact in object space accuracy using self-calibration. However, these works are using a central projective model.
Photogrammetric measurements using wide-angle lenses require preceding processing approaches. Since the imaging process does not follow a central projective model, a pre-correction of the initial distortion could be applied followed by a camera calibration procedure. Alternatively a specific fish-eye model can be introduced into the bundle adjustment process. Different approaches on the calibration of fish-eye lenses are e.g. discussed by Schneider et al. (2009). Further investigations are published by Schneider et al. (2016). They present an approach on the epipolar, equidistant rectification of distorted images to be applied to central projective dense image matching algorithms for UAVs.
Nowadays action cameras, like GoPro cameras, are equipped with fish-eye lenses. Balletti et al. (2014) discuss results on the evaluation of a GoPro Hero3 camera for photogrammetric purposes. They recommend the use of the highest possible resolution. A calibration procedure using a chessboard pattern and OpenCV algorithms are applied in order to generate distortion-free images. Comparative analyses with Agisoft PhotoScan are done by applying the original and resampled images. An increase in accuracy of factor 2 could be achieved using the undistorted images. Similar investigations using the GoPro Hero4 are presented by Teo (2015) using Photomodeler and Agisoft PhotoScan. The application of an action camera on a low-cost aerial vehicle is analysed by Ballarin et al. (2015). They compare different acquisition modes, but further investigations only include full resolution images, not video mode acquisition. They refer to difficulties in the step-by-step processing. Interior orientation by a previous calibration is applied as initial values.
The following investigations focus on the analysis of the GoPro Hero4 Black Edition applied to photogrammetric tests. Different calibration procedures are tested in order to estimate the applicability of standard central projective approaches. The authors refer to preliminary work on camera calibration effects comparing photogrammetric and computer vision approaches (Hastedt 2015, Hastedt & Luhmann 2015. A pre-correction of the initial distortion is applied to the data by using GoPro Studio and subsequently the remaining distortion is estimated. The calibration results are analysed with respect to their accuracy potential in object space. Finally an experimental setup is used to prospect the influence of the interior orientation parameters by pre-calibration and self-calibration using Agisoft PhotoScan.

Camera specifications
For these investigations an action camera GoPro Hero4 Black ( Figure 1) with a rolling shutter was used. The camera is characterized by a high video resolution of up to double HD (4K) combined with an ultra-wide field-of-view (FOV) of up to 170°, thus a so-called fish-eye lens is used. Different video and photo acquisition modes can be selected. These modes include settings of the acquisition frequency, the FOV and different other imaging settings (see GoPro 2016). For calibration purposes the sensor size resp. the pixel pitch in the metric system needs to be specified. The manufacturer gives no information about it for the Hero4 Black. Since the resolutions in different modes are the same compared to its legacy version Hero3 Black, a sensor type 1/2.3" is defined (GoPro 2016a). Due to its highest image resolution of 4000 x 3000 pixels (12MP) with a 4:3 aspect ratio the sensor size is set to 6.16mm x 4.62mm (Holst et al. 2011), which leads to a pixel pitch of 1.54µm (Table 1). In order to assign the sensor size resp. the used sensor pixels in other acquisition modes the camera was set-up in front of a projected chessboard pattern on a planar wall. The camera was set to different modes taking an image/video in each case. The imaged chessboard in every mode was compared to the FOV of the 12MP acquisition mode. For the 4K video mode a slightly bounded horizontal and a significantly bounded vertical FOV was identified comparable to its 16:9 aspect ratio. This leads to the assumption that in 4K the pixel pitch is the same as in 12MP mode. For HD resolution the same FOV as for 4K was identified. Therefore the same sensor size is used, which indicates a fourfold pixel pitch. The results are summarized in Table 1

Definition of interior orientation
In photogrammetry as well as in computer vision the functional model of 3D reconstruction is based on the pinhole camera model, namely the central projection. The modelling of the interior orientation includes three main groups of parameter sets: a. principal distance c and principal point x'0, y'0 b. radial-symmetric lens distortion (rad) c. decentring distortion (tan).
The pinhole camera model and resulting standard observation equations are defined in Luhmann et al. (2014). The distortion parameters are defined with respect to the principal point. An unbalanced form of radial-symmetric lens distortion, based on Brown (1971) is chosen (1): The decentring distortion follows equation (2): When applying equality of the principal distance and pixel size in x-and y-direction in image space as well as the distortion model in (1) and (2) the results between the photogrammetric approach (AXIOS Ax.Ori) and those in computer vision (Agisoft PhotoScan or OpenCV) are comparable. Hastedt et al. (2015) give an overview on the conversion instructions. Table 2 summarizes the conversion scheme from computer vision approaches (defined in pixel coordinate system) to photogrammetric approaches (defined in metric image coordinate system).

Calibration setup and processing
For camera calibration different setups and workflows are introduced. In a first step standard approaches for camera The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B1, 2016 XXIII ISPRS Congress, 12-19 July 2016, Prague, Czech Republic calibration using planar testfields are analysed. Due to the ultrawide FOV of the camera's lens a central projective mathematical model, that can be found in most approaches in photogrammetry and computer vision, might not fulfill the requirements for precise 3D reconstruction. However, as a central projective model is implemented to standard UAV software packages and well known (and e.g. for free using OpenCV) it is reasonable to compare the different results and applicability to practical work. In addition, subsequent use of the photogrammetric data between different systems require equality of the functional model, therefore a central projective model will be the first choice.
Comparable setups include image blocks over different testfields that have to be considered as planar testfields in terms of the bundle approaches and the usage of an ultra-wide FOV: a semi-spherical testfield for a photogrammetric approach and a planar chessboard for approaches in computer vision (see Figure 2). Within these investigations different acquisition modes of the GoPro Hero4 are analysed. For processing, standard central projective approaches from photogrammetry like AICON 3D Studio combined with AXIOS Ax.Ori bundle adjustment as well as from computer vision like OpenCV 2.4.7 and Agisoft Lens are used.
In a second step a volumetric testfield based on the German Guideline VDI/VDE 2634.1 (2002) is used ( Figure 5 left). This setup allows for the estimation of an absolute accuracy potential in a 1 m³ volume of the Hero4 in 4K mode based on calibrated length within the volume. By this the processing is done by using the original data as well as pre-corrected data by GoPro Studio. The pre-correction generates distortion-free images by using initial values implemented to the software. This leads to images based on a central projective model.
UAV software packages like Agisoft PhotoScan or Pix4D and OpenCV offer fish-eye camera models, too. The approaches are based on different mathematical models. Therefore, their effectiveness can only be estimated when applying them to selfcalibration bundles.
The numerical analyses of the calibration results using different approaches include statistical analyses as well as the estimation of the accuracy potential in object space by forward intersection or length measurements against calibrated lengths. Additionally a prospective flight scenario estimates the impact of the calibration results in object space. Besides the numerical analyses, distortion-free images are calculated for visual interpretation of the calibration results.

Calibration results on planar testfields
In general, within the first setups comparable image blocks are taken by using the GoPro Hero4 in different acquisition modes ( Figure 3) over different testfields. Image blocks include rolled imaged around the optical axis. For a photogrammetric analysis using AICON 3D Studio and AXIOS Ax.Ori a semihemispheric testfield is used (Figure 2, left). This testfield has to be regarded as a planar testfield in terms of the use of an ultra-wide lens. It is taken into account 1) due to the comparability to OpenCV 2.4.7 and Agisoft Lens, where a planar chessboard pattern (Figure 2 right) is used, and 2) in order to allow for non-distorted imaging of points in the image space corners.
The results of interior orientation are summarized in Table 3. For 12MP mode nearly equality in the distortion parameters between all three processing procedures can be determined. Besides differences of up to 14 pixels in principal distance the principal point differs in sign and value. In 4K mode it has to be paid attention to the differing distortion parameters. In HD mode the results show high differences especially between the approaches in photogrammetry and computer vision. Repeating calibrations show the instability of parameter determination in HD mode using the photogrammetric approach. In general, the consideration of the decentering distortion parameters has to be handled carefully and their comparability throughout the different approaches is difficult. The principal distance and principal point estimated with OpenCV differs most compared to the others. In order to get statistical information about the parameter estimation and comparable results, a chessboard calibration using Agisoft Lens should be preferred.   The calibration bundles for the chessboard calibration in 4K mode result with a reprojection error of 1.4 pixels in Agisoft and 1.2 pixels in OpenCV. The photogrammetric approach results in a standard deviation of unit weight of 0.4µm resp. 1/10 pixel. The estimation of the object points by forward intersection using the resulting exterior and interior orientation parameters and the input image measurements show remaining deviations to their reference values of 0.01mm to 0.43mm for the results of OpenCV and 0.05mm to 0.85mm for the photogrammetric approach. The photogrammetric approach is defined as free-network adjustment with 6 degrees of freedom, whereas OpenCV considers the input of object coordinates of the chessboard as control points.
Visual interpretation of the different distortion effects is given by the analysis of undistorted images. Therefore for three chessboard images their distortion-free images are generated using the interior orientation results in 4K mode from 1) photogrammetric approach, 2) OpenCV 2.4.7 and 3) Agisoft Lens. Difference images, likewise shown in Figure 4, show deviations of up to 11 pixels in transverse direction. The distortion free images are generated by using the in-house tool PhoX (Luhmann 2016). The undistort-function of OpenCV can also be used, but it has to be considered that the images are then not corrected for their principal point.

Calibration results in volumetric testfield
For the second setup a volumetric testfield based on the German Guideline VDI/VDE 2634.1 (2002) is used, see Figure 5 left. Using GoPro Hero4 camera in 4K mode a typical photogrammetric image block is taken over the volumetric testfield ( Figure 5 right). A pre-correction of the images is used in a further step. Videos and original images can be corrected for their initial distortion using GoPro Studio. The remaining distortion and parameters of interior orientation are then estimated using a standard calibration approach, as it is done by using the original images.
In order to estimate the absolute accuracy in 1 m³ volume lengths between defined object points are tested against their calibrated lengths. As characteristic the length measurement error (LME), calculated as measured value minus target value, is chosen. The analyses and bundle adjustments are done using AICON 3D Studio and AXIOS Ax.Ori.

Results of uncorrected image blocks:
The bundle adjustment results in a standard deviation of unit weight of 0.3µm, 1/5 pixel respectively. The RMS 1-sigma values in object space remain to RMSX = 50µm, RMSY = 54µm and RMSZ = 43µm. The estimation of the interior orientation parameters leads to problems in the decentering distortion, especially in B1. A closer look to the correlations between the interior orientation parameters show almost no correlation between the decentering distortion and the principal point. This correlation should remain due to the functional model. It can be stated that this is probably superimposed by other correlations or effects within the bundle. The resulting maximum length measurement error is 0.143mm with an overall relative accuracy of 1:22,900 (Table 4).

Results of pre-corrected image blocks:
The resulting statistics of the bundled pre-corrected image blocks can be summarized to a standard deviation of unit weight of 0.2µm, 1/7 pixel respectively. The RMS 1-sigma values in object space give RMSX = 133µm, RMSY = 132µm and RMSZ = 113µm. The estimation of the interior orientation parameters is successful, all are estimated significantly. The remaining correlations in the bundle results fulfill the expectations. Therefore the applied central projective approach is reliable. The resulting maximum length measurement error is 0.161mm with an overall relative accuracy of 1:15,200 (Table 4).

Evaluation of calibration results:
The results of the uncorrected and pre-corrected image blocks show a similar accuracy potential when looking at the length measurement error. The statistics of the uncorrected data refer to an expected higher precision ( Table 4). However, these results are influenced by the bundle approach itself. Due to the central projective model the image measurement outliers are eliminated during the bundle approach and its data snooping algorithms. Figure 6 shows the used image measurements and their position on the image sensor for the uncorrected and pre-corrected data. The interior orientation parameters are only valid for these sensor parts that are covered by the image measurements. For the uncorrected image block only image measurements in sensor parts of low distortion remain in the bundle adjustment. This is not true for the pre-corrected data, all sensor parts and almost all image measurements are used for the bundle adjustment. Remondino & Fraser (2006) refer to an accuracy increase if better prospects are available to model the departures from collinearity throughout the full image format.  Table 4. Choice of results of calibration in volumetric testfield with uncorrected and pre-corrected data As conclusion the usage of pre-corrected data combined with a subsequent camera calibration using standard calibration methods is recommended. The results in statistics, precision, accuracy and correlations are reliable and plausible. All sensor parts are considered within the calibration procedure which leads to significant, reliable and valid interior orientation parameters for subsequent applications. An overall higher accuracy in terms of independent measurements can be expected.

Impact in object space:
An independent setup is used to estimate the impact of the calibration results in object space by creating a prospective flight scenario. Therefore the camera is moved along in two rows while adjusted towards a wall testfield (Figure 7) with an acquisition distance of ~1m. An overlap of 80-90% along the acquisition direction in one row is reached. The images are pre-corrected using the initial correction by GoPro Studio.
Two estimates are calculated for 14 signalized points (Agisoft coded targets) on the wall testfield (see Figure 7): a. Forward intersections for object coordinates based on previously estimated interior and exterior orientation parameters using the overlapping images (as it would be in flight, too), and transformed to their control point coordinates from independent photogrammetric measurement. b. Forward intersections for object coordinates based on previously estimated exterior orientation parameters using the overlapping images and pre-calibrated interior orientation parameters, and transformed to their control point coordinates from independent photogrammetric measurement.   The results for the forward intersections of specified object points to their control points are summarized in Table 5. The maximum and minimum deviations prove the assumption of more reliable and accurate results using fixed pre-calibrated interior orientation data connected to prospective flight scenarios with overlapping but non-tilted images. Otherwise higher deviations of up to factor 30 have to be expected.

EXPERIMENTAL SETUP
For a final analysis using Agisoft PhotoScan an experimental setup is constructed. On the previously used wall testfield a photograph for better feature matching is projected (Figure 8). Based on a prospective flight scenario a video in 4K mode is taken. An acquisition distance of 1m is used that leads to a ground sample distance of about 0.5mm. Due to the ultra-wide FOV a different experimental setup with respect to a comparable acquisition distance to typical UAV flights could not be found. Therefore all following results in accuracy have to be taken as sample accuracy that have to be multiplied with the value of flying height in meters in order to be valid in object space.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B1, 2016 XXIII ISPRS Congress, 12-19 July 2016, Prague, Czech Republic Table 6 summarizes the achievable ground sample distances (GSD) in different flying heights using the GoPro Hero4. It has to be considered that due to the ultra-wide FOV a drop in resolution occurs with an increase in radial distance from image centre. For the assessment of the object space accuracy the image scale number for nadir views as well as for margin views has to be considered.   Table 6. Ground sample distance and image scale number for UAV flights using the GoPro Hero4

Rolling shutter
As action cameras like the GoPro Hero4 are video devices using CMOS sensor techniques a special effect in image motion has to be considered. Hedborg et al. (2012) state that CMOS sensors are reading images by rows. Furthermore, modern video devices usually come without mechanical shutter systems, mostly electronic rolling shutters are applied. Hedborg et al. (2012) state that this leads to the necessity of a rolling shutter camera model because structure-from-motion (SfM) approaches are based on global shutter camera models. This leads to the assumption that using pre-corrected images (initially corrected for their distortion in order to gain images in central projective model) should be the first choice to be used in common SfMbased software packages.
Forward motion effects during acquisition have to be minimized. Pix4D recommends to use the GoPro cameras in highest resolution with an imaging rate of 1-2 seconds (for flying with Phantom Vision) or 0.5-1 second (for walking). These investigations consider images of 1fps out of 30fps video material. Rarely, blurred images are resulting for walking parts.

Experiment results
The taken image blocks follow a straight overlap in walking direction without tilted images in two rows. The images are aligned using Agisoft PhotoScan in four projects 1) using the uncorrected images within a self-calibration bundle, 2) using the uncorrected images with fixed interior orientation parameters, 3) using the pre-corrected images within a selfcalibration bundle and 4) using the pre-corrected images with fixed interior orientation. 14 coded targets are placed on the wall, their coordinates are estimated by a photogrammetric bundle and set as control points. Six coded targets are used for coordinate system definition in Agisoft PhotoScan. The remaining control points are used for independent forward intersections using from each project: interior and exterior orientation as well as their image measurements. Forward intersections are calculated with PhoX. Figure 9. Experimental setup on wall testfield with texture projection and control point targets Table 7 and Table 8 list the results from Agisoft PhotoScan. The total error [m] defines the distance between the input coordinates to their estimates, the maximum error is also denoted. The total error [pix] represents the root mean square reprojection error over all photos. The interior orientation parameters are listed side by side. For the use of the uncorrected image data within a selfcalibration bundle a high deviation in principal distance and principal point to the external calibration results are obvious. A similar effect on the estimation of the principal distance within a planar object space is stated by Gerke & Przybilla (2016). They additionally constitute a high variation in principal point components using the same camera in different flight configurations on a planar object space. Better results can be obtained using the pre-corrected image data within a selfcalibration bundle. Applying pre-corrected data leads to an improvement in the bundle adjustment results. The total error in image space as well as the total error in object space can be improved. The introduction of pre-calibrated interior orientation parameters does not comply the expectations. Especially looking at the uncorrected data, the results are significantly influencing the bundle results.
Furthermore forward intersections are calculated in order to quantify the impact in object space. The results for the four projects are summarized in Table 9. The forward intersected object points are transformed to their control point coordinates.
The minimum and maximum deviations in X, Y and Z are listed as well as the RMS values. The results prove the previous statistics for the uncorrected data using fixed interior orientation parameters. The results do not show a significant improvement in object space accuracy using pre-corrected data, with or without fixed interior orientation parameters.   However, the transformation statistics lead to better standard deviations in translation and rotation estimation using precorrected data. The scale factor 1 is reached with slight scaling of 1.005, the uncorrected data transformation show a scaling factor of 0.982. In general a mean absolute sample accuracy of ± 20mm has to be expected, this equals 40*GSD. This leads to uncertainties in nadir of 0.6m for a flying height of 30m. The increase in object space accuracy using fixed interior orientation with pre-corrected image data, as shown in  Addressing the assessment of accuracy potential in object space by forward intersections, it has to be stated that the deviations of the estimates of the used six control points often result in maximum.
The estimation of dense point clouds is used for visual interpretation. Figure 10 and Figure 11 show similar sections of the setup. The resulting point cloud of the planar wall is more smooth and flat, representing a planar field, using the uncorrected data than using pre-corrected image data. Figure 10. Section of dense point could using uncorrected data with self-calibration Figure 11. Section of dense point could using pre-corrected data with self-calibration This leads to the conclusion that using the uncorrected data, the processing in Agisoft PhotoScan is more homogeneous within its functional model.
Using the fish-eye camera model within Agisoft processing yields to higher precision in the bundle statistics compared to all other results. Unfortunately it is not possible to use the data for subsequent processing as e.g. dense points cloud generation. Therefore this approach is neglected.

CONCLUSIONS
With this article investigations on the estimation of the interior orientation parameters, their quality and impact on the accuracy potential in object space using a GoPro Hero4 action camera are presented. Different calibration procedures and processing steps are used in order to provide comprehensive information on the behavior of the camera with an ultra-wide FOV. The calibration procedures consider the photogrammetric approach by AICON 3D Studio combined with AXIOS Ax.Ori bundle adjustment, OpenCV camera calibration and undistort functions as well as Agisoft Lens and Agisoft PhotoScan for subsequent processing. An experimental setup follows a prospective flight scenario for UAV to allow for the estimation of accuracy in object space.
First analyses summarize the estimation of the interior orientation parameters by using a planar testfield respectively a semi-spherical testfield that has to be considered as planar in terms of using a fish-eye lens. Different acquisition modes are tested and compared. The results refer to some instabilities in the determination of the interior orientation parameters. Sometimes sign changes occur even for the principal point. The decentring distortion often remains with different values even though using the same image block in OpenCV and Agisoft Lens. In HD mode high deviations are figured out. In general it has to be assumed that instabilities of the camera system are present that lead to only partially comparable results.
Pursued analyses on the estimation of the interior orientation parameters based on a volumetric testfield include the use of original images as well as the use of pre-corrected images. Precorrected images are corrected for their initial distortion and fulfill a subsequent central projective imaging. The results refer to problems in the calibration procedure by eliminating margin image measurements using the original images. The distortion parameters are therefore invalid with respect to all sensor parts. As a conclusion the use of pre-corrected images is recommended in order to provide valid parameters, to avoid correlations within the system and subsequent applications. Forward intersections refer to an increase in object space accuracy.
An experimental setup is chosen to analyse the alignment process in Agisoft PhotoScan using a GoPro Hero4. A wall testfield covered by different control and check points is used. Due to the ultra-wide FOV a setup representing a sample accuracy with 1m acquisition distance is chosen. The results have to be multiplied by the acquisition distance value in order to represent an accuracy potential for UAV flights in object space. The results refer to slightly better results just using the original images in a self-calibration bundle. However, it has to be considered that an idealized lab test setup is chosen for this analysis. Using the pre-corrected images a similar accuracy potential is reached. Using pre-calibrated interior orientation parameters with pre-corrected images most reliable, stable and accurate results are assumed to be determined.