A MODIFIED PROJECTIVE TRANSFORMATION SCHEME FOR MOSAICKING MULTI-CAMERA IMAGING SYSTEM EQUIPPED ON A LARGE PAYLOAD FIXED-WING UAS

In recent years, Unmanned Aerial System (UAS) has been applied to collect aerial images for mapping, disaster investigation, vegetation monitoring and etc. It is a higher mobility and lower risk platform for human operation, but the low payload and short operation time reduce the image collection efficiency. In this study, one nadir and four oblique consumer grade DSLR cameras composed multiple camera system is equipped on a large payload UAS, which is designed to collect large ground coverage images in an effective way. The field of view (FOV) is increased to 127 degree, which is thus suitable to collect disaster images in mountainous area. The synthetic acquired five images are registered and mosaicked as larger format virtual image for reducing the number of images, post processing time, and for easier stereo plotting. Instead of traditional image matching and applying bundle adjustment method to estimate transformation parameters, the IOPs and ROPs of multiple cameras are calibrated and derived the coefficients of modified projective transformation (MPT) model for image mosaicking. However, there are some uncertainty of indoor calibrated IOPs and ROPs since the different environment conditions as well as the vibration of UAS, which will cause misregistration effect of initial MPT results. Remaining residuals are analysed through tie points matching on overlapping area of initial MPT results, in which displacement and scale difference are introduced and corrected to modify the ROPs and IOPs for finer registration results. In this experiment, the internal accuracy of mosaic image is better than 0.5 pixels after correcting the systematic errors. Comparison between separate cameras and mosaic images through rigorous aerial triangulation are conducted, in which the RMSE of 5 control and 9 check points is less than 5 cm and 10 cm in planimetric and vertical directions, respectively, for all cases. It proves that the designed imaging system and the proposed scheme have potential to create large scale topographic map.


INTRODUCTION
When disaster happened, UAS is a higher mobility, lower risk platform than airplane and satellite based sensor platform.However, it is restricted by lower payload and short operation time that only small format camera is equipped for small area data collection.It is thus reducing the efficiency and increasing the cost that various missions and flight lines are needed to acquire large ground coverage images for mapping and disaster investigation.The post processing will be time consuming and be a tedious work during stereo plotting that changing images frequently.To conquer that, the multi-camera system is the solution to collect larger ground coverage images and mosaics synthetic images as single perspective image for reducing the number of images.
In category, the multi-camera imaging system can be used to (1) expand the overall FOV and increase the strength of imaging geometry for topographic mapping (Gruber and Ladstädter, 2011;Ladstädter et al., 2010); (2) collect multi-spectral imagery for vegetation and water stress monitoring (Stagakis et al., 2012;Suárez et al., 2009;Turner et al., 2014); or (3) acquire both vertical and oblique imagery for 3D city modelling or point cloud classification (Frueh et al., 2004;Rau et al., 2015;Xiao et al., 2012).Image registration is necessary for case (1) to generate larger format virtual image and for case (2) to perform band to band registration, where case (3) is not needed since no or small overlapping area among oblique views images.The image registration is to find corresponding features on overlapping area, and then estimate the coefficients of transformation model for registration, mosaicking, or texture mapping (Brown, 1992;Zitová and Flusser, 2003).
Related works of generating virtual image can be found in Microsoft/Vexcel UltraCam (Gruber and Ladstädter, 2011;Ladstädter et al., 2010) and ZI Imaging DMC (Zeitler et al., 2002), who mosaicking medium format CCD arrays to single perspective large format CCD.They all detected feature points on the overlapping area, and then estimating the projective transformation coefficients and calibrate the interior orientation parameters through bundle adjustment.In the end, the inconsistent lens distortion and focal lengths were corrected and all images are projected to a virtual image plane for mosaicking.It could reach internal accuracy of 0.2 -0.3 pixels and has been proved the result is suitable for category mapping.
Holtkamp and Goshtasby (2009) adopts similar approach for 6 consumer grade cameras composed multi-camera system, who also adopt image matching and estimate the projection coefficients for image mosaicking.However, traditional image matching methods needs well distributed and abundant points to estimate the coefficients, which may fail when the overlapping area among images is small or the content of image is texture less.
Instead of traditional image registration method that extracting redundant points on overlapping area and estimate the projection coefficients for image registration, Tommaselli et al. (2013) proposed using relative orientation constraint for image mosaicking.They designed a dual head camera system composed by two consumer grade cameras and calibrated the relative orientation parameters (ROPs) for generating rectified images in a virtual image space.The image registration is conducted from tie-point matching among two rectified images and registered as single one.However, misregistration effect is noticed since the different camera locations and the uncertainties in the ROPs and IOPs, which is thus introduced as translation and scale difference to optimize the results.They acquire 0.6 and 0.3 pixels accuracy in row and column directions, and show the RMSE of 1 GSD and 2 GSD after aerial triangulation of mosaicked images.
In this study, we design a multi-camera system composed of five consumer grade DSLR cameras to increase the total FOV for collecting large ground coverage image and is equipped on a large payload UAS.It is particularly suitable to apply on mountainous area for disaster mapping and data collection in an effective way.The synthetic images are mosaicked to reduce the total number of images.The proposed method also calibrated the interior orientation parameters (IOPs) and relative orientation parameters (ROPs) in advance and to derive the coefficients of modified perspective transformation (MPT) for image mosaicking.Considering the vibration of UAS and the different environment condition of indoor calibration field and in the air, the uncertainty of IOPs and ROPs will cause misregistration errors.A systematic errors correction scheme is thus adopted to extract the tie-points and analyse the errors by modifying the IOPs and ROPs for finer registration results.Comparison among multi individual cameras, three cameras mosaicked virtual image, and five cameras mosaicked virtual image are conducted through rigorous aerial triangulation.

EQUIPPMENTS AND STUDY AREA
The design of multi-camera imaging system, the specifications of the adopted UAS, the study site and mission planning are described here.

Multi-Camera System
The multi-camera system is composed by five consumer grade DSLR cameras (Sony A850, 6048×4032 pixels, 5.9 ).Each one equips 50 mm focal length that has FOV of 38 × 25 degrees.Five cameras are mounted on an aluminum frame that has total weight is 24 kg.As depicts in Figure 1, the multi-camera system is composed as 1 × 5 array is mounted under the belly of UAS, cameras are naming Cam1, Cam2, Cam3, Cam4, Cam5 with tilt angles of -50, -30, 0, 30, and 50 degrees, respectively.However, with the restriction space of UAS, the Cam1 and Cam 5 are rotated by 90 degree to avoid taking the wheels.Cam 3 is the traditional nadir camera and the perspective center is also considered as virtual image center.
In the view of this design, the FOV could increase to 127 degrees, and has ground coverage of 3.1 km when flight height is 900 m. Figure 2 illustrates the ground coverage and overlapping area among cameras, where the green dotted polygon denotes the image mosaicked area.It thus has ability to increase the strength of imaging geometry, overall FOV and reduce the number of flight lines.
A position and orientation system (POS), i.e.SPAN CPT, containing a tactical grade IMU and a dual frequency GPS are also integrated together on top of Cam3.It provides initial values of positions and attitudes with accuracy better than 0.015 °/0.015°/0.03° in roll/pitch/heading direction.Once the system has been calibrated, it could provide near real-time image acquisition with position and orientation for fast disaster investigation through direct-georeferencing (Chiang et al., 2012;Skaloud and Legat, 2008).
Another Automatic Image Control System (AICS) is also integrated with a U-blox Lea-6 GPS receiver for estimating the travel distance and sending trigger events to acquire synchronise images and record the POS value.Through a predefined parameters setting, it could acquire similar image overlap ratio and record the time tag at the same time.

AL-150 UAS
The fixed-wing UAS adopted in this study is AL-150, which is designed and manufactured by Aeroland UAV Inc., Taiwan.It has 40 kg payload, 8 hours endurance time, and 100 km operation distance, which is suitable for equipping the multicamera system and executing long distance and long term mission.The appearance of the AL-150 and its specifications can be found in Figure 3 and Table 1, respectively.Table 1.Specification of AL 150 UAS

Study Area and Mission Planning
The study site is a 5 km 2 control field where 46 control points are well distributed in 6 x 7 grids.Each one was surveyed by eGPS system based on Virtual Base Station Real-Time Kinematic (VBS-RTK) technology with planimetric and vertical accuracies about 3 cm and 5 cm, respectively.
The mission planning adopts double block trajectory with flight height of 900 m above mean sea level that resulting ground sample distance (GSD) of nadir images is about 10 cm.The overlap percentage of nadir camera (Cam3) for end lap and side lap are 80% and 50%, respectively.Figure 4 illustrates the double block UAS trajectory, in which a total of 1296 events were triggered but only about 500 image groups were used in this study for mosaicking and accuracy analysis.Figure 5 shows an example of multi-camera image group acquired in a single shot over this area.

METHODOLOGY
As illustrated in Figure 6, instead of using image matching for finding conjugate features between overlapping areas and estimating their transformation coefficients, the parameters of modified projective transformation are estimated from IOPs and ROPs.However, considering the in-situ environment in the air is different from the laboratory, the systematic errors correction is adopted to correct the scale and displacement errors and updating the IOPs and ROPs for finer registration results.They were estimated by the residuals among overlapping area of neighbour camera which are computed from SURF (Bay et al., 2008) features.After systematic error corrections, the multiviews images were registered to a virtual image plane.The results were analysed and compared through aerial triangulation.In this study, the indoor calibration field was established in a room with size of 6 m x 4 m x 3 m that contains well distributed Australis© coded targets on the wall.By taking multi-view, multi-rotation and multi-positions images, the IOPs and EOPs can be estimated simultaneously through self-calibration bundle adjustment with additional parameters (Fraser, 1997).On the other hand, the ROPs among all 5 cameras can be estimated from the EOPs of all images.As shown in Figure 7, it illustrates the camera calibration field and 3D view of image positions.1) and ( 2): From equations ( 1) and (2), the relative rotation angles were calculated from      that denotes the rotation matrix between two cameras' image coordinate systems under a local mapping frame M, where C M and C S represent the master and slave cameras, respectively.The offset vector (  ,   ,   ) was derived from      which depicts the position vector between two cameras' perspective centers.The estimated ROPs were averaged through the multiple stations and their standard deviations represent the internal accuracies of calibration results.

Modified Projective Transformation (MPT)
Equations ( 3) and ( 4) are the general projective transformation model for image mosaicking, which needs at least four conjugate points and performs least square adjustment to solve the eight coefficients (  1 ~3 ,  1 ~3 ,  1 ,  2 ).However, this scheme may fail when overlapping area is small and no sufficient points are found on texture less images.
In this study, a modified projective transformation is adopted where required elements are derived from the IOPs and EOPs.
As equations ( 5) and ( 6) illustrated, they are basic function to generate the normalized stereo images (Cho and Schenk, 1992) which are derived from collinear equation.First, the IOPs are applied to correct the lens distortion effect (∆, ∆) in the original image space.Then, transfer the original image space (  ,   ) into the virtual image space (  ,   ) by using the rotation matrix acquired from relative rotation angles of ROPs.Meanwhile, the focal length of original image (  ) is transfer to the assigned focal length of virtual image(  ).Note the nadir camera is only required to correct the lens distortion effect and focal length since it is the assigned virtual image space.Through these equations, the different centers of cameras are not considered, since it only causes slightly misregistration errors over hundred meters of flight height.
However, since the uncertainty of IOPs and ROPs applying for image mosaicking will cause misregistration errors where the temperature, air pressure and distance are different to laboratory environment.The misregistration errors will be analysed and modifying the parameters of IOPs and ROPs for finer registration through systematic errors correction.Details are discussed in section 3.3.

Systematic Errors Correction
To analyse the systematic errors, SURF features are extracted on overlapping area of initial results, and calculated the misregistration residuals (dx, dy) to evaluate their consistency.
To reduce matching errors, a recursive blunder removal scheme is applied, which calculating the mean and standard deviation of dx and dy, and remove any points exceeding the mean ±3 times standard deviation.It is iteratively processed until no more points are removed.The accuracy is thus represented by the average length of residuals.
By observing the distribution of residuals (dx, dy) vs. image position (x, y), two systematic effects were introduced i.e. scale and displacement differences that can be corrected by modifying the focal length and rotation angle of ROPs.In more detail, an example of the step by step systematic errors correction among the overlapping area of initial results of Cam 2 vs. 3 and Cam 4 vs. 3 is illustrated in Figure 8.In (a) and (b), the linear distribution phenomena of initial results are observed, where the slope value meaning there are certain degree of scale difference and lines not passes through the origin point representing the displacement errors in x and y directions.After modified the ROPs no systematic effects could be observed as Figure 8 (e) and (f) shows.The approach shows reliable results and errors will be corrected when scale difference larger than 1.0e -4 and correcting the displacement when displacement larger than 0.1 pixels.These correction procedures can be conducted by image dependent or image independent approach.The former one is an ideal case assuming the multi-camera system is stable, meaning no vibration of UAS, weather condition difference, and signal delay of the trigger.The errors can be estimated by averaging from a few image groups and applied the same correction parameters for the others, which is an efficient way and each image has the same accuracy level.
In contrary, image independent treating the multi-camera system may be influenced by the vibration of UAV or time delay of trigger.Thus, every image group has its own systematic errors coefficients and it will consume more computer processing time to correct the systematic errors individually.The comparisons of these two approaches will be discussed in section 4.4.

Image Mosaicking
After image transformation and systematic errors correction, the multi cameras were transferred to the same image space and can be merged to obtain a single perspective image.The three images and five images mosaicked virtual image size is 19,500×3,400 and 29500×3400, respectively.Meanwhile, the FOV has increased from 40 degrees of nadir camera to 97 degrees of three images mosaicked and to 120 degrees of five images mosaicked virtual image.Figure 9 (a) and (b) illustrates the three images mosaicked and five images mosaicked virtual image.
Since the contrast of original images is different causing none continuously colour gap of seam line, which needs to be adjusted through image processing method, such as colour balancing and seamless mosaicking procedures.However, it is not necessary to adjust the colour difference for the analysis of aerial triangulation.The preliminarily mosaic result is thus to perform aerial triangulation to exam the performance.

ROPs
Through the indoor camera calibration and ROPs calculation, the internal accuracies of ROPs with respect to Cam3 are summarized in Table 2.It shows the calibrated angles are fits to the design angle, and the standard deviations (Std.) are all smaller than 0.04 degrees, meaning the results are accurate and reliable.However, it may cause 6 pixels uncertainty since the IFOV of camera is 0.006 degrees.On the other hand, one can notice the maximum offset vector is 39 cm that only contributes less than 4 pixels displacement comparing to 10 cm GSD over 900 m flight height, meaning the higher flight height it is the lower misregistration errors it introduced.All the remaining errors are analyzed and corrected through systematic errors correction.

Residuals Analysis of Overlapping Area
The internal accuracy is estimated from the average length of residuals between the overlapping areas.To further realize the remaining errors after systematic errors correction, the residual vectors among Cam 2 vs. 3 and Cam 4 vs. 3 are plotted in Figure 11.It shows no significant systematic effects, but the case of Cam 2 vs. 3 still has a slightly rotation effect implying some systematic errors exist and need to be further corrected.However, the average lengths of these cases are all smaller than 0.5 pixels meaning that the proposed method is feasible for multi-images mosaicking purpose.

Image Dependent vs. Image Independent
This section compares the results of image dependent and image independent approach.The image dependent approach used 10 images to calculate the averaged systematic errors coefficients and applied to other images.The Mean and Std.Dev. of scale and displacement values are summarized in Table 3. Their respective seam lines are shown in Figure 12.
The table and figure depict using image independent approach is better than image dependent.Meanwhile, the standard deviation of scale and displacement larger than the threshold i.e. 1.0e-4 and 0.1 pixels, which is the major reasons leading the remaining misregistration effect.It implies the multi-camera system is not stable.

Aerial Triangulation Accuracy Analysis
A rigorous aerial triangulation accuracy analysis is performed through Image Station Automatic Triangulation (ISAT © ) software.In which, the POS system provides the initial position and orientation values.The control points and tie-points were marked and matched by Pix4Dmapper © software and all imported to ISAT for bundle adjustment.The standard error (sigma0) of image measurements and RMSE of control points (GCP) and check points (CKP) are summarized in Table 4.In which, case A, B, C, D and E are nadir camera only, three cameras, all five cameras, virtual image mosaicked from three cameras and virtual image mosaicked from five cameras, respectively.The adopt GCP and CKP are 5 and 9 points, respectively.
From the table, it shows the sigma0 of separate cases (A, B, and C) all smaller than 0.5 pixels demonstrating the overall image measurement noise level.However, a slightly increased sigma0 is observed in case D and E which is caused by error propagation of image mosaicking.The RMSE of control points and check points of both cases are less than 5 cm in horizontal direction and 10 cm in vertical direction.The case of nadir camera shows the best precision among the others, since its spatial resolution is better than the other cameras.The accuracy among separate cameras and mosaicked images i.e. (B vs. D, C vs. E) has similar results meaning the mosaicked image is accurate and reliable for topographic mapping.However, the sigma0 of case D and E is bigger than case B and C, showing that minor misalignment effect or systematic errors might exist between neighboring cameras.Table 4. Accuracy assessment of aerial triangulation 5. CONCLUSIONS Multi-camera system is designed to increase the FOV and has ability to reduce the number of flight lines.The designed five small format composed multi-camera system is equipped on a large payload UAS and mosaic synthetic multiple images to one larger image, which can reduce the number of unknown parameters of EOPs and reduce the post processing time.
The image mosaicking method adopts modified projective transformation, in which the required coefficients are directly converted from IOPs and ROPs through indoor camera calibration.Since the calibration environment and object distance are different to the flight, the uncertainty of IOPs and ROPs will once modified though systematic errors correction.
The preliminary results show the accuracy of image mosaicking is better than 0.5 pixels through systematic error correction.Though comparisons, it shows image independent approach is better than the results of image dependent approach.It means the camera frame is not stable, which introduces systematic errors when high frequency UAS vibration exists.
In the rigorous aerial triangulation analysis, it shows the RMSE of control points and check points of mosaicked images are better than 5 cm and 10 cm in horizontal and vertical directions, respectively.Although slightly increased sigma0 of mosaic images is noticed which is caused by error propagation, the results shows mosaicked images are suitable for topographic mapping.
In the future, further systematic error effect will be analysed, i.e. the rotation effect will be corrected by modified the kappa angle.Meanwhile, the discontinuous color intensity across the seam lines will be adjusted through color balancing and seamless mosaicking processes.

Figure 1 .
Figure 1.Appearance of multi-camera system

Figure 4 .
Figure 4. UAS flight trajectory and distribution of control points

Figure 6 .
Figure 6.Image Mosaicking and Accuracy Analysis Workflow Orientation Parameters (IOPs): Camera calibration is to calibrate the IOPs of each camera and further for estimating the ROPs of four oblique cameras among nadir camera, which can be conducted in an indoor calibration field.
(a) Indoor camera calibration field (b) 3D distribution of camera positions and coded targets Figure 7. Camera calibration field 3.1.2Relative Orientation Parameters (ROPs): The ROPs depicts the relative rotation angles (∆, ∆, ∆) and spatial offset vectors (   ,   ,   ) among master and slave camera, where the nadir camera is chosen as master and the other four are slave ones.Due to the images of all five cameras are acquired at the same time, the relative orientation calibration can be conducted through equations ( By applying linear regression to estimate the slope values(  ,   ), the scale difference (  ,   ) can be calculated by (1 −   , 1 −   ).Then the two dimensions focal length are corrected to fit the best registration results, i.e.   =   ×   in x direction, and   =   ×   in y direction.Figure8 (c) and (d) shows the results after focal length correction, and only displacement errors can be observed.The displacements (   ��� ,   ���� ) are computed by averaging the value of residuals which can be seem as the slightly variation of rotation angle difference, i.e. 2 pixel difference in x direction meaning 0.012 degrees variation of phi angle since the IFOV is 0.006 degrees.

4. 2
Visual Comparison of Seam LineIn this part, the seam lines of Cam 2 vs. 3 and Cam 4 vs. 3 of each step after systematic errors correction are discussed.Figure10shows the seam line of Cam 2 vs. 3 and Cam 4 vs. 3 , from left to right shows the original results through MPT, results after scale correction, and results through both scale and displacement corrections.From the figure, it depicts no more misalignment effect could be found after systematic errors correction.(a) Seam line of Cam 2 vs. 3 (b) Seam line of Cam 4 vs. 3 Figure 10.Comparison of seam line in each step (a) Residuals of Cam 2 vs. 3 (Accuracy: 0.49 pixels) (b) Residuals of Cam 4 vs. 3 (Accuracy: 0.38 pixels) Figure 11.Error vectors of Cam 2 vs. 3 and Cam 4 vs. 3 Cam 2 vs. 3 Seam Line of Cam 4 vs. 3 Figure 12.Seam line comparison by image dependent and image independent approaches Statistics of the estimated ROPs and their internal accuracies

Table 3 .
Summarized systematic errors coefficients of image dependent and image independent approach