PUSHBROOM HYPERSPECTRAL IMAGING FROM AN UNMANNED AIRCRAFT SYSTEM ( UAS ) – GEOMETRIC PROCESSING WORKFLOW AND ACCURACY ASSESSMENT

In this study, we assess two push broom hyperspectral sensors as carried by small (10 – 15 kg) multi-rotor Unmanned Aircraft Systems (UAS). We used a Headwall Photonics micro-Hyperspec push broom sensor with 324 spectral bands (4 – 5 nm FWHM) and a Headwall Photonics nano-Hyperspec sensor with 270 spectral bands (6 nm FWHM) both in the VNIR spectral range (400 – 1000 nm). A gimbal was used to stabilise the sensors in relation to the aircraft flight dynamics, and for the microHyperspec a tightly coupled dual frequency Global Navigation Satellite System (GNSS) receiver, an Inertial Measurement Unit (IMU), and Machine Vision Camera (MVC) were used for attitude and position determination. For the nano-Hyperspec, a navigation grade GNSS system and IMU provided position and attitude data. This study presents the geometric results of one flight over a grass oval on which a dense Ground Control Point (GCP) network was deployed. The aim being to ascertain the geometric accuracy achievable with the system. Using the PARGE software package (ReSe – Remote Sensing Applications) we ortho-rectify the push broom hyperspectral image strips and then quantify the accuracy of the ortho-rectification by using the GCPs as check points. The orientation (roll, pitch, and yaw) of the sensor is measured by the IMU. Alternatively imagery from a MVC running at 15 Hz, with accurate camera position data can be processed with Structure from Motion (SfM) software to obtain an estimated camera orientation. In this study, we look at which of these data sources will yield a flight strip with the highest geometric accuracy.


INTRODUCTION
Recently the use of push broom hyperspectral scanners on Unmanned Aerial Systems (UAS) has become more common for assessing health of vegetation (Colomina and Molina 2014).Zarco-Tejada et al. (2013) used UAS borne hyperspectral system to estimate leaf carotenoid content in vineyards.Lucieer et al. (2014) and Hruska et al. (2012) described the development of a UAS hyperspectral system and detailed the methods used to geometrically correct and calibrate the data collected.Zarco-Tejada et al. (2012) described how UAS hyperspectral data was used to retrieve chlorophyll fluorescence of trees in a citrus orchard.
For these studies accurate radiometric calibration and geometric correction of the imagery was essential.Radiometric calibration is required such that the true radiance and/or reflectance of the vegetation being studied can be extracted, thus allowing accurate and repeatable calculation of vegetation indices.Geometric correction is essential for change detection and/or the locating of ground validation samples within the imagery.
When a push broom sensor is directly mounted to the airframe, every little change in pitch, roll and yaw made by the autopilot to maintain the pre-programmed flight path is transcribed into the data.This is especially important when using a push broom sensor.Mounting the push broom line scanner on a stabilised gimbal has the potential to reduce the geometric noise in the data that is induced by the airframe movements.
The established method for georeferencing push broom imagery is to use a GPS to record the position, and an IMU the orientation of the sensor at the time of scan line collection.This data in combination with an underlying Digital Surface Model (DSM) allows each scan line to be projected onto the orthorectified surface.Each line is then joined to form an orthorectified mosaic.When using this type of methodology, a low accuracy IMU can lead to geometric errors in the final product (Hruska et al. 2012).
An alternative to using an IMU to measure sensor orientation is to use a frame based camera to capture conventional imagery.If the camera location at the time of image capture is recorded, then the imagery can be processed with Structure from Motion (SfM) software which via a bundle adjustment estimates the orientation of each camera.With the pushbroom sensor mounted in the same alignment as the frame camera, the frame camera orientations can be used as sensor orientations Alignment of the frame camera and sensor may not be precise, and thus there is likely to be a small boresight angle difference between the two.Suomalainen et al. (2014) used a frame camera and SfM to determine camera orientations, and thus the orientation of their hyperspectral sensor.
In this study we have concentrated on geometric accuracy rather than radiometric calibration.To achieve a high geometric accuracy, in a similar manner to Suomalainen et al. (2014) we have used a frame camera to determine sensor orientation, however we run our camera at a much higher frame rate and we have used a higher accuracy GPS to determine airframe/sensor position.We also investigate the effect of varying the capture frequency of the frame camera, as well as demonstrating the benefits of lever arm and boresight corrections.

Sites
For geometric accuracy assessment a dense network of 46 Ground Control Points (GCPs) were laid out on the oval at the University of Tasmania (see Figure 1) in an area of approximately 70 x 70 m.We also included tape measures to mark out straight lines to provide a visual indicator of accurate correction of the pushbroom imagery.Two flights were conducted at 15 m above ground level over the experimental grid, each flight was made up of six overlapping passes, one flight in an approximately northeast / south-west orientation, the other flight in a southeast / north-west orientation, giving a total of 12 passes.

Airframes
The first UAS used was a DJI Matrice 600 (http://www.dji.com/matrice600) which is a heavy lift hexacopter multi-rotor capable of carrying a payload of up to 6kg (see Figure 3).The sensor payload is mounted on a stabilised gimbal, such that the sensor remains nadir facing throughout the flight.The missions were carried out in an autonomous mode where the waypoints at the end of each pass were pre-programmed and the airframe instructed to fly at 2 m/s between the waypoints and then make smooth rounded corners when changing flight direction.During the flight the heading was held constant, i.e. the airframe moved forwards along the first pass, then returned backwards along the second pass and so on.
The second UAS used was a FalconVIZ multirotor with a stabilised gimbal mount for the sensor package.It also flew a pre-programmed flight path, but unlike the first airframe, it changed heading at each waypoint and flew forwards along each of the passes.

Sensors
The sensor combination carried by each airframe can be found in Table 1.The specifications of the two hyperspectral cameras can be found in Table 2. Differences between the two systems that should be noted are spatial bands (across track resolution), and also that an accurate GPS and a Machine Vision Camera (MVC) are present on Airframe 1.The nano-hyperspec has its own proprietary system for logging the hyperspec frame data along with sensor positon/orientation.Whereas for the micro-hyperspec we developed our own data logging system based on an IO industries frame grabber (www.ioindustries.com).With data coming from multiple sensors, it is essential that the data-streams are synchronised.The benefits of collecting high accuracy position and orientation data are diminished significantly if it cannot be accurately synchronised with the image and line scan data.The system on Airframe 1 uses GPS time to synchronise the data-streams.The IO Industries frame grabber that records the hyperspectral line data has an internal clock that is set to GPS time during start up and kept in sync via a Pulse Per Second (PPS) signal from the GPS.The IO Industries is thus able to record the precise time of capture for each hyperspectral frame.As each of the frame camera images are collected, the time of capture is marked in the GPS data log, post processing allows the extraction of the position at the time of capture, enabling accurate geo-tagging of each image.The Xsens IMU also has a GPS and thus records all its data with a GPS timestamp.

Data selection
As the airframe navigates through the corners of the programmed flight path and sets up for the next pass it tends to speed up and make quick navigational movements which can cause the gimbal to lose stability for a brief time.The hyperspectral cube georeferencing software also works on one pass at a time, so it is necessary to split the data into parts based on the individual passes.It is also important to exclude the sections during which the airframe is moving too fast and the gimbal is unstable.Plotting airframe speed (based on the change in GPS position) versus the scan line number (see Figure 4) we can see the stable section of flight (when speed is close to the programmed 2 m/s speed) and thus extract those scan lines as an individual pass.
Figure 4 -Airframe speed plotted for a one pass, allowing stable section of flight to be selected.

Lever arm correction
There is a physical offset between the location of the GPS antenna that is being used to record the airframe position and the optical centre of the camera sensors (see Figure 3).It is thus necessary to apply a lever arm correction to the GPS positions recorded for each sensor such that a more accurate sensor position can then be supplied to the georeferencing software.We wrote some python code to apply a transformation vector to the GPS position, based on the orientation (in particular the heading) of the airframe at the time of acquisition and an offset between the GPS antenna and the sensors as measured in the lab.

Machine vision camera processing
The implemented of a high frame rate MVC on Airframe 1 served three purposes; to be a supplementary method to determine sensor orientation based, and to create a high resolution Digital Surface Model (DSM) that is required by the hyperspectral orthorectification software, and finally to provide an accurate reference image to enable boresight correction.
Once accurate positions for the camera at time of image capture is known (see Sections 2.3 and 2.5) we can process the images with a standard SfM package, in this case Agisoft Photoscan (www.agisoft.com).Detailed description of the Photoscan workflow can be found in Verhoeven (2011) and Turner et al. (2014).When processing the full frame rate of the MVC we subset the data into sections that only contain three passes (see Figure 5) otherwise the complete dataset (~10,000 images) is too large to process in a reasonable timeframe on a standard desktop computer.By doing three passes at a time, with the centre pass being the pass of interest for which we wish to obtain camera orientations, we reduce the processing time down to 5-6 hours per three pass set.The final outputs of the Photoscan process are a list of camera orientations for each camera frame, i.e. roll, pitch and yaw, a DSM of the area that has been imaged by the hyperspectral line scanner, and a orthophoto of the area beneath the hyperspectral pass.

Hyperspectral camera processing
The raw data collected by the hyperspectral camera is known as a hyperspectral cube.To georeferenced this data we need to know the position of the camera and the sensor orientation at the time exposure for each hyperspectral line within the cube.To project the line accurately onto the ground below, a DSM is also required and is provided by the workflow described in Section 2.6.All this data is then used as input for the PARGE (Schläpfer et al. 1998) line scanner software package.This software performs an ortho-rectification of hyperspectral scan line data based on the GPS position, airframe attitude, and the underlying DSM.
Once and hyperspectral data cube has been georeferenced, GCPs can be identified, either manually of via an automated process that correlates one band of the hyperspectral image cube with a previously generated orthophoto that has a high spatial accuracy.The GCPs can then be used to correct the flight path of the airframe (for example if the measured airframes positions were based on low accuracy GPS data) and/or determine the boresight differences between the senor measuring the camera orientation and the orientation of the line scanner itself.
For this study, we used the automated GCP tool and correlated the hyperspectral data with an accurate orthophoto as generated from the MVC imagery for which the spatial accuracy was optimised by using a series of manually identified GCPs.

Orthorectification of hyperspectral passes
A raw pass can be seen in Figure 6a, the tape measure that was laid out in the scene does not present as a straight line demonstrating that the data needs to be geometrically corrected.The results of the orthorectifcation process that was applied with the PARGE software (see Section 2.7) can be seen in Figure 6b, where 2 cm/pixel resolution orthorectified flight strips have been created.It can be seen in Figure 6b that this process has removed much of the distortion and has accurately orthorectified the passes.
Figure 6 -a) Raw hyperspectral data strip, b) 3 orthorectified passes overlaid on a visible orthophoto.

Orientation data and boresight corrections
As described in Section 2.6 the MVC can provide an alternative source of sensor orientation data.It can be seen in Figure 7 that the roll record by the IMU and calculated based on MVC imagery are similar but with a small offset.Thus it was considered that using MVC based orientation data was a viable alternative, but it is important to assess the accuracy of pass processed with MVC based orientation as comparted to IMU based orientation.Also of interest is what frequency of MVC data is required, computing camera orientation data from 15 Hz data can require a up to 6 hours of computing time for a set of three passes, whereas reducing the dataset to 1 Hz images reduces processing time to less than an hour.The offset that can be seen in Figure 7 is likely due to a small misalignment (known as a boresight difference) between the IMU and the MVC.However, there is also a boresight correction needed for the misalignment between each of these sensors and the hyperspectral line scanner.As described in Section 2.7, automatically generated GCPs were used to calculate the required boresight  for each type of orientation data and across two test passes.

Geometric accuracy
For Pass 3 there are 4 check points covered in the scene, and for Pass 4 there are 6 check points covered.To assess the accuracy for a geo-rectified hyperspectral pass, the positional error for each check point (see Figure 8) is measured and an RMSE calculated.The results of the passes tested with different sources of orientation data can be seen in Table 4.By way of comparison we also used PARGE to georectify push broom imagery collected by an alternative sensor that comes as a turnkey solution.That is, a system that has its own built in data logging and synchronisation system.It is however, based on a low accuracy (navigation grade) GPS and it is therefore of interest how well it performed.An example of the georectification of a pass collected by the nano-hyperspec and be seen in Figure 9.There were 6 GCPs located within this scene, the overall RMSE was 5.5313 m.
Figure 9 -Orthorectified hyperspectral pass from nanohyperspec system over an agricultural field in the Kingdom of Saudi Arabia

DISCUSSION
The accuracy assessment of the data processed in this study shows clearly that the boresight correction has significantly improved the geometric accuracy of the orthorectified passes.When using the IMU as the data source for sensor orientation the improvement in accuracy for both passes was around 50-80%.The accuracies for all the boresight corrected combinations are 3 -6 cm (see Table 4) which is around the same order of the accuracy of the RTK measured GCPs (2-4 cm) that are being used for the accuracy assessment.Essentially the accuracy of the boresight corrected passes is now the same as the GCP accuracy.
Replacing the IMU as the orientation data source with sensor orientations calculated from the machine vision camera has been shown to be a viable alternative.As the MVC data was degraded from 15 Hz to only 1 Hz the accuracy of the orthorectified passes did not suffer.Pass 3 did have one inconsistent result for the 3 Hz MVC data, however, this pass had only four GCPs, two of which are on the edge of the flight strip which may be effecting the accuracy assessment.
When looking at the boresight correction angles (see Table 3) it can be seen that the pitch and roll correction angles for pass 4 are quite consistent, remembering the boresight angles will be different for the MVC to the IMU.However, there is less consistency with pass 3.At this time the reason for this is not known, more data is required and processing the remaining passes from this dataset may help explain these differences.
The study undertaken by Suomalainen et al. (2014) has many similarities to this project in that we both used SfM to obtain sensors orientation.In their study Suomalainen et al. (2014) had difficulty processing the strips of images collected during a single pass to obtain sensor orientation, we have overcome this problem by processing multiple overlapping strips of image (see Figure 5) and thus producing a more reliable SfM solution.Suomalainen et al. (2014) identified that a higher accuracy GPS would be needed for accurate direct geo-referencing and we have a

b) a)
DGPS onboard our airframe.One critical difference is the fact that we have our sensor payload attached to a stabilising gimbal, whereas Suomalainen et al. (2014) have their sensors hard mounted to the airframe.MVC camera imagery collected at 1 Hz is quiet slow when compared to the hyperspectral scan line data that is collected at up to 100 Hz in this study.The ability to use this low frequency orientation data and still achieve accurate results is due the gimbal that is keeping the sensor payload pointing nadir and not transmitting every little movement made by the airframe into the sensor data.In a similar manner, Suomalainen et al. (2014) hypothesised that for determining heading from SfM a low frame rate (0.5 Hz in their case) was sufficient.
These preliminary results indicate that it may be possible to replace the IMU with a MVC in a UAS based hyperspectral system.A MVC in combination with a sufficient amount of GCPs allows the creation of an accurate DSM (required for the orthorectification process) and a reference orthophoto.The reference photo is used to automatically increase the accuracy of orthorectified hyperspectral passes.However, further validation of this hypothesis will require more data, in the first place the remainder of passes from the University oval will need to be processed to verify this assertion.In the longer term a larger test field with a more varied topography will be required to ensure a system without an IMU is a robust solution for accurately orthorectifying push broom hyperspectral imagery.
The need for an accurate on board GPS is clearly demonstrated when looking at the RMSE achieved when orthorectifying the nano-hyperspec imagery.The low accuracy positional data leads to an inaccurate orthorectified hyperspectral pass with an RMSE of more than 5 m.Further work on this system will look at using an accurate reference orthophoto and the PARGE software to attempt to automatically detect control points and thus interpolate a more accurate flight path.This could then lead to an orthorectified hyperspectral pass with a higher geometric accuracy.

CONCLUSIONS
In this study we have demonstrated that a UAS carrying a hyperspectral push broom sensor can be used to create accurate (~ 5 cm RMSE) orthorectified image strips.The PARGE software package was used to process the push broom line data.Factors critical to the accuracy of the orthorectified imagery are an accurate GPS to record sensor position, synchronisation of this position with the image data collected (both MVC and hyperspectral lines), an accurate lever arm correction, and using a reference image to correct for the boresight alignment between the orientation sensor and the hyperspectral imager.
The preliminary experiments carried out in this study have demonstrated that it is possible to use a MVC and SFM processing techniques as an alternative source of sensor orientation data.Initial tests have shown that MVC sensor orientation data can produce orthorectified hyperspectral passes with a similar geometric accuracy as passes that were processed using IMU based orientation data.
Results from processing hyperspectral imagery based on positional data from a low accuracy GPS demonstrate that a high accuracy GPS is essential for accurate orthorectification.To validate the hypothesis that a high accuracy GPS, coupled with a MVC will provide sufficient sensor positional and orientation data for orthorectification it will be necessary to process more data.Initially from these datasets and then later from new datasets that have a more variable topography.

Figure 1 -
Figure 1 -Layout of geometric test field, cricket oval, University of Tasmania.The second site flown was in the Kingdom of Saudi Arabia over a centre pivot irrigated agricultural crop at a locality called Todhia, near the township of Al Kharj.This site contained 20 GCPs that were measured with RTK GPS (see Figure2).

Figure 2 -
Figure 2 -Test site showing layout of GCPs at Todhia Farm, Kingdom of Saudi Arabia

Figure 3 -
Figure 3 -Matrice 600 airframe with sensor payload, yellow lines indicate the lever arm correction required.

Figure 5 -
Figure 5 -Screenshot for Photoscan showing the alignment of 3 passes of image data and the sparse pointcloud that was created.

Figure 7 -
Figure 7 -Data from a single pass comparing the roll measured by the IMU and the roll as calculated from the MVC camera data.

Figure 8
Figure 8 -a) Positional error of GCP before boresight correction and b) Error after boresight correction applied Pass number

Table 1 -
List of sensors carried by each airframe

Table 3 -
Table 3 for the two passes assessed in this study.List of calculated boresight corrections found