HIGH ACCURACY DIRECT GEOREFERENCING OF THE ALTUM MULTI-SPECTRAL UAV CAMERA AND ITS APPLICATION TO HIGH THROUGHPUT PLANT PHENOTYPING

With the appearance of cost effective, easy to fly Unmanned Aerial Vehicles (UAV), a new type of data collection has been enabled: super high resolution multi-spectral, precisely georeferenced imagery and point clouds, collected over high value targets. The high spatial resolution and precise georeferencing accuracy makes information extraction and advanced analytics possible both in the spatial and temporal domain at scales simply not possible to collect from manned aircraft, and at much greater efficiency than can be collected from the ground. One example of this is plant phenotyping for experimental research where a high-accuracy spatial reference needs to be assigned to each plot entry to enable accurate and efficient plot level statistics of plant phenotypic attributes. This paper presents results from an integration of the Trimble APX-15-EI UAV Direct Georeferencing system with the Micasense Altum multi-spectral camera to produce a highly accurate and efficient UAV based mapping solution for advanced spatial and temporal analytics without the use of Ground Control Points (GCP’s). Results from a series of flights over a test range outfitted with GNSS surveyed check points show an orthomap accuracy at the level of 3 cm RMSx,y horizontal can be achieved. The same system flown over a test field operated by researchers at the University of Guelph containing plots of soybean demonstrated pixel-level alignment of the directly georeferenced orthomosaic to the cm-level plot boundaries previously surveyed by the researchers, thus meeting the requirements for automated phenotyping.


INTRODUCTION
Direct Georeferencing (DG) of airborne imagers using GNSS-Aided Inertial systems as an alternative or complement to Aerial Triangulation (AT) has been a standard for manned applications since the early 2000's (Cramer et al. 2000, Hutton et al., 2005. More recently it has also been adopted for use with small and medium format cameras and LiDAR on Unmanned Aerial Vehicle (UAV) platforms (Mian et al, 2015). DG brings several advantages to airborne mapping that include eliminating the need to use Ground Control Points (GCP's) to do georeferencing, reducing sidelap and endlap (or even no sidelap or endlap) requirements, and sensor fusion of multiple imaging payloads in a single platform (such as a LiDAR with a camera). Each of these greatly reduces the cost and increases the overall efficiency of airborne mapping (Mostafa et al., 2001).
UAV platforms provide a new and powerful method of surveying using GNSS. Instead of being limited to the ground and measuring discrete points with a GNSS receiver operated by a human, a UAV with a Direct Georeferencing system moves the GNSS receiver into the sky to a vantage point where it can measure multiple points at once using an imager such as a LiDAR or a camera. With a UAV, GNSS becomes a much more viable and cost effective method to accurately georeference data, enabling activities such as temporal analysis to be conducted for applications where GNSS collected from the ground might not be practical or simply too costly. * Corresponding author One such application is plant phenotyping for experimental research. Plant phenotyping is generally defined as the quantitative assessment of specific characteristics and traits of plants such as: growth, yield, stress, development, maturation, tolerance, resistance and biomass (Pieruschka et al., 2019).
An important part of this assessment is how these characteristics and traits change over time. While there are many ways of performing plant phenotyping including destructive and nondestructive means, remote sensing using images collected from satellites or airplanes is a popular method of choice due to its productivity.
However, there are limitations to this approach, primarily the resolution of the imagery may not be sufficient to identify plot boundaries, and accurate georeferencing is required to perform objective temporal analysis. Data collected from UAV's can solve the issue of resolution, and the use of high accuracy georeferencing of the imagery allows for precise, repeatable alignment with pre-surveyed plot boundaries (Bruce et al. 2020, Khan, et al., 2019. This paper investigates the use of Direct Georoeferencing on a UAV based multi-spectral imager to efficiently achieve the cm level positioning accuracy required to align the imagery with presurveyed plot boundaries, all without the use of GCP's.

Trimble APX-15-EI UAV
The Trimble APX-15-EI UAV manufactured by Applanix, is a high-performance GNSS-Aided Inertial system designed specifically for Direct Georeferencing on UAV's. It is comprised of a single board GNSS-Inertial module with dual inertial measurement units (IMU) and POSPac UAV post-processing software. One IMU is embedded onto the main board (onboard IMU), while the second IMU is remote (remote IMU) and can be mounted externally onto an imaging sensor. A key functionality of the APX-15-EI UAV is the computation of the position of the imaging sensor origin (ie the camera perspective centre or LiDAR origin). This is achieved by using the orientation computed by the IMU and the fixed lever arm (3D offsets) from the sensor origin to the GNSS antenna phase centre (APC) to translate the GNSS computed position to the sensor origin.
However, when the sensor is mounted on a gimballed mount, which performs active stabilization to keep a sensor heading along the flight path and pointed down, two sets of orientation are required to do the translation: the orientation of the airframe and the orientation of the mount itself. By mounting the APX GNSS-IMU board on the UAV airframe along with the GNSS antenna, and the remote IMU on the sensor attached to the gimbal, the APX-15-EI computes both sets orientation required to do the lever arm translation.
The computed position of the sensor origin is as follows (1): where: = position of sensor origin in mapping frame, = position of GNSS APC in mapping frame, R = rotation matrix from vehicle frame to mapping frame computed by onboard IMU, − = lever arm from gimbal centre of rotation to GNSS APC in vehicle frame, R = rotation matrix from sensor frame to mapping frame computed by remote IMU, and − = lever arm from center of rotation to sensor origin in sensor frame.
The POSPac UAV post-processing software provides the following functions: • Generation of centimetre accuracy combined forward and reverse GNSS-Aided Inertial position and orientation of the sensor using GNSS augmentation data • Calibration of IMU to camera and IMU to LiDAR boresight angles • Calibration of Camera interior orientation • Computation of Exterior Orientation for each image or LiDAR point cloud in local mapping frame and datum.
Single GNSS base station 2.
Post-processed Trimble Centerpoint RTX (PP-RTX) For small local area collects, setting out a GNSS reference station like the Trimble SmartTarget base station is a simple way to achieve cm level accuracy.

Figure 2: Trimble SmartTarget Base Station
The Applanix SmartBase module uses existing continuously operated (CORS) networks that are automatically downloaded to generate a VRS without the need to set out a local base station.
The PP-RTX service enables cm level position without the use of base stations, and is ideal for areas that do not have CORS coverage (Mian et al, 2019)

Micasense Altum Multi-spectral Imager
The Altum is a small, high performance combined multi-spectral and thermal imager designed for UAV mapping applications. It collects imagery in 5 spectral bands (blue, green, red edge and near infrared), along with LWIR thermal, yet with a size of only 8.2 cm x 6.7 cm x 6.45 cm and weight of only 407 grams, the Altum is small enough to fit on even the smallest of UAV's. With a field of view of 48 deg x 37 deg for the multispectral channels, the Altum can collect imagery with a Ground Sample Distance (GSD) of 5.2 cm at 120 m height above ground, making it ideal for supporting plant phenotyping. The Altum also uses a global shutter and supports up to a 1 second capture rate ensuring crisp and aligned imagery.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition)

Integration on DJI M600 and Ronin Mount
The APX-15-EI UAV and Altum imager were integrated onto a DJI M600 test platform using a Ronin 3 axis stabilized mount. The APX-15-EI board set was mounted to the M600 airframe and the external IMU was attached to the Altum sensor that was then attached to the Ronin mount.
Also mounted on the airframe was the survey grade, compact GNSS antenna supplied with the APX-15-EI (AV-18 antenna).
The antenna was mounted to be the highest point on the airframe, well above any sources of potential interference. The lever arm from the Ronin mount centre of rotation to the external IMU and to the GNSS antenna were measured and entered into the system, along with the lever arm from the onboard IMU to the GNSS antenna. Lever arms were measured to better than 1 cm accuracy.
A key requirement to use DG is the precise time alignment of the image exposures with the GNSS-Aided Inertial solution, in order to accurately interpolate the high rate position and orientation solution to the image times. This was achieved by connecting the mid-exposure pulse from the Altum as an event input to the APX-15-EI. For each image the APX-15-EI was then able to record the exact time of exposure in GPS time.

Test Description
On July 31 2019, Applanix performed two flights of the M600 with Altum and APX-15 EI payload (Flight1 and Flight 2) over the Applanix test field. The test field is located approximately 60 km North-east of Toronto, and is comprised of an area of approximately 300m x 400m with 15 surveyed GCP's. The GCP's were surveyed using carrier phase differential GNSS with respect to the ITRF00 reference frame and UTM projection. Accuracy of the GCP's is estimated to be better than 2 cm in each component.
The test field contains a permanent GNSS base station that is used for POSPac post-processing. For the tests, a Trimble SmartTarget Base station was also set out.

Figure 4: Applanix Test Field Showing GCP Layout
The two flights were flown at a height above ground of approximately 100 meters, producing a GSD of approximately 4 cm in the imagery.
The flights were comprised of 8 East-West strips overlaid with 3 North-South strips, and were flown with 60% endlap and 40% sidelap.

Processing Methodology
The raw data logged by the APX-15-EI for Flight 1 were processed in POSPac UAV along with the data recorded by the dedicated base station to produce a 200 Hz navigation solution with position accuracy at the cm level.
This solution was then interpolated to the recoded mid-exposure times and transformed into the Exterior Orientation (EO) for each image with respect to the ITRF00 datum and UTM projection.
The EO and Altum Band 1 images were processed in the POSPac CalQC module to solve for the IMU boresight angles and to calibrate the camera ( Figure 6). As a first step, CalQC runs point matching on the imagery to generate pass and tie points using the a-priori EO to speed up and simplify the matching process.
A least squares bundle adjustment is then run using the matched points, EO, and GCP's set as control to solve for the IMU boresight and basic camera parameters such as focal length and principle point offset. The calibration results from Flight 1 are summarized in Table 1.

Accuracy Assessment
The data from Flight 2 were then processed in POSPac UAV using the dedicated base station to generate the EO, which was then loaded into CalQC along with the imagery.
This time the IMU boresight angles and camera model computed from Flight 1 were entered and held fixed (locked), and the GCP's were used as checkpoints to validate the calibration ( Table  2). The checkpoint residuals were 3.2 cm and 4.5 cm RMS horizontally (approximately 1 pixel), and 11 cm RMS vertically (approximately 3 pixels). The EO generated for Flight 2 with the IMU boresight applied was then loaded into the Agisoft Photoscan software along with the Altum images (Figure 7) to generate a self-extract DEM and orthomosaic at full resolution. The orthomosaic was then loaded into Global Mapper and the checkpoints were measured against their known values, producing residuals within 3 cm RMSx,y, as shown in Table 3.
Note that the number of checkpoints assessed in the orthomosaic was limited to those that could be clearly identified (listed in Table 3), resulting in fewer points being measured than in CalQC (Table 2).

Residuals (m) Point ID
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition)

Figure 8: Location of SmartTarget and GCP Tile
The flights were flown at an above ground altitude of 60 m resulting an approximate GSD of 2.6 cm, and with an endlap and sidelap of approximately 60%.
Due to extreme wind conditions, the Ronin gimbal mount malfunctioned on the first flight preventing useful data from being collected. However, for the second flight the mount functioned correctly and proper data was collected.

Accuracy Evaluation Against GCP's
Using the calibration obtained from the previous flight tests over the calibration field, POSPac UAV was used to generate the EO for each image, this time in the NAD83 datum and UTM projection to be consistent with the GCP coordinates provided by the University. The EO along with the images were loaded into Agisoft Photoscan where a self-extracted DEM and full resolution orthomosaic were generated without the use of the GCP's.
The DEM and orthomosaic were then loaded into Global Mapper (Figures 9 and 10), and the GCP's were measured to produce the accuracy statistics shown in Table 4. For this flight the RMSE of the residuals ranged from 3 cm to 6 cm horizontally (approximately 1 -2 pixels), and 11 cm vertically (approximately 4 pixels).

Evaluation Against Pre-Surveyed Plot Boundaries
The Soybean field was planted into individual research plots on June 8, 2019. The plot sizes were 1.4 m in length and 5 m in width, with a 35 cm separation. During planting a Trimble TMX-2050 display with built in RTK GNSS was used to record the spatial positions of the plots to within 1 -3 cm accuracy, from which a plot shape file was produced by researchers at the University of Guelph (Bruce et. al, 2020).
The orthomosaic produced from the Altum sensor and the APX-15-EI UAV was provided to the researchers who overlaid the plot boundaries onto the mosaic. A visual inspection showed accurate alignment of the crops identified in the orthomosaic with the plot boundaries to within a few pixels, easily allowing the plants in a given plot to be correctly identified (Figures 11 and 12).

CONCLUSIONS
The Micasense Altum integrated with the Trimble APX-15-EI UAV Direct Georeferencing system on a DJI M600 UAV can consistently produce georeferenced orthomosaics without the use of GPC's to an absolute accuracy level of a few pixels. Flight tests over a soybean research field showed pre-surveyed plot boundaries using RTK could be overlaid on the orthomosaic to an accuracy sufficient to identify the correct plots and plants, thus enabling faster, lower cost and more efficient plant phenotyping.
Furthermore, the accuracy of the DEM extracted from the Altum imagery is at the level of 11 cm RMS, which is more than adequate for many types of research requiring height information.
The results prove that the MicaSense Altum multi-spectral UAV Camera integrated with the Trimble APX-15-EI UAV Direct Georeferencing solution can produce highly-accurate, repeatable large scale map products without the use of GCP's, ideal for applications such as plant phenotyping. The benefits of such a solution include: • better plot-level geometry for machine learning of phenotyping analysis, resulting in greater accuracy and efficiency • lower cost by removing the laborious processing of having to redefine the plot boundaries for every flight throughout the season • lower cost and faster results by eliminating the need to survey GCP's and reducing the image overlap requirements for georeferencing • more accurate height and volume information leading to better biophysical attributes of the plant canopy

FUTURE WORK
Future work will include using the Altum and APX-15-EI payload to perform repeated flights over the research plots to investigate automatic temporal analysis.