WHY THE HORIZON IS IMPORTANT FOR AIRBORNE SENSE AND AVOID APPLICATIONS

The utility of the horizon for airborne sense-and-avoid (ABSAA) applications is explored in this work. The horizon is a feature boundary across which an airborne scene can be separated into surface and sky and serves as a salient, heading-independent feature that may be mapped into an electro-optical sensor. The virtual horizon as established in this paper represents the horizon that would be seen assuming a featureless earth model and infinite visibility and is distinct from the apparent horizon in an imaging sensor or the pilot’s eye. For level flight, non-maneuvering collision course trajectories, it is expected that targets of interest will appear in close proximity to this virtual horizon. This paper presents a model for establishing the virtual horizon and its projection into a camera reference plane as part of the sensing element in an ABSAA system. Evaluation of the model was performed on a benchmark dataset of airborne collision geometries flown at the National Research Council (NRC) using the Cerberus camera array. The model was compared against ground truth flight test data collected using high accuracy inertial navigation systems aboard aircraft on several ’near-miss’ intercepts. The paper establishes the concept of ’virtual horizon proximity’ (VHP), the minimum distance from a detected target and the virtual horizon, and investigates the utility of using this metric as a means of rejecting false positive detections, and increasing range at first detection through the use of a region of interest (ROI) mask centred on the virtual horizon. The use of this horizon-centred ROI was shown to increase the range at first detection by an average factor of two, and was shown to reduce false positives for six popular feature detector algorithms applied across the suite of flight test imagery.


INTRODUCTION
The global market for small unmanned aircraft systems (UAS) is expected to grow significantly over the next decade [Mendelson, 2014].Building inspections, pipeline monitoring and aerial photography/videography production are among the success stories for commercial applications utilizing UAS, as they can be performed within visual line of sight of the pilot.It is more difficult to gain approval of flight operations beyond the visual line of sight (BVLOS), owing to the regulatory requirement to see and avoid conflicting air traffic.This requirement either restricts BV-LOS operations to segregated, or unpopular airspace (e.g.low level over the ocean, or arctic), or necessitates the use of a sense and avoid (SAA) system with an equivalent level of safety to the manned aircraft see and avoid principle.Regulatory bodies have yet to establish a consensus on the required performance standard for a sense and avoid system, however several groups are currently working on solutions to this issue, with the most recent developments revolving around a specified target level of safety that is comparable to that of the current mid-air collision rate for manned aircraft.Despite the lack of performance standards, the SAA problem has been the subject of a vast amount of academic, commercial and regulatory activity and still remains an open problem for operations in civil airspace.

BACKGROUND
An airborne SAA (ABSAA) system based on electro-optical (EO) or infrared (IR) cameras must transform a sequence of images into an accurate and reliable estimate of collision-course targets.The standard approach to the problem is a detection and tracking pipeline consisting of the extraction of 'good' features, followed by tracking of those features across the image sequence, and finally, applying target discrimination criteria to remove false positives.* Correspondence author

Electro-Optical SAA Techniques
Vision-based approaches to solve SAA, utilizing a variety of image processing and machine learning techniques, are extensively explored in literature.Detection and extraction of actual airborne collision-course targets has been presented in [Fasano et al., 2008,Dey et al., 2009,Dey et al., 2011,Lai et al., 2013,Salazar et al., 2013, Fasano et al., 2014].The authors in [Dey et al., 2009,Dey et al., 2011] present a novel approach for aircraft detection incorporating a processing pipeline utilizing trained classifiers.The approach was tested against airborne targets using multiple sensor and lens configurations with relative success.However the system was affixed to the ground and observed a fixed portion of the sky.As such, it was completely absent of ground clutter and platform vibration.Further testing is needed to ensure that it will extend to the ABSAA paradigm.A similar classification approach was employed in [Carnie et al., 2006].The authors in [Fasano et al., 2008, Fasano et al., 2014] conducted airborne collision trials with fused sensor methodologies (EO/IR, EO/Radar) and tested target detection and tracking algorithms in real-time with encouraging results.The work in [Salazar et al., 2013] describes an ABSAA system fusing EO, IR, millimeterwave radar and laser radar sensing modalities as well as avoidance algorithms utilizing a minimum-distance bubble.The algorithms were tested in a simulation environment and noise models were introduced to approximate sensor data in a more realistic fashion.The work in [Lai et al., 2013] presents airborne flight tests with an EO-only system.The authors present an image processing pipeline utilizing a morphology-based feature detector.The use of morphology for target detection was tested by the National Research Council (NRC) against the NRC dataset, but was found to be prone to false detections, especially in the presence of ground clutter.Use of a weighted ROI mask such as the one described in this paper to localize the target should dramatically improve performance.The authors in [Forlenza et al., 2012] described an air-to-ground target detection approach implemented in hardware on an unmanned helicopter.They compared the performance of Harris and Shi-Tomasi feature detectors against synthetic targets.Here, we present the results of those corner detectors, among others, against real targets acquired from airborne near-collision flights.

MODEL
This section describes the theoretical horizon model and its projection, defined as the "virtual horizon" into an electro-optical sense calibrated with respect to the platform INS.The accuracy of the projection is a function of the fidelity of the model, and the precision of the calibration and the INS.

Inertial Reference Horizon
The theoretical horizon defines an omnipresent feature that may be used for image stabilization, attitude registration and target localization.Note that it is distinct from a 'horizon' that may be visible in the image.In forward-facing imagery acquired at general aviation altitudes under clear sky assumptions, the horizon is routinely observed to possess a smooth gradient rather than a sharp discontinuity near the ground-sky boundary.This 'gray' region is a direct consequence of atmospheric effects and illustrates the complexity of extracting the horizon contour using purely image processing techniques.Without an obvious boundary, image processing techniques may be influenced by scene structure and cloud cover to detect a false horizon contour, and are not considered as reliable as a theoretical horizon computed from the platform INS measurements.Previous efforts based on machine learning techniques extracted the contour defined by the groundhorizon boundary [Minwalla et al., 2011].Such a contour can reveal geological and man-made topological features that a theoretical horizon will not capture, and may prove useful when flying at low altitudes in a city or near rocky terrain.The approach described in this paper does not tackle those issues and is more suited for general aviation altitudes at 1000 feet or above over roughly level terrain, where the horizon is distant and indistinct.

Target Behaviour Under ABSAA
A typical intruder for ABSAA is an aircraft encountered in nonsegregated airspace that will spend the majority of flight time under non-maneuvering and non-accelerating conditions.Such aircraft are likely to appear at or near the horizon.Targets that appear above the virtual horizon are sky-lit and are expected to have greater contrast than those that appear at or below the virtual horizon.Targets that appear below the virtual horizon are difficult to distinguish from ground clutter, and prove to be the most challenging targets.However, for all oncoming cases where one aircraft appears below the horizon, the other aircraft will observe the host as a high-contrast sky-lit target above the horizon.This is not the case in descending overtake scenarios, where the aircraft being overtaken cannot see the overtaking aircraft due to a limited cockpit field of view.This suggests that descending overtake scenarios are the most difficult to detect.Looking at these non-co-altitude cases will form part of future work.

Derivation
We start the description of the model with the assumption that the Earth is an ellipsoid devoid of topological features.At a given platform altitude, the distance to the horizon (Dp) is defined by a line from the aircraft centre of gravity to a point tangent to the ellipsoid.This line forms a right-angled triangle between the aircraft centre, the tangent point and the centre of the Earth.Fig. 1(a) depicts the direct linear distance to the horizon as Dp, while Dsp depicts the corresponding ground-track distance and Ap is the platform local above ground level (AGL).The INS reference frame origin is placed at the aircraft's centre of gravity.
Ap and the local earth radius, Rp, are the only two quantities required to ascertain Dp, Dsp and the declination angle (θp) (Eq.1).
(1) Note that atmospheric scattering will limit the visibility to well within Dp, referred to as D p .By contrast, atmospheric refraction will extend the distance to beyond Dp, referred to as D p .Therefore, D p < Dp < D p .Since atmospheric scattering is the limiting factor, the apparent horizon in real, forward-facing imagery is typically closer, at D p , and by extension, positioned lower in pixel coordinates.Note that Dsp can be treated similarly.
If perceived from a birds-eye configuration, the horizon projects a circle of radius Dsp on the earth's ellipsoidal surface, centred at the aircraft position (Fig. 1(b)).Assume that the position, Xp = [Xp, Yp, Zp, 1] and the attitude, [θp, φp, ψp], of the host platform are known in a local Cartesian coordinate system.One such Cartesian system is the local Universal Transverse Mercator (UTM).Since the radius of the circle, Dsp, is strictly a function of the platform altitude, points on the horizon circle (X h ) are relative to the platform position and can be represented in the host aircraft inertial reference frame (Eq.2).
, where α = (−π, π) (2) Here, ψp denotes the heading of the aircraft.The azimuth, α, spans the full circle with the plane of the horizon.Any camera model may be used to project body axis coordinates into image space.For convenience, we use the pinhole camera model in our work.Although not explicitly explored, this model may be augmented for distortion correction if necessary.Eq. 3 denotes the projection matrix, where K is the intrinsic matrix, [Rc| − Rctc] the fixed calibration between the camera and the aircraft INS, and [Rv| − Rvtv] the INS to world transformation.
x = PX, where The world horizon points may be projected into the camera plane as per Eq.4: The horizon circle may be sampled at equidistant intervals of α and projected into image coordinates.The points of interest, denoted by x h (Eq.4), are those that possess valid image coordinates and are in front of the camera, the latter property defined by a positive magnitude in w.Note that nrows and ncols refers to the row and column pixel counts of the focal plane array respectively.At least two suitable points are needed, therefore the sampling interval for α, denoted as τα must be at least F OV ≥ 2τα where F OV is the camera's field of view.
Note that this solution is applicable regardless of the camera orientation and is readily expanded to multiple cameras, independently or serially calibrated to the aircraft INS reference frame.

Virtual Horizon Proximity (VHP) -∆Htgt
Given the projection of a virtual horizon line in image coordinates, the virtual horizon proximity (VHP) is defined as the distance between a potential target and the virtual horizon line.The VHP concept exploits a characteristic unique to level-flight, collision course targets, namely their quasi-constant separation from the horizon for the duration of the course.This cue is strictly a function of the altitude difference between the host and intruder platforms and is invariant to the host and target attitudes.Given the horizon line, ∆Htgt may be computed by the distance equation (Eq.5), where xt is the target point, x h 1 and x h 2 are two points from the projected horizon line.
To validate the VHP hypothesis, the measured intruder proximity (Htgt) must be compared to a ground-truth estimate from known host and intruder positions.This estimate can be generated as follows: Let us assume a plane parallel to the camera plane located at the intruder that defines the background.As the intruder is much closer than the theoretical horizon, the distant horizon line must be projected onto this background plane coincident with the intruder.Retaining the ellipsoid assumption, let Hp = Rp + Ap and Hi = Ri + Ai, where Rp and Ap are as before, Ai is the intruder altitude and Ri is the local ellipsoidal radius at the intruder.Then the ground-truth proximity in radians, ∆ Ĥtgt, can be derived via similar triangles and the cosine law (Eq.6), Here, Dp is the distance from the platform to the horizon (from Eq. 1) and Di the distance between the host and the intruder.This Ĥtgt model is utilized for ground-truth estimates in Section 5..Note that Dp >> Di for general aviation altitudes.For instance, at an altitude of 300 m (Ap = 1000 ft), distance to the horizon is ≈ 60 km.Furthermore, the projective relationship is reasonable at long ranges, where Di >> f and f is the lens focal length.At close ranges, the assumptions begin to break down and parallax effects limit the efficacy of this technique.

Horizon-weighted Region-of-Interest Mask
The primary purpose of a region-of-interest (ROI) mask is to reduce clutter and false positives by isolating the pixels where the target of interest is likely to appear.A typical ROI is a rectangular crop with a step function weighting, such that pixels outside the region are eliminated.Although the rectangular ROI is fast to compute, it has the unfortunate side-effect of culling all targets that may occur outside the expected region of interest.We propose the use of a horizon-weighted ROI mask as a target likelihood estimator.This weighted ROI can be implemented as a spatial probability density function (PDF) centred at the virtual horizon and applied perpendicularly to the direction of the horizon line.
The concept of this idea arises naturally from biological vision and the likelihood of expected/typical level-flight collision course targets, as pixel intensities are attenuated based on their distance from the ROI centre.By not discarding the pixels beyond the ROI bounds, the ability to detect a strong target in unlikely locations of the image is retained.At the same time, strong false

Airborne Flight Tests
Flight tests were flown with a Bell 205 helicopter acting as a surrogate UAS (Fig. 3(a)) [Ellis and Gubbels, 2005].A Bell 206 rotorcraft (Fig. 3(b)) and a Harvard Mark IV trainer fixed-wing aircraft (Fig. 3(c)) acted as the intruders.Flights were flown in controlled airspace near the Flight Research Laboratory at Ottawa International Airport (CYOW).Common test conditions consisted of visual meteorological conditions (VMC) at high visibility in excess of 15 statute miles (24 km).Tests were conducted in the morning and afternoon to encounter different solar zenith angles.Each sortie consisted of 6-10 collision geometries, where each collision geometry was flown from ∼ 15 km separation to within 100 meters.Data was collected on head-on, azimuthal offset, chase, descending head-on and descending overtake trajectories.The aircraft were instrumented with an automatic dependent surveillance-broadcast (ADS-B) transceiver and an INS [Leach et al., 2003].An "Intercept Display", parsing the on-board ADS-B

Instrument Configuration
A multi-camera array, dubbed Cerberus, was rigidly mounted to the port-side external mount of the Bell 205.Each imager consisted of a 5 megapixel monochrome sensor operating at 15 fps.Monochrome sensors were chosen to enhance focal plane resolution at the expense of colour fidelity.A fixed-focal-length lens was installed per camera.The images were streamed over Ethernet to a local rack-mount PC, where each frame was synchronized via GPS time-stamp to the precision INS on board the Bell 205 and recorded for offline analysis.The total instrument field-ofview nominally spanned 90-degrees from the forward look direction, with subsequent flights conducted with a variety of sensor and lens configurations.

Selection of Benchmark Data
Although a considerable number of trajectories were flown, only a selected subset of the data is presented for analysis and comparison.These benchmark datasets were chosen for their clarity of the imagery, similar sensor configuration and favourable scene conditions (high visibility, minimal cloud cover and occlusions).A total of four collision trajectories were chosen for analysis: three head-on and one azimuthal offset.Head-on collisions represent the scenario limiting critical problem-space, due to a maximal closure rate and a minimal observable intruder footprint.All frames of interest were acquired with the same forwardfacing camera to correlate the sensor noise characteristics across datasets.Specific parameters for the flight tests are outlined in

Temporal VHP -∆Htgt(t)
The target VHP measured over a sequence of images is expected to indicate co-altitude collision-course behaviour.Fig. 4 presents the behaviour of a MATLAB model simulating level -flight collision -course intruders approaching a host UAS at various altitude differentials.The abscissa denotes range while the ordinate depicts the VHP as a function of range.Here, range is used interchangeably with time by assuming a constant closing velocity.It can be observed that the absolute VHP is directly correlated to the altitude differential, with larger VHP indicating larger corrections.Furthermore, as the intruder approaches the host, the VHP grows non-linearly for all cases except the co-altitude one.This indicates that a small absolute VHP value and minimal temporal variation are both indicators of collision-course behaviour.
To experimentally verify this phenomenon, we compare the measured VHP (Htgt) to the ground-truth VHP ( Ĥtgt) derived in Eq. 6. Recall that the ground-truth VHP is the height differential extracted from the INS data recorded on-board host and intruder aircraft.Fig. 5 contains the VHP plots acquired from the July and March datasets respectively.In each curve, the abscissa is the range and the ordinate is the proximity to the projected horizon measured in degrees.Points represent the per-frame VHP (Eq.5) converted into degrees via the known camera scale factor.A trend-line is depicted to enhance readability.The dotted grey line illustrates the modeled proximity derived in Section 3.4, with error bars indicating the ±2σ = 0.1 deg error in the on-board INS attitude measurements of the host and intruders [Leach et al., 2003].
It is observed that the measured VHP tracks the model consistently for the duration of the flight geometry in all four plots.Plots for both intruder types are on track, demonstrating the robustness of the cue against variations in the intruder radiance profile.In addition, the heading offset has no impact on the measurement (Figs. .It is speculated that these perturbations are tied to INS drift correction, suggesting that optical methods could be utilized to augment attitude measurements.Once again, the observed effect is minor and the peak-peak variation remains well within the 2σ bounds of the INS error. Note that the deviation between measured and modeled proximity curves will increase at close ranges due to parallax.A breakdown of the assumption is visible in Fig. 5(d) but not obvious in the other plots.Table 5.1 summarizes the plot results.Note that the absolute deviation is miniscule, with the maximum peak-to-peak variation limited to 0.51 degrees (observed in March Head-On 1).The maximum deviation from the model is 0.35 degrees observed in March Head-On 2 (Fig. 5(b)).For all intents and purposes, ∆Htgt(R) is roughly constant.In summary, ∆Htgt(R) is a quasi-constant cue for collision-course targets, consistent between trajectories and is robust against changes in the intruder type, radiance profile and background clutter.

Comparison to False Positives
To observe the proximity effect, we compare the temporal VHP of the known, true target against other false positives detected in the scene.Target detection and tracking was implemented via algorithms developed locally at NRC.These algorithms rely on visual cues unique to collision course targets augmented by Kalman filtering for a robust, real-time estimation.intruders, one co-altitude and one not, is required to demonstrate the utility of temporal VPH in discriminating between a collisioncourse and a non-collision-course intruder.However, such data is not available yet and is relegated to future work.Note that the overlap of multiple targets at close range in Fig. 6(c) is an artifact of the track estimation algorithm currently under development and has little bearing on the analysis.

Detection Performance of Weighted ROI Mask
Since a region-of-interest limits the scene to areas of interest, it stands to reason that horizon-weighted ROI masks can lead to fewer false positives and consequently a greater range at earliest detection (R det ).Six popular feature detectors were compared to determine the impact of ROI weighting on R det .Seven feature detectors were identified in prior work [Tulpan et al., 2014], of which the following six were chosen for comparison: FAST [Rosten et al., 2010], Shi-Tomasi [Shi and Tomasi, 1994], Harris-Plessey [Harris and Stephens, 1988], MSER [Matas et al., 2004], SIFT [Lowe, 2004] and SURF [Bay et al., 2006].Detector implementations were sourced from the OpenCV library version 2.4.9 (64-bit) and interfaced into MATLAB via the mexopencv API [Yamaguchi, 2015].The actual range values are expected to be slightly different from prior work due to the differences in feature detector settings and the OpenCV version.Nonetheless, the overall trend is expected to be consistent.
Table 5.3 contains the results of this comparison.All four benchmark datasets were tested against the six algorithms.The number of features detected per frame was restricted to 50, and the ROI mask was chosen to be a Gaussian PDF at a half-width of 150 pixels centred at the virtual horizon line.Each element of the table denotes the range at earliest detection, R det , in units of kilometres, as observed for that dataset-algorithm-weighted/unweighted combination.The performance improvement is generally positive across all chosen algorithms with a weighted ROI.The effect is significant in some cases (July 10 cases where the algorithm could not pick up the true target at all.The highest incidence of failure was observed with MSER, suggesting that it is a remarkably poor candidate for airborne target detection.The SURF algorithm also failed for the March Headon 2 dataset.It is noted that use of the ROI mask shifted MSER performance from failure to marginal success in two out of four cases.

CONCLUSION AND FUTURE WORK
Presented here is the rationale that the horizon is an important feature for ABSAA applications.A model for the virtual horizon line was presented as a projection of the INS horizon into image coordinates.Proximity to the virtual horizon could then be calculated and denoted as the virtual horizon proximity (VHP).
The model was tested against simulation results and augmented by local Earth radius corrections for small-angle sensitivity.The predictions of the model were experimentally verified against a benchmark dataset of collision-course trajectories flown at the National Research Council's Flight Research Laboratory.An ROI mask, centred at the horizon, was explored as a mechanism for improving the earliest detection range, R det .This hypothesis was experimentally verified by measuring the performance of six popular feature detectors with and without the ROI mask.It was observed that the use of the ROI universally improved R det across all feature detectors and all datasets, sometimes by a factor of two, and in two cases, recovered targets where the feature detector failed in the unweighted case.
and Fazel Famili for their work on the original analysis.

Figure 1 :
Figure 1: Side and birds-eye views of the theoretical horizon on the Earth's surface.

Figure 2 :
Figure 2: An image from a head-on collision couse with the March 2012 dataset with and without a ROI applied.The Bell 206 acted as the intruder.The image dimensions are 2448 columns by 2050 rows for reference.The ROI mask was generated by a Gaussian PDF centred at the horizon with a half-width of 150 pixels.

Figure 3 :
Figure 3: Host and intruder aircraft used for airborne nearcollision flight tests.positivesfrom the scene clutter have a reduced impact.Fig.2(a) illustrates a raw capture during a head-on collision run.In Fig.2(b), a Gaussian PDF with a half-width of 150 pixels and centred at the horizon line, is applied as a weighting mask.The choice of function, bias and half-width are user-configurable parameters and can be tailored to the host platform's performance envelope, although care must be taken as too narrow an ROI or too steep a gradient may result in target drop-outs.

Figure 4 :
Figure 4: Modeled temporal VHP for collision course targets at different relative altitudes to the host platform.Nominal altitude was set to 1000 ft."Target Miss Distance" indicates the altitude differential in metres between the host and each intruder.data,provided range and navigation guidance to the flight crew for beyond visual line of sight operation[Keillor et al., 2011].
5(a) and 5(c)).However, two different head-on collision trajectories flown within a few minutes of each other can exhibit different temporal VHP profiles (Figs.5(b) and 5(d)).Periodic oscillations are present in the measured proximity values for July Head-on and March Head-on 1 (Figs.5(a) and 5(b)) that are not explained by the models, or visible in July Offset or March Head-on 2 geometries (Figs.5(c) and 5(d))

Fig. 5 .Figure 5 :
Fig.5.3 illustrates the proximity plots of the ground-truth target and four other highest-likelihood false positives.The abscissa denotes the range while the ordinate denotes the virtual horizon proximity (VHP) measured in pixel coordinates.The diamond markers show the true target while false positives are indicated by coloured lines.It is observed that all detected false positives are below the horizon, corresponding to false targets embedded in ground clutter.The true target has the minimum distance to the virtual horizon line (smallest | ∆Htgt |) and maintains quasistatic distance to the horizon line in all four datasets.However, it is observed that false targets also maintain a relatively constant separation from the horizon line.This may be partially explained by the difference in closing rates between the true target (fast) and ground clutter (slow).The results suggest that VHP is a useful discriminator of collision course targets from other false positives, and furthermore that the minimum absolute separation is a better discriminator than the temporal VHP.Testing against two

Table 4
imuthal angles relative to the aircraft nose.The overlap between adjacent cameras was nominally set at 5%.The March 2012 collision trajectories were flown against the Bell 206 as the intruder.In this particular dataset, the instrument configuration consisted of two cameras, a forward-looking (0 degree) camera equipped with a 23 mm lens and a 45 degree camera equipped with an 8 mm lens.The increased forward angular resolution translated

Table 1 :
Benchmark Datasets from July 2009 and March 2012

Table 3 :
Peak-Peak variation between predicted and measured virtual horizon proximity (VFP)

Table 4 :
Comparison of true target proximity to false positives tracked across the image sequence.The abscissa denotes the range while the ordinate denotes the virtual horizon proximity (VHP) measured in pixel coordinates.Top five targets are presented.The true target, indicated by diamond markers, is observed to have the smallest | ∆Htgt | and maintains quasi-static proximity to the virtual horizon in all four datasets.The false positives have a greater | ∆Htgt |, but are also quasi-static with thscene.The range at earliest detection, R det , with and without a weighted ROI Mask.Each row indicates an R det value kilometres for that feature detector.Columns indicate the datasets, evaluated with and without the ROI mask.A Gaussian PDF with a half-width of 150-pixels was used to generate the ROI.
• -offset and March Headon 1) and almost negligible in others (March Head-on 2).The degree of variability bears further investigation.The FAST algorithm performed the best and posted the greatest R det values in both unweighted and weighted categories and across all four datasets, which is consistent with earlier work.Dashes indicate