SIMULTANEOUS ORIENTATION AND CALIBRATION OF IMAGES AND LASER POINT CLOUDS WITH STRAIGHT SEGMENTS

This paper presents a method based on the use of straight lines as tie features for simultaneous orientation and calibration of Frame Camera (FC) and Airborne Laser Scanner (ALS) images. For the orientation and calibration of FC images, various methods have been investigated in the past. From those, FC Integrated Sensor Orientation (ISO) has established as the most accurate and robust one. More recently, ISO block adjustment techniques have been successfully applied to ALS orientation and calibration. As of today, for the combined ALS and FC image data sets there are no well-defined procedures as compared to the other two scenarios. Our proposed method essentially reduces to the use of straight line segments as tie features between FC and ALS images. The concept is similar and compatible to the use of tie points in FC ISO and of planar surfaces in ALS ISO thus allowing for the simultaneous orientation and calibration of ALS and FC images that involve, point, line and planar surface tie features. In the paper we describe and derive the two observation equations that relate (1) FC image point measurements to straight lines and (2) distance-parallelism measures between straight lines and planes. The concept and models are validated with a small FC and ALS block with real data. The preliminary results indicate the potential of the approach and confirm its feasibility.


INTRODUCTION
Orientation and calibration systems for photogrammetric and remote sensing instruments are fundamental components of modern mapping data processing systems.Orientation and calibration of frame and line cameras has been thoroughly investigated in the last decades.As a result, precise, accurate and reliable orientation and calibration procedures have been fielded and integrated in the photogrammetric and remote sensing production lines in companies and governmental agencies.With current INS, GNSS and image matching technologies, the best results are obtained with the well known Integrated Sensor Orientation (ISO) method.Typically, in addition to the ground and aerial control observations, in ISO, image measurements of ground pointsthe so-called tie points-are used.ISO, and in particular, sensor orientation and calibration with the bundle adjustment method using "tie straight lines" has also been proven to be feasible and efficient (Tommaselli, 1996, Habib et al., 2002).Analogously, though more recently, ISO has been proven to be the bestin terms of precision, accuracy and reliability-method for the orientation and calibration of ALS scenes (Friess, 2006, Kager, 2004, Skaloud and Lichti, 2006).These three methods use planar surfaces as tie and control features.
Today, many mapping projects combine photographic and laser scanner data.As an example, the Spanish "Plan Nacional de Ortofotografía Aérea" (PNOA) seeks to generate and/or update orthophotos and digital elevation models for Spain with an update/revisit cycle of two years with airborne digital cameras and laser scanners.Usually, there are discrepancies between the point clouds derived from the photographs and those derived by the laser scanner data.The data may originate from the same or different missions, but even in the case of simultaneous photographic and laser data acquisition, discrepancies remain.Thus, it is usually the case that, in a same airplane, two different GNSS antenna-GNSS receiver-IMU setups, one for the camera and one for the laser scanner, coexist.The navigation data may later on be processed with different software systems.This, together with the system and sensor calibration errors, ends up with inconsistent data sets that require manual editing and are suboptimal from the accuracy and productivity points of view.It is therefore necessary that methods be found and procedures be fielded to guarantee the correctness and mutual consistency of data derived from camera and laser scanner images.
We investigate the feasibility of combined orientation and calibration of camera and laser scanner images by identifying features in the object space that can be "camera-to-laser scanner" tie features.We envision an ISO method where points, planar surfaces and straight line segments link all types of images, from cameras and laser scanners.Co-registration between photographs and laser scanner point clouds using lines as common tie feature have already been investigated (Ronholm, 2011, Habib et al., 2005).(Habib et al., 2005) proposed two approaches for coregistration.The first approach uses the laser scanner straight lines as control lines in the photogrammetric ISO.The second approach derives a point cloud from photographs and then both point clouds are co-registered using straight lines.In this paper we focus on the orientation and calibration aspects rather than on the co-registration ones and, consequently, on the use of raw laser scanner range-angular data rather than on the point clouds.In the rest of the paper we use the terms "FC images" and "ALS images" when referring to photographic images captured with frame cameras and to range-angle data sets captured with airborne laser scanners, respectively.
The paper is organized as follows.Firstly, the main ideas behind the simultaneous network adjustment are introduced.Then, the proposed models as well as some mathematical concepts behind the uses of tie lines are explained in detail.Concept validation section presents the preliminary results of an adjustment obtained with real data.The last section summarizes the main conclusions of the proposed approach and discusses future improvements.

PROPOSED APPROACH
We propose a method where a simultaneous FC, ALS and FC-ALS ISO is performed.In it, the imaging observations are image coordinates measured on FC data (the digital FC images) and selected measurements of the ALS images (ranges and scanangles).The other observations are surveyed ground control points and aerial INS/GNSS-derived time-Position-Attitude (tPA) control.Ground control points can be used to derive indirect control observations like ground control lines and planes.The tie features are points (FC ISO), planar surfaces (ALS ISO) and straight line segments (FC-ALS ISO).Points, planes and lines are defined by 3, 3 and 4 independent parameters, respectively.
The classical FC ISO approach is well known.From the observations mentioned above, estimates for the tie points (TP), for the exterior orientation parameters of the FC images (EO), for the self-calibration parameters of the camera, for the INS/GNSS shifts and possibly for other system calibration parameters are computed.In ALS ISO, the unknown parameters of the tie planar bounded surfaces -tie planes (TPL)-are estimated together with INS/GNSS shifts, ALS self-calibration parameters and possibly other system calibration parameters.
In the proposed FC-ALS ISO method, as mentioned, straight line segments -Tie Lines (TL)-are used as tie features between FC and ALS images.TLs tie FC and ALS images as follows.Assume that s is a TL that can be recognized in the FC image c and in the point cloud of an ALS image a, and also assume that s is the intersection of two planar surfaces p1, p2 determined by points of the ALS image a.Given s, c, a, p1 and p2 we will then use three types of observation equations.The first type of observation equation (a coplanarity equation) relates image measurements of the s TL on the c FC image with s TL parameters, with the EO and self-calibration parameters of c.At least two of these observation equations are needed.The second type of observation equation (a line-in-plane relationship) compels s to lie on planes.In general, there will be two observation equations of this type, for p1 and p2, although it is also possible to use just one TPL.The third type of observation equation relates TPLsin this case p1 and p2to the measured ALS points, i.e., relate the ALS range and scan-angle measurements, to the ALS selfcalibration parameters, the INS/GNSS-derived tPA aerial control, the INS/GNSS shift parameters and possibly other system calibration ones.There will be, for each p1 and p2, as many of these observation equations as ALS points that define the plane.Details of the models are given in the next section.
In our combined FC-ALS ISO concept, ALS raw data instead of processed point clouds are used.3D coordinates of point-cloud points are not explicitly computed: the ALS points that define the TPLs are parameterized with raw range and scan-angle measurements.This is a key aspect of the concept as it allows for self-calibration of ALS measurements.Self-calibration of FC and ALS image observations is the key to correct and consistent results.
It is out of the scope of this paper to discuss the measuring (TLs and TPLs extraction) and matching techniques that precede the ISO network adjustment stage.We summarize them for the sake of completeness.TLs are extracted from FC images by combining edge detectors and the Hough transform (Vosselman et al., 2004).Once they are extracted, a matching procedure is performed to identify homologous TLs in different images.

MATHEMATICAL MODELS
In this section we discuss and/or introduce the observation equations that are more relevant to the proposed approach.Ground control and INS/GNSS-derived aerial control observations will not be discussed as they do not change from what is already known and published (Blázquez and Colomina, 2012).We begin by presenting some notation conventions and then develop the observation equations.

Some naming and notation conventions
The Coordinate Reference Frames (CRFs) involved in the mathematical functional models of the paper are detailed in Table 1.
The CRF l of a variable X is denoted by a superscript symbol like X l .For a rotation matrix R l b , the subscript indicates the origin CRF while the superscript indicates the final CRF so it can be written For the sake of simplicity, X e = (x, y, z) e is used instead of the formally correct X e = [(x, y, z) e ] T .

FC point collinearity observation equations
In FC ISO, the known collinearity observation equations are used to relate the FC image coordinate observations with the EO, tie point and self-calibration parameters.No modifications are required to use them in our method.

ALS plane observation equation
Again, no modifications are required to use the also known ALS plane observation equation (Skaloud and Lichti, 2006) in our method.In this equation, the observations are the two ALS image measurements (range and scan-angle) and the six tPA aerial control (position and attitude) ones.The ALS plane observation equation is where and where For an ALS point, eq. 1 enforces it to belong to a planar surface through a direct georeferencing implicit step (eq.3).The model extends the one proposed by (Skaloud and Lichti, 2006) with an additional scale factor self-calibration parameter for the scan-angle observation because our experience reveals its significance.The Hessian form of a plane is used to parametrize the planar surface.The plane is then characterized by a unit normal vector n l = (nx, ny, nz) l (eq.2) and d, the orthogonal distance between the plane and the CRF origin.rotation matrices from e to l and l to e CRFs, respectively n l 0 = (n0 x , n0 y , n0 z ) l constant approximation of the unit normal vector of tie plane P l 0 = (P0 x , P0 y , P0 z ) l local origin coordinates for tie plane.
Table 2: Symbols in the ALS plane observation equation.

3D straight line parameterization
Straight lines are elementary mathematical objects with many possible parameterizations.We propose yet another one which is convenient for numerical computations and whose four universal parameters (p, q, α, β) can be used regardless of the line location and direction.This is achieved with the help of, for each line, some auxiliary values: an auxiliary CRF origin P l L 0 close to the straight line and rough approximations α0, β0 of the straight line director vector.With this simple strategy, numerical singularities are avoided and a unique analytical formulation is used.
The point P l L 0 is taken, for instance as one of the points in the interest area, or the mean value of several points in this area.α0 and β0 are auxiliary constant angles so that the rotation R(α0, β0) = Rz(α0) • Ry(β0) brings the vector u = (1, 0, 0) to an approximation of the director vector of L (Figure 1).
The intersection point γ = (0, p, q) between the plane {X = 0} and the line is obtained by applying the previous translation and the inverse of the rotation R(α0, β0) to L. A point P l L of L can thus be obtained, for some µ as where (p, q, α, β) are the characteristic parameters of the straight line.The angles α, β use to be quite small by construction.

FC line coplanarity observation equation
This model relates the parameters (p, q, α, β) of a 3D line, the EO parameters of an image, and the image observations (x, y) of Observations Table 3: Symbols in the FC line coplanarity equations.
a point of the line.It has been derived from the collinearity condition and the 3D straight line parameterization presented in the previous section 3.4.Eq. 6, and its equivalent eq. 7, are the coplanarity equations where the straight lines are used as tie features.Table 3 details the mathematical symbols of these equations that are not introduced in section 3.4.
For some λ, µ λ and µ exists, if and only if Eq 7 is the FC line coplanarity observation equation.In practice, for this model to be of any value, the FC self-calibration function s, s = (sx, sy), has to be included to replace x c by its calibrated value (sx(x, y), sy(x, y), 0) c .

Line-in-plane observation equations
Given the parameterization of section 3.4, the observation equations that express that a straight line belongs to a plane can be easily deduced and result in the two following equations where the covariance matrix of the "observation" vector (0, 0) and the residual vector (vo, v d ) are a mechanism to respectively tune and asses the actual degree of coincidence of the actual line and plane.

CONCEPT VALIDATION
In order to validate the proposed concept, real data were collected and software was developed.The data originate from two flight campaigns over an area near Castellbisbal (Catalonia, Spain).In these campaigns high-end airborne laser scanners and digital metric camera systems were flown.As for the software, an experimental "model toolbox" with the models discussed before was developed at the Institute of Geomatics and run on the generic network adjustment platform GENA (Colomina et al., 2012).
Since the aim of this paper is not an exhaustive study, a small subset of the original data has been used to validate the feasibility of simultaneous block adjustment for calibration and orientation of FC and ALS images.The main characteristics of the camera and laser scanner systems as well as the test configurations are detailed in Table 4.The precision of the observations can be found in Table 5. Figure 2 shows the block layout.Notice that from the FC-derived data, 4 tie lines have been extracted.These tie lines correspond to building ridges.On the other hand, from the ALSderived data, 6 planes have been extracted: 4 of them are our tie planes and the other 2 are ground control planes.The tie planes correspond to 2 building roofs and the ground control planes to a football field that was surveyed with standard differential GNSS techniques.The straight lines that result from the intersection of the ALS tie planes of the same building roof are FC tie lines.
With the given data, the configurations described in Table 6 have been processed.Block configuration FC TP performs a classical FC ISO using all available tie points and ground control points (GCPs).Test FC TL adds 4 straight lines as tie lines.This configuration allows to validate the coplanarity equation model with tie lines.As for the ALS tests, an ALS orientation and calibration adjustment without FC data is performed using planes as tie features with two ground control planes (GCPLs).The ALL configuration combines FC data with ALS data using tie points, ground control points for the FC ISO, tie planes and ground control planes for the ALS ISO and tie lines for the FC-ALS ISO.This test allows to test simultaneously all developed models.Test ALL nc seeks to explore the simultaneous adjustment without using ground control planes in the ALS ISO adjustment.It also helps in understanding how laser scanning can benefit from photogrammetry in order to estimate the ALS self-calibration parameters.
Beside the EO parameters, one INS/GNSS shift for the FC subblock is also estimated as well as corrections in the boresight matrix between the IMU and the camera (Table 7).In the FC ISO, self-calibration parameters are not estimated due to the small number of photographs.As for the ALS ISO calibration, note that the geometry of the sub-block is not strong enough with only one scan line.For the proposed test, self-calibration of ALS refers to the estimation of the self-calibration parameters ∆r, ∆θ and S. In this case, an INS/GNSS shift cannot be estimated because it is strongly correlated with ∆r as well as with the offset between the ALS scan mirror and the IMU.The boresight matrix cannot be estimated because there is only one scan line and the boresight matrix is strongly correlated with ALS self-calibration parameters.
The test results are shown in Table 8 and Table 9.As it was expected, due to the small data set and GCP configuration, FC TP and FC TL results show a similar performance in terms of accuracy (Figure 3) and precision ( small differences and the small size of the experiment (Table 9).In general, the tests show the potential of using tie lines to link the camera and laser scanner images.In test ALL nc, the simultaneous adjustment is carried out without using ground control planes.The results indicate that it is possible to estimate ∆θ, S and ∆r, with the same performance of ALS case.

CONCLUSIONS AND FURTHER RESEARCH
A joint ISO method for FC and ALS images that uses straight lines as tie features between FC and ALS images has been proposed.It consists of a FC (with tie points), ALS (with tie planes)   and FC-ALS ISO (with tie lines) combined adjustment.Mathematical models to support the use of tie lines have been introduced and tested: a 3D straight line parameterization, a coplanarity observation equation and a line-belongs-to-plane one.In the ALS models, raw measurements are used to allow for system and sensor calibration.
The results of this paper are only preliminary because of the size of the data sets.Photogrammetry results show that with this combined FC, ALS and FC-ALS ISO concept, the performance in terms of precision and accuracy is maintained.The results suggest also that the ALS self-calibration can benefit from a combination with photogrammetric data.
The next step is to validate the presented concept and models with larger data sets and to understand the behavior of FC and ALS self-calibration parameters in combined adjustments as well as the requirements in terms of ground control points and planes.

Figure 1 :
Figure 1: Straight line characterization in object space.

Table 1 :
Reference frames and coordinate systems.
In the ALS ISO case, in this research, only planar surfaces belonging to building roofs are extracted.For the purpose, a region growing algorithm has been used.

Table 8
) for the EO parameters and tie points.The results of tests cases ALS and ALL show that the simultaneous block adjustment works and, apparently, allows for a slightly better determination of ALS self-calibration parameters in terms of precision although this result is inconclusive given the

Table 6 :
Observations of block configurations.

Table 7 :
Estimated parameters of block configurations.