EXPERIMENTS ON CALIBRATING TILT-SHIFT LENSES FOR CLOSE-RANGE PHOTOGRAMMETRY

One of the strongest limiting factors in close range photogrammetry (CRP) is the depth of field (DOF), especially at very small object distance. When using standard digital cameras and lens, for a specific camera – lens combination, the only way to control the extent of the zone of sharp focus in object space is to reduce the aperture of the lens. However, this strategy is often not sufficient; moreover, in many cases it is not fully advisable. In fact, when the aperture is closed down, images lose sharpness because of diffraction. Furthermore, the exposure time must be lowered (susceptibility to vibrations) and the ISO increased (electronic noise may increase). In order to adapt the shape of the DOF to the subject of interest, the Scheimpflug rule is to be applied, requiring that the optical axis must be no longer perpendicular to the image plane. Nowadays, specific lenses exist that allow inclining the optical axis to modify the DOF: they are called tilt-shift lenses. In this paper, an investigation on the applicability of the classic photogrammetric model (pinhole camera coupled with Brown’s distortion model) to these lenses is presented. Tests were carried out in an environmentally controlled metrology laboratory at the National Research Council (NRC) Canada and the results are hereafter described in detail.


INTRODUCTION
Close-range photogrammetry (CRP) is a powerful technology because of its flexibility for speedy reconfiguration and the possibility to accomplish very demanding tasks while keeping the costs reasonable.It is an established technology worldwide for 3D recording in industrial/engineering as well as in cultural heritage applications.Nevertheless, when the imaging distance is small (less than 2 m), the depth of field (DOF), i.e. the axial (longitudinal) resolving power of the employed optical system, becomes a strong limiting factor.DOF extends in front of the camera and can be visualized as a prism with two faces parallel to the sensor plane.The portion of object that falls into the solid will be rendered adequately sharp, or in other words with enough detail in the image.A way for controlling and adapting the DOF to the scene of interest is to use the so called camera movements (Ray, 2002), i.e. translational and rotational movements both of the lens and image plane.Camera movements are possible when using view or technical cameras, which are large format cameras where the lens, usually mounted on a folding baseboard or a monorail (Figure 1a), projects the image on a ground glass to allow for focusing.Once the focus is set, the ground glass is replaced by the film holder.View cameras are mainly used by professional photographers in studio or for architectural photography.Even if adapters are now available for digital single-lens reflex (DSLR) cameras (Figure 1b), these systems are not very suitable for photogrammetry due to their low flexibility and portability.A more practicable solution is represented by tilt-shift lenses (Figure 1c), also called perspective control (PC) lenses, which are available for smaller camera format (e.g., 35 mm or full frame) and allow for some of the view camera movements.In particular, by rotating the tilt-shift lens of a proper angle, an inclined subject may be fitted within the optical system DOF, according to the Scheimpflug rule.This paper presents the first results of a wide experimental study conducted in an environmentally controlled metrology laboratory at the National Research Council (NRC) Canada to investigate the metric capability and benefits of tilt-shift lenses.Several experiments evaluated the overall performance of a DSLR camera equipped with a tilt-shift lens, using quality parameters for measurement uncertainties, i.e. accuracy and precision.To rigorously model the image formation when the Scheimpflug principle is employed, the actual angle between the optical axis and the image plane should be included in the mathematical formulation.Neglecting this angle would certainly introduce systematic errors in the photogrammetric measurement process.Nevertheless, if this angle is bounded to a range of ±4 degrees for most practical uses, the error introduced might be limited and compensated by calibration parameters.Such evaluation has not been done until now and therefore the present investigation aims at evaluating if the most popular photogrammetric model, i.e. the pinhole camera coupled with Brown's formulation for lens distortion, can be considered still valid for tilt-shift lenses and to which extent.The rest of the paper is structured as follows.Section 2 reports the motivations behind the study, along with a discussion on DOF, Scheimpflug condition and tilt-shift lenses.Section 3 presents the experiments, with particular emphasis on the photographic equipment, the calibration fixture specifically designed for the tests and the acquisition protocol adopted.Finally (Section 4), the results of the experimental investigation are presented and discussed.

MOTIVATION AND BACKGROUND
The aim of the collaborative research between FBK and NRC is to explore the use of Scheimpflug rule, a well-known principle commonly exploited in "artistic" and professional photography (Merklinger, 1996;Evens, 2008) as well as active triangulation scanning technique (Dremel et al., 1986;Rioux et al., 1987;Albers et al., 2015) and particle image velocimetry -PIV (Fournel et al., 2004;Louhichi et al., 2007;Astarita, 2012;Hamrouni et al., 2012), but not yet adequately investigated in CRP.

DOF and Scheimpflug rule
DOF is the distance between the nearest and farthest object planes still in focus, i.e. the distance between the nearest and farthest objects in a scene that appear acceptably sharp in an image.Sharpness can be defined as the quality for which adequate resolved details and, hence, information content, are provided by the image.In absence of other degrading factors (aberrations, camera shake, etc.) sharpness only depends on correctly focusing the subject of interest.From a physical point of view, a lens can precisely focus at only one distance: there exists only one specific distance where a point in object space is imaged in the image plane as a sharp point.At any other distances, an object point produces a blur circular spot, which means that the point is defocused.When the diameter of the circular spot is sufficiently small, the spot is indistinguishable from a point, and appears to be in focus.The decrease in sharpness is gradual on either side of the focused distance, so that it can be considered that the object is "acceptably sharp" within the DOF.The diameter of the circle increases with distance from the plane of focus; the largest circle that is indistinguishable from a point is known as the acceptable circle of confusion (COC) (Figure 2).The concept of COC goes back to analogue photography and to the printing and enlargement (or magnification) of film.The value of COC or blur circle is originally defined according to the finer detail that is visible by human eye at a comfortable vision distance on a printed and magnified photograph.It depends on the camera format and lens used.COC typical values for different analogue camera formats are provided in Ray (2002).For digital cameras, COC is usually taken as around 1-3 pixels (Menna et al., 2012;Luhmann et al., 2014).According to Ray (2002, page 218), DOF can be computed according to equation ( 1): where: and are the far and near limits of DOF, respectively; -is the lens focal length; -is the focusing distance; -is lens relative aperture.For large object distance, equation (1) can be simplified to (Ray, 2002;Luhmann et al., 2014): In this case, it can be easily argued that the DOF increases with the COC, the f-number and the square of focusing distance, while decreases when the focal length grows.When working in close up photogrammetry, which corresponds typically to a magnification1 range of 0.1-1 and object distance range of 1000 mm -10 mm (Ray, 2002), DOF is very limited.
In an ordinary camera setup, the points in space that are in focus ideally lie in a plane parallel to the image plane and, once decided the focal length, the desired DOF is generally accomplished changing the f-stop value.A smaller aperture yields a greater DOF but may lead to diffraction that in turn leads to a loss of image detail (Luhmann et al., 2014).In CRP, usually the objects to be surveyed are three-dimensional and of varying depth, but a perfectly sharp image is obtained only for the part of the object that is within the DOF.According to the Scheimpflug rule, by tilting the lens so that its axis is no longer perpendicular to the plane of the sensitive media, film or digital sensor (Figure 3), the DOF can be optimized.This condition is realised when the image plane, the lens plane and the plane of sharp focus meet in a single line.Through the tilting of the DOF zone at the proper angle, the spatial extent of the DOF can be adapted to the real shape and extent of the scene (Figure 4).When the Scheimpflug condition is applied, the far and near planes that limit the DOF are no longer parallel, but they intersect, along with the plane of sharp focus, in the so-called hinge line (Merklinger, 1996).

Tilt-shift lenses
In most DSLR camera systems, the sensor plane and lens axis are fixed perpendicularly to each other, limiting the DOF during the image acquisition.In order to apply the Scheimpflug principle and adapt the DOF to the shape and size of the object, specifically designed tilt-shift lenses have to be used (Figure 1c).The movements allowed by tilt-shift lenses, shown in Figure 5, are here described: tilt (T)¸ also known as swing, is the rotation of the optical axis around a point that can coincide with either the exit pupil or the centre of the sensor plane or an arbitrary construction pivot point (Eastcott, 1997); shift (S) is the translation movement of the optical axis; rotation (R) is the rotation around the optical axis.Let's consider a camera with the image plane vertical (Figure 4).
When R=0° (in this study indicated as R0), the tilt movement takes place in the horizontal plane and the shift in the vertical; on the contrary, when the lens is rotated (R= ± 90° or R90), tilt occurs in the vertical plane and shift in the horizontal.From now on, a counter-clockwise rotation is considered positive and when all the movements are zero (T= 0°, S=0 mm, R=0° or in the convention here adopted T0S0R0) the lens is said in "normal condition".A lens projects on the image plane a circle, called image circle, which in normal lenses is usually slightly larger than the sensor.A peculiarity of tilt-shift lenses is that the image circle is much larger, to allow the sensor plane to remain within the circle when movements are used.Indeed, tilt-shift lenses have mechanical limitations in the range of motions in order to guarantee that the image area is always covered by the image circle.

EXPERIMENTS
As already stated, the core purpose of this research is to investigate the applicability of the pinhole camera model coupled with Brown's distortion model to tilt-shift lenses mounted on DSLR cameras ("tilt-shift photogrammetry").The approach adopted to perform the assessment consists in comparing the 3D coordinates of coded targets measured with tilt-shift photogrammetry and the coordinates of the same targets obtained with classic or normal photogrammetry.Here classic or normal photogrammetry refers to the use of standard, i.e. non tilt-shift, lens.Actually, the experiments conducted at the NRC laboratory were not restricted to this specific tests, rather preliminary efforts were made with a twofold aim: (i) to design a proper calibration fixture and (ii) to assess the quality (precision or repeatability and accuracy) of classic photogrammetry, considered the reference method.However, the paper will particularly focus on the comparison between tilt-shift and standard photogrammetry.

Equipment and material
The photogrammetric equipment employed for the tests is summarised in Table 1, whereas Table 2 reports information about the system set-up and measurement volume.In the present study, the photogrammetric system composed of the Nikon D750 with 50mm lens (from now on D750-50mm) is considered the standard reference photogrammetric system, since the 50mm is a prime (i.e., fixed focal length) standard lens, with simple lens schema that makes practically valid the classical photogrammetric model (pinhole camera coupled with Brown's formulation).The aperture value and focus distances were selected to assure a reasonably useful DOF also with standard lens, i.e. the reference measuring system.Two photogrammetric software applications were used for the calibration of the optical systems, namely EOS PhotoModeler (http://www.photomodeler.com/index.html) and Photometrix Australis (http://www.photometrix.com.au/australis).

3D photogrammetric calibration test object
An ad-hoc volumetric testfield, called 3D photogrammetric calibration test object (3D PCTO, Figure 5a and Figure 5b), was built for calibrating the optical system.The 3D PCTO features: -a size of 700 mm × 700 mm in plane and 115 mm in height; -5 mm diameter photogrammetric contrast coded targets on aluminium surface and blocks of different heights; -2 invar scale bars (Brunson, http://www.brunson.us)measured with an optical coordinate measurement machine (CMM); -12 nests where half spheres are placed.The half spheres can be substituted with spherically mounted retro-reflectors (SMRs), i.e. targets specifically built to be measured with laser trackers.The centre of the spheres coincides with the centre of the SMR, i.e. with the point measured by the laser tracker.

Testing environment and acquisition protocol
Several experiments were specifically designed and realised in the NRC facility dedicated to research in the areas of metrology, sensor calibration, certification and performance evaluation of 3D imaging systems.The 100-square meter laboratory is characterised by a controlled environment, i.e. temperature at 20°C±0.1°C and humidity at 50%±10%.Image acquisitions of the 3D PCTO were carried out according to a high redundancy photogrammetric camera network (Figure 5c), both with conventional and with tilt-shift lenses.The convergent multi-station camera configuration includes up to 48 images, incorporating also pictures with orthogonal camera orientation to assure a proper roll diversity suitable for selfcalibration.To avoid undesirable camera shake, a photographic tripod and remote control were used; moreover, the mirror lockup function was enabled.The shutter speed was manually set to guarantee the proper exposure of the 3D PCTO at an ISO value of 200.The lenses were locked with hot melt glue to assure mechanical stability and prevent involuntary changes of the focusing setting.The image acquisition with the tilt-shift lens was realised in three configurations: (i) normal condition (T0S0R90), (ii) with a tilt angle in the vertical plane of 4 degrees (T-4S0R90) and (iii) with the maximum allowable shift movement of 11 mm in the vertical plane (T0S11R0).The images were acquired in the proprietary raw format at 12-bit (NEF) and converted into high quality JPG using Adobe Camera Raw without further image processing or enhancement (neither sharpening, noise reduction nor histogram stretching).

RESULTS
Table 3 and Table 4 summarise the relevant outcomes of the selfcalibrating bundle adjustment for the tested photogrammetric systems.The mathematical model adopted comprises the classic pinhole camera model with Brown's lens formulation plus affinity and shear that is recalled in equation ( 2).

CAMERA
where: -and are the corrected image coordinates; -and are the measured image coordinates referred to the principal points, whose coordinates are respectively and ; -Δ and Δ are the radial distortion corrections: where = √ + , and , , are the radial distortion parameters; -Δ and Δ are the decentring or tangential distortion corrections: with P , P the decentring distortion parameters; -Δ and Δ are the affinity and shear distortion corrections: with B and B the affinity and shear parameters, respectively.
For the tilted system (D750-45mm T-4S0R90), two different processing are shown, i.e. with and without including the affinity term in the calibration model.The shear parameter B2, as well as the radial distortion parameter k3, are not computed since they were found to be not significant.Table 5 reports the difference in the 3D coordinates of the photogrammetric targets measured with the reference system D750-50mm and the system D750-45mm in the different tested configurations.To calculate the differences, a Helmert rigid transformation without scale factor is computed initially to obtain the two sets of points in the same coordinate reference system.In figure 6, the distortions of the different configurations tested are visualized according to a colour or distortion map, where the colour represents the difference between the ideal pixel position (no distortion) and the actual pixel position due to the influence of distortion parameters as computed through the self-calibration.
The first interesting outcome is that the tilt-shift lens in normal condition (T0S0R90) provides results essentially equivalent to the standard 50 mm lens, confirming the good optical and mechanical manufacturing of the lens.Also the effect of a shift movement of the optical axis is well modelled and absorbed by the classic photogrammetric model through a significant movement of the principal point along the sensor height, as it is easily visible in Figure 6b.When a tilt angle is applied, the affinity term B1 becomes essential, even if it is not sufficient to assure precision and accuracy fully comparable with the normal condition.The individual contribution of B1 on the whole distortion corrections is shown in figure 6d: an absolute correction value up to almost 0.04 mm is reached at the borders of the image.

DISCUSSION AND OUTLOOK
The experiments described in the previous sections reveal that, when the optical axis is no longer perpendicular to the image plane with a non-zero tilt angle, the classic pinhole camera model and Brown's lens formulation can be adopted if: (i) the affinity factor is computed along with radial and decentring distortion components, (ii) a relative accuracy of some 1:70.000 is sufficient.
In case of a shift movement of the optical axis and consequently of the principal point in the image plane, standard photogrammetric mathematical model is basically valid.The next step of the study will be to test a theoretical framework that explicitly incorporate the tilt angle of optical axis into the camera model.The following topics were neither reported nor discussed for this paper: (i) tests to assess dense image matching procedure with tilt CRP, (ii) the creation of a proper 3D test object suitable to address the quality of the calibration results (stability and repeatability), (iii) the analysis and comparison between photogrammetry and laser tracker data.The next steps of the study will present these topics and complete the theoretical framework that explicitly incorporate the tilt angle of optical axis into the camera model.

Figure 2 .
Figure 2. COC and DOF for a lens focused at finite distance.

Figure 3 -Figure 4 -
Figure3-With an ordinary camera lens the plane of focus is parallel to the image plane (a).Scheimpflug principle: plane of focus of an optical system tilts when the lens plane is not parallel to the image plane (b).

Figure 4 .
Figure 4. Movements of tilt-shift lens mounted on a DSLR camera with the image plane vertical and R=0°.

Table 3 .
Results of self-calibrating bundle adjustment for the tested photogrammetric systems.Interior orientation and additional parameters are reported along with internal assessment in image and object space.

Table 4 .
External assessment of the self-calibrating bundle adjustment for the photogrammetric systems tested: length error on reference scale bars.