A MODULAR GEOMETRIC MODEL FOR UNDERWATER PHOTOGRAMMETRY

Underwater applications of photogrammetric measurement techniques usually need to deal with multimedia photogrammetry aspects, which are characterized by the necessity of handling optical rays that are broken at interfaces between optical media with different refrative indices according to Snell’s Law. This so-called multimedia geometry has to be incorporated into geometric models in order to achieve correct measurement results. The paper shows a flexible yet strict geometric model for the handling of refraction effects on the optical path, which can be implemented as a module into photogrammetric standard tools such as spatial resection, spatial intersection, bundle adjustment or epipolar line computation. The module is especially well suited for applications, where an object in water is observed by cameras in air through one or more plane parallel glass interfaces, as it allows for some simplifications here.


INTRODUCTION
Photogrammetric tasks requiring the tracing of optical rays through multiple optical media with different refractive indices are called 'multi-media photogrammetry' in the photogrammetric literature.This denomination has already been coined at a time, when the term 'multimedia' meaning the combined use of digital media such as text, images, film, animation and audio was completely unknown.
Multi-media photogrammetry is characterized by the refraction of optical rays at the transition between media with different refractive index, which can be modeled by Snell's Law.An early treatment of this issue can be found in (Rinner, 1948), who worked on the relative orientation of stereo aerial images of underwater scenes at analogue plotters and coined the term 'two-media photogrammetry '. Höhle (1971) and Okamoto/ Höhle (1972) showed an analytical solution and took the step from two-to multi-media photogrammetry, replacing the imaging rays by polygons.Kotowski (1988) developed a raytracing method for tracing rays through an arbitrary number of parameterized interfaces, which was implemented in a bundle adjustment.It allows for handling both image invariant and object invariant interfaces.
In photogrammetry, we can distinguish three major categories of applications of multi-media techniques:  In aerial photogrammetry, photo bathymetry is a technique to derive models of the sea floor from stereo imagery, provided limited depth and sufficient water transparency (e.g.Okamoto/Höhle 1972, Vanderhaven 1978).The airwater transition can be modelled on the basis of Snell's Law.Most implementations herein assume the water surface to be horizontal and planar, with waves on the water surface leading to significant errors (Okamoto 1982).
 In underwater photogrammetry, cameras (with suitable housing) are used underwater.Some of these cameras are equipped with lenses specially designed for underwater imaging.As an alternative, cameras may be equipped with a plane parallel front window, which can geometrically be treated as an image invariant interface.Typical application examples are in archaeology (Drap et al., 2002), in the recording of ship wrecks, in marine biology (Shortis et al., 2009), in measurements in nuclear power stations (Przybilla et al., 1988) or in the measurement of the shape of fishing nets (Zwart, 1987). Many applications in industrial/technical close range photogrammetry deal with objects or processes in liquids, which are observed by cameras situated outside the observation vessel, imaging the scene through a plane parallel glass window (e.g.3D flow measurement techniques, Maas 1992, Maas/Grün 1995).The ray path herein is a twice-broken beam, which is refracted when passing through the three optical media interfaces air/glass/liquid (or vice versa).Also in lidar bathymetry, which is used to determine water body topographies by airborne laser scanning (e.g.Irish/Lillycrop 1999or Mandlburger et al. 2011), geometric models are used which can be derived from the above categories.
In the following, a multi-media model first introduced by (Maas, 1992) will be discussed, which can flexibly be integrated as a module in standard tools of photogrammetry.Subsequently, several extensions of the model will be discussed.

A STANDARD MODEL FOR MULTI-MEDIA CLOSE RANGE PHOTOGRAMMETRY
Many applications of multi-media close range photogrammetry in industrial-technical applications require the observation of objects or processes in a liquid through a glass window, which can be considered a plane parallel plate.This constellation allows for some algorithmic and computational simplifications, which form the basis for a flexible -yet strict -multi-media photogrammetry model (see also Maas, 1992).The model can be integrated as a module into the collinearity equations and can thus be used in photogrammetric standard procedures such as spatial resection, forward intersection, bundle adjustment or epipolar line computation.Like almost all approaches shown in the literature, the model assumes homogeneity and isotropy of the optical media.
The collinearity condition per definition connects image coordinates, camera projection center and object point coordinates.Its basic assumption, that image point, camera projection center and object point form a straight line, is not fulfilled any more in multi-media photogrammetry due to the refraction of the rays at the multi-media interfaces.The approach proposes a radial shift of an underwater object point with respect to the camera nadir point in a way that the collinearity condition if re-fulfilled.This radial shift is implemented as a correction term into observation equations derived from the collinearity equation.Simplifications can be achieved when choosing the coordinate system in a way that the X/Y-plane is identical with one of the interface planes glass/water or air/glass (see.Fig. 1).
Figure 1.Radial shift for multi-media effect compensation (Maas, 1992) The procedure can be explained as shown in Fig. 1: An object point P is imaged onto image point p through the water/glass and glass/air interfaces, obeying to Snell's Law.If P were radially shifted to P in a plane parallel to the X/Y-plane, the collinearity condition could be applied with P like in the standard one-media case.Therefore the goal is to compute the radial shift R  relative to the nadir point N ( R  >0 if n 2 > n 1 and n 3 > n 1 ).This will, for instance, allow to use the radially shifted point in a spatial resection for camera orientation and calibration.
The calculation of the radial shift R  can be derived from Fig. 1: Snell's Law says: The thickness of the glass plate t and its refractive index n 2 are usually assumed to be known and fix.The refractive index of water depends on the optical wavelength as well as water temperature, salinity and depth and can be obtained from an empirical formula as used in (Höhle, 1971): A closed solution of the above equation system is not possible due to the trigonometric functions.Therefore an iterative procedure is being used, wherein P is chosen as a first approximation of P : For the 1. iteration we get the incidence angle in medium 1 Switching back from polar to cartesian coordinates after the last iteration, we get the coordinates of the radially shifted point P can then be used in the collinearity equation instead of P , so that the equation can be used as an observation equation in spatial resection, forward intersection (with two or more images) or bundle adjustment.This offers the great advantage that existing photogrammetric software solutions can be extended by the multi-media module handling the radial shift procedure, without any modification in the core software tools.That means that the whole multi-media problem is out-sourced into the radial shift module.The epipolar line will usually not be a straight line anymore in multi-media photogrammetry.A possible solution by approximating the epipolar line by a polygon obtained by intersection a twice-broken image path with multiple discrete depth layers is shown in (Maas, 1992).
The procedure can easily be extended to an arbitrary number of parallel interfaces.It can also be derived from the generalized model shown in (Kotowski, 1988) with the simplifications shown here.A related approach is has been shown in (Philips, 1981), who very vividly connects it with the 'apparent places' known from astronomical geodesy.

A STANDARD MODEL FOR MULTI-MEDIA CLOSE RANGE PHOTOGRAMMETRY
The iterations cost some computation time.If time is an issue, an over-compensation factor as shown in (Maas, 1992) may be used, reducing the number of iterations by about 50%.A much more efficient reduction of the computational effort can be achieved by outsourcing the multi-media calculations into a lookup-table.This may for instance be relevant in photogrammetric 3D-PTV (particle tracking velocimetry) systems (Maas et al., 1993), where the coordinates of several thousand neutrally buoyant tracer particles in a liquid flow have to be determined from image sequences of three or four cameras over several seconds or minutes at 25 Hz.In the processing of these image sequences, millions of forward intersections have to be computed, each of them requiring the iterative multi-media shift procedure.In a lookup-table based solution, the problem can be reduced to the initialization of a two-dimensional lookup-table with the depth P Z and the radial nadir point distance distance R of a point P as entry parameters and the radial shift ratio R R / as a result.Due to the reference to the nadir point, one lookup-table per camera has to be established.The lookup-table entries can be generated in a ) , ( R Z P raster using the iterative model shown above.If the lookup-tables are initialized at a sufficient density, the relative radial shift of each point can easily be obtained by bilinear interpolation in the lookup-tables.The loss of accuracy caused by this interpotaion-based procedure depends on the density of the initialization of the lookup-tables.(Maas, 1992) shows that less than 2000 lookup-table entries provide a basis for handling the multi-media geometry without significant loss of accuracy in a typical 3D-PTV constellation.

CONCLUSION
The multi-media module shown above allows for a flexible integration into standard geometric toold of photogrammetry.
Thus it has the potential to turn any standard photogrammetry package into a mult-imedia photogrammetry package.As an extra option, even the rafractive index of the liquid may be introduced as an extra unknown into the bundle adjustment, provided a geometrically sufficiently stable configuration (Maas, 1995).

R
we get new incidence and refractive angles  1 , 2 , 3 , which can be used to compute a new R  etc., until   R (e.g. with  = 0.0001mm).