UNDERWATER PHOTOGRAMMETRY, CODED TARGET AND PLENOPTICTECHNOLOGY: A SET OF TOOLS FOR MONITORING RED CORAL INMEDITERRANEAN SEA IN THE FRAMEWORK OF THE ”PERFECT” PROJECT

Abstract. PErfECT "Photogrammetry, gEnetic, Ecology for red coral ConservaTion" is a project leaded by the Laboratoire des Sciences de lInformation et des Systmes (LSIS - UMR 7296 CNRS) from the Aix-Marseille University (France) in collaboration with the Spanish National Agency for Scientific Research (CSIC, Spain). The main objective of the project is to develop innovative Tools for the conservation of the Mediterranean red coral, Corallium rubrum. PErfECT was funded by the Total Fundation. The adaptation of digital photogrammetric techniques for use in submarine is rapidly increasing in recent years. In fact, these techniques are particularly well suited for use in underwater environments. PErfECT developed different photogrammetry tools to enhance the red coral population surveys based in: (i) automatic orientation on coded quadrats, (ii) use of NPR (Non Photo realistic Rendering) techniques, (iii) the calculation of distances between colonies within local populations and finally (iv) the use of plenoptic approaches in underwater conditions.


INTRODUCTION 1.1 Project background and main goals
Human activities are threatening the conservation of marine ecosystems worldwide.The Mediterranean is considered a hot spot for marine biodiversity.The communities dominated by red coral Corallium rubrum are included among the coralligenous assemblages one of the richest assemblages found in the Mediterranean (Ballesteros, 2006) .The coralligenous are affected by the global change.The definition of conservation actions based on an accurate understanding of its functioning is therefore essential to preserve it and ensure its persistence.The project PEr-fECT (Photogrammetry, gEnetics, Ecology for red coral Con-servaTion) 2015-2018, uses a novel multidisciplinary approach based on the latest developments of underwater photogrammetry to support population ecology and genetics study.The ultimate goal of PErfECT is to define new and more efficient conservation plans for the coralligenous community.Focused on the red coral, an emblematic species of the coralligenous, we will address two complementary objectives: 1) to define the conservation goals; 2) to characterize the functioning of the populations and its implication for restoration actions.In the first objective, we will provide baseline by characterizing data on well preserved populations as well as from large sized colonies formerly abundant and now rarely found in nature.In the second objective, we will characterize the mode of reproduction of the red coral in order to improve the transplantation methods, which are essential to the restoration of the populations (Montero-Serra et al., 2017).Considering that the red coral is an engineer species, this project will benefit from the associated community.These actions will

The ecological context
The Mediterranean red coral Corallium rubrum (Linnaeus, 1758) is a slow-growing coral which can reach 50 cm in height and live for up to 100 years (Garrabou and Harmelin, 2002), (Marschal et al., 2004).The red coral is mainly distributed in the Mediterranean Sea and forms dense populations in dim-light habitats between 10 and 1000m.Its red calcareous skeleton has been appreciated for its use in jewelry since ancient times.Current populations are dominated by small sized colonies, except in populations within marine protected areas (MPAs) or in unexploited populations usually in deep water habitats (Garrabou et al., in press), where harvesting activities are limited or never occurred.Moreover, red coral is also suffering from the effects of climate warming, which can severely increase mortality rates.Monitoring the current state of red coral populations is a basic step in designing sound management and conservation plans for this threatened species.In situ surveys in red coral populations are difficult since they usually develop in deep water (> 20 m depth), limiting the diving time and in semi-dark habitats (overhangs, cave entrances, vertical walls).In addition, red coral is fragile, making the application of direct measurements difficult, as it is currently done for other benthic species.Underwater photogrammetry has been successfully applied to the monitoring of red coral populations (Ledoux et al., 2010) (Linares et al., 2010) (Drap et al., 2013).However, the measurement of colonies at different times is time-consuming and difficult for large-scale applications.Surveying red coral in situ is both a necessity and a challenge.We need efficient tools, accurate, non-invasive, absolutely remote sensing, minimizing both bottom diving times and contact with the seabed.

Previous work
In recent years, various teams of biologists have experimented with photogrammetry or computer vision (the difference is more subtle) to measure, count, describe living organisms or the seabed itself.From a light photogrammetric approach (Sanchez et al., 2009)in deep water to the use of a specifically-designed, sophisticated underwater vehicles (Singh et al., 1999) or, for a long time, using specific stereoscopic photography equipment (and since a long time) (Hollister, 1984), (Fryer, 1983) biologists have worked hard to measure benthic communities using photogrammetry (Abdo et al., 2006) (Madjidi and Nagahdaripour, 2003) (Shortis et al., 2007) (Leon et al., 2015a) and also (Zapata-Ramrez et al., 2013).Shortis (Shortis et al., 2009) provides a clear and synthetic overview.
The diffusion of 3D measurement in marine biology converges naturaly toward the use of GIS.Even if absolute geopositioning is almost impossible, a local reference system can be defined (Canese et al., 2009) (Giusti et al., 2009).Photogrammetry and GIS approaches can be used to characterize reproductive units within the populations studying family relationships of combining spatially explicit positions red coral colonies with genetic analysis (Ledoux, 2010) page 10.The use of photogrammetry allows a better approach to study spatial distribution compared to the traditional approach used in this community (Bythell et al., 2001).For a few years, new commercial photogrammetric software such as Photoscan from Agisoft company (Agisoft, n.d.), change the way to handle photogrammetry and make this technique more democratic and accessible to a non-specialist team (Leon et al., 2015b) .Indeed, photogrammetry is becoming a common tool that a scientific team can use by themselves.We can see for example the work of John Burn using photogrammetry in relationship with GIS software (Burns et al., 2015a), (Burns et al., 2015b), (Burns et al., 2016), (Figueira et al., 2015), as well (Bennecke et al., 2016) about deep coral surveys.

Surveys supported by photogrammetry tools
Within PErfECT we developed photogrammetric tools to support three kinds of red coral population surveys.
1. Characterization of red coral populations.Based in random quadrat sampling, scuba-divers use a set of photographs on which are visible colonies surrounded by a 20x20 cm quadrat.The analysis of the photographs provides reliable measurements on colony sizes (basal diameter and maximum height), occurrence of breakage of colonies and the occurrence of necrosis.To minimize the divers tasks during the acquisition phase, we opted for multi-view photogrammetric acquisition using a single device to easily adapt the measurement procedure to the scene configuration.The material is quite light: one reflex digital camera with housing, two electronic strobes and a simple procedure with a few photographs taken for each site.We developed an automatic exterior orientation procedure based on a known (and 3D printed) quadrat with coded target (Drap et al., 2014).2. Temporal surveys and fine scale spatial distribution of red coral colonies using permanent plots surveys.Permanent plots are set up using PVC screws fastened to holes in the rocky substratum.Each plot was variable in length, depending on the complexity of the substratum, and 40 cm wide.The total area covered in each site was about 2 m 2 , which allowed monitoring hundreds of colonies through time.Transects were photographed using different reflex cameras (from Nikon D70 to Nikon D700) with a housing and two electronic strobes.Until now, in each sampling a cord is deployed between the screws to help positioning the 20x20 cm quadrats sequentially along the cord (above and below the cord length).At each position, two to three photographs from each quadrat (using two slightly different angles, approximately 30) were obtained.This approach was used until last year but will be drastically changed in the future: indeed, moving a quadrat in a global scene is no more possible when equipped with coded target.We will use a global orientation of the whole transect and move a virtual 20x20cm quadrat on the photogrammetric model for the analyse.3. Characterizing emblematic large colonies red coral with high natural heritage value.Measurements on large colonies characterized by complex shapes and entanglement of the branches pose several constraints.Likewise, the proximity of the substrate on which the colony is fixed may sometimes hinder observations and thus the acquisition of data.The approaches used for population surveys are not suitable in this case.To address these difficulties, we have focused both on the shooting approaches and on the post processing methods..

Characterization of red coral population
The first approach is the exploration of a 20 x 20 cm zone of the seabed limited by a plastic frame, sometimes handled by the photographer or another diver (see fig. 1).The frame can be placed randomly on the seabed, all the measures done are made in a local reference system defined by the frame.Two to three images are done by the photographer (fig.2), when the other diver holds the frame and maintain it immobile with respect to the seabed.Within each quadrat, three measures are carried out: colony size (maximum colony height), other non-metric parameters on the state of each colony and the number of colonies (for density estimations at population level).This approach is, easy to use underwater, minimizing the number of needed photographs and post treatment processes (e.g.manual orientation of the photographs, scaling and quality control is done automatically within the Arpenteur module see below), which are usually very time consuming operations.Beyond taking the photographs the user only needs to identify some specific points on the colonies manually in order to obtain the size of the colonies.
We developed a specific photogrammetric chain of treatment, from the fabrication of a specific quadrat with two color coded targets, an automatic orientation and a user friendly GUI for plotting relevant points.
The first step is to create a 3D quadrat, with a clear coordinate systems, self-made with a 3D printer in order to be printable by every potential user of the whole photogrammetric system.The quadrat was designed with two kinds of coded targets: specific tagRuler target used by our photogrammetric platform (Drap et al., 2014) and the well-known circular targets used both by Photomodeler and Agisoft company in order to be fully compatible with these commercial software.
Quadrat external dimensions are 23 x 27 x 1,5 cm (internal dimensions are fixed at 20 x 20 cm) according to the most common 3D printers size.The quadrat (see figure 3) , designed with CAD technic, is especially made for the divers: no air pockets inside, thickness of 2 mm and cellular framework to balance stiffness, weight and handling.
The two kinds of coded targets are printed on top using a dual head 3D printers and PLA material, black and white.
Figure 3.The standard frame called "Quadra1" during the 3D printing process.
The automatic orientation can be done with any of the aforementioned programs.The coordinates of all the targets (18 coded targets of the two type so 36 targets on the quadrat) are defined in the CAD software and can be used as control point in the photogrammetric orientation.
The orientation of a set of unordered photographs seeing the same set of control points is easy.We just compute the pose of each photograph independently and use this result as an approximate value for a global bundle adjustment.The global accuracy is provided by the residual on the control point and working at a distance around 20 cm, the residual on the 3 coordinated of the control point is generally less than one millimeter.
Once the orientation is done specific plotting operations can start.An input module was developed for measurements of coral colonies to allow the operator to obtain specific measurements as quickly as possible (see figure 4).Only a few particular areas of the colonies have to be measured, the module ensures the integrity of the observations by a series of real-time controls.The user has the possibility to measure some 3D points on predefined areas of the red coral colonies in order to obtain the measurements required by the biologists: basis diameter, height evaluation, approximate volume, as well as to link to these data some visual observations as for example the percentage of observed necrosis.
Figure 4.The GUI dedicated to the specific coral measurement.

Quantifying temporal changes and fine scale spatial distribution of colonies in red coral populations
The goal of this survey is to quantify temporal changes in the populations.To do this, permanent plots are set up, using PVC screws fastened to holes in the rocky substratum with epoxy putty.The number of transects and distance between screws vary depending on the complexity of the substratum.In general, the total area covered in each site is about 2 m 2 , which allows monitoring hundreds of colonies through time.Transects are photographed using different reflex cameras (Nikon D70, Nikon D300) with a housing and two electronic strobes.In each sampling, an elastic cord was deployed between the screws to help positioning the 20x20 cm quadrats sequentially along the cord (above and below the cord length).At each position, two/three photographs from each quadrat (using two slightly different angles, approximately 30) are obtained.
Nevertheless, according to what we said previously, this approach is limited and is changing.Moreover, we need to consider and visualize a global representation of the covered area.Indeed, for the monitoring process, end users wish to evaluate distances between colonies and quantifying temporal changes related to the red coral populations is much easier if they can see the full area.
To achieve this task, we chose to construct and provide orthophotos of the site (fig.5).
First, we have to shoot many photographs of the area (usually, hundreds to some thousands).This set of photographs can then be oriented in a given global coordinate system and we are able to reconstruct, in a first time, a sparse 3D cloud of the site and next, a dense cloud, which is scaled thanks to the quadrats.This dense cloud can be meshed, leading to a 3D model of the site.The final step of that process is to perform an orthophoto, which is an orthographic projection of the model on a chosen projection plane.Orthophotos exhibit noticeable advantages.It is easier to work with them than with complex 3D meshes and they benefit from a coordinate system, which is the same as the one used in the scaled model.In this framework, we have developed a tool to perform measurements on orthophotos 5. Interesting points for the user can be added by a simple mouse click on the orthophoto.
Then, the point list can be managed in order to fit the specific needs.When ready, the end user starts the computation of the true distances between the whole set of points.An example of the GUI is provided on fig.6.
Figure 6.Example of orthophoto measurements.
To provide accurate results, we compute the geodesic distances on the mesh (even if other methods are available).Indeed, the seabed is rugged and a basic Euclidean distance would lead to poor results.To calculate a true 3D distance, taking into account the deformations of the seabed, we need to project back the chosen 2D points on the mesh.The process can be time consuming, so space partitioning optimizations were implemented to accelerate ray-triangle intersections.Finally, all of the distance computation process is, consequently, hidden to the user.The geodesic algorithm is iterative and the path goes through the faces of the triangles (Seinturier et al., 2013).It is based on the Darboux formulas, which provide a relationship between 3 base vectors and 3 scalar values, called normal curvature γn, geodesic curvature γg and relative torsion τg.This relation can be written under a matrix form, for each sample i belonging to the path: s the tangent vector to the path and the geodesic normal vector g s , which is both tangent to the curve and orthogonal to t (i) s .We aim at determining the geodesic curvature.Indeed, the geodesic path is given by (Lai et al., 2011): To determine γg, Eq.( 1) is rewritten Inverting matrix C, γg is estimated.The geodesic path is then computed for each sample thanks to the following finite difference equation with c is small constant (for example c = 0.001) and sign(a) is a function returning 1 if a > 0, −1 if a < 0 and 0 otherwise.In order to get a fast convergence, (which means the geodesic curvature is close to 0), a pertinent initial guess should be provided.We chose to use the path returned by the Dijkstra algorithm (Dijkstra, 1959) as initial value.This path is more restrictive as it follows the edges of the triangles, but it is a good start before applying the geodesic method.For each sample, we compute the 3 base vectors n s and g s as well as their derivative with respect to the path.These derivatives are numerically estimated.

Inventorying natural heritage of red coral colonies
The third goal is to build a catalogue on the red coral colonies display large size and complex forms.These colonies are very rare in the field at present.To do this a global 3D and metric representation of these colonies is needed.This particular case brings two kinds of problems: acquiring data and extracting the relevant information.Acquiring data can be done with a standard photogrammetric process (see figure 7), obtaining hundreds of images with high overlap and we are in the framework of a wellknown Structure From Motion approach.Due to the common complex topography where red coral populations develop such as cave ceilings, narrow overhangs obtaining good overlap in situ, it can be challenging and very time consuming.
Except for the particular case in figure 7 these surveys have to be carried out in situ.We will applied a new 3D approach offered by the plenoptic technology (Lumsdaine and Georgiev, 2009) (Georgiev, 2009) (Georgiev, 2011).Using a Nikon D800 camera customized by Raytrix, we can have an accurate and properly scaled 3D model from only one shot.However, the model is not complete, but with only one shot, we have enough data on the colonies.An example of an obtained 3D model is shown in figure 14.The next step is to extract relevant 3D data from the entire 3D mesh computed on the 3D point cloud coming from the 3D process.Currently an expert, a marine biologist, uses a 3D software to extract the data he needs.We are developing an automatic  et al., 2003).The proximity and the presence of the substrat (resulting from the photogrammetric acquisition of the colony) prevents observations.NPR approach enables to highlight the colony structure and automatically identify noticeable points.This approach also enables to detect edges on 3D models, or ridges and valleys.They mainly resort on using an information not available on images: surface normals.The basic principle reads as following.Considering a 3D model and a specific point of view of this model, for each vertex t, we define the vector v as the normalized vector from t to the camera looking at the model.We also define n as the normalized normal at t.Then, we define: The set of vertice of I which satisfy I(t) = 0 represents the contour.To be less restrictive and to keep suggestive contours, we add a threshold, say th, so that represents the suggestive contours we are looking for.
These contours can be drawn, projecting the vertice on the camera plane, to get a traditional image.To obtain an accurate 3D contour of the model, it is then possible to oversample it, using technics from the raytracing domain.Casting a ray from the camera optical center, through contour pixels, and tracking the intersection between the ray and the triangle leads to a contour 3D point.
If we subdivide the pixel, we oversample the contour points and get a more accurate representation.The next step, which is still a work in progress, consists in efficiently meshing the new point cloud in order to get a precise, but simplified, representation.
Figure 8. Color Per Vertex rendering and Non Photorealistic Rendering.

Origin
For a decade, plenoptic or light field cameras have recently developed strongly (Oberdrster and Lensch, 2013), because the increase in computational power and the improvement in image resolution have enabled the development and manufacture of highquality cameras by Raytrix (Raytrix, n.d.) and Lytro (Lytro, n.d.).
Instead of a single traditional lens, plenoptic cameras have a matrix of micro lenses behind the main lens that capture the position and direction of each light beam in the image field, enabling the scene to be refocused or 'Get different views from a single image lens and this after capturing the image.
From these data, it is possible to extract depth information as well as a high resolution image and a focus at any point of the image, all this being obtained from a single optical sensor (Adelson and Wang, 1992) (Perwass and Wietzke, 2012).Previous work on 3D reconstruction using plenoptic cameras has focused on the static reconstruction of a single scene (Kim et al., 2013).Tao et al. (Tao et al., 2013) exploited both defocus and correspondence indices to create high-quality depth maps of single-image images that could then be interpolated to form a 3D surface reconstruction.Our first objective is to perform the 3D reconstruction of a zone of the importance of a quadrat in order to minimize the time of intervention on site and to obtain a 3D model using a "simple" shot.

3D reconstruction
Figure 9 shows the basic geometry of a plenoptic camera, which, unlike traditional cameras, has a Micro Lens Array (MLA) located between the image plane and the main lens.The process of obtaining a 3D image is as follows (see the scientific documentation from Raytrix (Raytrix, n.d.)) : 1) Calibration: The MLA calibration calculates the parameters of the intrinsic camera, including parameters of the structure of the micro lens for the plenoptic camera.The uncalibrated raw image returned by the light field camera displays the raw intensity values read by the sensor during shooting.This can be monochrome or, for our use, in color.As shown in figure 11, the individual micro lenses can still be distinguished in the raw image.To remove the micro lens segmentation from the raw image, a gray image is taken during MLA calibration with a white diffusive filter generating continuous illumination to highlight the edges and vignetting of each micro lens.This provides a template to eliminate the micro vignetting of the raw image and to produce a processed image.Finally, a metric calibration must also be performed to convert units from pixels to metric units.The results of the calibration provide intrinsic parameters which include microlens diameter and offset, field of view, depth of field, shooting distance (actual focusing distance within which an object Can move and remain focused), as well as lateral resolution and depth.
2) MLA Imaging: Each of these micro lenses captures a small fraction of the total image.These micro images can be synthesized to form the processed image in a plane of focus (see figure 10).To do this, the depth (the distance from the object to the micro lens) must be taken for each pixel.For each point (x, y) in the object space, a set of micro lenses is selected which captures this point in their image plane.Then, the projection of this point (x, y) on the complete image plane by each micro lens is determined.
The color value for the pixel in the full image plane is taken as the (weighted) average of the micro image values.
3) Total focus processing: With a plenoptic camera, the depth of field increases, the working distance -or the depth range that can be focused in a scene -is much greater than with a traditional camera.In addition, plenoptic cameras can be designed as multifocus cameras, where different types of micro lenses are present with a different focus for each type throughout the MLA.Each group of similar micro lenses is considered as a sub-array.An image with "full focus" or an image in which a large depth of field is maintained can be synthesized using several sub-series of different focal lengths to compute a sharp image over a large distance.Figure 10 shows a total focus image.For more details on the total focusing process, the reader is referred to Perwass and Wietzke (Perwass and Wietzke, 2012).
4) Depth map processing: Plenoptic cameras are designed so that each micro-image slightly covers its neighbor, providing correspondences between the micro-images which, when the MLA is calibrated, are used to determine the geometrical parameters between the micro lenses, Then are used as a match signal to determine the depth by triangulation.Note that the best triangulation performance is obtained in high contrast regions.Figure 12 shows a depth map obtained on the measurement of a red coral colony.The generation of the 3D model is done somewhat brutally by Raytrix software on the same image we can see the first filter that must be implemented to isolate the relevant parts in the 3D model.

First results
We use a Nikon D800 customized by Raytrix and embedded in a simple Ikelite housing.We change strobe by continuous LED light system which are smaller and easier to use underwater.An example of rough 3D model obtained on a large colony is presented on figure 13.After filtering, the background is removed, which gives the result shown on figure 14.Built with only one shot, this model is obviously less complete than the one estimated thanks to photogrammetry, on figure 7. We can clearly identify missing parts.Another drawback is the price.Currently, a full plenoptic processing chain, including the camera, computer and software to post-process raw images, is sold around tens of thouands dollars.On the other hand, plenoptic approach leads to satisfying results in a few minutes of processing time, whereas photogrammetry needs days or weeks to build the model, mainly depending on the number of photos and the precision we want to reach.

Conclusion
This work results from a more than ten years interdisciplinary collaboration between computer science/photogrammetry and    marine biologists researchers.We have presented innovative tools that help preserve red coral.To achieve the two main objectives -collect morphological data to set conservation goals and improve knowledge on the functioning of the red coral populations.We provide a full chain of treatment, from the methodology of underwater acquisition to the computer processings.All the processing were implemented to take into account the constraints (underwater photogrammetry, easy to carry out, different kinds of measures, with different natures) and to fit the needs of the marine biologists.On the other hand, biologists had to adapt and change their usual methodology of measures.We have considered 3 (main) research axis.
In the first one, using 20x20 quadrats, colony morphological data are characterized and precisely determined, enabling end-users to post-process for more global information about the state of red coral.
In the second one, we are able to provide information about fine scale spatial distribution of colonies to support the survey of temporal changes in a given area and to provide new insights on the functioning of populations.
Finally, we tried different methods (standard photogrammetry, plenoptic, NPR) to build a library of well preserved big colonies, with their associated 3D model and size data.

Future work
This study is dedicated to red coral and many tools have been provided to better understand and characterize the functioning of benthic communities.This approach could be easily extended to other species with similar arborescent growth forms.
Moreover, studying and measuring red coral is a challenging task, that should be automated as much as possible, limiting the role of the end-user to a supervising one in an ideal case.For this purpose, we wish to extend the NPR toolkit we have developed in order to extract morphological and dimensional data.More precisely, white corals can be really thick and dense, so that it is really hard to count the number of branches, for example, even for an expert human eye.Traditional work on images would probably not lead to good results, as the confusion of the branches is important.We aim at simplifying the 3D model and determining the skeleton of coral, so that the branch counting could be automated in this example.

Figure 1 .
Figure 1.In situ observation: the frame is sometimes handled by a diver.

Figure 2 .
Figure 2. Three photographs made on the seabed with the stable frame called "Quadra1"

s
the normal to the interpolated face, t

Figure 7 .
Figure 7. Large red coral colony.More than 50cm, survey by 800 photographs taken in air, in the shop where the colony is for sale.Two days process by photogrammetry.

Figure 11 .
Figure 11.Micro lenses perfectly visible in the raw image.