THE EFFECT OF UNDERWATER IMAGERY RADIOMETRY ON 3D RECONSTRUCTION AND ORTHOIMAGERY

The work presented in this paper investigates the effect of the radiometry of the underwater imagery on automating the 3D reconstruction and the produced orthoimagery. Main aim is to investigate whether pre-processing of the underwater imagery improves the 3D reconstruction using automated SfM MVS software or not. Since the processing of images either separately or in batch is a time-consuming procedure, it is critical to determine the necessity of implementing colour correction and enhancement before the SfM MVS procedure or directly to the final orthoimage when the orthoimagery is the deliverable. Two different test sites were used to capture imagery ensuring different environmental conditions, depth and complexity. Three different image correction methods are applied: A very simple automated method using Adobe Photoshop, a developed colour correction algorithm using the CLAHE (Zuiderveld, 1994) method and an implementation of the algorithm described in Bianco et al., (2015). The produced point clouds using the initial and the corrected imagery are then being compared and evaluated.


INTRODUCTION
Underwater 3D modelling and mapping techniques are based on various systems and methodologies and most of these methods are based on RGB imagery as primary data (Drap 2012, Henderson et al., 2013, Johnson-Roberson et al., 2016).However, despite the relative low cost of these methods in relation to others, they present a major drawback; optical properties and illumination conditions of water severely affect underwater imagery.Colours are lost as the depth increases, resulting in a green-blue image due to light absorption, which affects mainly red wavelength.Therefore, red channel histogram has fewer values compared to green and blue.Water also absorbs light energy and scatters optical rays creating blurred images.
During the last decades, the recovery of the correct or at least realistic underwater colour imagery is a very challenging and promising research field which affects the 3D modelling and mapping techniques.To overcome these problems, two different approaches for underwater image processing are found in the literature.
The first one is image restoration.It is a strict method that attempts to restore true colours and correct the image using suitable models which parameterize adverse effects such as contrast degradation and backscattering, using image formation process and environmental factors with respect to depth (Hou et al., 2007, Treibitz andSchechner, 2009).
The second one uses image enhancement techniques based on qualitative criteria such as contrast and histogram matching (Ghani and Isa, 2014, Iqbal et al., 2007and Hitam et al., 2013).Image enhancement techniques do not consider image formation process and do not require environmental factors to be known a priori.

Motivation
The main aim of the work presented here is to investigate whether preprocessing of the underwater imagery improve the 3D reconstruction using automated SfM -MVS software or not.Visual computing in underwater settings is particularly affected by the optical properties of the surrounding medium (von Lukas 2016).Since the processing of images either separately or in batch is a time-consuming procedure, it is critical to determine the necessity of implementing colour correction and enhancement before the SfM -MVS procedure or directly to the final orthoimage when this is the deliverable.

METHODOLOGY
In order to address the above research issues, two different test sites were used to capture underwater imagery ensuring different environmental conditions, depth and complexity.In addition, three different image correction methods are applied to these datasets: A very simple automated method using Adobe Photoshop, a developed colour correction algorithm using the CLAHE (Zuiderveld, 1994) method and an implementation of the algorithm described in Bianco et al., (2015).Subsequently, dense 3D point clouds (3Dpc), 3D meshes and orthoimages were generated for each dataset.The produced 3D point clouds were then compared using Cloud Compare freeware.

Shallow waters
The dataset created for shallow waters (Figure 1a) is a nearshore underwater site situated at depths varying from 2 to 3 meters.Image acquisition took place with a Nikon DSLR D5200, with 24 MP sensor and pixel size of 3.92μm and by using an 18-55mm lens set at 18mm, and an Ikelite dome housing.No artificial light sources were used due to the small depth.Due to the wind, the wavy surface of the water creates dynamic sun flicker (caustics) on the seabed.Waves also resulted into water turbidity and thus very poor visibility conditions.

Deep waters
The amphorae dataset is an artificial reef constructed using 1m long amphorae, replicas from Mazotos shipwreck (Demesticha 2010).They are positioned in the sea bottom at 23 meters depth, stacked together in two layers, with 25 and 16 amphorae respectively (Figure 1b).Their placement is assumed to be similar to their original position in the cargo area of the ship.Photography took place with an action camera, Garmin Virb XE camera, with 12MP sensor with 1.5μm physical pixel size.Four LED video lights were mounted next to the camera, to enhance recorded colour information.
As it is expected due to it's depth for the shallow waters dataset, green is the dominant colour while red colour is also present.In contrast, in the deep waters dataset, the blue is the dominant colour.Although red channel absorption is strong in such depth, red is present in the histogram of the photos, depending on the lights and on the camera to object distance.(Zuiderveld, 1994) algorithm partitions the image into contextual regions and applies the histogram equalization to each one, while it limits contrast.This evens out the distribution of used grey values and thus makes hidden features of the image more visible.CLAHE algorithm has been used extensively in underwater image correction in the literature (i.e.Kumar Rai, et al., 2012, Yussof, et al., 2013, Singh, et al., 2011and Hitam, et al., 2013) The implemented image enhancement algorithm separates image channels of RGB colour space.Then a histogram equalization process is applied to each channel.A Rayleigh Cumulative Distribution Function (CDF) was created for the equalization.The CDF range is from 0 to 255 in order to match the pixel intensity values and its maximum value was appointed to one third of total range (76.5).Afterwards the CLAHE (Zuiderveld, 1994) algorithm was applied to each channel.Partition size and contrast clipping were defined experimentally.
Rayleigh distribution was used again as transformation function parameter.Finally, the algorithm composes all three channels and the output is the colour corrected image.
In shallow waters dataset applying histogram equalization using a Rayleigh CDF on each colour channel results in histograms of practically identical shape.This dramatically improves image sharpness and colours seem to be restored.In deep waters dataset the algorithm was modified in order to cope with the large depth conditions.Red channel was not equalized before CLAHE algorithm correction while green and blue channels were equalized with a different CDF in order to restrict their intensity into lower values.The imagery was also processed using the algorithm presented by Bianco et al., (2015), where colour correction of underwater images is performed by using lab colour space.In more detail, the chromatic components are changed moving their distributions around the white point (white balancing) and histogram cut-off and stretching of the luminance component is performed to improve image contrast.Main constrains of this method are the grey-world assumption and the uniform illumination of the scene (Bianco et al., 2015).
I shallow water dataset, the corrected imagery looks very realistic as all the colours are correctly enhanced.However, the sharpness of the imagery is not well improved.In deep waters dataset the corrected image presents enhanced contrast and despite the fact that the image looks similar to a greyscale image due to the absence of the red colour at such depths.According to the software this algorithm analyzes the image in order to find dark and light colours and uses them as the shadow and highlight colours.The option "Snap Neutral Midtones" was also checked.This adjusts the midtones so that colours close to neutral are mapped to the target neutral colour.
In shallow waters the dataset image blurriness is reduced and colours become more realistic.In contrast the correction has little effect on the imagery acquired in deep waters.Contrast is slightly increased while colours remain unaffected.For the deep waters dataset this method was proven ineffective and a different algorithm was used.It clips colour channels identically in order to increase contrast while it preserves the original colours.The final image is improved partially.Red prevails on the image corners because of the chromatic aberration effect provoked by the fish-eye lens of the action camera.To sum up, for the shallow waters dataset, all the correction algorithms improve the colours of the imagery.The CLAHE based algorithm improves more the image sharpness, however, the more realistic colours are resulting by using the algorithm of Bianco et al., (2015).For the deep waters dataset, the results of Adobe Photoshop and CLAHE based algorithms present more percentage of red values and thus the imagery looks more appealing to the human eye.However again the results of the algorithm of Bianco et al., (2015) resemble more to underwater environment reality, even if the red colours are undervalued.

SfM -MVS procedure
These image enhancement methods were evaluated by visual inspection and histogram comparison.Corrected images were processed using SfM -MVS with Agisoft's Photoscan software.Four different 3D projects were created for each test site.
One with the original uncorrected imagery which is considered the initial solution, a second one using the developed correction algorithm applying CLAHE (Zuiderveld, 1994), a third one using the imagery which resulted implementing the colour correction algorithm presented in Bianco et al., (2015) and a fourth one using Adobe Photoshop enhanced imagery.
All three channels of the images used for these processes.For the created projects of each test site, the alignment parameters of the original (uncorrected) dataset were used as the corrected images were replacing the uncorrected ones.To this end, the alignment parameters were the same in order to test only the number of points extracted for the dense cloud.In order to scale the 3Dpc, predefined GCPs were used for the shallow water projects.Scalebars used for scaling of the deep water dataset projects.Subsequently, point clouds of medium quality and density were created for each data set.No filtering during this process was performed in order to get the total number of dense could points, also the noise.
Table 1 sums up the results of the aforementioned processing for medium quality dense cloud generation.For the shallow waters dataset, an area of 21.3m 2 was covered by 155 images, having an average camera to object distance 1.57m and thus resulting in a ground resolution of 0.304mm/pixel (Figure 5).
For the deep water dataset, an area of 8.01m 2 was covered by 89 images, having an average camera to object distance 1.68m and thus resulting in a ground resolution of 0.743 mm/pixel (Figure 8).It must be noted that the percentages of the differences of dense point cloud number from the original ones are not important.These differences are magnified when it comes to the deep waters dataset and this is probably due to the complexity of the object in respect with image correction methodology.

Orthoimage Generation
Orthoimages were created for both test sites.The main aim of this procedure is to determine the necessity of implementing colour correction and enhancement before the SfM -MVS procedure or directly to the final orthoimage when the orthoimagery is the deliverable.To this end, orthoimages were generated for every different project using the original imagery (Figure 6a, 9a), the imagery corrected with Adobe Photoshop (Figure 6b, 9b), the imagery corrected by the CLAHE based algorithm (Figure 6c, 9c) and the imagery corrected by the algorithm of Bianco et al., (2015) (Figure 6d, 9d).For the two datasets, GSD was selected to be 0.005m and disabled the colour correction mode.
Additionally to this process, the two final orthophotos resulted from the projects of the two test sites using the original imagery (Figure 7a, 10a), were processed with the three colour correction methods (Figure 7b,c,d and Figure 10b,c,d) and the results compared and evaluated and the results are illustrated in Section 3.

Evaluation of the resulting Orthoimages
In order to address the necessity of implementing colour correction and enhancement before the SfM -MVS procedure or directly to the final orthoimage when it is the deliverable, orthoimages using the correcting imagery were generated for each dataset.On the contrary, the orthoimages created using the original uncorrected imagery were then corrected using the aforementioned algorithms.
By comparing both the visual appearance and the histograms of the results illustrated in Figures 6-7 and Figures 9-10, occurs that the implementation of the specific image enhancement techniques do not affect in a remarkable way the produced orthoimagery.There are minor differences between the orthoimages resulting from the corrected imagery and the respective corrected orthoimage.
However, it must be noted that the orthoimages resulting from the corrected imagery of the CLAHE implementation and the Bianco et al., (2015) algorithm are more sharp and high contrasted than the respective directly corrected orthoimages.The opposite happens for the Adobe Photoshop orthoimages for the shallow waters dataset only.These results are confirmed both from tests performed by using the shallow waters dataset and the deep waters dataset.
Additionally, orthophotos produced were compared through image subtraction in QGis software.Orthophotos were inserted to QGis software and subtracted in respect to the colour correction methods, however the results are not illustrated in this paper.Results also suggests that there is not any notable geometric difference between the orthoimages.

Test and Evaluation of 3D point clouds
The generated dense point clouds were compared within Cloud Compare freeware.Differences between 3D point clouds were illustrated with a colour scale bar (Figures11a,b and Table 2).In order to ignore outliers, the maximum distance between the respective compared points of each 3D point cloud was set at 0.02m and the colour scale was divided into 8 levels of 0.0025m.
Figures 11c-h present the approximate number of points of the 3D dense point clouds created using the corrected images that deviate from the 3D dense point cloud resulting from the original images in relation to the value of that deviation.In Figures 11c,d,e it is observed that over 91% of the points resulted from the corrected images of all algorithms deviates less than 0.0025m from the point cloud resulting from the original images.However, all the points deviated less than 0.005m.
In Figures 11f,g,h it is observed that for the 3D point clouds of the CLAHE corrected imagery, over 34% of the points resulted from the corrected images of all algorithms deviates less than 0.0025m from the point cloud resulting from the original images.The respective percentage for Bianco et al., (2015) is 26% and for Adobe Photoshop corrected images 23%.The rest 49% of the 3D point clouds of the CLAHE images represents deviations from 0.0025m to 0.01m while the same percentages for Bianco et al., (2015) is 72.5% and for Adobe Photoshop corrected images 69.8%.The rest 17 %, 1.5% and 7.2% of points respectively represent errors from 0.01 to 0.02m.
As it is observed the results of the comparison of the 3D point clouds of the deep waters dataset have larger percentages of errors in relation with the 3D point clouds of the shallow water dataset.This is because of the different complexity of the objects in the dataset well the impact of the increased depth to the original colours.
Additionally, it is noted that for shallow depth dataset, the 3D point cloud comparisons resulted in almost the same percentages and this is due to the small depth and mainly due to the non-complex captured environment.These are also confirmed by Table 2 showing that the mean and the sigma of the differences between 3D point clouds in Cloud Compare freeware are smaller for the shallow test site.Moreover, even if the 3D point cloud of the CLAHE based corrected images presents the largest percentage of points that differ less than 0.0025m from the point cloud resulting from the original images, the 3D point cloud of the images resulting from the correction of Bianco et al., (2015) has most of it's points between 0.0025 and 0.01m while it presents the smallest percentage of deviations larger than 0.01m for deep waters dataset.
However, the 3D point clouds resulting from all the correction algorithms do not present any important deviation from the 3D point cloud of the original imagery.Most of these insignificant differences are resulting mainly due to noise introduced by the alteration of the image radiometry, a fact which may misleads the image matching algorithm.As already mentioned, the underwater environment affects the underwater image radiometry, even in shallow waters.Results show that the number of automatically generated points in the point clouds is altered for each image enchancment method, without providing useful extra information and in some cases adding noise to the object.The specific image enhancement techniques on shallow and deep depth underwater imagery do not seem to affect the 3D reconstruction.Colour correction before automated photogrammetric procedure does not seem to have any important impact on the final orthophoto and as such, the stage of using the specific image enhancement processes is subjective.

CONCLUDING REMARKS
Concluding, the tested enhancement and correction methods do not seem to improve significantly 3D reconstruction effectiveness in Agisoft's Photoscan software and for the specific types of test sites and depths.Point cloud comparisons in Cloud Compare software showed minor differences in relevant accuracy, when the noise is ignored.In addition, orthoimages subtraction did not suggest any important differences.image enhancement methods mentioned above improve image visual quality and make them more for human the human eye.However, they do not improve feature detection on the process and overall 3D reconstruction in the specific SfM -MVS software.
When it comes to underwater orthoimage production applications, the initial underwater image processing seems not necessary since simply the colour enhancement and correction of the produced orthoimagery is sufficient and time efficient.However, more tests must be performed for more test sites of different depths and complexity and by using more colour correction algorithms and SfM -MVS software.
Sample images from shallow waters dataset (a) and deep waters dataset (b) corrected (a) imagery with CLAHE based algorithm for shallow and deep (b) water 2.2.2 Algorithm of Bianco et al. (2015) Corrected (a) imagery with Lαβ for shallow and deep (b) water 2.2.3 Adobe Photoshop ® Additionally, images were processed with Adobe Photoshop in order to enhance contrast and sharpness.Automated algorithm "Find Dark and Light Colours" was used for the correction.

Figure 4 .
Figure 4. .Corrected (a) imagery with Adobe Photoshop ® for shallow and deep (b) water Medium point cloud for shallow waters dataset from the original imagery (a), the imagery corrected by the CLAHE based algorithm (b), the imagery corrected by the algorithm of Bianco et al., (2015) (c) and the imagery corrected with Adobe Photoshop The orthoimage of the shallow dataset created by using the original images (a), by using the imagery corrected by the CLAHE based algorithm (b), by using the imagery corrected by the algorithm of Bianco et al., (2015) (c) and by using the imagery corrected with Adobe Photoshop (d) The orthoimage of the shallow dataset created by using the original images (a), by correcting the (a) using the CLAHE based algorithm (b), by correcting the (a) using the algorithm of Bianco et al., (c), (2015) and by correcting the (a) using the Adobe Photoshop Figure 8. Medium quality point cloud for deep waters dataset from the original imagery (a), the imagery corrected with Adobe Photoshop (b), the imagery corrected by the algorithm of Bianco et al., (2015) (c) and the imagery corrected by the CLAHE based algorithm (d) The orthoimage of the deep waters dataset created using the original images (a), by using the imagery corrected by the CLAHE based (b), by using the imagery corrected by the algorithm of Bianco et al., (2015) (c), and by using the imagery corrected with Adobe Photoshop (d) The orthoimage of the deep dataset created by using the original images (a), by correcting the (a) using the CLAHE based algorithm (b), by correcting the (a) using the algorithm of Bianco et al., (2015) (c) and by correcting the (a) using the Adobe Photoshop (d) 3Dpc comparisons in Cloud Compare: Examples of the results of the comparison of the 3Dpc of the shallow and deep site (a,b) and histograms of deviations from the 3Dpc of the original images for shallow waters: CLAHE-shallow(c), Bianco et al. (2015)(d), Photoshop (e) and for deep waters: : CLAHE-shallow(f), Bianco et al. (2015)(g), Photoshop (h)

Table 1 :
Results of the SfM -MVS procedure for medium quality dense cloud generation

Table 2
Differences between 3Dpcs in Cloud Compare freeware