FOVEON VS BAYER : COMPARISON OF 3 D RECONSTRUCTION PERFORMANCES

The main idea of this particular study was to validate if the new FOVEON technology implemented by sigma cameras can provide better overall results and outperform the traditional Bayer pattern sensor cameras regarding the radiometric information that records as well as the photogrammetric point cloud quality that can provide. Based on that, the scope of this paper is separated into two evaluations. First task is to evaluate the quality of information reconstructed during de-mosaicking step for Bayer pattern cameras by detecting potential additional colour distortion added during the de-mosaicking step, and second task is the geometric comparisons of point clouds generated by the photos by Bayer and FOVEON sensors against a reference point cloud. The first phase of the study is done using various de-mosaicking algorithms to process various artificial Bayern pattern images and then compare them with reference FOVEON images. The second phase of the study is carried on by reconstructing 3D point clouds of the same objects captured by a Bayer and a FOVEON sensor respectively and then comparing the various point clouds with a reference one, generated by a structured light hand-held scanner. The comparison is separated into two parts, where initially we evaluate five separate point clouds (RGB, Gray, Red, Green, Blue) for each camera sensor per site and then a second comparison is evaluated on colour classified RGB point cloud segments.


INTRODUCTION
All modern digital cameras capture radiometric information using a colour filter array named Bayer pattern.This way, each individual pixel of the sensor records only one colour channel.For a complete RGB image, the missing colour values are interpolated to neighbouring pixels through a processing step called de-mosaicking.The only exception to this trend are digital cameras using the FOVEON sensor, exclusively used by Sigma cameras.This sensor uses the physical properties of silicone to catch all colour information in each pixel of the sensor, hence there is no need for post processing to reconstruct full coloured images.The most commonly used cameras in photogrammetry are single sensor cameras using Bayer technology.They are more affordable than 3-CCD (Wootton, 2005) cameras and offer good results in terms of accuracy and precision for 3D modelling techniques.However, new Sigma's camera manages to capture as much information in Red, Green and Blue than 3-CCD cameras for a way much more affordable price.
When it comes to geosciences, quality of the final product directly relies on raw data quality, which is guaranteed by the use of high quality and well calibrated equipment.In photogrammetry that means good lighting conditions, proper setup of camera settings and raw image recording to ensure maximum detail capturing (Stamatopoulos et al., 2012).
Therefore, as FOVEON technology inherently records more information than Bayer in blue, green and red ("Foveon -Direct Image Sensors," n.d.), it is legitimate to investigate if, whether or not, this kind of technology can enhance geometric quality of final photogrammetric products and 3D point clouds in particular.In fact, given that the Bayern pattern consist of 50% green sensitive pixels, 25% red sensitive pixels and 25% blue * dimitrios.skarlatos@cut.ac.cy; phone +357 25002360; fax +357 25002806; www.photogrammetric-vision.weebly.comsensitive pixels, it is expected that the performance of the FOVEON should be considerably better on red and blue channels and marginally better in green.
This study aims to compare the capability of both technologies to build accurate and precise 3D models, against reference 3D point clouds acquired by hand held structure light scanner (Faro Freestyle 3D X).This comparison is based in a number of colourful graffities following a colour classification and comparison of SfM & MVS point clouds generated by two different cameras against the structure light scanner.The devices used to lead this comparison are the Nikon D90 APS-C sensor with Bayer technology and the Sigma Sd1 Merill APS-C FOVEON sensor.
The study is carried out in two phases.First one aims to evaluate the quality of information reconstructed during de-mosaicking step by detecting potential additional colour distortion added during the de-mosaicking step.The de-mosaicking phase is done using Rawpy ("RawPy" 2014), an open source python wrapper for libraw ("LibRaw" 2018) library.It offers the possibility to process some de-mosaicking algorithms like Adaptive Homogeneity-Directed method (AHD) (Hirakawa and Parks, 2005), Discrete Hartley Transform method (DHT) (N.Bracewell, 1983), Bilinear algorithm (DCB), Various number of gradient method (VNG) (Chang et al., 1999), Pixel Grouping method (PPG) (Lin, n.d.), AAHD algorithm.The original FOVEON photos are used as reference (Figure 1) for this test.Artificial Bayer pattern images are created and the Rawpy is used to demosaick the photos, which are then compared against the original FOVEON photos.The differences are quantified and represent a measure of missing information.
The second phase consists in point cloud geometric comparisons against a reference point cloud.First a comparison of five point clouds per sensor, per site is done.There, the RGB, Gray, Red, Green and Blue point clouds derived from each sensor's imagery are compared with a structured light handheld scanner generated 3D point cloud.As a secondary comparison, segments from each sites RGB point clouds are extracted for both sensors and then are compared with the reference hand held scanner point cloud.For each test site only one segment is exported per sensor which refers to each site's dominant colour i.e. for one site the blue points are extracted for the second the green points and for the third the red points are extracted.Perko et al., (2002) presented various alternatives for the creation of digital colour images from an aerial or space sensor.They also explained the issue of "pan-sharpened" images using higher resolution panchromatic images and lower resolution colour images as they addressed the different coloration schemes combining high resolution black and white pixels with the lower resolution colour pixels and observed the differences between the colour images obtained from various approaches.Additionally, an experiment in their work is described to classify the errors committed in de-mosaicking Bayer pattern colour images.Perko et al., (2005) gave an evaluation concept to assess the geometrical accuracy of resulting colour images.In that particular concept, only if no geometrical distortions were created during the de-mosaicking process, it is allowed to use the images for 3D reconstruction or arbitrary metrical photogrammetric processing.Li et al., (2008) made a comparative study evaluating 11 selective inter-channel de-mosaicking algorithms on two different benchmark data sets.The two benchmark data sets that were used are Kodak PhotoCD images and high-quality IMAX images.During their study the authors stated that in order to reduce the risk of mismatches and thus the presence of artefacts in the images, a fuse of de-mosaicking results by different algorithms is recommended.Stamatopoulos et al., (2012) took advantage of the camera's higher dynamic range without applying any pre-processing steps and applied various de-mosaicking algorithms to the RAW images of a Bayer sensor.Following that, the authors examined the extent of variation in the mean positional standard error of object target point coordinates.The authors concluded that all tested de-mosaicking algorithms performed better than the standard in-camera JPEG image formation.Riutort-Mayol et al., (2012) introduced a practical, comprehensive and flexible laboratory procedure to analyse the radiometric values and the uncertainty effects due to FOVEON sensor system.Their procedure was performed on the grey level output signal using image raw units and was entirely based on statistical and experimental techniques.

PREVIOUS WORK
Journes, (2014) proposed to study the impact of the different components of the acquisition chain on the image quality.The author also studies some of the state-of-the-art algorithms and the different quality metrics for image quality assessment.Author states that the choice of an appropriate algorithm is dependent on the frequency content which evolves all over the acquisition chain.
Fent and Meldrum, (2016) Showcased a method using the FOVEON sensor in full spectrum mode in combination with green-pass filters to reproduce the red-magenta hues of healthy vegetation in aerial images without resorting to multi-exposures or channel swaps.During their study the authors developed a colour model for a FOVEON/green-pass filter using quantum efficiencies to explain the colour effects observed.

DE-MOSAICKING PROTOCOL
The de-mosaicking testing procedure, wishes to quantify how much information is omitted in Bayern pattern colour images and how effective the tested algorithms are.The main concept is that the images taken from the FOVEON camera can be used as a reference, which is then converted in an artificial Bayer pattern file and then de-mosaicked with several de-mosaicking algorithms.The recreated image from the de-mosaicking process, can be compared against the FOVEON original image, to evaluate the loss of information.In technical terms, Rawpy Python module, was used for creating the artificial Bayern pattern from original FOVEON images and recreating the images using several de-mosaicking algorithms.The conversion of the FOVEON reference jpeg image I(ref) is done by replacing every pixel value of the Bayer sensor raw image with the radiometric values of FOVEON and respecting a GRBG Bayer pattern and then, de-mosaicking step is followed using the aforementioned 6 algorithms for every I(Bayer) that was derived.The demosaicked resulting images, I(demo), are then compared to reference image to see how different from I(ref), I(demo) is.   Figure 3 shows Red, Green and Blue map of differences for Athena I(ref).To illustrate, Green map is obtained by subtracting Green value from de-mosaicked Athena to Green value from reference Athena.This computation is done for Red, Green and Blue channel and for the 7 different references (Table2, Figure 8).In total 21 difference maps for every de-mosaicking algorithm are produced.

POINT CLOUDS COMPARISON PROTOCOL
The second phase of our study was to evaluate which camera delivers the best datasets to produce accurate 3D point clouds, in correlation with each colour channel.Three different testing sites were chosen for testing each one giving a large emphasis to each colour channel.As reference, point clouds captured by a structured light hand-held scanner sensor, which provides an accuracy better of 1mm at 1m object-scanner distance were used.The capture protocol was designed so that to have a digital image GSD of approx.2.25 mm and a corresponding accuracy on the camera lens axis of approx.4.5 mm.To achieve that, distance from the objects should have been 11m for the FOVEON and 10m for the Bayer as their physical pixel sizes differ slightly (Table 1), and it was necessary to have exactly similar GSD, to be able to compare their geometric accuracies.With the structure light scanner accuracy being 4 times better that the photogrammetric one, the structured light point served as reference.Both the hand-held scanner and photogrammetric point clouds were georeferenced using local Ground Control Point networks in each site separately.

SIGMA SD1 NIKON D90
Focal lens 24mm 24mm Effective pixels 15M 4700x3200 12.2M 4288x2848 The base between captures has been computed for an overlap of 80% and resulting datasets are composed of three photos where the working section is totally visible (Figure 4).Nine Ground control points have been measured with total station to build a common local coordinate system in every test site separately.The photogrammetric pipeline was executed using Agisoft Photoscan.From RGB Bayer and FOVEON datasets red, green and blue radiometric values were extracted to build a Red, a Green and a Blue dataset for each camera.Grey-scale datasets were also computed for each camera, which resulted in 10 blocks in total (RGB, R, G, B and grey-scale for each camera) for each site.According to software's reports, the effective acquisition distance is 10.80m for FOVEON camera and 10m for Bayer Camera, which resulted to a GSD of 2.24mm for FOVEON camera and 2.26mm for Bayer camera, as obtained from bundle adjustment results.

De-Mosaicking Results
The final results of the de-mosaicking algorithms are demonstrated below in Figure 6.For the three sites all algorithms produce I(demo) images with a slight red tone across the images.
To eliminate that and obtain I(demo) images with the same colours as the I(ref), a white balance and a brightness adjustment was applied to try reproducing the colour from the reference image.Unfortunately, after that step the I(demo) images appeared still to have a red tone as it can be seen below in Figure 6.An assumption is that FOVEON channel brightness is affected by the silicon layers.Hence the Blue is more bright and red is less bright, as an average.During the post-processing within the FOVEON camera, this is balanced somehow.Unfortunately, in the six de-mosaicking algorithms that were used, this phenomenon is not taken into account, and that's the reason we are ending up with de-mosaicked images with red tone.
Assuming the above to correct that phenomenon during the demosaicking, the for white balance and brightness must be adjusted accordingly in order to reproduce an I(demo) image visually identical to the I(ref).It is obvious that there is no visual changes or differences with the naked eye between the I(demo) images produced from the 6 different de-mosaicking algorithms.Additionally, this is validated below with the evaluation of the statistical analysis for every algorithm as the average differences between I(ref) and I(demo) are almost identical between the 6 algorithms (Figure 8).
Table 2 shows the Mean of the average differences in radiometric values for Red, Green and Blue channels between I(ref) and I(demo) for seven different images which have been demosaicked with the 6 different algorithms.The results below represent the average from all 6 algorithms per image.The resulting differences to the reference highly depend on the de-mosaicked images.This is linked to the way white balance and brightness adjustment are processed.White balance and brightness don't represent the capture conditions in which I(ref)s have been shot.The results shown in Taking into consideration the "MEAN" row in Tables 2 and 3, it can be noticed that de-mosaicking results are coherent with the Bayer pattern theory.As a Bayer sensor gathers twice more green information than blue or red information, the de-mosaicking interpolation step should reconstruct green missing values better than red and blue ones.As is shown in Figure 3 where the blue difference map is brighter and red is less bright on average, the assumption that FOVEON channel brightness is affected by the silicon layers is verified by the "MEAN" row in Table 2.
The standard deviation calculation statistically eliminates, previously mentioned, white balance and brightness heterogeneous effects.The standard deviation computed varies among the images and the values for different algorithms are comparable between images.Additionally, based on Figure 9 of the Appendix section, values are really close between them and it cannot be decided whether an algorithm is better than another with the applied methodology.Even from the Standard deviation section, Green channel seems to be better reconstructed than Blue and Red, which is coherent with Bayer filter theory.

Point Cloud Comparisons Results
Because of the red tone that the de-mosaicked images appeared to have, for the 3D photogrammetric point cloud generation and comparison, the raw camera files converted into 16-bit tiff files, were used in point cloud generation, instead of the de-mosaicking algorithms data sets.
For the point cloud comparisons two methodologies were followed.The first methodology was the comparison of five different point clouds in their entirety (RGB, Grayscale, Red, Green and Blue) per camera system with the reference.The second comparison was made on the RGB point clouds by classification of points based on the radiometric attributes of specific segments and then comparing those segments with the reference point cloud following a similar procedure and analysis.As it is shown in Tables 4, 5 and 6 Additionally, as we see from the comparisons, the Bayer Red point clouds for all the graffities were by far the most erroneous.

5.2.1
That could be explained by the fact that while blue and green channels have specific wavelengths, that is not the case for the red.In some sensors, information about the Red Edge channel might be recorded within the red channel, where in other sensors only the strict red band width is recorded.This difference might affect the point cloud extraction procedure.The bandwidth that both sensors record the radiometric information in the Red channel is unknown to this study thus we can only make a hypothesis for the above statements.Further tests could prove the aforementioned but that is not the focus of this study.
Based on the above comparison, FOVEON appears to be a superior sensor than Bayer regarding the photogrammetric point cloud generation with less noise as the standard deviation and RMS error values show in most of the comparisons.That is what was initially expected, since FOVEON records in all pixels original radiometric information of Red, Green and Blue, instead of interpolating two channels in every pixel.

Cloud Segmentation and classification
In this phase, a classification of the RGB point clouds based on specific point selection was made.More specifically, a manual selection was made on specific regions to extract only points with radiometric attributes similar to the dominant band of each site.
After the point classification a comparison was carried on against the reference point cloud provided by the hand-held laser scanner.In the case of Athena and Random Red graffities it is clear that Blue and Red classified points of both sensors have the better statistics where the Green classified points in mermaid are worse.

Athena
In the case of mermaid graffiti, the mean distance, standard deviation and RMS error for Green classified points are worse because of the lack of distinct green points.Additionally, Bayer sensor selected points have better statistics than FOVEON's in that particular scenario.
Even though in this comparison FOVEON outperforms Bayer for Red and Blue, we cannot consider this comparison as most reliable, since the two sensors record different radiometric information for the same objects.Because of that, a proper classification based on the radiometry of the reconstructed objects, cannot be made as different areas and points with different radiometric properties will be selected respectively.This is shown in the case of the blue classified points of the Athena graffiti where many of the points selected by the classifier were closer to red as well as many points in the Random Red graffiti are classified as red but in reality, are white (Figure 7).
For the case of Mermaid, the number of green classified points for FOVEON is quite large in comparison with Athena potentially due to the lack of distinct green points as blue and green are almost equally dominant.

CONCLUSIONS AND FUTURE WORK
This paper describes the difficulty to reconstruct missing colour values of a Bayer pattern image and tries to evaluate and compare overall quality of 3D point clouds obtained with datasets resulting from different sensor colour capturing technologies.Additionally, our test using 6 selected de-mosaicking algorithms showcased the difficulties of true colour reconstruction from a Bayer pattern raw file.
FOVEON technology is simpler in terms of post-process.It captures more information and less processing is needed for the final image, but more time is needed for recording data from sensor to card.One of the main limitations is its use to low light conditions as high ISO shots produce a lot of noise.It still needs white balance and brightness adjustment to deliver sharp and properly coloured images.On the other hand, Bayer technology needs additional process steps to reconstruct images like the demosaicking step plus a white balance and brightness adjustment are necessary.
In a terrestrial photogrammetric context, both technologies do well.FOVEON technology's performance regarding the accuracy and noise is superior based on the results of this study.
By evaluating the single channel point clouds, the difference in the radiometric information between the two technologies become more noticeable as Bayer's resulting geometric quality is on average inferior to FOVEON's.
As it is easier, and less challenging to test and compare these technologies in a terrestrial photogrammetric context, a future study can be made to compare the performance of Bayer and FOVEON technologies in aerial or underwater imaging.In order to perform the aforementioned tests, the heavy cameras, need to be fitted on an octacopter, which is feasible.For the underwater environment though, although Nikon D90 has several commercial underwater housings, Sigma lacks such accessory.

Figure 1 :
Figure 1: Array of images containing I(ref)s used for the testing (From left to right, top to bottom): Athena, aZoomed, pinkGraf, Miami, redWhistle, sLips, Sirena.

Figure 3 :
Figure 3: From Top to bottom: Athena, Red, Green and Blue difference maps for AHD algorithm

Figure 4 :
Figure 4: Capture plan of images as is shown from the UI of Agisoft Photoscan
, for both sensors and the C2C comparisons, the mean distances from the reference for the RGB and grayscale point clouds are as expected, i.e FOVEON has the smaller mean distance from the reference in every single comparison.In the individual colour comparison, the mean distance between the FOVEON PCs and the reference ones are -0.624mm(Athena Blue) to -0.073mm (Random Red Green) which is much smaller (on absolute values) than the GSD of the acquired images.Regarding the Standard deviation and the RMSE of FOVEON the values are varying from 2.4mm to 3.6mm.As for the Bayer sensors the standard deviation and RMS error are overall worse than FOVEON varying from 3.2mm to 10.2mm and 3.2mm to 10.5mm respectively.Regarding the Bayer sensor the mean distances, StDev and RMS error from the reference PC are not similar to all channels.The Green PC has the smallest values regarding the individual colour comparisons in the Athena and Mermaid Graffities where the blue PC has the smallest values and thus the better quality in the case of the Random Red Graffiti for both sensors.

Table 1 :
Camera Specifications

Table 2 :
Table 2 & 3 should be interpreted carefully as in the derived images' white balance and brightness correction parameters may carry heterogeneous errors from one image to another.The analytic results for every algorithm's performance are shown in Figures 8 & 9 in the Appendix section.Mean of Average Differences in radiometric values for Red, Green, Blue of the 6 De-mosaicking algorithms for every dataset.

Table 3 :
Mean of Standard Deviations in Red, Green, Blue of the 6 De-mosaicking algorithms for every dataset.

Table 4 :
Cloud to Cloud Comparisons Mean Distance, Standard Deviation and RMS error of every point cloud from the reference point cloud for Athena graffiti.

Table 5 :
Mean Distance, Standard Deviation and RMS error of every point cloud from the reference point cloud for Mermaid graffiti.

Table 6 :
Mean Distance, Standard Deviation and RMS error of every point cloud from the reference point cloud for Random Red graffiti.

Table 7 :
Mean Distance, Standard Deviation and RMS error of Red, Green and Blue classified points for every respective site.