GEOMETRIC PERFORMANCE OF A CAMERA WITH SINGLE SENSOR AND MULTIPLE HEADS

Remote sensing with lightweight optical sensors is becoming a powerful tool to solve many problems in agriculture. Achieving the level of spatial and spectral resolutions required for this type of detection at an acceptable cost-benefit ratio has motivated the development of new sensors which must be lightweight to be carried by mobile robots either aerial or terrestrial. One new type of multiple head cameras has been developed by Agrowing, an Israeli company developing technology for digital agriculture. The aim of this paper is to analyse the geometric features of an Agrowing dual head camera trough calibration experiments. The sensor was calibrated following two options, depending on the cropping technique used to produce the 4 spectral bands. Different calibration techniques were also used and very accurate results were achieved. Experiments with data collected with a UAV also confirmed the results achieved with close range calibration. * Corresponding author


INTRODUCTION
Proximal remote sensing is emerging as a powerful technique for precision agriculture, especially for detection of crop diseases and quality of plant nutrition (Nansen, 2016) (Murray et al., 2019). Achieving the level of spatial and spectral resolutions required for this type of detection is also challenging. Unmanned Aerial Vehicles (UAV) or Mobile Mapping Systems are recognized as suitable platforms to carry lightweight sensors with detection capabilities, like multispectral or hyperspectral cameras. There are many models of lightweight multispectral cameras, with some differences in geometry, weight and costs (Nebiker et al., 2016). Multispectral, light-weight and low-cost systems usually combine multiple cameras to acquire different spectral bands. Cameras such as MCA_Tetracam, MAIA, MicaSense and Parrot Sequoia have become common alternatives in the UAV applications. There are some drawbacks when using multiple cameras to acquire multispectral images, such as the triggering synchronization, the relative calibration and bands registration and relative exposures. New solutions are being proposed combining commercial available cameras with special optics. One new type of multiple head cameras has been developed by Agrowing, an Israeli company developing technology for digital agriculture.
The aim of this paper is to analyse the geometric features of an Agrowing dual head camera trough calibration results. This camera will be used in several agricultural applications and the knowledge of its internal features and a rigorous calibration procedure will enable optimizing the photogrammetric pipeline.

CAMERA FEATURES
Agrowing (Agrowing, 2019) design principle uses existing and well proven sensors, such as from Sony adapting these cameras with filters and special multi-lenses mounts. Following this concept, they offer two groups of multi-lenses systems: a dual head system with different filter sets to produce 4 spectral bands for Normalized Difference Vegetation Index (NDVI) or Red Edge configurations ( Fig. 1.a) and; a four heads system ( Fig. 1.b) which can produce up to 10 spectral bands. These cameras are compatible with modified Sony ILCE QX1, UMC-R10C, Alpha 5000, Alpha 5100, NEX5, Alpha 6000, Alpha 6300 and Alpha 7Rxx (Agrowing, 2020).
All bands are acquired instantaneously because the lenses redirect the bundles to specific parts of the sensor (see Fig. 1.c for a schematic concept of the dual head system and Fig. 2.a for an example of the original frame acquired). The spectral bands are generated by combining the camera Bayer filter and the customized lenses filters, after removing the original infrared filter from the camera. Image acquisition is performed with a single shot, generating a compound frame, with several tiles, depending on the number of heads. Fig. 2.a shows an example of image acquired by a dual head system and Fig. 2.b an image acquired by the 4 heads system (10 bands). The spectral bands can be generated by cropping the original frames and splitting them into spectral bands. Agrowing proprietary software crops the original image frame and splits it into selected spectral bands. Bands co-registration is ensured by an algorithm which extracts interest points in bands of the original tiles for different heads, computing an optimal crop origin and size, thus avoiding image resampling for rectification. This also compensates for slight misalignments of the optical axis of different lenses. Another option is to define a fixed crop origin and size for all images. Fig. 2.c shows the software screen which can save spectral bands or products of spectral indexes, such as NDVI and Soil Adjusted Vegetation Index (SAVI) (Fig. 2.d and e). The spectral bands can be combined in multipage tiff and used in commercial software like Agisoft Metashape (Agisoft, 2020). Considering these novel features of these sensors, it is relevant to analyse their geometric consistence by applying conventional photogrammetric procedures. In this paper the images acquired with the dual head camera and split in 4 spectral bands (450 nm, 550 nm, 710 nm, and 850 nm) will be used in a camera calibration procedure with multi-camera concept, as presented by Tommaselli et al. (2013).

Type
Dual lens

MULTICAMERA CALIBRATION
Camera calibration is a technique for determination of inner orientation parameters (IOPs), usually, focal length, principal point coordinates and lens distortion coefficients (Brown, 1971) (Clarke and Fryer, 1998). The estimation of such set of parameters have to be done by indirect observations of points or linear features from several images, a process known as bundle adjustment, which uses Least Squares Method (LSM). Collinearity equations added with lens distortion model is the mathematical model usually used. Using this model besides the IOPs, also the exterior orientation parameters (EOP) and object coordinates of photogrammetric points are simultaneously estimated by the LSM from the image observations. Additional constraints can be introduced to define the object reference frame, enabling the estimation without any control points (Clarke and Fryer, 1998).
When using several cameras tightly attached is a stable mount the IOPs of all cameras can be determined simultaneously. There are many approaches for multi-camera calibration: some of them use modified collinearity equations (King, 1994) whilst others introduces some constraints in the relative orientation between cameras (Detchev et al., 2014) (Detchev et al., 2018).
In this paper we are using weighted constraints on the stability of relative orientation parameters, as described in Tommaselli et al. (2012), Tommaselli et al. (2013) and Campos et al. (2018). Similar approaches were also used by Lichti et al. (2015).

METHODOLOGY AND EXPERIMENTS
The analysis of the geometric performance of Agrowing dualhead camera was done in several steps. The first step is to analyse the results of automatic image crop. Then, in a second step, the camera was calibrated using techniques of multicamera calibration, with two different crop techniques: automatic and fixed. Finally, independent assessment with bundle adjustment of aerial images was performed.
First, 22 images of a terrestrial calibration field were collected. Each frame was processed to crop the individual bands by using Agrowing proprietary software (Fig. 2.c). Five scenes with four bands were selected and analysed. The cropped image bands were compared with respect to the original image to determine the crop origin. The coordinates of the interest points were detected and matched with feature-based matching both in the cropped image bands and in the original frame with Agisoft Metshape (Agisoft, 2020). The differences in image coordinates of point's pairs (rows and columns) were computed and averaged along with the standard deviations of the average. With these matched pairs, the origins of each cropped band with respect to the original image frame were computed. From the results it was observed that the standard deviations of the average translations were always smaller than 0.5 pixels, which is compatible with the precision of feature based matching.  Table 2. Average values of the origin of crops automatically generated.
Two sets of images were then produced to be used in the camera calibration step. The first set was composed of those image bands automatically generated and cropped by the Agrowing proprietary software and the second one with images automatically cropped using fixed origins (column 252, row 9, for the three visible bands and, column 3235, row 14 for the infrared band). Thus, in some images there will be a slight displacement of 1 pixel between the different crops. As it can be seen from the crop origins, the difference in y origin (9 pixels for visible head and 14 pixels for infrared head) compensates optical heads misalignment.
Then, each camera head was independently calibrated using a calibration field composed of coded targets of ARUCO type and circular coded targets (Fig. 3). Each target in the calibration field consists of a rectangular external crown and 5x5 internal squares arranged in five rows and five columns. The four targets' corners are automatically located and the target label is identified, enabling the association with ground coordinates and points extracted in other images. To improve precision the centroids of the four corners are calculated and used as virtual points. Image coordinates of circular targets were also extracted with Agisoft Metashape and both sets of observations were merged. Self-calibrating bundle adjustment was performed using a minimum of seven absolute constraints, defined by the 3D coordinates of two points and the Z coordinate of a third point. The distance between the two 3D points was accurately measured using a precision rule defining the scale of the photogrammetric network. The coordinates of the remaining ground control points (GCPs) were introduced in the bundle adjustment calculation as weighted constraints with a standard deviation of 50 cm, since they were known from a previous trial. The IOPs, the EOPs and the ground coordinates of the tie points were simultaneously estimated in the bundle adjustment The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B1-2020XXIV ISPRS Congress (2020 based only on internal information. In this study, the IOPs comprised the coordinates of the principal point, the focal length, radial and decentering distortion coefficients. Affine parameters were included in the preliminary tests but they were later removed since their effects were lower than image measurement errors. The self-calibrating bundle adjustment was performed using the in-house-developed software CMC (Calibration with Multiple Cameras), which uses a least-squares combined model with constraints (Tommaselli et al., 2012). Stability constraints on the relative orientation parameters were also used as constraint in the bundle block adjustment with different weights.  After the calibrations trials with the characteristics presented in Table 3, results were analysed and discussed.

RESULTS AND DISCUSSION
The first set of analysis was performed in the values of the estimated IOPs and estimated standard deviations with the automatically cropped images processed by single camera calibration (Configuration A in Table 3, results in Table 4 and  Table 5). As previously mentioned, affine coefficients were considered insignificant and thus removed from the set of IOPs to be estimated in all experiments. Table 4 presents the estimated values of the IOPs with images from the visible bands (columns 2, 3 and 4), the average values of IOPs standard deviations estimated in the bundle adjustment (column 5) and standard deviations of the average values from the 3 bands (column 6). It can be seen that the variations in focal length and principal points coordinates are smaller than one pixel. The effects of variations in the distortion coefficients were also smaller than the measurement errors indicating that a common value could be used for the visible bands. The sigma naught of each calibration trial is also presented in the last row and it can be seen that the values are similar (approximately 0.0008 mm, which corresponds to 0.2 pixels).   Table 5. Results obtained with the automatically cropped images: estimated values for the IOPs of images of the infrared, estimated standard deviations and differences with respect to the average value of visible bands.
The second set of analysis was performed in the values of the estimated IOPs and estimated standard deviations with the crops performed with a fixed origin and the images also processed by single camera calibration (Configuration AF in Table 3, results in Table 6 and Table 7). The results are similar to those achieved with the automatically cropped images although some minor differences can be noted. Comparing values estimated for the focal lengths and principal point coordinates using both sets of cropped images, it can be seen differences are smaller than 1 pixel, except for focal length of IR band and y o coordinate of bands 550 nm, 710 nm and 850 nm. These differences could be explained by the variations in the crops origin in some images automatically cropped.  Table 7. Results obtained with the images cropped with a fixed value for the crops origins: estimated values for the IOPs of images of the infrared band, estimated standard deviations and differences with respect to the average value of visible bands.
Since the IOPs of images of the visible bands are similar, the next group of experiments were performed with two sets of images from two image bands: red band (710 nm) and IR band (850 nm). In these experiments the two camera heads are calibrated simultaneously. In experiments B and BF the simultaneous calibration was solved with any constraints. In experiments C, CF, D, DF, D and DF stability constraints were applied, according to configurations presented in Table 3.  It is clear that residuals in the simultaneous adjustment of two image bands taken from different heads are slightly higher than single camera calibration, but still subpixel. This was expected since in the simultaneous calibration the object point coordinates are computed together with the IOPs and EOPs of both camera heads. Comparing results achieved with fixed crop (Experiments AF, BF, CF, DF and EF) with the corresponding ones performed with automatically cropped images, it can be seen that residuals are smaller, probably reflecting the changes in crop origin for some images, as previously presented.
The estimated standard deviations for focal length, principal point coordinates of both camera heads for all experiments are presented in Figure 5. The estimated standard deviations of lenses distortions were similar for all experiments and will not be detailed in this paper. From the analysis of Figure 5, it can be seen that the estimated standard deviation for the IOPs were slightly smaller when using fixed crops. In both groups (automatic and fixed crop) the results with simultaneous adjustment of two heads were better than single camera calibration. The results with stability constraints were better and stable for the fixed crop; for the automatic crop imposing stability constraints caused worse results.  In Figure 6, the differences in estimated IOPs (focal lengths and coordinates of principal points) of two bands for all experiments are computed with respect to IOPs estimated in Experiment BF. In Figure 6, index (1) refers to band 710 (first head, visible bands) and index (2) refers to band 850 (second head, IR band). It can be seen that there are higher variations when using the automatically cropped images and that differences were reduced for the fixed cropped images. The differences in the IOPs when imposing stability constraints were also smaller than 0.5 pixels for the fixed cropped images. These results show that imposing this kind of constraints does not affect significantly the IOPs with the advantage of getting very accurate relative orientation parameters.
The set of graphics grouped in Figure 7 shows the behaviour of the relative orientation parameters in the experiments. It is important to emphasize that these values were not inserted as unknowns in the bundle adjustment but, rather, they were computed from the estimated EOPs from pairs of images taken with both heads at the instant. The average values for the relative orientation elements are then computed along with the standard deviation of the average.   Figure 7.c and 7.d represents the differences with respect to IOPs values estimated in CF experiment. In this representation the results with stability constraints were choose as reference since they provided the best results for the relative orientation parameters. It can be seen that the estimated values for b x reflects the physical displacement between heads (1.177 cm in experiment CF), b y and b z (0.013 cm and 0.02 cm, respectively, in experiment CF) are small but not negligible because they compensate the physical offsets. The average rotation  is practically null, but  and  have noteworthy values (3' 24.1'' and 5' 20.4'' respectively), with a standard deviation of the average small than 0.03'', which is practically null. Table 8 presents the average values and the standard deviation of the average for the experiment CF, whilst the values for all experiments are depicted in Figure 7 and the standard deviations in Figure 8. Figure 8 presents the standard deviation of the average values for the relative orientation parameters and it is clear that the best results were achieved with stability constraints in experiment CF. Considering all the results, it can conclude that these values are reliable since they reflect the stability of the physical mount. Since the crops and, thus, the origin of the image systems are displaced to each other, the values of the relative orientation not only reflects the orientation of these systems but are also affected by other errors in the role process, due to several correlations.   Table 8. Average values and standard deviation of the average for the relative orientation elements in experiment CF.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition) To assess the estimated IOPs several flights were conducted with a quadcopter UAV over an agricultural test area planted with millet (Pennisetum glaucum), with a flight height of 120 m resulting in a block with seven strips and 93 images and a GSD (Ground Sample Distance) of 1.8 cm. Six ground control points were signalized in the area for the bundle adjustment which was performed in the Agisoft Metashape using navigation GPS data as weighted constraints with a standard deviation of 5 m. Figure  9 shows the configuration of the photogrammetric block. Tests were performed with 4 sets of IOPs generated in the experiments (B, BF, C, CF). Due to the reduced number of control points, three trials were performed in Metashape: (1) all the 6 GCPs as control without check points; (2) 5 GCP used as control and one as check; and (3) 5 GCP as control and changing the check point. The 4 GCP in the block corners were the same in trials (2) and (3). The residuals in the ground coordinates of GCP and check points were computed and the planimetric and height values were used for analysis. For the trial (1) only the RMSE (Root Mean Square Error) of the ground control points were considered. For trials (2) and (3) the errors in the check points and the RMSE in the check points were averaged. The results are presented in Figure 10.  From the analysis of Figure 10, it can be concluded that the option for fixed crop provided results slightly better. The sets of IOPs generated with stability constraints generated similar results when compared to the IOPs computed without constraints. Considering that stability constraints lead to more accurate relative orientation parameters, this option can be used without loss of accuracy. However, for the case studied there is no significant advantage since the heads are quite near and, thus, when using direct georeferencing the values could be used for both heads.
Future studies will use more control points to improve the assessment of the IOPs generated with different cropping and calibration techniques. In addition, the effects in the Digital Surface Models and orthomosaics and the bands co-registration will be evaluated. Preliminary analysis with NDVI and SAVI indexes indicated areas in the millet crop with stress condition and evidenced planting line failures. Regardless of the slight differences achieved when using two cropping techniques, this sensor provided very accurate results at a very attractive costbenefit ratio.

CONCLUSIONS
In this paper the geometric accuracy of a novel type of multispectral camera was assessed. The Agrowing two heads camera provides a powerful and light sensor for UAV and proximal remote sensing at a very competitive cost-benefit ratio. Further features are the use of a single sensor and the simultaneous exposure of multiple bands which is advantageous in comparison with multi-sensor technologies.
The sensor was calibrated following two forks, depending on the cropping technique used to produce the 4 spectral bands. Different calibration techniques were also used and very accurate results were achieved. In general, better results were achieved when imposing fixed crop origins basically because this strategy maintains the same geometry of the cropped image with respect to the perspective centre. Experiments with data collected with a UAV also confirmed the results achieved with close range calibration.