USING THERMAL AND RGB UAV IMAGERY TO MEASURE SURFACE FLOW VELOCITIES OF RIVERS

This study assesses the suitability to use RGB and thermal infrared imagery acquired from an UAV to measure surface flow velocities of rivers. The reach of a medium-scale river in Hungary is investigated. Image sequences with a frame rate of 2 Hz were captured with two sensors, a RGB and an uncooled thermal camera, at a flying height that ensures the visibility of both shores. The interior geometry of both cameras were calibrated with an in-house designed target field. The image sequences were automatically co-registered to account for UAV movements during the image acquisition. The TIR data was processed to keep loss-free image information solely in the water area and to enhance the signal to noise ratio. Image velocimetry with PIV applied to the TIR data and PTV applied to the RGB data was utilised to retrieve surface flow velocities. Comparison between RGB and TIR data reveal an average deviation of about 0.01 m/s. Future studies are needed to evaluate the transferability to other non-regulated river reaches.


INTRODUCTION
The application of image sequences to measure flow velocities and eventually retrieve river discharge has received a recent increase in interest in the field of hydrometry due to the possibility to extend the application to unoccupied aerial vehicle (UAV) data. This can be especially useful in scenarios of observations in remote and inaccessible areas as well as during large flood events. The suitability to receive hydrometric information from videos has been illustrated for terrestrial cameras by various studies (e.g. Muste et al., 2008). And the extension to UAV imagery has been demonstrated in the last five years (e.g. Fujita et al., 2015).
The exploitation of UAVs to measure hydrometric parameters has so far been focused on the application of RGB imagery as most UAVs are equipped with it. However, for instance in the case of the retrieval of surface flow velocities it is difficult to acquire the information across the entire river cross-section with RGB data due to potential clustering of particles to track and their usual concentration to the main flow. Nevertheless, especially at irregular river profiles complete velocity information is necessary to estimate the discharge reliably, which is a key parameter amongst others for hydrological and hydraulic modelling. Artificial seeding can circumvent this circumstance (Detert et al., 2017), but this is usually not possible in remote or extreme flood scenarios. Furthermore, the effect of different seeding methods on the accuracy of velocity estimations has been studied only in laminar, slow flow conditions (Pearce et al., 2020). * Corresponding author A possibility to densify the surface velocity measurements can be the application of thermal infrared (TIR) cameras. Lin et al. (2019) demonstrated the suitability of TIR imagery for surface flow velocity estimation with accuracies of up to 1 m/s in a terrestrial application, however relying on thermal tracers. The suitability to solely use the natural thermal signature of the river has been demonstrated by Kinzel & Legleiter et al. (2019) for UAV data.
In this study, thermal and RGB image sequences, captured together from an UAV, are used to estimate river surface velocities to measure entire river cross-sections without the need for seeding. Particle image tracking methods (particle tracking velocity -PTVand particle image velocimetry -PIV) are applied. Results are compared to each other. First, the study area is introduced. Afterwards, the workflow to acquire the data is described. Then, the applied methods are explained and finally the results are displayed and discussed.

STUDY AREA
The case study is located in North East of Hungary at the Sajó River near the city of Nagycsécs. The Sajó River is a meandering sand-bed river, where 40 % of the total river are under natural channel conditions without any river management works (Bertalan et al., 2019). The lack of an extensive levee system and bank protection led to the continuous change of channel morphodynamics resulting in bank erosion rates of 4-7 meters per year along the studied sub-reach (Bertalan et al., 2018). A detailed reconstruction of local flow conditions is vital to reveal the natural driving factors of these erosion processes. Surveys were performed along a 650 m long double bend-system of the river (Figure 1). A nearly straight transition zone between two long meanders was chosen in anticipation of laminar flow conditions (water flows in layers that do not mix and therefore no turbulences occur) without strong secondary flow. The chosen segment of the river reach has an approximate width of 20 m and maximum depth of about 1.2 m.

DATA ACQUISITION
The data had been acquired during a field campaign in March 2019. Imagery in the visible and thermal infrared range of the electromagnetic spectrum had been captured with an octocopter.

RGB and TIR image sequences
The image data acquisition was carried out with an UAV equipped with a RGB camera and a thermal camera ( Figure 2). The frame of the CADMIC Goliath Coax 8 has four cantilever arms each with two coaxial mounted rotors. For direct georeferencing of the sensors the inertial navigation system Spatial from Advanced Navigation provides GNSS positions and orientation angles using a navigation filter 1 . The RGB camera is an industrial camera from Allied Vision Technologies with a resolution of 2048 px × 2048 px, a pixel pitch of 5.5 µm and a fixed focal length of 12.5 mm. As thermal camera a FLIR A65 with a resolution of 640 px × 512 px, a pixel pitch of 17 µm and a fixed focal length of 13 mm was used.
For data acquisition, the UAV was flown to a defined waypoint above the river at a height of about 60 m and kept as stable as possible to capture image sequences with a frequency of about 2 Hz. This results in images of the river with a ground sample distance of 2.64 cm for the RGB camera and 7.8 cm for the thermal camera.
Due to a missing stabilisation camera mount and the fact that the cameras are not synchronized to each other, both shores have to 1 https://www.advancednavigation.com/products/spatial be visible in the images of both cameras for a pre-processing steps of stabilization and registration. Furthermore, eight ground control points (GCPs) were distributed along both shorelines for data georeferencing. The targets are designed specifically for thermal imagery. The targets are made of silver heat foil circles in front of black velvet tape squares to increase the contrast in the images for a better visibility (Westfeld et al., 2015). Thus, in the TIR images cold circles are shown in front of warm rectangles because the cold background radiation is reflected from the heat foil and the black velvet tape leads to high emission rates of thermal radiation. The same targets are also used as GCPs in the RGB images. The reference points are measured with dGNSS (STONEX S9I) with accuracies between 2 cm to 3 cm.

DATA PROCESSING
To achieve the goal of automatic flow velocity measurement the following processing steps are needed. The image sequences were co-registered to account for UAV movements during the image acquisition. The thermal infrared data was further preprocessed with a local histogram equalization algorithm to enhance the contrast of the measured river temperature ( Figure 2). Thereby, the shore needed to be masked to avoid information loss in the water area due to a shore with significantly higher thermal contrast compared to the more homogeneous temperature values at the water surface. Subsequently, decreased signal to noise ratios, caused by the histogram equalization, are increased with low-pass filtering. Finally, image velocimetry was utilized considering particle image velocimetry for the thermal infrared data and particle tracking velocimetry for the RGB data. The data has been processed with commercial tools and with in-house implementations in Python and Matlab.

Camera calibration
If tracks of particles only cover short distances within the image, distortions of the camera exhibit a low impact at the estimated flow velocity. However, in this study the frame rate is low (2 Hz) and therefore large tracking distances are expected necessitating calibration of distortion parameters of the RGB and thermal camera. Furthermore, focal length estimates are needed for a correct scaling of the tracks.
The calibration of the thermal camera required special markers, because the temperature gradient of black-white-paper makers is not sufficient for image point measurement. For this purpose, a combination of silver heat foil and black velour foil (same material as for the GCPs) is used for the target field ( Figure 3). To increase the contrast, the calibration images of the target field have to be captured outside to ensure low grey values of the silver foil (i.e. cold background radiation). The transformation from the 3D object point into the image coordinates is realized by the collinearity equation. The model is extended by the distortion correction parameters considering the radial symmetric and decentring distortion parameters (Brown, 1971) as well as the affinity and shear parameters (El-Hakim, 1986). Further information can be found in Westfeld et al. (2015). The results of the calibration are shown in Table 1.
While the majority of the calibration parameters of the RGB camera could be estimated significantly, i.e. with the threefold of the standard deviation (with exception of C1 and C2), it is only the case for the focal length for the thermal camera. A similar effect for that thermal camera was also noticed in Lin et al. (2018).

Referencing and co-registration of each image series
In the first image of each image sequence, GCPs were measured manually in order to retrieve the exterior orientation of the image (EOR), which is needed later to scale the velocity tracks (section 4.4). First, a direct linear transformation is utilized to get approximate values of the position and orientation parameters. Then, adjusted spatial resection is used, with fixed interior orientation parameters, to estimate the EOR also calculating the accuracy for each estimated parameter. The co-registration of all subsequent images of the TIR and RGB sequence is executed to the initial frame of each series using an approach introduced in Eltner et al. (2018). This is necessary because the UAV was not equipped with a stabilizing camera mount and hence octocopter movements have to be compensated digitally for each image (Ljubičić et al., 2021). Tie points are detected as Harris features (Harris et al., 1988) and afterwards matched with the SIFT algorithm (Lowe, 2004). The matched features are used to estimate the homography and thus transformation matrix between each image and the initial image. For the RGB image sequence the estimated homography parameters are applied to transform and hence co-register each image to the first. For the thermal imagery a different approach is chosen and the calculated homography parameters are saved for further processing in the next step.

Improving thermal signal to noise ratio
Retrieving the thermal signal from the TIR images is difficult in riverine environments due to a weak contrast in the river and a The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B2-2021XXIV ISPRS Congress (2021 high contrast to the shore line as temperature differences are small in the water but strong between river and shoreline. The images are captured with 14-bit. However, the tracking of particles is performed with 8-bit images. Therefore, 14-bit images have to be converted into 8-bit images. Solely the relevant temperature range has to be determined to allow for a conversion without information loss and a high radiometric resolution ( Figure 4).
The water area is masked in the initial thermal image to enable that grey values are stretched to the temperature range only within the river. In a next step, the binary image is transformed to match the orientation of each subsequent (yet not coregistered) image of the sequence using the inverse of each corresponding homography matrix. Then, the water area is clipped in the 14-bit image with the transformed masks to stretch the values to the minimum and maximum values within the water area in the 8-bit image to enhance the contrast and thus potential texture. These new images are transformed with the parameters of each homography to co-register them to the initial image.
Although pixel values only cover water temperatures, image contrast is low because the thermal signal is weak in the water area. To further improve the signal-to-noise-ratio (SNR) several image processing tools from OpenCV are applied (Bradski, 2000). First, an adaptive histogram equalization is used to improve the contrast locally (Pizer et al., 1987), amongst others to mitigate vignetting impacts. However, the contrast enhancement also increases the fixed pattern noise (FPN) significantly as the temperature differences in the river are close to the thermal sensitivity of the camera.
Simple image processing steps are applied to decrease the noise. First, a fast Fourier transformation (FFT) is used to remove high frequencies to filter stripe noise. Afterwards, salt and pepper noise is eliminated with a median filter. Finally, the image is smoothed with a Gaussian filter to make the later PIV approach more reliable. In the future, more sophisticated spatial and temporal non-uniformity correction (NUC) approaches will be tested (e.g. Lin et al., 2018) but are not considered here for a first evaluation of how well RGB and TIR data can be used together for surface velocity estimation.

Image velocimetry
The filtered TIR images and the RGB image sequence are processed using different image velocimetry approaches implemented in the FlowVelo tool introduced by Eltner et al.
(2020) to measure river surface velocities. PIV is applied to the TIR imagery. Therefore, a grid is generated that covers the area of interest with a sampling distance and template size of 30 pixels. The Lucas-Kanade algorithm (Lucas & Kanade, 1981) is used to perform area based matching, minimising grey value differences within an optimization approach, to track thermal signals at the water surface. Resulting track vectors are filtered at each grid cell considering track directions and lengths (Eltner et al 2020).
PTV is applied to the RGB imagery. First, features are detected with the Shi Tomasi algorithm (Shi & Tomasi, 1994) looking for points of large contrast to the surrounding. Afterwards, the detected features are searched for in the subsequent frames also using the Lucas Kanade algorithm. Finally, a local filter is applied that eliminates outliers via median filtering of the estimated velocities within a specified search radius of each tracked particle, which allows for the preservation of more complex velocity patterns.

RESULTS AND DISCUSSION
The accuracies of the spatial resection to estimate the poses of the RGB and thermal sensors during the acquisition of the first image of each sequence used for tracking are shown in table 2.
The EOR of the thermal camera is estimated with a higher error mainly due to more difficulties during the identification of the thermal GCPs in the image. The image resolution is significantly lower compared to the RGB image resolution and thus targets are less reliable measured. The low frame rate is challenging for pattern or feature tracking for both, TIR and RGB, datasets due to a fast change of appearance of the water surface. Furthermore, the thermal signal is very weak, especially in the areas close to the left shore. However, in the river centre and near the right shore the SNR was sufficient to allow for velocity retrievals. The RGB imagery reveals a larger concentrations of trackable features in the centre of the river, which corresponds to the main flow, and near the left shore. The range of velocity values of the thermal data are smaller than of the RGB data, i.e. 0.84 m/s to 1.21 m/s and 0.45 m/s and 1.41 m/s, respectively ( Figure 5). However, the informative value of range comparison is limited because the PIV and PTV approach retrieve velocity information partly at different locations in the river reach. The average PIV velocities are marginally larger than the PTV results, which is in contrast to other studies revealing that areabased tracking approaches (i.e. PIV) underestimate velocities compared to feature-based approaches (i.e. PTV) because the latter method tracks single particles instead of relying on areaaveraged methods as in the former method (e.g. Lin et al., 2019, Tauro et al., 2017. However, these studies compare PIV and PTV applied to the same data source, which is contrast to this study. Furthermore, the high standard deviation of velocity differences indicates that it is not possible to state if either method over-or underestimates velocities.

CONCLUSION
In this study, the surface flow velocities of the river were measured with thermal image sequences using PIV and with RGB imagery using PTV, even with frame rates as slow as two images per second. However, challenging conditions due to very low contrast in the thermal images because of small temperature differences at the river surface limited the application of a TIR camera significantly. In future studies the signal to noise ratio in the thermal data should be further improved, amongst others considering systematic camera based noise removal with radiometric calibration approaches. In this study, the river reach revealed complex flow patterns due to secondary flows and nonstationary conditions making the assessment with the reference less reliable, which is further indicated by the low average but high standard deviation of velocity differences between the RGBand TIR-based tracking.