FUSION OF THE OPTICAL AND RADAR IMAGES OF THE LAND SURFACE IN MULTI-POSITION ON-BOARD SYSTEMS FOR OPERATIONAL MONITORING

The issues of image fusion in a two-position small-size radar on-board operational monitoring system are considered. The aim of present research is to develop a method for fusion of images of the land surface based on data obtained from a multi-sensor spatially distributed on-board location system implemented on the basis of a UAV. The method of combining different-angle location images is implemented iteratively. At each iteration, there is a search for pairs of points of the contour of the isolated areas. Areas are highlighted using two image segmentation methods. The final result is a proposed method for information fusion from a two-position on-board small-sized radar system and an optical location system. The implemented method of fusion makes it possible to increase the information content, quality and reliability of information about the observed underlying surfaces and the physical objects detected on them. The practical significance of the results obtained lies in the formation of integral information in real time in the interests of environmental reconnaissance, monitoring in hard-to-reach and dangerous places for human life, as well as in order to promptly prevent natural and man-made emergencies. * Corresponding author


INTRODUCTION
On-board multi-position small-sized systems are widely used in many areas where there is demend to monitor the land surface. These locating systems are usually based on small unmanned aerial vehicles (UAVs) [1]. At the same time, the UAV flight time and the ability to use a rather large payload in terms of weight and dimensions have significantly increased recently.
However, for the purposes of prompt notification of emergencies and other environmental disasters, it is advisable to use a multi-position system [2][3][4] consisting of several spatially distributed UAVs. In this case, due to the use of a group of UAVs, the unnecessary time to search for an emergency or disaster site is reduced, which is a key factor in the implementation of such operations. For these reasons, the development of multi-position on-board systems and methods of operational monitoring implemented on their basis is currently an urgent task.
In this regard, the present research solves the problem of developing a method for forming a complex image based on multi-angle images of different nature, formed by the on-board equipment of a multi-position system of radar and optical stations.
The paper discloses a method for forming a complex image of the land surface from a pair of multi-angle location images using segmentation algorithms.

METHODS OF IMAGE SEGMENTATION
Segmentation is the process of dividing an image into its constituent parts or objects. An object is a group of pixels, united by similarity of brightness or color. Segmentation refers to the preliminary stage of image processing, on the results of which the subsequent tasks of image recognition and analysis depend.
One of the most common problems with image segmentation is the choice of the number of colors (gray levels) to generate the final partitioning. With a small number of them, the pixels are combined into large areas. The resulting partition is rough. Small details are lost, merging with large areas. Such an image is called an under-segmented image. Conversely, with a large number of colors (gray levels), the resulting subdivision contains many details and may differ slightly from the original image. This is called an over-segmented image. In this regard, according to the number of partitions at the output of the segmentation algorithm, the algorithms are divided into a category that returns a single partition, and a category that returns a set of partitions at the output of the algorithm. The advantage of the second category of methods is that from the generated series of partitions, one or several suitable for further processing can be selected.
There is no single generally accepted method of image segmentation. The method is chosen depending on the task to be performed. In this study, two image segmentation algorithms are used, each of which provides a specific result. With the help of multithreshold processing, a series of partitions of varying degrees of detail into characteristic areas The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-2/W1-2021 4th Int. Worksh. on "Photogrammetric & computer vision techniques for video surveillance, biometrics and biomedicine", 26-28 April 2021, Moscow, Russia is obtained. Then, using the filters for selecting breaks, the boundaries on the selected partition are isolated. The use of such a combination of algorithms helps to avoid the effects of both over-segmentation and under-segmentation when isolating the boundaries of characteristic areas to find the key points on contours.

Multilevel thresholding
The idea of the multithreshold processing method described in [5] is to sequentially combine adjacent pairs of clusters on the brightness histogram. Initially, each column of the brightness histogram is a separate cluster. All adjacent pairs of clusters are viewed in one pass along the brightness histogram. The distance function is calculated for each pair of clusters. At the end of each pass, a pair of clusters with the minimum value of the distance function are combined. Each cluster is characterized by the number of pixels included in it and the average brightness value.
In general, all clustering is generated in 256 passes. In the special case, when the number of non-empty gray levels is equal to K < 256 and it is required to find the partition into t < K of the gray level, (Kt) passes over the brightness histogram are required.
Let Сk be the k-th cluster of gray levels. Cluster Сk contains gray levels in a continuous range. The cluster merging algorithm is as follows: 1) Let the target brightness histogram contain K different non-empty gray levels. At the beginning of the merge process, each cluster is assigned to each gray level. The number of clusters is K and each cluster contains only one gray level. 2) For t-level thresholding, the following steps a), b), c) are repeated (Kt) times: a) For each adjacent pair of clusters, the distance Dist is calculated, which indicates the difference between adjacent clusters. b) The pair of adjacent clusters with the smallest distance is combined into one cluster. c) Cluster indices are reassigned because the merge decreases the number of clusters by one.
The modified version of the multithreshold processing method retains the idea and calculation strategy of the original method [5], but the function of calculating the distance between pairs of adjacent clusters changes. Instead of the product of the interclass and intraclass variances, the increment in the total squared error ΔE [6][7][8] is used.
Modification is justified for several reasons. First, it halves the number of accompanying operations for calculating the distance between the next pair of adjacent clusters, which greatly facilitates the programming work. The increment in the total squared error ΔE arising from the merging of two adjacent clusters 1 and 2 is calculated by the formula: where n1 and n2 = number of pixels in clusters 1 and 2 I1 and I2 = average brightness values for clusters 1 and 2 || || 2 = the square of the Euclidean distance Second, the accumulated value of the total squared error E serves as an indicator of the quality of the clustering. Its value for the current partition step is calculated using the values of the previous step: Index i indicates the sequence number of the step of sequential merging of adjacent pairs of clusters. It accepts integer values in the range from 1 to K-1: i [1, K-1], i = 1,2,3…. The total squared error of the initial partition is 0: Estep 0 = 0. The total squared error E is related to the standard deviation σ by the equality: where N = the number of pixels in the image.
Both values E and σ equally characterize the quality of dividing the image into clusters. However, the first is convenient to use when calculating the distance between pairs of adjacent clusters, and the second is to display the splitting quality, since for grayscale images the tolerance region of the value of standard deviation σ is limited by the range [0; 127.5].
Third, the application of the increment of the total squared error ΔE as a distance function also changes the order of the sequence of combining pairs of adjacent clusters and the values of the average brightness over the formed clusters of partitions. It was found that the series of piecewise constant partitions generated by the modified multithreshold processing method is optimal. This is evidenced by the convex curve of the dependence of the values of the total squared errors E on the number of clusters N in the partition.

Edge detection
Border isolation methods convert the original image to an edge image using grayscale changes. An edge is a place of local change in the intensity of an image that occurs at the border between two regions, where the smooth transition of brightness is interrupted. Operators of Prewett, Roberts, Sobel, Canny [9] are among the simplest methods of boundary detection. Among the soft edge detection approaches, there are fuzzy logic based approach [10], genetic algorithm approach [11], neural network approach [12].

TECHNOLOGY OF IMAGE FUSION WITH USE OF SEGMENTATION AS PREPROCESSING
The method for the fusion of images of the land surface from a pair of location images is implemented using segmentation algorithms. The initial location images of the land surface formed by the on-board equipment of multi-position location systems are multi-angle and heterogeneous (Figures 1a and  1b). The fusion method includes the operation of joint preprocessing of a pair of original images docked into a single composite image. Using the pixel clustering algorithm, the docked image is reduced to several gray levels, which allows isolating characteristic areas while preserving boundaries. Further, the border isolation on the clustered image is carried out by an appropriate filter.
The proposed method for fusion of images based on the selection of pairs of points of contours consists in preliminary reduction of the number of clusters of the original images using the pixel clustering algorithm, which generates many partitions into clusters. A pair of multi-angle images is docked together into a single image and clustered using multithreshold processing method [5]. A one suitable is selected from a series of partitions, on which the contours are highlighted with an edge detection filter [9]. After applying this bundle of segmentation algorithms to a pair of images of the docked image, the search for pairs of reference points on the isolated contours is carried out. For these selected pairs of points, a functional transformation is determined and after it is applied to the original images, the degree of correlation of the fused image is estimated. The functional transformation itself is selected from the reduced images, which is then applied to the original images. This process is repeated for clustered images with greater detail in case if the assessment of the quality of the integration is not acceptable. A larger number of clusters corresponds to a greater extent of the boundaries of the contours, on which a larger number of pairs of reference points on the contours can be selected. On a new pair from a series of clustered images, the position of the reference pairs of the points of the contour and the functional transformation sought for by them are refined until the assessment of the integration quality is acceptable.
The essence of the technique for synthesizing a complex image from heterogeneous and multi-angle source images is the preliminary application of two segmentation methods. The first pixel clustering method, generating a series of partitions, avoids the problems of under-segmentation and oversegmentation. From a series of partitions, one or several suitable for their further processing partitions can be chosen. If the first method returns characteristic areas on the original images, then the second method of segmentation allows isolating the boundaries of characteristic areas. Without the first method of clustering pixels, the second method returns inadequate, overly segmented results, which already make it impossible to determine pairs of contour points for merging two original images into one.

EXPERIMENTAL RESULTS
The initial location images of the earth's surface formed by the on-board equipment of multi-position location systems are multi-angle and heterogeneous (Figures 1a and 1b). The method of combining includes the operation of joint preprocessing of a pair of original images joined into a single composite image. Using the pixel clustering algorithm, the stitched image is reduced to several shades of gray, which allows you to select characteristic areas while preserving their boundaries. Further, the isolation of borders on the clustered image is carried out by an appropriate filter. Figure 2 shows parts of the sequence of splitting the original optical and radar images into clusters using the multithreshold processing method. In order to isolate similar areas in different images in a similar way, the technique of joint processing of two images was used. The original docked image is shown in the upper left corner. It is formed from the original optical image of the land surface converted into shades of gray and a color-inverted radar image [13] of the same area. Conversion to grayscale and color inversion made it possible to bring dissimilar images to the same gamut. To the right of the original matched image is a part of the sequence of partitions into clusters. Each partition is labeled with the number of clusters into which all pixels of the original image are divided, and the value of the standard deviation σ, which characterizes the quality of the subdivision. The lower the value of σ for the same number of clusters, the better the quality of the partition. It is clearly seen that the same objects are distinguished in a similar way in two different docked images already with two clusters. A further increase in the number of clusters details the internal structure of the imprinted objects.
a) optical image b) radio image In Figure 3, the boundaries of the isolated areas are marked in turquoise using the Canny operator. Figure 4 shows the result of fusion images of the land surface with a preliminary application of the clustering algorithm for pixels of location-based heterogeneous images captured by the UAV equipment in a multi-position system.
In Figure 4, the original heterogeneous images are combined into a single complex image by selecting the most important various components from the original images. The contours of the characteristic areas are highlighted in turquoise. The complex image contains two layers: optical and radar. Such an image is complex due to the fact that these two dissimilar components complement each other. Depending on the problem to be solved, each of the layers can be strengthened if necessary. Figure 4 shows the case when the radar layer is partially reinforced in the built-up and forest zones, and in the remaining zones -the optical layer. Such a complex representation of the image of the land surface in the future makes it possible to better classify the characteristic zones and objects on it, as well as to monitor the structural changes of the imprinted objects of various subjects.

CONCLUSION
The paper proposes a method for image fusion, a distinctive feature of which is an increase in the number of pairs of points of the contour with an increase in the number of image clusters. At each subsequent iteration, the degree of detail of the image increases, the number of pairs of contour points increases, and, as a consequence, the type of the desired functional transformation is refined.
The use of an iterative approach in the implementation of the developed method allows improving the quality of synthesis, to increase the accuracy of integration, as well as the information content of the final complex image.
The method for synthesizing a fused image proposed in present research can be adapted to develop algorithms for pixel clustering and fusion of the generated images of a stream of frames recorded during field tests with heterogeneous devices for generating location information. The prospect of the developed technique is the use of multi-format images (for example, optical, radar, infrared, etc.) in the fusion task, as well as spaced in time for monitoring structural changes in captured objects of various subjects.