IMAGE AND SHAPE COMPARISON VIA MORPHOLOGICAL CORRELATION

A lot of image matching applications require image comparison to be invariant relative to intensity values variations. The Pyt’ev theory for Morphological Image Analysis (MIA) was developed based on image-to-shape matching with mosaic shape models. Within the framework of this theory, the problem of shape-to-shape comparison was previously considered too. The most sophisticated and weakest part of MIA theory is the comparison of mosaic shapes with some arbitrary restrictions described by graphs or relations. In this paper we consider the possible options for comparing images and shapes using morphological projection and morphological correlation. Our contribution is a new scheme of morphological shape-to-image projection and, correspondingly, the new morphological correlation coefficient (MCC) for shape-to-image correlation with restricted mosaic models. We also refine the expressions for shape-to-shape comparison.


INTRODUCTION
A lot of image matching applications require image comparison to be invariant relative to intensity values variations (so called "template matching" task). In this context, the Pyt'ev theory for Morphological Image Analysis (MIA) was developed based on image-to-shape matching with mosaic shape models (Pyt'ev, Chulichkov, 2010). Within the framework of this theory, the problem of shape-to-shape comparison is considered too (Vizilter et al., 2015). The most sophisticated and weakest part of MIA theory is the comparison of mosaic shapes with some arbitrary restrictions described by graphs or relations. Recently, a new motivation has appeared for the development of this section of MIA theory. It is related to the need for "transparent" analysis of decision making in deep neural networks. One of the possible approaches for design of "transparent" AI concerning the scene understanding tasks is to perform the morphological analysis of mosaic image models obtained by CNN-based semantic segmentation. In such applications one uses the scene graphs, which are analogues to the mosaic models with restrictions in the Pyt'ev morphology. In this paper we consider the possible options for comparing images and shapes using morphological projection and morphological correlation.

IMAGE-TO-SHAPE COMPARISON (TEMPLATE MATCHING)
In the framework of Pyt'ev morphology, we consider images as piecewise-constant 2D functions where n is a number of tessellation F on frame  with area S into connected regions of constant intensity, F={F 1 ,…,F n } is described by support functions is an intensity value vector. Such images we call the mosaic images. Image class with shape F has a following form: Morphological projector P F to shape F is determined by solution of optimization problem The solution of (3) has a following evident form: . ,..., 1 , In this framework we could compare the image g(x,y) to the shape F of image f(x,y) via the morphological background normalization This MCC is a main MIA tool for the image-to-shape comparison (template matching). MCC is asymmetrical K M (g,F)K M (f,G) and normalized K M (g,F)[0,1], because the morphological projector P F is the non-decreasing by norm operator.

DISSFUSION MORPHOLOGY AND DIFFUSION MOSAIC SHAPES
In (Vizilter et al., 2014) we propose the following generalization of Pyt'ev projective morphology in the form of diffusion morphology. The relational model of the diffusion shape F of image f(x,y) is the pairwise image point similarity function and there is an unambiguous similarity function , such that The operator model of the diffusion shape F is the diffusion transformation operator P F , with normalized kernel p F (x,y,u,v), such that: It is easy to see that any diffusion relational model can be matched by its operator model: The diffusion shape F of the image f(x,y) is the Eigen space of the diffusion transformation operator P F : where { 1 ,…, n } are the eigenvalues and { 1 (x,y),…, n (x,y)}the Eigen functions of P F : In particular, the choice of leads to the standard Pyt'ev morphology, which describes the shape as a tessellation of the frame into regions of equal brightness. In this case, the similarity relation b F (x,y,u,v) becomes the equivalence relation between the points of the frame, which separates them into disjoint regions F={F 1 ,…,F n }, where n is the number of regions of the tessellation F of the frame . As a result, the diffusion operator becomes a projector, All eigenvalues of it are singular, and Eigen functions are the characteristic functions of frame tessellation regions: Accordingly, in this particular case, the diffusion shape becomes the standard Pyt'ev shape of the form So, for each image g(x,y)L 2 () the projection onto the shape F can be expressed as: It is also interesting to note that the equivalence relation on image points generates here an equivalence relation on images (equivalent images belong to the same shape). Diffusion mosaic shapes could be produced from usual mosaic shapes by diffusion mosaic filters. The diffusion mosaic filter to the image of the same shape, such that We meet such diffusion (fuzzy) shapes in the next section.

MOSAIC SHAPE COMPARISON (SHAPE MATCHING)
Morphological comparison of mosaic shapes by complexity is traditionally implemented in terms of a partial order relation "not more complex by shape". The set of mosaic shapes has an algebraic lattice structure: for any shapes F and G we can find the more complex shape FG and less complex shape FG.
More complex shapes are obtained by region splitting, and less complex shapes are obtained by regions merging. In terms of sets (classes) of images, F is not more complex than G, if F  G. In terms of morphological projectors, F is not more complex than G if P G P F = P F . Consider the image comparison by shape, which is traditionally determined using a morphological projector (4). If image g(x,y) is mosaic too, then Thus, we obtain two alternative descriptions of the projection of g F (x,y): based on a usual mosaic model  F ,g F  and a fuzzy model of  GF ,g G . Hence, the P F operator can be considered no longer as an operator in image space, but as an operator in the space of mosaic shapes. In such scheme, the fuzzy or diffusion (Vizilter et al., 2014) model G F automatically arises as a projection of the mosaic model onto another mosaic model: So, now we can define all standard Pyt'ev morphological tools, but for shape-to-shape comparison instead of image-to-shape one. We introduce the Morphological Shape Background Normalization: the Morphological Shape Pseudo Distance: Then we substitute (6) into (7) and obtain the evident formula for K M (G,F): where p ij = S ij / S, p Fi = S Fi / S. Note that was previously received the expression (8) from completely different, statistical considerations (Vizilter, Zheltov, 2012). We supposed that mosaic images from F and G of the form .,n f i S ij )/S j , j=1,..,m.
These assumptions directly lead to conclusion that all mathematical expectations of intensity values are zero-valued f i  =0, and their pairwise covariations have the simple form: where  is a dispersion of p(f i ) distribution. Based on this, the average square of the norm of image f from F takes the form: Correspondingly, the average square of the norm of image f from F to the shape G is We determined the mean square effective morphological correlation coefficient (MSEMCC) as the root of the ration of average square of the projection norm of the image shape F to the shape G to the average square of the projected image norm: The formula for the MSEMCC based on assumption of the mutual independence of the region intensities on G, turned out to be exactly the same as the expression we obtained here for the MSCC (8).
In conclusion of this subsection note that instead of G F we can use the following absolute morphological shape difference map which is close the morphological difference map. The greater difference between  G (x,y) and  GF (x,y) at some (x,y) point, the greater the |G| F value at this point. So, this operation highlights those regions from mosaic shape G, which have no the close matches by shape in the set of regions from the mosaic shape F.

IMAGE AND SHAPE COMPARISON (RESTRICTED TEMPLETE MATCHING)
Consider an example of mosaic shapes with ordered intensity values as the simplest case of morphological models with oriented relationships. In this case, we suppose that the regions of shape F are ordered for a particular image f such that: .,n f Fi  Fi (x,y), The shape class for such modes has a following form Morphological projector P F(+) to shape F (+) is determined by solution of optimization problem g F(+) = P F(+) g = arg min(g F(+) F (+) ) || g(x,y)g F(+) (x,y) || 2 . (11) Level functions  Fi (x,y) play here the role of support functions:  Fi (x,y) =  k=i,..,n  Fi (x,y), i=1,..,n. The image reconstruction based on intensities and level functions (f F , F ) is performed as The projection (11) is calculated via the monotonous least squares approximation (MLSA). This algorithm is based on the construction of the bottom convex cover of data. We describe the MLSA-projection (11) in terms of level functions. Let The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-2/W1-2021 4th Int. Worksh. on "Photogrammetric & computer vision techniques for video surveillance, biometrics and biomedicine", 26-28 April 2021, Moscow, Russia mean (i,ki) (g,F (+) ) = mean(g(x,y), Fi (x,y) - Fk+1 (x,y)) = =  j=i,..,k mean(g(x,y), Fj (x,y)) S Fj /  j=i,..,k S Fj , .
The Algorithm 1 iteratively forms the level representation of projection g F(+) as a set of level functions { GFj (x,y)} j=1,..,m . and corresponding intensity values {g GFj } j=1,..,m . In result, we obtain both the image projection g F(+) , and the shape of projection G F(+) ={ GFj (x,y)} j=1,..,m . The projection shape is always simpler than the class shape, which we project to. We select the level functions from the same set, but not all of them will be of use. All functions will be preserved just in case of gF (+) . The MCC based on such projection (11) has the following obvious form: K M(+) (g,F (+) ) = K M(+) (g, P F(+) ) = || P F(+) g || / || g ||. (13) Algorithm 1: Approx(F (+) ,mean).
End of Algorithm 1.
The analysis of MLSA projection algorithm demonstrates that P F(+) g(x,y) = P GF g(x,y) = =  j=1,..,m mean(g(x,y), GFj (x,y))  GFj (x,y), where P GF is a standard projector of g(x,y) to the (non-ordered) shape G GF , which is a basic mosaic shape for ordered shape G F(+) . Moreover, we can introduce the projection of shape F (+) to the image g(x,y): which is obviously the idempotent and non-increasing (by norm) operation. So, it is a new morphological filter in the MIA framework.
Note that expression (15) describes the new scheme of twostage mutual adaptive filtering of image g and shape F.
Step 1: Project shape F + onto the image g; Step 2: Project image g onto the shape projection P (g) F (+) . It seems that we did not meet such mutual correction of shapes in classical MIA: shape F was always the constant part of statement, and the image was always the subject of modification (projection). But there is one special case of MIA standard projection, which presumes the modification (simplification) of the target shape. If intensities of some neighbor regions have the same values in projection, then these regions of target shape could be merged, thus the shape of projection will be simpler. If some constant valued region of projected mosaic image g contains two or more regions of target shape F, then these regions will be merged in projection, and the resultant projected image g F will be of shape F g  F. So, in traditional and ordered MIA we have the same effect: image and shape always mutually simplify each other. This effect was not so obvious in nonordered MIA due to the fact of very often target shape preserving and too rare cases of shape simplification. In the morphology of ordered mosaic shapes we see the opposite situation. Really, it is easy to demonstrate that G GF  G F  || P F(+) g ||  || P F g ||   K M(+) (g,F (+) )  K M (g,F), so, the operator P F(+) is always the stronger simplifier than P F . This is just due to the fact that F (+)  F, and the cardinalities of these classes differ very dramatically: where n! = P n = A n n is a number of permutations of region orders (ranks) in the shape F. So, the each mosaic shape with n regions corresponds to n! ordered shapes with equal occasion probabilities, and only the one of them does not require the simplification of g F and corresponding projected shape g F(+) . Thus, we can perform not only image-to-shape comparison, but also the shape-to-image comparison using the following altered MCC: K M(+) (F (+) ,g) = || P (g) F (+) || / || F ||.
This tool (16) is new and was not considered earlier in the framework of MIA.

CONCLUSIONS
In this paper, we consider the possible options for comparing images and shapes using morphological projection and morphological correlation. Our contribution is a new scheme of morphological shape-to-image projection and, correspondingly, the new MCC for shape-to-image correlation with restricted mosaic models. We also refine the expressions for shape-toshape comparison. We hope the further development of these approaches will make them applicable to morphological comparison of semantically segmented images and scene graphs.