MUTUAL COMPARATIVE FILTERING FOR CHANGE DETECTION IN VIDEOS WITH UNSTABLE ILLUMINATION CONDITIONS

: In this paper we propose a new approach for change detection and moving objects detection in videos with unstable, abrupt illumination changes. This approach is based on mutual comparative filters and background normalization. We give the definitions of mutual comparative filters and outline their strong advantage for change detection purposes. Presented approach allows us to deal with changing illumination conditions in a simple and efficient way and does not have drawbacks, which exist in models that assume different color transformation laws. The proposed procedure can be used to improve a number of background modelling methods, which are not specifically designed to work under illumination changes.


INTRODUCTION
Recently many algorithms have been proposed to solve the problem of video change detection.Most of these algorithms are designed for moving object detection in surveillance systems and rely on background subtraction techniques to segment the scene into foreground and background.Existing methods are trying to cope with the challenges that can be found in a real-world scenarios such as high variation in environments conditions, illumination changes, shadows, camera-induced distortions, low camera frame rate, camera movement and so on.Change detection algorithms have been demonstrated to perform well on some categories of videos but in despite of the progress that was done there is no single algorithm that is able to deal with all challenges in a robust and computationally efficient way.This can be seen in the change detection benchmark (Wang, 2014), where change detection algorithms are evaluated on a common dataset composed of different types of videos sequences and classified according to their performance.According to this benchmark and the latest paper (Bianco, 2015) it is suggested to combine several fast, simple or «weak» change detection algorithms to create a new robust algorithm.Therefore, the idea was to rely on individual strong properties of weak algorithms to build a robust algorithm by Genetic Programming Method.Thus, the further improvement of the simple change detection algorithms remains an important task and this paper is about how to enhance their robustness to changing illumination conditions.If someone will pay his attention to change detection benchmark (Wang, 2014) he will see that there is a lack of videos with various illumination changes in different categories, and what is more, there is no «illumination changes» category at all.In the same time, practical computer vision systems are placed in environments where the illumination conditions vary through time.For example, indoor scenes frequently exhibit light switching, while passing clouds affect outdoor cameras.We consider the problem of illumination changes to be very important because the majority of the known «simple» algorithms (Tian, 2005), (Heikkila, 2006), (KaewTraKulPong, 2001), (Zivkovic, 2004) experience problems when they deal with fast, abrupt illumination changes and depend on some predefined values.This problem is more noticeable in low frame rate videos.Thus, this work is dedicated to the design of a new effective and efficient illumination change detection system, which works along an existing (chosen) background modelling algorithm.
The main advantage of our approach with respect to other illumination change detection algorithms is that there is no specific law of the color transformation to be expected when an illumination change takes place.As a result, our proposal does not depend on the physical properties of the lights that are present in the scene.Therefore, it can be used in a wide range of situations where the lighting conditions vary in an uncertain way.It must be highlighted that most current approaches assume some model of pixel color variation under illumination changes either explicitly or implicitly (Van, 2007), (Farcas, 2012), (Xiangdong, 2013), (Huerta, 2013).
In our proposal, the idea of using morphological filtering (Pytiev, 1993) (Vizilter, 2014) is taken as our basis.One type of the known morphological filtering is diffusion morphological filtering that was proposed in (Vizilter, 2014) and was applied by (Vishnyakov, 2015) to the creation of diffusion regression background model.A key feature of the diffusion regression background model was diffusion filtering of the current image based on the accumulated background model.In the case of well-chosen parameters, diffusion filter is resistant to illumination changes and blurs objects that did not exist in the background model while keeping the original background contours of the current image.Fast implementation of the diffusion filter is based on local binary patterns (LBP descriptors), LBP binary codes are strongly dictated by the predefined threshold, that makes difficult to specify universal settings of diffusion filtering for different lightning conditions.Another drawback of the approach (Vishnyakov, 2015) is that the resulting filtered image was used incorrectly, i.e. the difference was calculated between the accumulated background and the filtered image that in fact only led to some noise filtering in addition to the original regression background model and did not help in the fight against illumination changes.
To solve these problems, we propose to use mutual comparative morphological filtering.Comparative filtering, as well as diffusion filtering, is resistant to changes in illumination conditions, but does not require the exact selection of "thresholds" and is individual for each video.
Moreover, change detection of regions that contain the «true exact changes» in the our method is based on the background normalization, i.e. we calculate the difference between the "filtered by the accumulated background model" current image and the "original" current image.Such a method of calculating the difference provides the required robustness to illumination changes.Since the size of the found regions depends on the size of the filtering window, in order to avoid fragmentation of foreground objects into unconnected zones it is suggested to use the image pyramid followed by intelligent aggregation of found regions between the pyramid layers and refinement of foreground objects boundaries inside these regions.Therefore, the found regions («big changes») help to select and specify regions found by usual background modelling algorithm.These selected regions are taken as final changes.The proposed procedure can be used to improve a number of known background modelling methods (Wang, 2014), which are not specifically designed to work under illumination changes.To determine the effectiveness of the proposed approach, video clips with different illumination conditions were obtained from GTILT dataset (Bales, 2011) and marked up.

Contributions.
The main contributions of this paper can be summarized as follows: (1) Mutual comparative filters for image comparison are introduced, this allows us to compare images by their shape without complex image region segmentation procedures; (2) Change detection approach based on mutual comparative filters and background normalization is introduced.This approach allows us to deal with illumination changes in a simple and efficient way.

PYTIEV MORPHOLOGICAL IMAGE ANALYSIS
The basic ideas of the proposed approach is closely related to Morphological Image Analysis (MIA) proposed by Pyt'ev (Pyt'ev, 1993), (Vizilter, 2014).Let the image be a 2D function where Rset of real numbers, R 2image plane, rectangular frame region of image plane.Images are elements of Hilbert space L 2 () with scalar product (f,g) and norm In the framework of MIA, images (1) are considered as piecewise-constant 2D functions ( 2 , where nnumber of non-intersecting connected regions of tessellation F of the frame , F = {F1,…,Fn}; f = (f1,…,fn)corresponding vector of real-valued region intensities; This tessellation of image is supposed to be obtained by some image segmentation procedure.Set of images with the same tessellation F (4) is a convex and close subspace F L 2 () called shape-tessellation, mosaic shape or simply shape: For any image g(x,y) L 2 () the projection gF(x,y) = PFg(x,y) (5) onto the shape F is determined as Then Pytiev morphological comparison of images f(x,y) and g(x,y) is performed using the normalized morphological correlation coefficients (MCC) of the following form ( , ) , ( , ) The left part of ( 6) estimates the closeness of image g to the "shape" of image f.The right part of ( 6) measures the closeness of image f to the "shape" of image g.
For elimination of constant non-informative part of image brightness following image normalization is usually performed: ( , ) , ( , ) , where POgprojection of image g onto the "empty" shape O (constant intensity image).This projection is also a constantvalued image filled by mean value of projected image.
Change detection between images g and f is carried out based on morphological background normalizationdifference calculation between an image g and its projection: To summarize, in MIA: 1) there is a way to produce a description of the image shape using the given model (model is an image which is segmented into regions); 2) comparison of the image with a shape is carried out as comparison of the image with his projection to that shape; 3) numerical measure of similarity of the image and a shape is the morphological correlation coefficient (MCC) calculated as the ratio of image projection norm to image norm;

MUTUAL COMPARATIVE FILTERS AND THEIR APPLICATION TO CHANGE DETECTION
where (x,y) = image coordinates, g0(x,y) = mean value of ( , ) g x y , K(f,g) = normalized linear correlation coefficient: where 0 0 ( , ) Mutual comparative filtering within a window w is more applicable in practice.The window w is parametrized by its center coordinates inside the image, its size s (width and height) w(x,y,s) = w(x,y.width,height)but for short we will use w(x,y) notation.0 0 ( , ) ( , ) ( ) ( , ), ( , ) ( , ); ( , ) 0, ( , ) ( , ); where go w(x y)  mean(go w(x,y) (x,y))mean value g(x,y) within the window w(x,y), K(f w ,g w )local normalized correlation coefficient within the window w(x,y).
In addition, this mutual filter  w (f, g) strongly smooths out such local parts of the image g, the shape of which is not similar to the shape of the corresponding fragments of template f.Such comparative filters may be useful for change detection in the images.
Even more useful can be mutual comparative filter based on search image correlation: , 0 where p(x,y)region for matching g patches to f.Such filter  w,p (f,g) can provide detection of changes in the images that differ not only in presence/absence of some objects but also have small differences in shooting conditions (small shaking of the camera).
Let's give a general definition to mutual comparative filter.
Comparison of images f(x,y) and g(x,y) is performed by analogy to (7): Detection of relative changes in the scene is based on the background normalization, depends on the size of the window and can be carried out as follows: ( , ) .
The main advantage of comparative filters over the classical morphological filters (Pyt'ev, Yu., 1993), (Vizilter, 2014) that are used for image shape comparison is that comparative filters do not require segmentation of images into semantic areas, and accordingly the result of image shape comparison is no longer dependent on the quality of the segmentation.In addition to the above local (4) and search (6) correlation coefficients for a(f,g w(x,y) ) can be used: -local and search morphological correlation coefficients proposed by Pyt'ev (Pyt'ev, 1993); -various local and search coefficients of geometric shape correlation (Vizilter, 2014 ); -heat kernel (Belkin, 2001) of mutual similarity of image fragments that are based on comparison of local feature vectors.
Examples of local mutual comparative filtration with normalized linear correlation coefficient and with different window size are presented on the figure 1.The more window size is, the more local parts of the test image g are smoothed and difference responses are high.As can be seen, the change contours are not precise for big window sizes but it is much easy to threshold such difference images to obtain change (foreground) masks.On the other hand, small window sizes can be used to detect contours of changes.
In the next section, we propose a general scheme for combination of any basic background modelling / subtraction method (Wang, 2014) and mutual comparative filtering.This scheme allows to detect foreground pixels in unstable illumination conditions.the same as 2 nd row, window size is 8×8. 4 th row: the same as 2 nd row, window size is 16×16.5 th row: the same as 2 nd row, window size is 32×32

THE PROPOSED APPROACH
The flowchart of our approach to change detection is presented on the figure 2. For every new image It that we obtain in the moment t, we construct the image pyramid {Is,t}.The image pyramid allows us to detect regions with changes of sizes s by using the window of constant small size and saves for us processing time.For every layer in the pyramid we build background models ms,t based on previous n-observations …,.The trade off between background model complexity and number of levels should be maintained to achieve computational efficiency.The size of the filtration window w defines the minimum rate of changes that can be detected.In our approach, we set w to 4×4.The same size window on every image layer should be used.
The detection of changes in every pyramid layer follows next: We use a high threshold to separate changes from the actual background.That allows us lo locate big change regions with a labelling algorithm.
In the same time, the foreground mask for the image It (1 st layer from pyramid) based on the 1 st layer background model should be obtained by the usual for chosen background model way.
The foreground mask adjustment based on connected regions M(x,y) (big adjustment) follows next: we take only those foreground pixels, which belong to connected regions with changes.
Now we need to refine the foreground mask by using only borders of real changes (small adjustment).The borders of true foreground objects usually are in the first layer of changes ∆I1,t,m(x,y) due to small window size.So, we shrink the foreground mask to borders from ∆I1,t,m(x,y) which are highlighted for us by non-maximum suppression algorithm a la (Canny, 1986).After that, opening and closing mathematical morphology operations are used to remove holes and very small regions that correspond to noise.
To make presented approach more efficient in terms of speed we use the following trick: we fix the maximum size of changes that can be detected by down sampling the original image to several times smaller image (in three times, in our tests) and keep our small window.Then fast approach uses only two layers of the pyramid: 1 storiginal size image, and 2 nddown sampled image.Strong difference responses still can be found by maintaining all filtration procedures on the 2 nd layer.Of course, we sacrifice some precision in detection of high difference responses if we skip some intermediate layers but we will achieve big boost in terms of speed.Figure 3 illustrates main steps of the fast version of the proposed approach to change detection in changing illumination conditions.
Figure 2. The flowchart of our approach to change detection

EXPERIMENTS
In experimental part of this paper, we use approach (KaewTraKulPong, 2001) for background modelling of every image layer and detection of the first layer foreground mask.(KaewTraKulPong, 2001) is another version of adaptive Gaussian mixture model (GMM) for background subtraction (Zivkovic, 2004).Since (KaewTraKulPong, 2001) is an example of the fast and simple background modelling algorithm that have problems with illumination changes and is used as a building block in (Bianco, 2015) algorithm, it suits well our needs.
In our tests background modelling is done in RGB color space and comparative filtration is done for gray versions of corresponding images.We also stick to normalized linear correlation coefficient (10) due to its simplicity, when computing mutual comparative filters.
In order to show the quality characteristics of the proposed approach we compare the original version of (KaewTraKulPong, 2001) algorithm against its modified version.For this study, various sequences were obtained from GTILT dataset (Bales, 2011) and marked up.GTILT dataset contain low frame rate realistic, camera-captured video scenarios with moving objects and strong illumination changes.
We follow the methodology from (Wang, 2014) to estimate the quality of the proposed approach.Thus the following metrics are used: Re -Recall, SP -(Specificity), FPR -False Positive Rate, FNR -False Negative Rate, F -F-Measure, PR -Precision, TP -True Positive, FP -False Positive, FN -False Negative, TN -True Negative.All videos have been down sampled to 320 × 240 pixels before testing.
Quantitative results of original algorithm (KaewTraKulPong, 2001) were computed for comparison and are shown in Table 1, 2, 5.
Proposed algorithm implemented on the basis of (KaewTraKulPong, 2001) is named «modified».Quantitative results of modified algorithm are shown in Table 3,   It can be observed from tables 5, 6 that with our modification change detection algorithm is capable of removing a significant amount of false positives and almost preserves recall value.The difference between the performance achieved by our proposal and that of the original version in terms of F-measure is 32% and in terms of precision is 38,4%.

CONCLUSION
The newest results in change detection field of study tell us that there are not many simple algorithms that are robust to some challenges that can be found in real-world scenarios.This paper addresses one of such challengesillumination changes, especially in low frame rate videos.In this paper we propose a new approach for change detection problem on the basis of the mutual comparative filters that allow us to locate changes in the morphological framework, but doesn't require the scene (background model) segmentation.Presented approach provides a simple way to deal with illumination changes and doesn't have drawbacks which exist in diffusion background modelling approach.The definitions of mutual comparative filters are given and their strong property is outlined.It is shown that popular background modelling could benefit from our proposal.

Figure 1 .
Figure 1.Comparative filtration with different window sizes. 1 st row from left to right: background model (template image f) and image with object (test image g). 2 nd row: filtered test image f(g) and detected relative changes ∆gf, i.e. difference between filtered test image and test image, window size is 4×4. 3 rd row:the same as 2 nd row, window size is 8×8. 4 th row: the same as 2 nd row, window size is 16×16.5 th row: the same as 2 nd row, window size is 32×32 4, 5.And comparative results are shown in figures 4, 5, 6.

Figure 3 .
Figure 3. Change detection in challenging illumination conditions.First row: leftimage of the scene that experience illumination changes, rightcurrent image with foreground object.Second row: left -detected changes in the 1 st pyramid layer, rightthresholded detected changes in the 2 nd pyramid layer.Third row: leftforeground detection based on first layer background model, right -foreground mask adjusted based on detected changes from 1 st and the 2 nd pyramid layers

Figure 4 .
Figure 4. Experimental results for video Bank.Top row: leftcurrent frame, rightground truth.Bottom row: leftoriginal algorithm, rightour modified algorithm Let us generalize MIA based on comparative filters.Comparative filter takes as input two images for comparison: model or template f and test image g.Comparative filter is mutual (f,g) if it is created from images f and g to filter image g.I.e. it is necessary to process a pair of corresponding images f and g (or pair of their fragments) in every image point to filter image g.

Table 2 .
Original algorithm results for each video in terms of metrics

Table 3 .
Modified algorithm results for each video in terms of pixels

Table 4 .
Modified algorithm results for each video in terms of metrics

Table 5 .
Comparison of algorithms results for all videos

Table 6 .
Relative comparison of Modified algorithm to Original algorithm for all videos