CLASSIFICATION OF FEATURES OF IMAGE SEQUENCES

Processing of image sequences is a very actual trend now. This is confirmed with a vast amount of researches in that area. The possibility of an image sequence processing and pattern recognition became available because of increased computer capabilities and better photo and video cameras. The feature extraction is one of the main steps during image processing and pattern recognition. This paper presents a novel classification of features of image sequences. The proposed classification has three groups: 1) features of a single image, 2) features of an image sequence, 3) semantic features of an observed scene. The first group includes features extracted from a single image. The second group consists of features of any kinds of image sequences. The third group contains semantic features. Reverse feature clarification method is the iterative method when on each iteration we use higher level features to extract lower level features more precisely. The proposed classification of features of image sequences solves a problem of decomposition of the source feature space into several groups. Reverse feature clarification method allows to increase the quality of image processing during iterative process.


INTRODUCTION
Today we can see the growing popularity of methods of processing of image sequences.This is due to several factors.At first, the increasing capabilities of computer systems allow to process more images for the constant time.Secondly, more powerful sensors can provide several images at the same time.For example, during multispectral photography cameras give several image of a single place in different frequency spectrums.Also we can consider here video recording instead of simple photography.Thirdly, complex methods and algorithms of image processing evolve.Some of them requires creating or generating new images based on current image during their work.This is done for extracting more image features from a single image.For example, during wavelet analysis three additional images are generated.Forthly, in the most cases an image sequence contain more information than a single image.Therefore, processing of the whole sequence should give us more precise results.A lot of publications in this area only confirms its actuality.
Feature extraction is one of the main steps during image processing and pattern recognition.It can also be performed during processing of image sequences.While single image features and ways of their extraction are well-known, features of image sequences is not studied well enough.With the growth of the number of features there is a problem of their ordering.This paper presents a general mathematical model of feature extraction from digital image sequences and a novel classification of features of image sequences.The reverse feature clarification method is described.It allows to increase results of image processing and pattern recognition during an iterative process.
In chapter 1 we show the evolution of methods of image sequences processing.Chapter 2 briefly describes the main kinds of image sequences.The general mathematical models of image sequences are given in chapter 3. The classification of image sequences features and the reverse feature clarification method are described in chapter 4.

THE EVOLUTION OF IMAGE SEQUENCES PROCESSING
The first references of image sequences appeared almost at the same time after the digital image processing was born in the 1950-1960s (Pratt, 1974, Meyer-Eppler and Darius, 1956, Anuta, 1969, Barnea and Silverman, 1972).In that time there was no term of image sequences, because computers were very limited and the scientists were focused on processing of a single image.With the growth of capabilities of computers and cameras the number of publications related to image sequences increased.
In (Pratt, 1974, Meyer-Eppler and Darius, 1956, Anuta, 1969) a problem of comparing shifted objects on two images was described.Authors used a correlation link between objects.The images had the same object came from different sensors or from single sensor but with time delay.In (Barnea and Silverman, 1972) authors used the sequential approach instead of correlational.The optimization of the given methods were described.
The correlational approach uses the finding of a maximum of a correlation function.The sequential approach uses the iteration over the image pixels and their comparison with the neighborhood of the first object.
In (Jain and Jain, 1981, Jain, 1981, Jane et al., 1995) the methods that uses motion in spatial area for segmentation were given.Authors used the approaches of building of image differences.Mathematical model of images of a sequence looks like f (x, y, ti), where we can see the third argument -the time of the frame.This description is suitable for describing video sequences.Authors of (Rajala et al., 1983, Shariat and Price, 1990, Cumani et al., 1991, Dellen and Wrgtter, 2011) also studied the problems of using motion during segmentation, but in spectral area using the Fourier transform.Moreover the motion estimation is described in (Shapiro andStockman, 2001, Sonka et al., 1999).

TYPES OF IMAGE SEQUENCES
There are a lot of kinds of image sequences.The main are video sequences and image sequences created during multiscale transforms (Zhiznyakov, 2007).They are image pyramids and images created during wavelet analysis.Besides there are some more types of image sequences.They are processed with the same methods as used with main kinds of image sequences.The other types are: 1. Images from multispectral photography.
2. Color images.During processing of color images the sources pixels are intensity values of the each color.
3. Image sequences generated with some processing operators (filters etc.).
Detailed review of different types of image sequences can be found in (Pugin, 2014).

MATHEMATICAL MODELS OF IMAGE SEQUENCES
There are a lot of mathematical models of image sequences today.
Because the number of problems in this area is great, authors often choose well-suited models for their purposes or simply do a verbal description of the image sequence model.In this case they assume that image sequence is a set of two or more images related to each other by some criteria, operator or semantics.For example, during video sequences processing the next formula is used (Shapiro and Stockman, 2001): where F -the image sequence, f (x, y, ti) -the frame in time ti (i = 1, 2, ..., n).
Color image in RGB format can be described as The sequence of generated images during processing can be presented as where Ti[•] -some operator (i = (1, 2, ..., n)).
Another approach of mathematical model of image sequences is given in (Zhiznyakov, 2008).An image sequence {fn} is an ordered set of images f .Every image has an index {fn} = {f0, f1, ..., fn}.
The expression ( 4) is well suited for many kinds of image sequences, so we will be using it in the next chapters.
Each image fi(x, y) of the sequence has its own set of features Si.So, we can build the next sequence {Sn} = {S0, S1, ..., Sn}. (5)

CLASSIFICATION OF FEATURES OF IMAGE SEQUENCES
The proposed classification has three groups: 1. "classic" image features or features of a single image, 2. features of an image sequence, 3. semantic features of an observed scene.
The first group includes features extracted from a single image f (x, y).Let Oi[•] be an operator of feature extraction.And the feature extraction process can be described as where X1 -the set of features of the image of the first group.This group includes statistical, fractal (Zhiznyakov et al., 2014), geometric, topological, color, spectral features.
The second group consists of features of any kinds of image sequences.The feature extraction can be done as The third group contains semantic features.These features are usually selected empirically and based on the features of previous groups.Such features are used for understanding of 3D-scenes and objects, their analysis, behavior estimation.The extraction of semantic features is where X3 -a set of semantic features.Sometimes it is possible to extract semantic features from each other or from all groups of features Examples of semantic features are speed of objects on the scene, movement vector in 3D-space, other scene characteristics.
The general algorithm of features extraction from an image sequence can be described as: 1. Extract features from every single image in sequence and add them to common multidimensional array.
2. Extract sequence features from the given array.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-5/W6, 2015 Photogrammetric techniques for video surveillance, biometrics and biomedicine, 25-27 May 2015, Moscow, Russia 3. Extract semantic features from the features of two first steps.
To increase the quality of image processing and pattern recognition it is necessary to study the influence of proposed groups of features on each other.Obviously that the features of the lower groups directly affect the higher level feature.Many research papers aims to study such influence.But in the same time it is interesting to investigate the backward influence of higher level features on the lower level features.There is a lack of research papers in that area.This task can solve the following question: "If we know semantics of the scene, can we improve the quality of recognition of the separate objects on it?" Let's call such technique as "reverse feature clarification method".It is the iterative method when on each iteration we use higher level features to extract lower level features more precisely.For example this method can be used in reconstruction of the location of the hidden object on early frames of video sequence in case if this object has predictable speed and stable forms (find a car in the car stream).
In general case we can present the reverse feature clarification method as where X k i -features of i group of image sequence on iteration k; i, j -numbers of features groups, k -the iteration number, Q[•] -the operator of clarification.

CONCLUSIONS
In this paper we did a review of evolution, types and mathematical models of image sequences.We presented a novel classification of image features extracted from image sequences.The proposed classification of features of image sequences solves a problem of decomposition of the source feature space into several well defined groups.The reverse feature clarification method is given.It is based on presented classification.Reverse feature clarification method allows us to increase the quality of image processing during an iterative process.
where X2, X 2 -extracted features, O2[•], O 2 [•] -the extraction operators, fn(x, y) -the set of images or the image sequence, {X1} -the set of first group features of the image sequence.The first form is used for extracting image features from the image sequence fn(x, y).The extraction can use only several images of the sequence.The second form extracts features using first and second feature groups.Image sequence features are used in motion and change detection, motion analysis, object detection and tracking and in some other areas.Feature examples of video sequences are the focus of expansion, the focus of contraction, the motion field, speed of changing of features of the first group during the sequence.