DETECTING EVENTS IN VIDEO SEQUENCE OF VIDEO-EEG MONITORING

In this paper, an algorithm for automated detecting diagnostic events in video channel of video and electroencephalographic (EEG) monitoring data is presented. The analysis of video sequences is focused on identifying a group of frames with high or very low (depending on the type of seizure) dynamics of informative areas according to a criterion calculated during processing of the optical flow. The preliminary results of the analysis of real clinical data are given and compared with data obtained from the synchronous EEG. The results showed the possibility in principle of reliable diagnosing epileptic seizures and distinguishing them from nonepileptic events. * Corresponding author


INTRODUCTION
This paper deals with the problem for automated detecting diagnostic events in video channel of video and electroencephalographic monitoring data is presented. Videoelectroencephalographic (Video-EEG) monitoring is a method of long -term synchronous registration of electroencephalography (EEG) and video image. Simultaneous video recording of the clinical condition of the patient and the bioelectric activity of the brain (i.e. EEG) allows one to diagnose epileptic seizures reliably and differentiate them with events of non-epileptic nature.
The duration of EEG monitoring is usually 24 hours or more, and if used in intensive care units it can last for weeks. Visual analysis of large amounts of data obtained during long-term video-EEG monitoring requires huge labour costs and special training of clinical neurophysiologists. This determines the urgency of development new methods for detection, quantitative analysis and metric classification of diagnostic objects and removal of artifacts in long-term video-EEG monitoring of patients.
The methodology of EEG analysis is traditionally based on the visual analysis of curves, for the selection of non-artifact record fragments and the analysis of its background structure, single (epileptiform) graphical elements and their specific patterns with specificity in relation to various clinical states (Hirsch, 2010). In most cases, the algorithmic capabilities of the software for video EEG instruments are limited to preprocessing multi-channel EEG signals, indicating the likelihood of record artifacts, calculating inter-channel coherence and sources of electrical activity. To simplify the assessment of large volumes of visual information, a mathematical analysis of the oscillations is used with a graphical presentation of the results -a quantitative EEG (qEEG) (Duffy, 1994). However, the method of visual presentation of quantitative EEG in the form of trends and histograms does not take into account many artifacts, in particular, chewing and movement of the patient's head. When detecting high-amplitude plots on the qEEG histogram, physician needs to revise the in video record a fragment of interest for visual assessment and differentiation of an epileptic and artifact event. For this purpose, they use not only the electrographic pattern, but also the analysis of video.
Currently, several methods of automatic detection of seizures according to EEG data have been proposed (Hirsch, 2010;Tzallas, 2007;Antsiperov., 2017;. However, all of them use only the native EEG without taking into account the video image, and their accuracy is insufficient for widespread clinical practice. The analysis of publications in periodicals and monographs in the subject domain showed the lack of publications on methods for automatic recognition of epileptic seizures in video sequences obtained during the video-EEG monitoring. Therefore, it is necessary to develop methods and algorithms for automated detection of diagnostic events in long-term video-EEG recordings, which will improve the reliability of their classification and significantly reduce the time for analyzing large amounts of video EEG data and increase their diagnostic significance.
In this paper, we propose an algorithm for detecting two types of diagnostic events in the video channel of the video-EEG monitoring data. The first type of events is observed for patients in coma. The second type of events is fixed for active patients and can be observed in the form of activity freezing for a few seconds.

ALGORITHM FOR DETECTING DIAGNOSTIC EVENTS
When creating methods and algorithms for automated detection of diagnostic events in video-EEG recordings, it is assumed that the decision on event detection is made when an event is detected simultaneously in two channels -EEG and video.

Detecting Events for Patients in Coma
In this case, it is necessary to analyze informative areas in patient images with visible particular muscle contractions. Informative areas are usually associated with the details of a person's face (eyes, nose, and mouth). Analysis of video sequences, which recorded seizures, showed a variety of appearance of these seizures. For example, for a non-convulsive attack, only rather weak muscular contractions are observed in the region of the patient's mouth. At the same time the rest of the facial muscles remain motionless. In another case, there are more intense contractions of the muscles of the mouth and periodic movements of the head with immobile muscles in the eye area. In a number of patients, intense contractions of the muscles are observed all over the face, and contractions of the neck muscles and head movements are also possible. In the absence of seizures, the frames of the video data are quite static for the studied group of patients.
One of the possible ways of detecting events on a video channel can be associated with an analysis of the dynamics of small areas associated with the details of a person's face (eyes, nose, mouth). The literature presents a wide range of methods for localizing such areas (Yow, 1997;Viola, 2004;Zhang, 2016;Singh, 2017;Shi, 2018;Baroffio, 2018;Araujo, 2015). It should be noted that the images of video sequences obtained during video-EEG monitoring have the following features. First, arbitrary angle of video recording the patient's face. This feature eliminates the use of methods based on the property of face symmetry, and methods that require full frontal image of the face. Second, presence of medical equipment, partially covering the details of the face. This circumstance also complicates the task of localizing diagnostically important areas. Third, Informative areas may not be associated with characteristic points of the face (eyes, corners of the mouth, etc.). Such areas, for example, may be neck areas. Therefore, traditional methods of localization of characteristic points and details of the face may not be applicable.
In this paper, we propose to detect diagnostic events by the magnitude of the criterion characterizing the degree of activity of the region of interest. The area of interest will be the portion of the frame that includes patient's face, head, and neck areas (see Figure 1). As a criterion of the activity of the region of interest, the total optical flow calculated for each frame of the video sequence will be used:  (Kalman, 1961). We apply the Kalman-Bucy algorithm, since it provides the optimal estimate in the sense of minimum error variance. The decision to fix a diagnostic event will be made according to the threshold rule. To avoid false alarms of the detector due to short-term spikes, the decision about the occurrence of an event will be made if the value of   Ji exceeds a predetermined threshold in a sequence of frames of length not less than M . Thus, the decision rule is formulated as follows: where Event is an event indicator, T is a threshold value, 0 i is a frame number from which the inequality taking place, M is the length of the sequence of frames required to make a decision about the appearance of a diagnostic event. The threshold value is defined as:   (2) and (3). If the condition   J i T  is satisfied, the current frame number 0 ii  is stored. If the condition is not satisfied, go to step 1.

 
J i T  and 0 i i M  are satisfied, it is decided that an event is detected. In the opposite case, go to step 1.

Detecting Events for Active Patients
Diagnostic events of a different type manifest as freezing for a few seconds of active patients (see Figure 2). As in the previous case, we propose to fix events by the value of the criterion (1). In contrast to the case considered above, the occurrence of an event corresponds to a minimum of the activity index. The decision rule is as follows: and the threshold value is calculated by the formula: where 1 k is the coefficient. Algorithm for event detecting is the same as presented in subsection 2.1, but the decision rule (2), (3) is replaced with rule (4), (5).
In the next section, a computational experiment is conducted to evaluate the effectiveness of the proposed algorithms.  (Lucas, 1981) is applied. This algorithm for computing the optical flow is chosen from the condition of the highest performance in comparison with other methods. The magnitude of the smoothed mobility index is calculated using a discrete version of the Kalman-Bucy filtering algorithm (Kalman, 1961). The values of the filtering algorithm parameters are selected when processing test video sequences, and based on the best ratio of the error and speed values.
The experiment includes two stages. At the first stage, the detection algorithm was applied to five videos of patients in coma. In three records epileptic seizures are detected, including non-convulsive one. For each of the five video sequences, the parameters of the decision rule (2), (3) were determined from fragments with low scene dynamics. The number of frames corresponding to the shortest event duration in condition (2) was chosen equal to 75, which corresponds to a time interval of 2.5 seconds. The value of the coefficient in (3) was selected as 1.1 k  . Examples of graphs of the criterion   Jt and its smoothed value   Jt, as well as the event indicator Event for two fragments of the video-EEG monitoring of the same patient are shown in Figures 3 and 6. Here and below, instead of the variable i designating the frame number, we use the variable , where t is the time, and FrameRate is the frame rate of the video. Figure 3 demonstrates detection of diagnostic events. The graphs in Figure 6 correspond to video record without seizures. For the videos, corresponding to the graphs shown in Figures 3 and 6, the parameters in expression (3) for determining the threshold value T were found to be equal to 0 1604 J  and 127.9   .  Figure 4 shows the projection of the ridge of the wavelet spectrogram on the power spectral density (PSD) and time axes. Using adaptive threshold, the ridge points (maximum values of the power spectral density at each time point), which lie above this threshold, were calculated. Close points of the ridge, lying above the threshold, were combined into clusters -fragments of the ridge, marked in black on the graph. These ridge fragments are interpreted as episodes of suspicious activity, similar to an epileptic seizure.  At the second stage of the experiment, the data of video-EEG monitoring of active patient were analyzed. In this case, an epileptic seizure is observed as the patient's freezing for a few seconds. The parameters of the decision rule (4), (5) were determined from the video record fragments. For the video, the analysis results of which are shown in Figure 7, the parameters of the expression (5) were found as 0 4483 J  and 797.2   . The value of the number of frames corresponding to the shortest duration of the detected event in condition (4), as at the first stage of the experiment, was chosen equal to 75, which corresponds to a time interval of 2.5 seconds. The value of the coefficient in (5) was selected as 2 k  . The graphs of the criterion   Jt, its smoothed value   Jt, and the event indicator Event for a fragment of the video sequence of the video-EEG monitoring are shown in Figure 7. In Figure 7 it can be seen that in the interval between 30 and 35 seconds, corresponding to the freezing of the patient in the frame, a seizure was detected.  9 show graphs of the processed EEG signals from synchronous recording of video EEG monitoring. Figure 8 shows the projection of the ridge of the wavelet spectrogram on the axes PSD, time. This fragment is interpreted as an episode of suspicious activity. From Figures 7-9, you can see the interval of synchronous low muscle and high brain activity between, which may correspond to an epileptic seizure. Figure 9. EEG signal in one of the channels, filtered by a Butterworth filter of the 8th order and notch filters

CONCLUSIONS
As part of the development of the technology for detecting epileptic seizures from video-EEG monitoring data, an algorithm for automatic detection of diagnostic events based on the analysis of the quantitative characteristics of the patient's activity in video records is proposed. Analysis of video sequence is focused on identifying a group of frames with high or low scene dynamics using a criterion calculated from the optical flow magnitude. The preliminary results of the analysis of real clinical data for patients in coma and active patients are presented. The results of the analysis showed the efficiency of the proposed algorithm at different angles of shooting and partial occlusion of the patient's face with the details of medical equipment. A comparison of the results of diagnostic event detection from the video record with data obtained from the synchronous EEG was made. The comparison showed the possibility in principle of reliable diagnosing epileptic seizures and distinguishing them from non-epileptic events. Further research will be aimed at developing a method for detecting epileptic seizures based on a joint analysis of synchronous EEG and video channels.