KR101262352B1 - Analysys method for video stream data using evolutionary particle filtering - Google Patents

Abstract

PURPOSE: An analysis method for video stream data using evolutionary particle filtering is provided to analyze multi-modal video stream in real time, by extracting dependency relation of video stream. CONSTITUTION: A video stream analysis system generates a particle set showing dominant feature for the video stream data and divides video stream data into segment sets(S1110, S1120). The system performs evolutionary particles filtering the segment set(S1130). The system converts the filtered particle set into a transition probability matrix, by estimating sequence-dependent structure for the filtered particle set(S1140). [Reference numerals] (AA) Start; (BB) End; (S1110) Generate a particle set for video stream data; (S1120) Divide the video stream data to segments using the particle set; (S1130) Perform evolutionary particle filtering for the segment set; (S1140) Convert into a transition probability matrix by estimating a sequence-dependent structure

Description

ANALYSYS METHOD FOR VIDEO STREAM DATA USING EVOLUTIONARY PARTICLE FILTERING}

The present invention relates to a filtering technique for video stream data. More particularly, the present invention relates to an evolutionary particle filtering that divides stream data into a set of segments represented by a set of particles and flexibly analyzes the video stream through evolutionary learning. It relates to a video stream analysis method used.

With the development of the imaging technology, various analysis and learning on the video data is being performed. In particular, various attempts have recently been made to perform semantic classification on video streams.

However, the related arts merely detect the change in the expression (composition) of the screen and distinguish the images or compare the images. In other words, the analysis is only performed based on the variability of the image itself, and has a limitation in that it cannot perform processing such as classifying or filtering video stream data according to the value of multi-modal.

An object of the present invention is to provide a video stream analysis method using evolutionary particle filtering that can extract a dependency relationship of a video stream and analyze a predetermined multimodal video stream in real time.

In addition, the present invention is to provide a video stream analysis method using evolutionary particle filtering that can analyze the stream in a very flexible manner without using prior knowledge by dividing stream learning into dominant feature extraction and dependency learning.

Among the embodiments, the video stream analysis method comprises the steps of: (a) generating a set of particles representing a dominant feature with respect to the video stream data, dividing the video stream data into segment sets, and (b) the divided plurality of segments For the set, performing evolutionary particle filtering to generate an ordered feature, and (c) estimating an order-dependent structure for the set of particles on which the evolutionary particle filtering is performed and converting them into transition probability metrics.

In an embodiment, the step (a) may include (a-1) extracting a feature from an original image and generating a visual word set using the extracted feature, (a-2) randomly selected non-monthly moon Generating a particle set having position information by associating words, and (a-3) generating the segment set to correspond to the particle set.

In an embodiment, the step (a-1) may extract the characteristic using SHIF.

In one embodiment, the particle set may represent individual particles included in the set as a pair of nodes and edges, and the node may be selected from a visual word set.

In one embodiment, step (b) may comprise selecting particles using the goodness of fit as the critical distribution q.

In one embodiment, the goodness of fit may be calculated in consideration of the distance between the nodes constituting one particle, the ability to represent one particle, and the penalty.

In one embodiment, the evolutionary particle filtering samples the particles using the distribution of the goodness-of-fit values, and the number of particles constituting a set of particles at a particular point in time may be fixed.

In one embodiment, the evolutionary particle filtering may construct a new population using the goodness of fit of the genetic operator output and the parent generation population to prevent population degeneration due to resampling.

In an embodiment, the step (c) may include: (c-1) comparing and searching for different particle sets; and (c-2) comparing the two or more particle sets with a new reference image when the difference between the two sets is greater than or equal to a certain reference value. And (c-3) adding a new dominant image, if present, to the transition probability data structure.

In one embodiment, the video stream analysis method may further comprise (d) regenerating an image representing the segment using the particle set.

According to the present invention, it is possible to extract a dependency relationship of a video stream to analyze a predetermined multimodal video stream in real time.

In addition, according to the present invention, by dividing stream learning into dominant feature extraction and dependency learning, there is an effect that the stream can be analyzed in a very flexible manner without using prior knowledge.

1 is a reference diagram showing an example of each step of the image representation method according to the present invention.
2 is a reference diagram illustrating an algorithm for an evolutionary particle filtering step according to the present invention.
3 is a reference diagram illustrating an algorithm for sequential dependency learning according to the present invention.
4 is a reference diagram for explaining the above-described evolution concept.
5 is a graph showing 19 evaluation results for one evaluation episode.
6 shows the distribution of the evaluation results by the person.
7 is a reference diagram comparing the evaluation by the person and the result according to the present invention.
8 is a graph showing the goodness-of-fit curves of the individuals with the highest suitability among the population constituting the section.
9 is an example of an image reproduced in an evolved population.
10 is a reference diagram illustrating an estimated image order based on an estimated transition probability matrix.
11 is a flowchart illustrating an embodiment of a video stream analysis method according to the present invention.

Description of the present invention is only an embodiment for structural or functional description, the scope of the present invention should not be construed as limited by the embodiments described in the text. That is, the embodiments are to be construed as being variously embodied and having various forms, so that the scope of the present invention should be understood to include equivalents capable of realizing technical ideas.

On the other hand, the meaning of the terms described in the present invention will be understood as follows.

The terms "first "," second ", and the like are intended to distinguish one element from another, and the scope of the right should not be limited by these terms. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" to another element, it may be directly connected to the other element, but there may be other elements in between. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that there are no other elements in between. On the other hand, other expressions describing the relationship between the components, such as "between" and "immediately between" or "neighboring to" and "directly neighboring to", should be interpreted as well.

It should be understood that the singular " include "or" have "are to be construed as including a stated feature, number, step, operation, component, It is to be understood that the combination is intended to specify that it does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

In each step, the identification code (e.g., a, b, c, etc.) is used for convenience of explanation, the identification code does not describe the order of each step, Unless otherwise stated, it may occur differently from the stated order. That is, each step may occur in the same order as described, may be performed substantially concurrently, or may be performed in reverse order.

The present invention can be embodied as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all kinds of recording devices for storing data that can be read by a computer system . Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and also implemented in the form of a carrier wave (for example, transmission over the Internet) . In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Commonly used predefined terms should be interpreted to be consistent with the meanings in the context of the related art and can not be interpreted as having ideal or overly formal meaning unless explicitly defined in the present invention.

The present invention introduces the dependency learning method embedded in the video stream. In the present invention, it is intended to convert the given sequential stream data into a dependency structure composed of particle populations. Each particle cluster is responsible for summarizing its segments. It is a feature of the present invention that segment summarization and dependency learning proceed with non-supervised-online learning without assuming prior knowledge. In the present invention, learning proceeds to this stage. In the first step, based on the evolutionary particle methodology, we find the dominant image that is common to one segment and estimate the segment through the change of the dominant image. Each dominant image is represented by a combination of image descriptors. Choose a feature that explains the dominant image through evolution. Genetic operators play a role in introducing diversity into clusters to prevent sample impoverishment. In the second step, the probability of transition between the estimated segments is calculated and stored. The present invention was used to extract dependencies that existed in episodes of TV drama. In the present invention, the performance of the present invention was confirmed by comparison with the evaluation result of the human evaluator.

I. Introduction

The present invention can observe stream data to confirm the order of streams, and analyze stream data more efficiently using the identified stream order. The present invention discloses a method of analyzing a multimodal sequential stream and learning the dependencies inherent therein.

The present invention divides the continuous stream data into a series of segment sets, and then extracts the dependency relationships inherent in the generated segment sets. Each segment is summarized through a set of particles representing the dominant characteristics of that segment. Through segmentation, a given sequential stream is transformed into a permutation of particle populations and the transition probability between segments can be calculated. Segment summary The dependency relationship between a set of particles is used as another way to store sequential streams.

In order to extract the dependency relationship, the present invention may use an evolutionary particle filtering (PF) method. Instead of assuming a prior distribution for the hidden variable that describes it for a segment, the present invention can approximate that distribution through a group of particles.

Unlike the traditional particle filter method, the present invention can express the particle set using a predominant image pattern that can describe a segment. Since the actual data constituting the video stream is affected by irregular distortion and the length of one segment is not fixed, dividing the video stream into segments is not easy. However, the present invention solves this difficulty by using two-step learning. The two-stage learning described above may include an evolutionary step and a dependency learning step. In the evolution stage, the dominant feature can be extracted through particle evolution. In the dependency learning stage, the dependency structure can be estimated based on the evolution stage result, and the estimation result can be stored as a transition probability matrix. The stream learning process according to the disclosed technique can be divided into dominant feature extraction and dependency learning stages, so that the stream can be analyzed in a very flexible manner without using prior knowledge.

II. related research

There is a layered mixture model for unsupervised stream clustering based on a multi-modal (multi-modality) characteristic distribution as a method of dividing a dominant image section from stream data. This dynamic mixing model makes it possible to discover story boundaries relatively easily because it uses repetitive shots that can be observed in news broadcasts. However, such a dynamic mixing model cannot perform meaningful segmentation on stream pictures (eg, TV dramas, etc.) that are not news broadcasts or the like, that is, expecting no repeated fixed frames.

The disclosed technique assumes the existence of a hidden variable that can produce all frames that share a dominant image, and can approximate the hidden distribution using a set of particles. In addition, the disclosed technique uses a method of selecting a main feature from a preprocessed image and selecting a main feature again when it is determined that the predominant image has changed in order to process the visual feature configuration according to the segmentation process of the TV drama. Can be.

The disclosed technique can use a particle filtering method to construct a generation model for a given stream. Particle filtering has good analogy in the latent state. In other words, because of the simplicity of local independence (there can be statistical independence of observed variables under the condition that they are known if they are hidden under a set of observed variables), the hidden variable model has great potential for explaining observed variables. Because it has.

In the present invention, a modified particle filtering method is proposed. In typical genetic algorithm approaches, the best chromosomes dominate the population, so the approach of the present invention can take a different approach than the conventional genetic algorithm approach. Thus, the present invention can maintain genetic diversity using genetic operators. In the present invention, a set of particles represents an image in a cooperative manner. In other words, by using a particle set that combines partial representations of a single particle, the present invention can overcome the limited representation ability of a single particle.

)이 존재하는 스트림에서 해당 구간이 서로 독립적일 경우, 시간 1에서 T에서 관찰된 스트림은 [수학식 1]과 같이 표현할 수 있다. Since the information content of real world mission is not only static or spatial but also depends on temporal order, it is multimodal. Therefore, the study of dependency modeling of sequential data is one of the important factors. The vector value correlation structure based on the sparse Gaussian graphical model applies an unsupervised analysis of sequential stream processing with multiple turning points. That is, K unknown segments (

When the corresponding sections are independent of each other in the stream in which) exists, the stream observed in T at time 1 may be expressed as Equation 1 below.

Here, the probability that the data from time t to time s belong to the same section may be defined as shown in [Equation 2].

Existing techniques assume that the prior distribution of time between successive points is a negative binomial distribution. Unlike the prior art, the aim of the present invention is not to find a regression model that can best represent a given stream, but to estimate the interval structure of a stream whose characteristics are not practical to assume a base distribution. Therefore, instead of calculating Equation 2, the present invention can avoid calculating Equation 2 by calculating the likelihood of a new frame given the current particle population.

To this end, the present invention can capture and use changes in the dominant image. The present invention can use a Gaussian mixture model in the image change modeling process. In the present invention, a pixel satisfying a given confidence level in a Gaussian mixture model may be regarded as a foreground pixel. According to the present invention, a visual word made through a scale invariant feature transform (SIFT) method may be used without distinguishing between foreground and background pixels.

III. Evolution particle filtering and dependency learning

Hereinafter, the dependency learning method according to the present invention will be described.

The present invention can segment the given stream through the estimation of the dominant image change. The evolved particles provided by the present invention serve as a generation model that summarizes the associated sections and can regenerate the sections.

The present invention can provide a learning method comprising an evolutionary particle filtering step (algorithm 1) and an order dependency learning step (algorithm 2) to generate a particle set using a dominant image and convert the generated particle set into order information. have.

1 is a reference diagram showing an example of each step of the image representation method according to the present invention, as described above, the present invention extracts characteristics from the original image using SIFT and the like <Figure (a)> Using the extracted features, a visual word set can be generated (Figure (b)). After that, particles with location information can be generated using randomly selected visual words (Figure (c)). Here, the particles can be made to cover a portion of the image based on the feature and the location of the feature <Figure (d)>.

1 illustrates a chromosome structure according to the present invention. In the present invention, to extract the dominant characteristic of any one section, in order to cooperatively express this, each particle may be composed of a set of unlimited SIFT characteristics. If this is expressed as an equation, Equation 3 is obtained.

)을 획득할 수 있다.Each circle illustrated in FIG. 1 may represent a specific characteristic of an image, and a rectangle corresponds to a SIFT characteristic representing a specific image characteristic group. When filtering particles through evolution, a set of particles expressing only the essential characteristics of a given image (

) Can be obtained.

2 is a reference diagram illustrating an algorithm for evolutionary particle filtering according to the present invention. Hereinafter, the evolutionary particle filtering will be described with reference to FIG. 2.

는 현재 구간의 첫번째 프레임을 의미한다.The goal of the evolutionary particle filtering step is to create a particle filter that captures the essential characteristics of the interval. If this goal is expressed in a manner that takes into account time, it may be expressed as shown in Equation (3) below. From here,

Means the first frame of the current section.

When a new frame is input, the present invention may complement the existing particle population by reflecting the characteristics of the new frame. However, since the possibility of the new section cannot be ignored, the present invention can supplement the existing particle population by using the likelihood to generate a new frame in the current population.

In the present invention, similar to a hyper network, individual particles may be expressed as <N, E> (N: node, E: edge connecting nodes). Nodes can be selected from a set of visual words, and edges can connect between nodes. What's new here is that multiple particles represent images in a collaborative way. It is impractical to assume a single data type that can represent an entire image because the dimensions of the image are so large, while the data structure representing a single feature (single SIFT feature) may be too limited. Therefore, the present invention is intended in the middle, it is possible to use a dominant pattern as a set of particles. In other words, although individual particles have limited expressive power, it is possible to capture a dominant pattern with a set of particles that represent part of the entire image. Even in the entire population, a group of highly suitable particles can be used to express the image.

와 그 구조적 특성을 반영할 수 있도록 탐색 프로세스에 탐색 바이어스를 적용할 수 있다. 여기에서, 적합도를 계산하는 것을 수식으로 나타내면, 도 2의 수식(4) 또는 아래의 [수학식 5]로 표현될 수 있다.The search process according to the present invention is the current particle set

The search bias can be applied to the search process to reflect the and its structural characteristics. Here, when the calculation of the goodness of fit is represented by a formula, it can be represented by the formula (4) of FIG. 2 or the following [Equation 5].

는 파티클 하나를 구성하는 노드간의 거리를 평가한다. 파티클 하나를 구성하는 노드 간의 거리가 너무 클 경우 프레임이 변할 때 파티클 노드의 일부 노드가 사라질 가능성이 있기 때문에 해당 파티클이 충분히 견고하지 않을 가능성이 있다. 그러므로

는 파티클 하나를 구성하는 노드의 평균 거리가 짧은 파티클을 더욱 선호하도록 할 수 있다.

는 파티클 하나의 이미지 표현 능력을 나타낸다.

는 페널티를 의미한다. 만약 어느 한 파티클이 너무 많은 노드를 지니고 있다면 파티클이 너무 자세한 정보를 보유하고 있을 가능성이 있다. 보다 개괄적인 표현 수준을 유지하기 위하여 본 발명은 구성 노드의 수가 적은 파티클을 보다 적극적으로 이용할 수 있다. 협력적인 방식으로 파티클 집합이 이미지를 표현하기 때문에 도 2에 도시된 식 (5)는 파티클 집합

의 함수이다. 반면 적합도는 개별 크로모좀에 적용되기 때문에 식 4는 파티클의

함수이다.Equation 5, which calculates the goodness of fit, consists of three sub-criteria.

Evaluates the distance between nodes that make up a particle. If the distances between the nodes that make up a particle are too large, it is possible that the particles will not be strong enough because some nodes of the particle node will disappear when the frame changes. therefore

May further favor particles having a short average distance of nodes constituting one particle.

Represents the ability of the particle to represent one image.

Means penalty. If a particle has too many nodes, it is possible that the particle has too much information. In order to maintain a more general presentation level, the present invention can more actively use particles having a small number of configuration nodes. Since the particle set represents an image in a cooperative manner, equation (5) shown in FIG.

. On the other hand, since fitness is applied to individual chromosomes, Equation 4

Function.

가 요구될 수 있다. 본 발명에서는

가 아닌 적합도 값[수학식 5]의 분포에 기반하여 파티클을 샘플링할 수 있다. 여기에서, 특정 시점에서 어느 한 파티클 개체군을 구성하는 파티클의 수

는 고정될 수 있다. In order to approximate the actual basis distribution, the particle filtering method has a significant distribution.

May be required. In the present invention,

Particles can be sampled based on the distribution of goodness-of-fit values (Equation 5). Where the number of particles that make up a particle population at any given time

Can be fixed.

Since new particles are created through evolution, excessive resampling in particle filtering can lead to degeneracy. In order to solve this problem, the present invention can solve the problem of population degeneration by introducing new particles through genetic operators. If this is expressed as an equation, it may be expressed as shown in [Equation 6].

)를 유지하기 위하여 유전 연산자 결과물과 부모 세대 개체군의 적합도를 이용하여 새로운 개체군이 구성된다.Fixed population size (

In order to maintain), a new population is constructed using the fit of the genetic operator output and the parent generation population.

Because particle sets have the main characteristics of an image, the process of regenerating an image can be described as an image search process in which a given particle set maximizes the likelihood to recreate an image. When the search process is expressed by an equation, it may be expressed as shown in [Equation 7].

에 의해 표현될 수 있다.In this approach, excessive computational load may be generated, and thus, the present invention may directly generate an image by utilizing the SIFT feature possessed by each node. This approach is

Can be represented by

을 생성해야 한다. 이를 위하여, 본 발명은 계산된 우도 값이 기준값보다 작다면 새로운 구간이라고 판단할 수 있다. 여기에서, 새로운 프레임

의 우도는 [수학식 8]을 이용해서 추정될 수 있다.An evolved particle represents the dominant image of a particular segment, but when a frame belonging to the new segment is observed, a new particle population

You need to create To this end, the present invention may determine that the calculated likelihood value is a new interval if it is smaller than the reference value. Here, the new frame

The likelihood of can be estimated using Equation 8.

는 사전 지식의 하이퍼 인자를 의미한다.
here

Means hyperfactor of prior knowledge.

3 is a reference diagram illustrating an algorithm for sequential dependency learning according to the present invention. Hereinafter, sequential dependency learning will be described with reference to FIG. 3.

이 획득된다. 두 번째 단계(알고리즘 2)에서는 Pf를 전이 확률 매트릭스로 전환할 수 있다. 알고리즘 2에서 제안된 전이 방법을 설명한다.After evolution particle filtering is complete, the set of particle populations, Pf =

Is obtained. In the second step (Algorithm 2), we can convert Pf into a transition probability matrix. The transition method proposed in Algorithm 2 is described.

를 비교하여 동일한 우점 이미지

를 갖고 있다고 추정할 수 있는 비슷한

를 찾는 것으로 이 문제를 해결하였다. 도 3에 도시된 식을 통하여 특정

에 대응되는

를 찾는 방법을 설명할 수 있다. 즉 서로 다른 파티클 개체군

와

를 비교한 후, 두 개체군 사이의 차이가 특정 기준값 이상일 경우 새로운 우점 이미지라고 추정하고, 전이 확률 자료 구조 T에 추가하는 방식으로 예측된 파티클 개체군의 종류를 줄일 수 있다.You cannot know | Pf | beforehand, but the actual number of segments is estimated | Pf | Will be less than This is because characters with limited drama episodes play in a limited background. Therefore, the important point in this learning step is to find the segments with the same dominant image. The present invention

Compare the same dominant image

Similar to assuming you have

Finding this solves this problem. Specific through the equation shown in FIG.

Corresponding to

Explain how to find Different particle populations

Wow

After comparing, we can reduce the predicted particle population by estimating a new dominant image if the difference between two populations is above a certain reference value and adding it to the transition probability data structure T.

Since each segment has a particle population after evolution, the present invention can reproduce an image representing the segment using the particle population. The pre-acquired SIFT characteristic remains at the node, making it simple to recreate the image. Each particle can be interpreted as a set of SIFT properties that has an image patch and the location of that patch (the center point of each image patch). Each particle represents a small portion of the image, but if there are many particles, the image patch and location information stored in each node are accumulated enough to reconstruct a human-identifiable image.

4 is a reference diagram for explaining the above-described evolution concept.

Referring to FIG. 4, it can be seen that the present invention extracts a series of SIFT features from an interval start frame and then uses them as chromosome nodes. After selecting the initial population, evolution can be repeated whenever a new frame is observed. However, when a new frame is observed, the likelihood for regenerating the given frame in the current population is calculated and used as the basis for the new interval (the beginning of the interval). A particle set represents a corresponding section and may be used to regenerate the dominant visual feature of the corresponding section.

IV. Experiment result

In the following, the results of the present invention are compared with those of 19 people for a given actual drama series. The total duration of the evaluation episode was 125 minutes and 30 seconds, and a total of 7530 frames were extracted to process the test data.

Before dealing with the estimation result according to the present invention, the results of judgment by the person are summarized as shown in FIGS. 5 and 6 to Table 1.

Fig. 5 is a graph showing the evaluation results of 19 persons for one evaluation episode, and Fig. 6 shows a distribution diagram of the evaluation results by humans. Table 1 shows the statistical summary of the change intervals.

Average Standard Deviation at least maximum Episode 1 3.85 seconds 2.33 seconds 1.00 sec 9.00 sec Episode 2 3.48 s 2.37 seconds 1.00 sec 9.00 sec Episode 3 3.17 seconds 2.05 seconds 1.00 sec 7.00 seconds

As shown in the figure, the evaluation by people does not agree with any one of the conversion points, and thus, a plurality of adjacent conversion points are collected to form a conversion interval (FIG. 5). Table 1 summarizes the statistical characteristics of the newly created sections. The average length of each interval is 3.85 seconds, 3.48 seconds, and 3.17 seconds, respectively. The minimum value of 1 second of the conversion interval is reasonable, but the maximum length of the interval of 9 seconds may be interpreted as being too long. The maximum section corresponds to a scene in which the city skyline is continuously changing, and corresponds to a section in which it is very difficult to determine that the dominant image changes at one point in time. Although the number of segments can't be known in advance, image analysis is required before the basic components such as visual words can be processed. The image was preprocessed using Scale Invariant Feature Transform (SIFT) to acquire robust characteristics.

7 and Table 2 are the data comparing the evaluation by the person and the result according to the present invention.

Episode 1 Episode 2 Person rating The number of changes 169 191 Number of bins 41 50 Invention Number of conversion points 763 744 Precision 0.076 0.101 Recall 0.343 0.393

로 계산된 우도는 개선의 여지가 많다. 예를 들어 우도의 경향에 집중하여 변환지점을 추정하는 것도 가능하다. 비록 인간 평가 결과보다 매우 저조하지만 본 발명은 SIFT 변환 특성만 사용했다는 사실을 고려했을 때 상당히 유의미한 결과를 달성하였다.As shown in Table 2, in Episode 1, 19 evaluators evaluated 169 conversion points, and 41 conversion point intervals were constructed based on the conversion points. In episode 2, people assessed 191 transformation points and constructed 50 transition points. In contrast, the present invention estimates 763 transform points for episode 1 and 744 transform points for episodic 2. In video stream analysis, it is not reasonable to expect that the human evaluator's score and the computer-supplied score will match exactly. Therefore, if the estimated conversion point belongs to the human-converted conversion point interval, it is regarded as an accurate estimation. In order to present the performance in a more traditional manner, the present invention also discloses accuracy and recall results. The recall was defined as the ratio of computer estimated transform points to the human estimated transform points included in the set of computer estimated transform points. Accuracy was defined as the ratio of the estimated change point to the accurately estimated conversion point. In terms of accuracy, the present invention achieved 0.076 in episode 1 and 0.101 in episode 2, respectively. Recall performance is 0.3431 and 0.3926 respectively. In terms of improving accuracy and recall

The likelihood, calculated as, has much room for improvement. For example, it is possible to estimate the conversion point by focusing on the likelihood of likelihood. Although much lower than the human evaluation results, the present invention achieved significantly significant results considering the fact that only SIFT transform properties were used.

In FIG. 7, the human evaluation result of the estimation result of the proposed method was compared between 35 minutes 34 seconds and 37 minutes 34 seconds of episode 1. As shown in FIG. 7, it can be seen that the interval between the estimated conversion points is short.

8 is a graph showing the goodness-of-fit curves of the individuals with the highest suitability among the population constituting the section. The fitness curve of FIG. 8 shows a stepped shape because the chromosomes that make up the population are replaced only when there is an individual with higher fitness in the subsequent generation that is created. In the case of the section constituting Figure 8, a total of 60 generations of evolution occurred during the duration of the section. Since the present invention aims at a methodology for analyzing streams in real time, there is no limit to the number of evolutionary generations. Therefore, the present invention needs to find an optimal way to meet the diversity and convergence in video stream analysis for a limited time.

9 is an example of an image reproduced in an evolved population. This image is generated from 20 particles. Only 20 particles could produce readable images. Since each population has enough information to recreate the dominant image of the interval, the proposed method can be interpreted as a new way of summarizing a given stream.

10 is a reference diagram illustrating an estimated image order based on an estimated transition probability matrix.

The series of pictures shown in FIG. 10 are reproduced based on the transition probability matrix (matrix) estimated in Algorithm 2. In FIG. 10, the upper part is the image order of the original section, and the lower part is the image order of the estimated section.

10 was generated in the following manner. First, when a seed image is presented, the section corresponding to the next section of the seed image is estimated based on the matrix. After regenerating the image in the segment for the estimated segment, the regenerated image is set as another seed image, and the next segment of the segment is estimated based on the matrix. The present invention attempted to estimate four sections and reconstructed the representative image corresponding to the section to confirm the section estimation result. The second of the four sections of FIG. 10 is not an accurate section. But the second image is also partly accurate because the dominant character (the bald male) is present in both the original and the estimated intervals. If the second image is regarded as partially accurate, then the estimation method clearly estimates the correct interval order.

In other words, it can be seen that by constructing a transition probability matrix, the present invention can represent a video stream with a smaller and simpler data structure, and can represent a partial interval order based on the constructed transition probability matrix.

11 is a flowchart illustrating an embodiment of a video stream analysis method according to the present invention. In the following description, it is assumed that a predetermined video stream analysis system performs the video stream analysis method according to the present invention. Here, the video stream analysis system may be provided with a predetermined component for performing the video stream analysis method according to the present invention. For example, an evolutionary particle filter, matrix transformation means, or the like may be provided. However, since the configuration of the video stream analysis system has a configuration corresponding to each other in the following description of the video stream analysis method according to the present invention, a more detailed description of the video stream analysis system will be omitted. However, those skilled in the art will be able to easily understand the video stream analysis system from the following description of the video stream analysis method.

Referring to FIG. 11, the video stream analysis system may generate a particle set indicating a dominant feature with respect to the video stream data (step S1110), and divide the video stream data into segment sets (step S1120).

The video stream analysis system may perform evolutionary particle filtering to generate an ordered feature on the set of segments (step S1130), and estimate the order-dependent structure of the set of particles on which the evolutionary particle filtering has been performed and convert the transition probability matrix into a transition probability matrix. (Step S1140).

In an embodiment of steps S1110 through S1120, the video stream analysis system may extract a feature from the original image and generate a visual word set using the extracted feature. The video stream analysis system may generate a particle set having position information by associating randomly selected non-monthly words, and generate a segment set to correspond to the generated particle set.

Here, the video stream analysis system may extract the characteristic using the SHIF.

A particle set may represent individual particles included in the set as a pair of nodes and edges, where the node may be selected from a visual word set.

In one embodiment for step S1130, the video stream analysis system may select particles using the goodness of fit as the critical distribution q.

Here, the goodness of fit may be calculated in consideration of the distance between the nodes constituting one particle, the ability to express an image of one particle, and the penalty, as described above, and the evolutionary particle filtering samples the particles using a distribution of goodness values and specifies The number of particles constituting any one particle set at a viewpoint may be fixed.

In addition, evolutionary particle filtering may construct new populations using the fitness of the genetic operator output and the parental generation population to prevent population degeneration due to resampling.

In an embodiment of step S1140, the video stream analysis system compares and searches different sets of particles, and when the comparison results show that the difference between the two sets is greater than or equal to a certain reference value, the video stream analysis system may estimate a new predominant image. The video stream analysis system can add new dominant images to the transition probability data structure if they exist.

In one embodiment, the video stream analysis system may recreate an image representing a segment using a set of particles.

V. Conclusion

The present invention discloses a method that can segment a video stream and estimate its dependencies. For stream segmentation, the present invention modifies the particle filtering method to estimate the interval based on a set of particles operating in a cooperative manner. The evolution shows that the dominant visual word is represented in the particle. By re-expressing the interval based on the particle set, it is possible to overcome the slight distortion caused by camera work or lighting and represent the interval. It can be seen that the segmented intervals can be used to learn the temporal dependency in the video stream. After converting the interval order obtained by grouping similar intervals into states, the transition probability matrix was used to predict the next interval for a given queue using the transition probability matrix. Because the order of the results and the particle set are linked, the proposed method can be interpreted as a new way to compress a given video stream.

The performance of the proposed method was confirmed by comparison with human evaluation results. Compared with the results of human evaluation, the proposed method has the following characteristics. The proposed method estimated the transform point corresponding to four times the evaluation result of human evaluator and the estimation accuracy is between 0.07 and 0.1. Although the difficulty of a given problem is considered, it is a very meaningful result.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

Claims (10)

In the stream data analysis method for a video stream,
(a) generating a particle set indicative of a dominant feature with respect to the video stream data, and dividing the video stream data into segment sets;
(b) performing evolutionary particle filtering to generate ordered features for the set of segments; And
(c) estimating an order-dependent structure for the set of particles on which the evolutionary particle filtering has been performed and converting them into transition probability metrics.
The method of claim 1, wherein step (a)
(a-1) extracting a feature from the original image and generating a visual word set using the extracted feature;
(a-2) generating a particle set having location information by associating randomly selected non-monthly words; And
(a-3) generating the segment set to correspond to the particle set.
The method of claim 2, wherein step (a-1)
The video stream analysis method, characterized in that for extracting the characteristic using the SHIF.
The method of claim 1, wherein the particle set is
The individual particles included in the set may be represented by a pair of nodes and edges, and the node may be selected from a visual word set.
The method of claim 4, wherein step (b)
And selecting particles using the goodness-of-fit as the significant distribution q.
The method of claim 5, wherein the goodness of fit
The video stream analysis method is calculated by considering the distance between nodes constituting one particle, the ability to represent one image, and the penalty.
The method of claim 6, wherein the evolutionary particle filtering
The particle is sampled using the distribution of the goodness-of-fit values, and the number of particles constituting any one particle set at a specific time point is fixed.
8. The method of claim 7, wherein the evolutionary particle filtering is
A method of analyzing a video stream comprising constructing a new population using a goodness of fit of the genetic operator output and the parent generation population to prevent population degeneration due to resampling.
2. The method of claim 1, wherein step (c)
(c-1) comparing and searching different particle sets;
(c-2) estimating a new dominant image when the difference between the two sets is greater than or equal to a certain reference value as a result of the comparison search; And
(c-3) if a new dominant image exists, adding it to the transition probability data structure.
The method of claim 1, wherein the video stream analysis method is
(d) regenerating an image representing the segment using the particle set.

Images