KR20030090947A - Progressive real-time news video indexing method and system - Google Patents

Abstract

PURPOSE: A method and system for indexing news video in real time are provided to simultaneously carry out shot segmentation and shot clustering through processing of a compressed domain. CONSTITUTION: A new shot is detected using a shot segmentation engine(502), and section information and characteristic histogram of the new shot are detected. It is judged whether or not the new shot is an anchor shot through qualified matching(503). In the case that the detected new shot is not judged to be the anchor shot, it is judged whether or not the new shot is the anchor shot through suspicious matching(504). When it is judged that the new shot is not the anchor shot, anchor shot seed condition checking for registering the new shot as a suspicious anchor shot is carried out if the length of the new shot is longer than the length of the average shot of shots inputted during a clustering procedure by a predetermined level(505).

Description

Progressive real-time news video indexing method and system

The present invention relates to a video indexing method and system, and more particularly, by detecting anchor shots in a news video in real time, thereby dividing the shots of the news video into anchor shots and episode shots, and breaking the news video into scene units. The present invention relates to a real-time progressive news video indexing method and system for analyzing and generating the index in real time.

With the development of mass media and the generalization of digital broadcasting today, the amount of media that people encounter every day has become very large. As the multimedia content is enormous, there is a demand for an automated system that selects data desired by the user, and researches on ways to solve the problem have been actively conducted. The most basic research in this area is analyzing video content, and most of the nonlinear video browsing functions are based on shot segmentation and shot clustering techniques.

Here, a shot refers to a sequence of video frames obtained from one camera without interruption, which is the most basic unit for analyzing or constructing video. Based on the results of many studies, it is reported that the shot segmentation process can be fully automated, and that it is possible to design an engine with a high accuracy of 95% or more, depending on the method.

A typical news video indexing system detects shots that are editorial units of the video in the news video, modeling the characteristics of the anchor scene among each shot, or using anchor scene templates to detect anchor scenes and articles based on each anchor scene. Detect units. If necessary, an advertisement detection algorithm is used to detect and index the advertising portion separately, and the headline portion and the summary news section are detected and indexed using characteristics of the headline portion or the summary news section.

Recently, a video stream reproducing / recording system capable of recording and reproducing at the same time, for example, a personal video recorder (PVR) has been introduced. Such a PVR is basically equipped with a memory such as a hard disk, and the memory is used to store video content or to temporarily store some video data for a time shift. In addition, using the PVR set-top box, the user can use instant replay or forward skip while browsing content, and can freely use high-speed fast view and low-speed playback. . In recent years, the PVR set-top box simultaneously indexes the content being recorded, enabling various non-linear video browsing such as segment skipping using index information, video skimming, and storyboard interface using key frames. The research to make it happen is being actively conducted.

On the other hand, since news video generally has a typical structure, research has been conducted from an early stage of video indexing. However, previous studies on video indexing have focused on methods for improving the accuracy of the indexing engine for video archives. Therefore, there is no consideration of the performance speed of the algorithm for analyzing news video scene by scene, the features used, the efficient use of memory, and the constraints of online processing, so that a real-time video stream playback / recording system such as a real PVR is used. There is a problem that can not be used.

The proposed algorithms are two-pass algorithms in which the processing of the compression domain is impossible or clustering is performed after the shot segmentation for the video content is completed. That is, since the existing research has not considered the real-time video stream playback / recording system, the accuracy may be high but it is not applicable to the real-time system.

An object of the present invention is to provide a real-time progressive news video indexing method and system capable of simultaneously performing a shot segmentation process and a shot clustering process through processing on a compressed domain.

1 illustrates the structure of a typical news video.

2 is a diagram illustrating a general global color histogram and a local color histogram for news video clustering used in the present invention.

3 is a diagram illustrating a global color histogram and a modified local color histogram used in the present invention.

4 is a conceptual diagram illustrating a memory structure in which anchor shots are stored in a real-time progressive news video indexing method according to the present invention;

5 is a flowchart illustrating a process in which news video indexing is performed in real time by a real-time progressive news video indexing method according to the present invention.

6 is a view showing a state conversion process of a shot by a news clustering algorithm according to the present invention.

7 is a block diagram schematically showing the configuration of a real-time progressive news video indexing system according to the present invention.

Real-time progressive news video indexing method according to the present invention to achieve the above object,

A new shot detection step of detecting a new shot using a shot segmentation engine and detecting section information and a characteristic histogram of the detected new shot;

With respect to the detected new shot, the dissimilarity between shots with representative shots of the Qualified Anchor Shot Group registered by the previous clustering process is measured, and the dissimilarity between the measured shots is referred to. A Qualified Matching step of determining whether the new shot is an anchor shot;

If it is determined that the detected new shot is not an anchor shot, the shot with the shots registered as candidate anchor shots in a previous clustering process is determined. A Suspicious Matching step of measuring liver dissimilarity measurement and determining whether the new shot is an anchor shot with reference to the measured dissimilarity between shots; And

When the detected new shot is not determined to be an anchor shot as a result of the determination in the suspicion matching step, if the length of the new shot is greater than a predetermined level greater than the average shot length of the shots input during the clustering process, An anchor shot seed condition checking step of registering a new shot as a candidate anchor shot; Its features are to include.

Here, in the new shot detection step according to the present invention, the shot segmentation engine is characterized by dividing the multimedia stream into individual shots as editing units and extracting section information thereof.

Further, in the new shot detection step according to the present invention, the shot characteristic histogram output by the shot segmentation engine is characterized in that it is a global color histogram in which all pixels of the representative frame image for representing the shot are reflected. .

Further, in the new shot detection step according to the present invention, the shot characteristic histogram output by the shot segmentation engine is a local color histogram reflecting only pixels belonging to a specific region of an image of a representative frame to represent the shot. There is a characteristic.

Further, in the Qualified Matching step according to the present invention, the Qualified Anchor Shot Group is a group of shots whose characteristic histogram difference is equal to or less than a predetermined threshold by a previous clustering process, and shots belonging to the Anchor Shot Group are selected. This feature is regarded as an anchor shot.

In addition, in the qualification matching step according to the present invention, the representative shot of the qualified anchor shot group is characterized in that the shot detected last in time among the shots of the group.

Further, in the qualification matching step according to the present invention, the disparity measurement between shots uses a difference between characteristic histograms of two shots, and the difference between characteristic histograms of two shots is a difference between global color histograms of two shots. And / or a difference between a local color histogram of two shots.

Further, in the qualification matching step according to the present invention, when the similarity between the detected new shot and the anchor shot group is measured and there is an anchor shot group whose difference is smaller than the set threshold, the new shot is selected. The candidate anchor shots registered in the anchor shot group and registered after the registration time of the shot representing the anchor shot group in time among the shots registered in the candidate anchor shot group are deleted from the candidate anchor shot group. It has that feature.

Further, in the suspicion matching step according to the present invention, the candidate anchor shot is characterized in that the shot is registered as the candidate anchor shot by the previous anchor shot seed condition checking step.

Further, in the suspicion matching step according to the present invention, the disparity measurement between shots uses a difference between characteristic histograms of two shots, and the difference between characteristic histograms of two shots is determined between the global color histograms of two shots. It is characterized by the difference and / or difference between the local color histogram of the two shots.

Further, in the suspicion matching step according to the present invention, if there is a candidate anchor shot whose difference is smaller than the set threshold as a result of measuring the dissimilarity between the detected new shot and the candidate anchor shots, the corresponding candidate anchor is present. Registering a shot and the new shot as a new anchor shot group, and among the shots registered in the candidate anchor shot group, candidate anchor shots registered after the corresponding candidate anchor shot in time and the corresponding candidate anchor shot are the candidate anchors. It is characteristic to delete from the shot group.

In addition, the anchor shot detection method according to the present invention in order to achieve the above object,

In calculating dissimilarity between the input shot and the candidate anchor shot or the qualified anchor shot registered during the clustering operation, the difference between the global color histogram and the local color histogram is used simultaneously, and the difference between the global color histograms is used. Is smaller than the set threshold τ _g and the difference between local color histograms is smaller than the set threshold τ _l , the feature is that the input shot is registered as a qualified anchor shot.

Here, the global color histogram according to the present invention is characterized in that it is a color histogram in which all pixels of the image are reflected.

In addition, the local color histogram according to the present invention is characterized in that it is a color histogram in which only pixels belonging to a specific area of the image are reflected.

Further, the local color histogram according to the present invention is divided into a first modified local color histogram and a second modified local color histogram, each of which is divided into two equal parts by horizontally and vertically, respectively, and when the center of the image is the origin, respectively. When defining the histogram corresponding to the first, second, third, and fourth quadrants of L1, L2, L3, and L4, the first modified local color histogram is a histogram normalized using L2, L3, and L4. The bimodal local color histogram is characterized in that it is a histogram normalized using L1, L3, L4.

In addition, the difference between the local color histogram according to the present invention is characterized in that a small value is selected from the difference between the first modified local color histogram and the difference between the second modified local color histogram.

Further, when the difference between the global color histogram between the input shot and the registered candidate anchor shot is smaller than the set threshold τ _g and the difference between the local color histograms is smaller than the set threshold τ _l according to the present invention. The feature is that the candidate anchor shot is also registered as an anchor shot.

In addition, in order to achieve the above object, a real-time progressive news video indexing system according to the present invention,

A feature extractor for extracting frame features of the compressed domain from the video stream;

A shot index determiner which detects a shot which is a physical editing unit from frame features provided by the feature extractor, and detects section information and a characteristic histogram of the shot;

A qualified anchor shot memory for storing shots verified as anchor shots by performing a clustering process;

A suspicion anchor shot memory for storing shots detected as candidate anchor shots by performing a clustering process;

A quality matching unit for measuring dissimilarity between input shots and representative shots of anchor shot groups stored in the quality anchor shot memory;

A suspicion matching unit for measuring dissimilarity between input shots and candidate anchor shots stored in the suspicion anchor shot memory;

An anchor shot seed condition checking unit for checking whether an input shot satisfies the anchor shot seed condition by comparing an input shot with an average shot length of shots input in a clustering process; And

A clustering controller which controls the first component and determines whether an input shot is an anchor shot with reference to a result performed by the qualified matching unit, a suspicion matching unit and an anchor shot seed condition checking unit; Its features are to include.

The feature extractor according to the present invention extracts a global color histogram and a local color histogram for an I frame when the input video stream is MPEG content in extracting a frame feature. As for the feature, the macroblock type histogram is extracted.

In addition, the clustering controller according to the present invention, if there is an anchor shot group whose difference is smaller than the set threshold value according to the dissimilarity measurement result between the two shots performed in the qualified matching unit, the input shot is selected; Register to a corresponding anchor shot group stored in the qualified anchor shot memory, and register after a registration time point of a shot representing the matched anchor shot group in time among the shots stored in the suspicion anchor shot memory. The selected candidate anchor shots are characterized in that they are deleted from the suspicion anchor shot memory.

In addition, the clustering controller according to the present invention, if there is a candidate anchor shot whose difference is smaller than the set threshold value according to the dissimilarity measurement result between the two shots performed in the suspicious matching unit, the matched candidate anchor Registers a shot and the input shot as a new anchor shot group in the qualified anchor shot memory, and among candidates stored in the suspicion anchor shot memory, a candidate anchor registered after the matched candidate anchor shot in time The shots and the matched candidate anchor shots are characterized in that they are deleted from the suspicion anchor shot memory.

In addition, the clustering controller according to the present invention, according to the result of comparing the length between the shots performed in the anchor shot seed condition check unit, the length of the input shot is constant than the length of the average shot of the shots input in the clustering process If it is larger than the level, the input shot is registered in the suspicion anchor shot memory.

In addition, the clustering controller according to the present invention may limit the maximum sizes of the suspicion anchor shot memory and the qualified anchor shot memory so as to periodically perform the candidate anchor shot inputted first in the suspicion anchor shot memory in time. Or the anchor shot extracted first in time in the qualified anchor shot memory periodically.

According to the present invention, it is possible to detect the scene structure of the news video only by processing the compressed domain, so that the execution speed is high.

In addition, according to the present invention, it can be directly applied to a system using a limited memory, and the shot segmentation operation and the shot clustering operation for the news video can be performed simultaneously.

In addition, according to the present invention, there is an advantage that it is possible to provide the input of various non-linear video browsing by extracting the structural information about the news video.

As described above, the real-time news video indexing method and system according to the present invention are not only a video archive but also a video content summarization and navigation method using keyframes in a video stream playback / recording system such as a PVR, and automatic video skimming. And real-time video indexing may be performed to be used for providing highlights. Therefore, the real-time news video indexing method and system according to the present invention satisfy the following conditions.

1. Shot information extraction and anchor shot detection for news video.

2. The anchor shot detection process uses the results of shot segmentation.

3. Fast execution time (using processed data from the compression domain).

4. Use limited memory capacity.

5. Shot segmentation and shot clustering (anchor shot detection) processes are performed simultaneously in real time.

6. It adopts efficient anchor shot registration / deletion method to improve accuracy and detect anchor shot in real time.

In particular, 3, 4, 5, and 6 among the above features are features of the present invention, and are conditions that must be satisfied in order to apply the shot clustering technique to a video stream playback / recording system such as a PVR.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 illustrates the structure of a typical news video.

As shown in FIG. 1, a news video is defined as a series of articles, and a summary or advertisement may be inserted in the middle. In addition, one article in a news video consists of a link between an anchor shot and an episode scene. Here, 'anchor shot' refers to a shot in which an anchor appears in a news video to describe an article and provides a brief summary. An 'episode scene' is formed by connecting several shots to explain the contents of an article. The scene refers to an individual shot constituting the episode scene, and the episode scene is composed mainly by a reporter's work or editing.

In the present invention, 'news video indexing' is defined as automatic shot detection and anchor shot detection based thereon. Summary News section detection, advertisement detection, etc. will be excluded from the scope of the present invention. Since many algorithms related to automatic shot detection have been introduced to date, the present invention proposes an anchor shot detection algorithm that can be executed simultaneously with the shot segmentation process using the characteristic value information of the shot detected in the compressed domain. Here, anchor shot detection is a problem of determining whether the shot is anchor shot or not with respect to the detected shot.

The characteristics of representative anchor shots used in the conventional anchor shot detection algorithm are as follows.

#. [AP1]: An anchor appears in the anchor shot.

#. [AP2]: Anchor shots are long enough to summarize an article.

#. [AP3]: From a visual point of view, similar anchor shots appear several times in a news program.

#. [AP4]: Many anchors do not appear in a news program and the number is limited.

Existing news video clustering algorithms based on these characteristics may use face region detection algorithms to use [AP1] in detecting anchor shots. However, as of the state of the art, face area detection algorithms require a very high level of computational complexity compared to methods such as color histogram comparison.

Accordingly, the present invention does not apply a face detection algorithm as a method for anchor shot detection so that the present invention can be applied to a real-time video stream playback / recording system such as a PVR. In addition, unlike the conventional algorithm, since the face detection algorithm is not used in the present invention, the following [AP5] has been added to increase the accuracy of the present invention. In fact, there are also very short anchor shots among the anchor shots. ] Was modified to [AP2 '].

#. [AP5]: If two adjacent anchor shots are similar in visual point of view and the time difference between the two shots is not large, there is no two anchor shots and very different anchor shots in visual point of view between the two anchor shots. Do not.

#. [AP2 ']: When grouping similar anchor shots from a visual point of view, the length of the first anchor shot of the group is longer than the average shot length of the program.

Then, a method of measuring dissimilarity between two shots in the present invention using the characteristics of the anchor shot set as described above will be described.

The anchor shot detection algorithm of the present invention detects an anchor shot candidate or a previously detected anchor shot candidate in addition to the task of investigating whether the shot input by the shot segmentation process is an anchor shot using [AP2 ']. Investigate whether the anchor shot is similar to the visual characteristics.

Therefore, in order to detect anchor shots in the present invention, it is necessary to measure dissimilarity between two shots. A feature for measuring dissimilarity between two shots is the color histogram, which is input to the anchor shot detection algorithm along with the shot detected by the shot segmentation algorithm.

Although various methods can be used, the present invention uses shot histograms for measuring similarity and dissimilarity between two shots. In the present invention, the shot histogram is defined as "a color histogram of a representative image sampled on a predetermined basis in one shot". Thus, the color histogram for the image represents the distribution of colors in the image, and the shot histogram represents the distribution of colors in the shot. The reason we chose the shot histogram as the basis for measuring the similarity and dissimilarity between two shots is that the color histogram for the image is already extracted during the shot segmentation process, so that the shot histogram for each shot can be efficiently obtained. This is because the computation consists of only the addition / subtraction operations, so the execution speed is very fast. Equation 1 is for explaining a process of obtaining a difference between two color histograms H ₁ and H ₂ . In this case, the number of color levels of the color histogram is n.

Although a general shot segmentation engine uses a global color histogram, in the present invention, in addition to the global color histogram, a local color histogram is used together to increase the accuracy of the anchor shot detection algorithm. do.

In addition, the color histogram used in the present invention uses pixel values extracted from a thumbnail image, such as a DC image, not pixel values extracted from an image of an original size, for fast performance. In the global color histogram, all pixels existing in the image are reflected in the color histogram. As shown in FIG. 2, only pixels corresponding to a specific area in the image are reflected in the color histogram. 2 is a diagram illustrating a global color histogram and a local color histogram for news video clustering used in the present invention.

In the present invention, the image is divided into two equal parts horizontally and vertically, and the four color regions are managed, and the color histograms are extracted for each of the four regions to construct local color histograms (L1, L2, L3, and L4). That is, after dividing the image into 2 equal parts horizontally and 2 equal parts vertically, the first, second, third, and fourth quadrants are set based on the center point, and are defined as areas L1, L2, L3, and L4, respectively. .

The reason for setting four local color histograms as shown in FIG. 2 is that the shape of the anchor shot can be classified into three types as shown in FIG. 3 according to the position of the anchor. 3 is a view for explaining the global color histogram and the modified local color histogram used in the present invention.

In most news, anchors appear in the center, left or right of the screen, and when located on the left or right side, the news icon with pictures, text, video, graphics, etc. is used to summarize the news story. In addition, in this part, even if it is an anchor shot in which the same anchor appears, the color characteristic appears very differently.

At this time, the histogram difference using the global color histogram (G) is used to detect the anchor shot at the center of the anchor, and the first modified local using L2, L3, and L4 is used to detect the anchor shot at the left of the anchor. The histogram difference between the color histogram L1 'is used, and the histogram difference between the second modified local color histogram L2' using L1, L3, and L4 is used to detect the anchor shot on the right side of the anchor. Equation 2 is for explaining the first modified local color histogram and the second modified local color histogram.

Thus, dissimilarity between two shots is obtained as a function of the global color histogram difference and the local color histogram difference. Equation 3 shows the process of calculating the difference between the global color histogram and the local color histogram. Here, GDiff (S _i , S _j ) represents the global color histogram difference between the i th shot (S _i ) and the j th shot (S _j ), and LDiff (S _i , S _j ) represents the i th shot (S _i ) And the j-th shot (S _j ) mean the local color histogram difference. And, G _i means the global color histogram of the i-th shot (S _i ), L1 '[i] means the first modified local color histogram of the i-th shot (S _i ), L2' [i] It refers to a second modified local color histogram of the i-th shot (S _i), respectively.

In this case, as shown in Equation 3, the difference in the local color histogram is to select a smaller value from the difference between the first modified local color histogram and the difference between the second modified local color histogram.

4 is a diagram illustrating a memory structure for the news clustering algorithm of the present invention. In the algorithm of the present invention, a 'Qualified Anchor Shot Memory' for storing information about shots verified as anchor shots and a 'Suspicious anchor shot' in which shots satisfying an anchor shot candidate condition are temporarily stored. Memory (Suspicious AS Memory). Here, the qualified anchor shot memory stores a list of anchor shot groups, and each anchor shot group is a set of anchor shots having similar visual characteristics. At this time, the shots in the anchor shot group are arranged in the detected order. In addition, candidate anchor shots are arranged in a chronological order in the suspicion anchor shot memory, and the number of anchor shots to be temporarily stored in order to use a limited memory may be limited.

In the present invention, when a new shot is input to the news clustering engine, three steps of [QM: Qualified Matching], [SM: Supicious Matching], and [ASSCC: Anchor Shot SeedCondition Check] are sequentially performed. Hereinafter, the process performed in each step will be described in detail.

1. [QM]: Qualified Matching step between the shot stored in the Qualified Anchor Shot memory and the input shot.

Based on the above-described [AP3], if a shot similar to a shot already determined as an anchor shot appears in terms of visual characteristics, the shot is regarded as an anchor shot. For this operation, in this step, the dissimilarity between the representative shots of the anchor shot group existing in the input shot S _cur and the qualified AS shot memory is measured. In this case, the representative shot of the anchor shot group may determine the anchor shot most recently detected in the anchor shot group as the representative shot.

In addition, the measurement of dissimilarity between shots in this step may proceed from the representative shot of the last detected anchor shot group to the representative shot of the first anchor shot group detected in the reverse order for quick performance.

At this time, the anchor shots present in the qualified anchor shot memory are shots registered through a verification operation automatically by the execution of the clustering algorithm, and thus the analogy between the representative shot of the anchor shot group and the newly input shot S _cur . After the slope measurement, if the difference satisfies the quality matching condition, the newly input shot S _cur is processed to match the corresponding group Q _m that satisfies the quality matching condition. [Equation 4] is for explaining the qualification matching conditions.

Here, Qualified_matching (S _cur , Q _k ) is determined based on the _{dissimilarity} between the newly input shot (S _cur ) and the k-th anchor shot group (Q _k ) existing in the qualified anchor shot memory. Accordingly returns True or False. In addition, S _{k, r} represents a representative shot of the _kth anchor shot group Q _k , τ _qg is a threshold for a global color histogram for the quality matching process, and τ _ql is a quality matching Threshold for the local color histogram for the process. If the result of Qualified_matching (S _cur , Q _k ) is true, the matched group Q _m becomes the k-th anchor shot group Q _k .

At this time, if the kth anchor shot group Q _k is matched, "AP5", which is generally not very different from two adjacent anchor shots from a visual point of view, is applied, and in the order entered into the clustering engine. In view, the shots added to the suspicion anchor shot memory after the last shot of the matched anchor shot group (Q _k = Q _m ) present in the qualified anchor shot memory are deleted from the suspicion anchor shot memory. The newly input shot S _cur is added to the end of the anchor shot group Q _k = Q _m and the representative shot of the anchor shot group is changed.

2. [SM]: Suspicious Matching step between the shot stored in the suspicion anchor shot memory and the input shot (Suspicious Matching)

The work performed in this step is similar to the algorithm applied in the [QM] step. Based on the above-described [AP3], if a shot similar to the candidate anchor shot appears in terms of visual characteristics, both the candidate anchor shot and the corresponding shot are regarded as anchor shots.

For this task, this step measures the dissimilarity between the input shot S _cur and the shots present in the suspicion anchor shot memory. The inter-shot dissimilarity measurement in this step may start from the last detected candidate anchor shot to the first detected candidate anchor shot in reverse order for fast performance. At this time, after measuring the dissimilarity between the candidate anchor shot existing in the suspicion anchor shot memory and the newly input shot S _cur , and if the corresponding difference satisfies the _suspicion matching condition, the newly input shot S _cur is suspend. The candidate anchor shot (S _{m, suspicious} ) satisfying the Fisher matching condition is said to match. Equation 5 below is for explaining a suspicion matching condition.

Here, Suspicious_matching (S _cur , S _{k, suspicious} ) is determined based on the _{dissimilarity} between the newly entered shot (S _cur ) and the kth candidate anchor shot (S _{k, suspicious} ) present in the _suspicion anchor shot memory. Returns true or false depending on the result of the judgment.

In addition, S _{k, suspicious} represents the kth candidate anchor shot of the _suspicion anchor shot memory, τ _sg is the threshold for the global color histogram for the _suspicion matching process, and τ _sl represents the _suspicion matching process. Is the threshold for the local color histogram. If the result of the above Suspicious_matching (S _cur , S _{k, suspicious} ) is true, the matched shot becomes the kth candidate anchor shot (S _{k, suspicious} ) of the _Suspicus anchor shot memory.

In this case, a new anchor shot group is registered in the qualified anchor shot memory, and S _cur , S _{k, and suspicious} are added to the newly registered anchor shot group. The order entered into the clustering algorithm in time is S _{k, suspicious} before S _cur , so insert S _{k, suspicious} first and S _cur later. In addition, "In general, if two adjacent anchor shots are similar in visual point of view and the time difference between the two shots is not large, there are no two anchor shots and very different anchor shots in visual point of view between the two anchor shots." By applying [AP5], the shots of the order inserted into the _suspicion anchor shot memory after S _{k, suspicious} in the order input to the clustering engine are deleted from the _suspicion anchor shot memory.

3. [ASSCC]: Anchor Shot Seed Condition Check

Based on the above-described [AP2 '], it is checked whether the newly input shot can be an anchor shot. In other words, when performing a similar anchor shots grouped together in a visual perspective in most news programs, the length of the first appearance anchor shots from that group, so the average shot length longer than the characteristics of the program of the incoming shot (S _cur) If the length is greater than a predetermined range than the average length (Avg_Shot_Length) of shots input up to the present point in time, the shot is registered in the suspicion anchor shot memory. Here, the average length of input shots is a value that continuously changes during the indexing process. It is preferable that the ratio of the length of the currently input shot to the average length of the input shots is used, since some news may have short lengths of average anchor shots, and some may not, depending on the type of news.

Next, we will discuss how to use the memory efficiently by using a limited amount of memory. In order to apply the algorithm of the present invention to a real-time system, it is necessary to be able to determine the maximum required amount of memory. The memory used in the present invention is characterized in that it can be limited without degrading performance.

First, a characteristic histogram (global / local color histogram) for the clustering algorithm is stored in the suspicion anchor shot memory in addition to the section information of the shot, and the memory space occupied by the histogram is larger than the section information.

By the above-mentioned [AP3], similar anchor shots appear several times in one news program from the visual point of view, and by [AP5] two adjacent anchor shots are similar in visual point of view, the time between two shots If the difference is not large, there are no two anchor shots and a very different anchor shot from a visual point of view. In addition, if the length of the shot (S _cur ) input in the [ASSCC] step is greater than a predetermined range than the average length (Avg_Shot_Length) of the shots that have been input up to the present time point, the process of registering the shot in the Suspicion anchor shot memory Therefore, the number of shots registered in the suspicion anchor shot memory at a specific time point is not large.

That is, according to the threshold of the ratio of the length of the currently input shot to the average length of the input shots used in the [ASSCC] step, a method of limiting the maximum number that can be registered in the suspicion anchor shot memory is used. It is possible to limit the number of shots to be stored in the Fisher anchor shot memory.

In addition, anchor shots have a characteristic of being periodically repeated within a certain time period, so if the temporal offset between the starting point of the current shot and the last point of the anchor shot present in the suspicion anchor shot memory is greater than or equal to a certain threshold, the shot is suspended. By using the policy of removing from the anchor anchor shot memory, the algorithm can be smoothly executed above the limited memory. In addition, by designing the suspicion anchor shot memory in the form of a queue of first in first out (FIFO), which stores only a certain number of candidate anchor shots, the size of the suspicion anchor shot memory is limited. You can also

Meanwhile, the qualified anchor shot memory is a place for storing shots verified as anchor shots. As described above, since the information about the shot has a large memory space occupied by the characteristic histogram (global / local color histogram) for the clustering algorithm in addition to the interval information of the shot, one characteristic histogram is maintained for each anchor shot group. In addition, memory requirements can be reduced by measuring dissimilarity with newly entered shots in [QM] matches. In addition, since many anchors do not appear in one news program and the number thereof is limited by [AP4], the number of anchor shot groups that can be registered in the qualified anchor shot memory is fixed to a certain number (for example, 5 to 6). Limiting may limit the memory requirements without degrading the performance of the algorithm.

Then, with reference to FIG. 5, the bar described above will be described collectively. 5 is a flowchart illustrating a process in which news video indexing is performed in real time by the real-time progressive news video indexing method according to the present invention. As shown in FIG. 5, when a new shot is input to the clustering engine, three steps of [QM], [SM], and [ASSCC] are sequentially performed, where the length of the shot corresponds to Is reflected in the average length.

First, the shot clustering algorithm determines whether a new shot, waiting to be input (S _cur) by a shot segmentation and a new shot (S _cur) exists (step 501, 502). At this time, if a new shot (S _cur ) exists, the process branches to step 503 to perform the [QM] process, otherwise terminates the shot clustering process.

In performing the [QM] process in step 503, if a new shot S _cur is input to the [QM] process, a representative of the anchor shot group existing in the newly input shot S _cur and the qualified anchor shot memory is performed. Obtain the image difference based on the histogram difference from the characteristic histogram (global histogram and local color histogram). At this time, if the difference satisfies the quality matching condition, it is said to be a match, and the process branches to step 506 to register the quality in the qualified anchor shot memory. As a result of the determination in step 503, if the qualification matching condition is not satisfied, the process [SM] of step 504 is attempted.

In performing the [SM] process in operation 504, when a new shot S _cur is input to the [SM] process, the histogram of the newly input shot S _cur and the present in the suspicion anchor shot memory exist. Obtain image differences based on histogram differences from characteristic histograms of candidate anchor shots. At this time, if the car satisfies the suspicion matching condition, the process branches to step 506 to perform a registration operation in the qualified anchor shot memory. If it is determined in step 504 that the suspicion matching condition is not satisfied, the process branches to step 505 to perform an anchor shot seed condition check (ASSCC) process.

In performing the [ASSCC] process in step 505, if a new shot (S _cur ) is input to the [ASSCC] process, the anchor is based on the length of the input shot (S _cur ) and the average length of the shots so far input. Check whether the shot seed condition is satisfied. At this time, if the input shot S _cur satisfies the anchor shot seed condition, the process branches to step 508 to register the shot in the suspicion anchor shot memory. As a result of the determination in step 505, if the anchor shot seed condition is not satisfied, the shot S _cur does not become an anchor shot, and branches to step 501 to wait for input of the next shot.

On the other hand, in step 506, the operation to register the shot (S _cur ) input to the qualified anchor shot memory. Here, the step 506 is performed only when the quality matching or suspicion matching is generated by the [QM] process or the [SM] process. When a new shot S _cur is input to the qualified anchor shot registration operation (step 506), the execution details according to the registration operation differ depending on whether it is performed by the [QM] process or the [SM] process. You lose.

In this case, when the qualified anchor shot registration is performed by the [QM] process, the last shot added in time order in the matched qualified anchor shot group Q _m is S _m . Here, a newly input shot S _cur is added to the matched quality anchor shot group Q _m , and the representative shot of the matched quality anchor shot group is set as the newly input shot S _cur , Step 507 is performed to update the suspicion anchor shot memory.

When the qualified anchor shot registration is performed by the [SM] process, if the anchor shot matched in the susceptor anchor shot memory is S _m , a new anchor shot group is registered in the qualified anchor shot memory. S _m is moved to the newly registered anchor shot group in the Qualified Anchor Shot memory, and the newly entered shot S _cur is also added to the newly registered Qualified Anchor Shot group. Thereafter, the representative shot of the group is set to the newly input shot (S _cur ), and step 507 is performed to update the _suspicion anchor shot memory.

In operation 507, the newly input shot S _cur is registered in the qualified anchor shot memory, and then the suspicion anchor shot memory is updated. After the anchor shot S _m matched by the [QM] process or the [SM] process, information about shots input into the susceptor anchor shot memory is deleted from the suspicion anchor shot memory. When step 507 is performed, the process branches to step 501 to wait for the next shot input. In step 508, the newly input shot selected as the anchor shot seed by the [ASSCC] process is registered in the suspicion anchor shot memory, and then branches to the step 501 to wait for the next shot input.

Next, let's look at how the status of the shot changes.

According to the news video clustering algorithm of the present invention, one shot may be directly registered in a qualified anchor shot memory, may be registered in a suspicion anchor shot memory, and may be registered in a qualified anchor shot memory or a suspicion anchor shot memory. It can be rejected without being registered. Here, the anchor shots stored in the qualified anchor shot memory are shots that have passed the anchor shot verification operation, and thus are determined to be anchor shots unless a special subsequent process is performed, and the status thereof is invariant.

However, candidate anchor shots registered in the suspicion anchor shot memory may be deleted because anchor shot verification is denied by the anchor shot registration of other shots or by a policy for storing only a limited number of shots in the suspicion anchor shot memory. It may be. In addition, candidate anchor shots registered in the suspicion anchor shot memory may be moved to the qualified anchor shot memory by a visual characteristic value match process with the newly detected shot. 6 shows a state transition process of the shot by the news clustering algorithm according to the present invention.

Next, a real-time progressive news video indexing system for performing such an algorithm will be described with reference to FIG. 7. 7 is a block diagram schematically showing the configuration of a real-time progressive news video indexing system according to the present invention. 7 illustrates the core modules of the news video indexing engine and the relationships between them. Here, the solid line represents the flow of data, and the dotted line represents the flow of control signals, respectively.

Referring to FIG. 7, the real-time progressive news video indexing system 700 according to the present invention includes a clustering controller 701 for controlling the entire clustering algorithm, and a suspicion anchor for temporarily storing candidate anchor shots. Similarity between shot memory 702, a shot between a qualified anchor shot memory 703 for storing a verified anchor shot, and an shot shot and a representative shot group of anchor shot group representative shots in the qualified anchor shot memory 703 The QM unit 705 in charge of the measurement, the SM part 706 in charge of measuring the dissimilarity between the shot shot and the candidate anchor shot in the suspicion anchor shot memory 702, and the input shot An ASSCC unit 707 which checks anchor shot seed conditions, a feature extractor 709 for extracting frame characteristics of a compressed domain from a video stream, and physical editing from input frame features A shot index determiner 708 for detecting a shot which is a unit and sending section information and a characteristic histogram of the shot to the shot index input unit, and temporarily storing information about the newly extracted shot by the shot index determiner 708, A shot index input unit 704 for notifying the clustering controller 701 that a new shot has been input, and an index manager 710 for structurally storing an index in a storage device such as a disk by the clustering controller 701.

Hereinafter, the operation between each component will be described in detail.

In FIG. 7, the feature extractor 709 receives information about a media to be indexed by the video indexing system 700 from the clustering controller 701, extracts features required for video indexing, and prepares corresponding data. Data is sent to the shot index determiner 708. In this case, the feature extractor 709 extracts different features for each frame to perform a fast overall algorithm. If the input stream is MPEG content, the feature extractor 709 extracts a global color histogram and a local color histogram for an I frame. The macroblock type histogram is extracted from the P / B frame.

In addition, the shot index determiner 708 performs a shot segmentation operation on the basis of the features received from the feature extractor 709, and each time a new shot is detected, the shot index information and the shot information of the shot are determined. The characteristic histogram information is transmitted to the shot index input unit 704. The shot index determiner 708 also notifies the shot index input unit 704 when the shot segmentation for the media is finished.

Accordingly, when a new shot is input, the shot index input unit 704 registers a new shot, notifies the clustering controller 701, and notifies the clustering controller 701 that the shot is terminated even when the shot segmentation ends.

The QM unit 705 performs a qualified matching operation between a shot newly input to the shot index input unit 704 and a representative shot of an anchor shot group stored in the qualified anchor shot memory 703. The clustering control unit 701 is informed of the result.

In addition, the SM unit 706 performs a Suspicious Matching operation between a newly input shot in the shot index input unit 704 and candidate anchor shots stored in the suspicion anchor shot memory 702, and then, as a result, Informs the clustering control unit 701.

In addition, the ASSCC unit 707 determines whether the input shot satisfies the anchor shot Seed condition based on the length information of the shot newly input to the shot index input unit 704, and then determines the determined result. The clustering control unit 701 is informed.

In addition, the index manager 710 permanently stores section information on each shot / scene generated by the clustering controller 701, information on whether each shot is an anchor shot, and relationship information between the shot and the scene. It is responsible for storing the device as a disk.

In addition, the clustering controller 701 controls the shot index determiner 708, the shot index input unit 704, and the like to control feature extraction and shot index extraction, and the suspicious anchor shot memory 702 and quality. The memory, such as the fast anchor shot memory 703, is controlled to perform a real-time progressive news video clustering algorithm. The clustering control unit & 01 controls the QM unit 705, the SM unit 706, and the ASSCC unit 707 to respectively match the newly detected shots input to the shot index input unit 704. QM], Suspicious Matching [SM], and anchor shot seed condition check [ASSCC]. Accordingly, the clustering control unit 701 registers the shot in the suspicion anchor shot memory 702, or registers the shot in the qualified anchor shot memory 703 whenever a new shot is input. It is determined that the input shot is not an anchor shot.

In addition, the clustering controller 701 may use the qualified anchor shot memory 703 and the suspicion anchor shot memory 704 according to the types of return values of the QM unit 705 and the SM unit 706 for the new shot. ). In addition, the clustering control unit 701 is a new shot is input to the shot index input unit 704 to determine the type of shot using the [QM], [SM], [ASSCC] process, the next [ASSCC] operation Update the average length of shots entered.

In addition, the clustering control unit 701 may be configured to limit the maximum size of the suspicion anchor shot memory 702 and the Qualified anchor shot memory 703 in some cases. It periodically plays a role of periodically removing the candidate anchor shot inputted first in time, or periodically removing the anchor shot extracted first in time from the qualified anchor shot memory 703.

In addition, the clustering controller 701 transmits shot index information and anchor shot information of the qualified anchor shot memory 703 to the index manager 710, and the index manager 710 provides information on the shot / scene section. And control the index manager 710 to store whether the detected shot is an anchor shot, a relationship between a shot / scene, and the like on a storage medium.

As described above, when the news video indexing system of the present invention as shown in FIG. 7 is used, structural information on the news video can be extracted using an advanced algorithm. That is, the shot segmentation and the anchor shot detection work can be performed at the same time. The algorithm of the present invention is applicable to a video archive and also to a real-time video stream playback / recording system such as a PVR. When the clustering algorithm of the present invention is applied to a video stream playback / recording system such as a PVR, recording, playback, and index extraction can be simultaneously performed. In addition, the index information generated by the algorithm and the system of the present invention can be used as a main input means of non-linear video browsing (NLVB) such as a keyframe interface, structural information-based skipping, and structural information-based skimming.

As described above, according to the real-time progressive news video indexing method and system according to the present invention, the news video by extracting the relationship between the shot / scene information on the news video, whether the anchor shot of the detected shot, the scene / shot at the same time, An advantage of providing an index that can be used as an input of an application program such as keyframe summary, video skimming, and structured information-based skip.

In addition, according to the real-time progressive news video indexing method and system according to the present invention, the shot segmentation operation and the shot clustering operation is performed at the same time, can be applied to a real-time video stream playback / recording system such as a PVR, such as a broadcasting station or VOD server This has the advantage of greatly improving the video indexing performance of the video archive.

Claims (21)

A new shot detection step of detecting a new shot using a shot segmentation engine and detecting section information and a characteristic histogram of the detected new shot; With respect to the detected new shot, the dissimilarity between shots with representative shots of the Qualified Anchor Shot Group registered by the previous clustering process is measured, and the dissimilarity between the measured shots is referred to. A Qualified Matching step of determining whether the new shot is an anchor shot; If it is determined that the detected new shot is not an anchor shot, the shot with the shots registered as candidate anchor shots in a previous clustering process is determined. A Suspicious Matching step of measuring liver dissimilarity measurement and determining whether the new shot is an anchor shot with reference to the measured dissimilarity between shots; And When the detected new shot is not determined to be an anchor shot as a result of the determination in the suspicion matching step, if the length of the new shot is greater than a predetermined level greater than the average shot length of the shots input during the clustering process, An anchor shot seed condition checking step of registering a new shot as a candidate anchor shot; Real-time progressive news video indexing method comprising a. The method of claim 1, wherein in the new shot detection step, The shot segmentation engine divides the multimedia stream into individual shots as editing units and extracts section information thereof. The method of claim 1, wherein in the new shot detection step, The shot characteristic histogram output by the shot segmentation engine is a global color histogram in which all pixels of the representative frame image for representing the shot are reflected or only pixels belonging to a specific area of the image of the representative frame for representing the shot are reflected. Real-time progressive news video indexing method characterized in that the local color histogram. The method of claim 1, wherein in the qualified matching step, The Qualified anchor shot group is a group of shots whose difference between characteristic histograms of shots is less than or equal to a predetermined threshold by a previous clustering process, and the shots belonging to the anchor shot group are regarded as anchor shots. Way. The method of claim 1, wherein in the qualified matching step, The representative shot of the Qualified Anchor Shot group is a real-time progressive news video indexing method, characterized in that the shot detected last in time among the shots of the group. The method of claim 1, wherein in the qualified matching step, The disparity measurement between shots uses a difference between characteristic histograms of two shots, and the difference between characteristic histograms of two shots is a difference between a global color histogram of two shots and / or a difference between a local color histogram of two shots. Real-time progressive news video indexing method. The method of claim 1, wherein in the qualified matching step, When the similarity between the detected new shot and the anchor shot group is measured and there is an anchor shot group whose difference is smaller than the set threshold value, the new shot is registered in the corresponding anchor shot group and the candidate anchor shot group is registered. Among the registered shots, candidate anchor shots registered after a registration time of a shot representing the corresponding anchor shot group in time are deleted from the candidate anchor shot group. The method of claim 1, wherein in the suspicion matching step, Wherein the candidate anchor shot is a shot registered as a candidate anchor shot by a previous anchor shot seed condition checking step. The method of claim 1, wherein in the suspicion matching step, The disparity measurement between shots uses a difference between characteristic histograms of two shots, and the difference between characteristic histograms of two shots is a difference between a global color histogram of two shots and / or a difference between a local color histogram of two shots. Real-time progressive news video indexing method. The method of claim 1, wherein in the suspicion matching step, If there is a candidate anchor shot whose difference is less than a predetermined threshold as a result of measuring dissimilarity between the detected new shot and the candidate anchor shot, the candidate anchor shot and the new shot are registered as a new anchor shot group. Among the shots registered in the candidate anchor shot group, candidate anchor shots registered after the corresponding candidate anchor shot and the corresponding candidate anchor shot in time are deleted from the candidate anchor shot group. . In performing the dissimilarity calculation between the input shot and the candidate anchor shot or Qualified Anchor Shot registered during the clustering, the difference between the global color histogram and the local color histogram is calculated. At the same time, if the difference between global color histograms is smaller than the set threshold τ _g and the difference between local color histograms is smaller than the set threshold τ _l , registering the input shot as a qualified anchor shot. Anchor shot detection method characterized in that. The method of claim 11, The global color histogram is a color histogram in which all pixels of the image are reflected, and the local color histogram is a color histogram in which only pixels belonging to a specific area of the image are reflected. The method of claim 11, The local color histogram is divided into a first deformed local color histogram and a second deformed local color histogram. The local color histogram is divided into two equal parts horizontally and vertically, and divided into four parts. When the histogram corresponding to the third and fourth quadrants is defined as L1, L2, L3, and L4, the first modified local color histogram is a histogram normalized using L2, L3, and L4, and the second modified local color histogram is Anchor shot detection method characterized in that the histogram normalized using L1, L3, L4. The method of claim 13, And wherein the difference between the local color histograms is selected from a difference between the first modified local color histogram and a difference between the second modified local color histogram and a small value. The method of claim 11, If the difference between the input shot and the registered candidate anchor shot, the global color histogram is smaller than the set threshold τ _g , and the difference between the local color histograms is smaller than the set threshold τ _l , the candidate anchor shot. Anchor shot detection method characterized in that the registration as an anchor shot. A feature extractor for extracting frame features of the compressed domain from the video stream; A shot index determiner which detects a shot which is a physical editing unit from frame features provided by the feature extractor, and detects section information and a characteristic histogram of the shot; A Qualified Anchor Shot Memory for storing shots verified as anchor shots by performing a clustering process; A Suspicious Anchor Shot Memory for storing shots detected as candidate anchor shots by performing a clustering process; A Qualified Matching Part for measuring dissimilarity between an input shot and a representative shot of an anchor shot group stored in the qualified anchor shot memory; A Suspicious Matching Part for measuring dissimilarity between input shots and candidate anchor shots stored in the suspicion anchor shot memory; Anchor Shot Seed Condition Checking Part which checks whether the input shot satisfies the anchor shot seed condition by comparing the input shot with the average shot length of shots input in the clustering process. ; And A clustering controller which controls the first component and determines whether an input shot is an anchor shot with reference to a result performed by the qualified matching unit, a suspicion matching unit and an anchor shot seed condition checking unit; Real-time progressive news video indexing system comprising a. The method of claim 16, wherein the feature extractor extracts the frame feature. In the case where the input video stream is MPEG content, a global color histogram and a local color histogram are extracted for an I frame and a macroblock type histogram is extracted for a P / B frame. . The method of claim 16, The clustering controller determines that the input shot is the qualified anchor shot memory when there is an anchor shot group whose difference is smaller than a predetermined threshold value according to a dissimilarity measurement result between two shots performed by the qualified matching unit. Candidate anchor shots registered after the registration time of the shot representing the matched anchor shot group in time among the shots stored in the suspicion anchor shot memory, are registered in the anchor shot group stored in the And deleting the suspicion anchor shot memory from the real-time progressive news video indexing system. The method of claim 16, The clustering controller, if there is a candidate anchor shot whose difference is smaller than a predetermined threshold value according to the dissimilarity measurement result between the two shots performed by the suspicion matching unit, the matched candidate anchor shot and the input shot Registers a new anchor shot group in the qualified anchor shot memory, and among candidates stored in the suspicion anchor shot memory, candidate anchor shots registered after the matched candidate anchor shot in time and the matched candidate And anchor shots are deleted from said suspicion anchor shot memory. The method of claim 16, The clustering controller, if the length of the input shot is greater than a length of the average shots of shots input in the clustering process, according to a result of comparing the lengths of shots performed by the anchor shot seed condition checking unit, inputs the input. Real-time progressive news video indexing system, characterized in that it registers the shot to the suspicion anchor shot memory. The method of claim 16, The clustering controller may be configured to limit the maximum sizes of the suspicion anchor shot memory and the qualified anchor shot memory so as to periodically delete the candidate anchor shot that is inputted first in the suspicion anchor shot memory in time, or to the quality A real-time progressive news video indexing system, characterized by periodically deleting the first and second anchor shots extracted from the fixed anchor shot memory.