KR100435235B1 - Method of adaptive detecting a shot and indexing a key frame - Google Patents

Abstract

The present invention relates to a scene change detection method using a different physical feature extraction method for each category of a program and a representative frame indexing method accordingly. The scene change detection method according to the present invention includes detecting a category for a current program, determining a suitable physical feature extraction method for each category, and determining whether a scene has been changed for each frame using the determined physical feature extraction method. Determining whether or not.

According to the present invention, it is possible to detect the scene change more accurately by applying a different physical feature extraction method for each category to detect the scene change.

Description

Adaptive Scene Transition Detection Method and Representative Frame Indexing Method Using Them {METHOD OF ADAPTIVE DETECTING A SHOT AND INDEXING A KEY FRAME}

The present invention relates to a scene change detection method that applies different physical feature extraction methods according to a category of a broadcast program, and a method of extracting and indexing a representative frame of each scene using the scene change detection method.

In general, broadcast programs may be divided into various categories, that is, genres, and have different characteristics for each category. For example, music programs have a lot of light effects, while news programs have little light effect. As such, by ignoring the features unique to each category and uniformly detecting the scene change or indexing the representative frame, the scene change may be incorrectly detected or the indexing may not be performed properly. As a result, users may be inconvenient not to obtain accurate information due to wrong scene transition detection or indexing.

An object of the present invention is to provide a method for detecting a scene change by applying different physical feature extraction methods according to the current program category.

Another object of the present invention is to provide a method for extracting and indexing a representative frame of each scene when a scene change is detected.

1 is a flow chart generally showing an adaptive scene change detection method and a representative frame indexing method using the same according to the present invention.

2 is a flow chart illustrating an adaptive scene change detection method according to the present invention.

Figure 3 is a graph illustrating the histogram of the physical feature extraction method according to the present invention.

4 is a flow chart illustrating a representative frame indexing method in accordance with the present invention.

In order to achieve the above object, the adaptive scene change detection method of the present invention comprises: (a) pre-databaseting a physical feature extraction method applied for each category of a program, a threshold value according to each physical feature extraction method, ( b) extracting information about a category from an EPG for a broadcast program, (c) extracting at least one physical feature extraction method applied to the category from the database, (d) using the physical feature extraction methods Calculating a difference value between the previous frame and the current frame, (e) obtaining a total difference value that is a sum of at least one or more difference values calculated using at least one physical feature extraction method, and (f) from the database. The threshold value determined for each category is extracted, and the total difference value is equal to the Comparing the threshold value, and (g) determining that the scene has been switched if the total difference value is larger than the threshold value.

In this case, the database may further include information on weights for respective physical feature extraction methods according to categories, and in step (d), obtaining a difference value according to each physical feature extraction method, and calculating the corresponding weights from the database. The method may include extracting, multiplying the difference value according to each physical feature extraction method by a corresponding weight to obtain a weighted difference value, and adding a weighted difference value according to each physical feature extraction method to obtain a total difference value.

On the other hand, the physical feature extraction method is more preferably provided with at least one or more of local variance, histogram, entropy coding, camera movement.

More preferably, the physical feature extraction method comprises a camera movement, the database comprises a motion vector threshold for the camera movement, the adaptive scene change detection method, The method may further include obtaining a motion vector for the motion vector, comparing the motion vector with the motion vector threshold value, and excluding from the scene change detection when the motion vector exceeds the motion vector threshold value.

Another feature according to the present invention relates to a representative frame indexing method, comprising: (a) a database of feature tables for each category including an optimal physical feature extraction method for each category for a program and feature information for each physical feature extraction method; (B) detecting a representative frame between a previous detection frame and a current frame section when a scene change is detected for the broadcast program, and (c) a method of extracting a physical feature from the category-specific feature table. Extracting feature information, (d) detecting a feature from a representative frame according to the feature information, (e) comparing a representative feature of a pre-stored query image with a feature of the representative frame, and (f) the When the query image and the representative frame have similarities, the representative according to the representative feature of the query image. Characterized in that it comprises the step of indexing the frame.

In this case, the representative frame indexing method may further include a step of databaseting the summary image table in advance, and the summary image table preferably includes a query image having a representative feature for each category.

According to the present invention, it is possible to detect the scene change more accurately by applying a different physical feature extraction method according to the category of the program to detect the scene change.

Below. A preferred embodiment of the adaptive scene change detection method according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a scene change detection and representative frame indexing method according to the present invention as a whole. Hereinafter, a scene change detection and a representative frame indexing method according to the present invention will be described in detail with reference to FIG. 1, and the scene change detection method and the representative frame indexing method will be described below in more detail with reference to FIGS. 2 to 4. . The scene change detection method and the representative frame indexing method according to the present invention can be executed at either a broadcast station or a digital television receiver.

First, in step 100, the category for the current program is extracted from the program guide table (EPG). In general, images broadcast on digital television are classified into categories such as sports, news, and music videos. The information on the category of each image may be obtained from the category classification module of the EPG table.

Next, in step 110, a method of extracting at least one physical feature to be used for detecting a scene change for the extracted category is determined. Using the determined physical feature extraction methods, it is detected whether the current frame has been transitioned (step 120). In this case, when a scene change is detected, the scene is indexed using the previously stored query image and predetermined feature information (step 130). Then, the user inputs or stores a comment or an annotation on the indexed scene (step 140).

Meanwhile, if no scene change is detected in step 120 or step 140 is completed, it is determined whether there is a next frame (step 150), and if there is no next frame, the process ends.

If not, it moves to the next frame (step 160), and returns to step 120 to detect again whether to switch scenes for the next frame.

Accordingly, according to the present invention, it is possible to extract information about a category from a category classification module for each program, and detect scene transitions by applying the most suitable physical feature extraction method for each category by using the same.

Hereinafter, the adaptive scene change detection method according to the present invention will be described in more detail with reference to FIG. 2.

First, in step 200, a category for the current program is extracted from the EPG table. Next, physical feature extraction methods for detecting a scene change for the category are extracted from the category detection method table (step 210).

The detection method table for each category stores and stores the most suitable physical feature extraction methods according to each category. Methods of extracting physical features of each image include local variance, edge histogram, entropy coding, and camera motion detection. For example, images such as news images, talks, and discussions are images with small movements and small lighting effects, and it is preferable to use pixel histograms and regional dispersion methods to detect scene transitions of such images. On the other hand, the image of the music program with a lot of light effect and a lot of motion uses the edge histogram and entropy method.

Table 1 below shows, by way of example, the most suitable physical feature extraction methods for each category of images.

category Physical feature extraction method Regional dispersion Histogram Entropy coding Camera movement Soccer O O X O news O O X X music X O O O Documentation O X X X animation O X X X Show X O O X

Local dispersion, histogram, and entropy coding methods of the above-described physical feature extraction methods detect the similarity between each frame by using a difference, and the camera movement is Up, Down, Reft, Right, Zoom-In, Zoom- The feature of the image is extracted using the motion vector values to the zoom-out.

From Table 1, it can be seen that, in the case of a soccer game program, the most suitable physical feature extraction method for detecting scene transitions is to use regional variance, histogram, and camera movement. In addition, in the case of an animation program, it can be seen that it is most preferable to detect a scene change by using regional variance. Hereinafter, each of the above-described physical feature extraction methods will be described in detail.

(1) Local Variance

First, the local variance of the brightness signal component is obtained, and then the number of macro blocks exceeding the local variance threshold is obtained. Next, the ratio D _L of the total number of blocks and the number of macro blocks described above is extracted and the physical feature of the image is extracted using this value.

(2) Histogram

The histogram of the previous frame and the current frame is obtained, and the similarity is extracted through the intersection of the histogram. In this case, when the similarity is lower than a specific threshold, it is determined as a scene change.

3 is a graph for explaining the histogram. Hereinafter, a method of obtaining the intersection of the histogram will be described with reference to FIG. 3.

The X coordinate of FIG. 3 represents the brightness level, and the Y coordinate represents the number of pixels. The solid and dotted lines on the graph represent histograms for the previous frame and the current frame, and the parts shown in the bar graph represent histogram intersections between the two frames. Therefore, when the color Ci step is set to 16 steps, the small value portion becomes the histogram intersection in each step.

In this case, the intersection of histograms for the images of two frames is obtained according to Equation 1.

Here, F _m and F _{m + 1} are two consecutive frames, and H _Ci (F _m ) and H _Ci (F _{m + 1} ) are the number of pixels in the image for the color Ci.

The similarity of the entire image is determined by Equation 2 using the obtained value. Equation 2 shows the similarity between the two frames in the range of 0.0-1.0.

And the difference value _Dh is extracted by Formula (3).

(3) entropy

The entropy difference D _e of two consecutive frames is obtained using Equation 4.

(4) Camera Motion

It extracts the motion vectors of six camera movements, that is, Left, Right, Down, Up, Zoom In, and Zoom Out, and compares these values with a predetermined motion vector threshold to exceed the threshold. It is considered to be excluded from the scene change detection.

The four physical feature extraction methods described above are merely exemplary for the purpose of better understanding of the present invention, and it is natural that the physical feature extraction method may be further added.

Meanwhile, after extracting the physical feature extraction method as described above with respect to the corresponding category through step 210 of FIG. 2, the weights of the extracted physical feature extraction methods and the threshold value of the corresponding category are again obtained from the detection method table for each category. Is extracted (step 220). Next, the total difference value for the current frame is calculated using the physical feature extraction methods (step 230). Hereinafter, the process of calculating the total difference value will be described in more detail.

First, each weight for local variance, histogram and entropy coding for each category is first extracted. Table 2 exemplarily shows weights and thresholds set for each category.

category Weights and thresholds Regional dispersion Histogram Entropy coding Threshold Soccer w _l w _h X τ (soccer) news w _l w _h X τ (news) music X w _h w _e τ (music) documentary w _l X X τ (documentary) animation w _l X X τ (animation) Show X w _h w _e τ (show)

Next, D _L , D _h , and D _e of the corresponding item are obtained for each category by using the above-described equations. Table 3 calculates difference values of corresponding items for each category by using the above-described equations.

category Physical feature extraction method Regional dispersion Histogram Entropy coding Camera movement Soccer D _L D _h X O news D _L D _h X X music X D _h D _e O documentary D _L X X X animation D _L X X X Show X D _h D _e X

Next, the difference value is obtained by using specific physical feature extraction methods for each corresponding category of the image, the weighted difference values for each item are obtained by multiplying the difference value by the corresponding weight, and the total difference value is combined by combining the weighted difference values. _Find (D _total ). The case where a category is music is demonstrated to an example. The total difference D _total of the music category can be obtained as shown in Equation 5.

Here, W _h and W _e are respective weights for histogram and entropy coding when the category is music, as shown in Table 2, and D _h and D _e are extracted histogram and entropy coding values.

As described above, when the total difference value D _total for the current frame is obtained through step 230 of FIG. 2, the total difference value D _total is compared with the threshold value τ of the corresponding category (step 240). At this time, if the total difference does not exceed the threshold, it is determined that the scene has not been switched, and the process ends.

On the other hand, if the total difference value exceeds the threshold in step 240, it is determined whether camera movement is applied among the physical feature extraction methods for the category. If the category is not applied to the camera movement, it is determined that the scene has been changed.

However, in the case of the category to which the camera motion is applied, the motion vectors of the six cameras described above are calculated and it is determined whether any one of them exceeds the predetermined motion vector threshold (step 250). At this time, if it is exceeded, it is determined by the movement of the camera, not the scene change, and the scene change is not detected (step 270). If the motion vector of the camera does not exceed the motion vector threshold in step 250, it is determined that the scene has been switched (step 260).

Hereinafter, a representative frame indexing method according to the present invention using the aforementioned scene change detection method will be described in detail with reference to FIG. 4.

The representative frame indexing method according to the present invention is to detect similarity between representative frames representing each scene when a scene change is detected, and generate indexing information so that users can gather and view only similar scenes.

First, when a scene change is detected according to the above-described scene change detection method, a section between a previously detected frame and a current frame is extracted, and then a representative frame is extracted within the section (step 400). Next, the physical feature information according to the physical feature extraction method for the category is extracted from the feature table for each category (step 410). Next, the feature is detected from the representative frame using the extracted feature information (step 420).

Next, a query image having a representative feature for the category is extracted from the summary image table (step 430). Next, it is determined whether the representative feature of the extracted query image and the feature of the representative frame are similar to each other (step 440).

If they are not similar to each other, a description or annotation of a user is inputted and stored in the representative frame, and the representative frame is indexed according to the feature information.

If similar to each other, it is determined whether a user's description or annotation on the query image already exists in the summary image table (step 450). If the user description exists, the user description is extracted from the summary video table (step 460), stored, and the representative frame is indexed. If the user description does not exist, the user receives the description from the user, stores the feature of the representative frame, and indexes the feature of the representative frame (step 470).

According to the representative frame indexing method according to the present invention, feature information received from a service provider may be stored in a feature table for each category, and each scene may be classified and indexed using a query to automatically reproduce similar scenes to a user. Will be. In this case, the feature information may include values for a histogram distribution or a total histogram distribution at a specific position in one frame.

In addition, by the representative frame indexing method according to the present invention, users can view only a desired scene or a desired frame from a program of a specific category. For example, users can gather and view only the bench frames in a soccer game where an anchor appears in a news program.

According to another exemplary embodiment of the representative frame indexing method according to the present invention, database information is used as indexing information about a combination of physical feature values extracted according to each detection method and information about a length of a previous detection and a next detection interval during scene change detection. This can be used to index representative frames more accurately.

For example, in a soccer game program, when a scene change using local variance is detected, a homogeneous texture and histogram lead to an audience, bench, ground, etc. When appearing repeatedly, these scenes index into goal scenes or shooting scenes. Or in a news program, if representative frames with very similar histograms are distributed throughout, they can be indexed as anchor frames.

Meanwhile, when one of the representative screen of the Anker frame, the location information in the content, and the histogram feature is received from the service provider, only the anchor frame may be searched for the extracted representative frame to be classified and indexed.

The representative frame indexing method according to the present invention allows a user to add an annotation to a detected representative frame, so that when the user later watches the same category of content, the current frame is used as a query for the annotated frame. The similarity with the representative frame detected from is detected, and if the similarity is high, the annotation can be automatically indexed.

As described above, in the detailed description of the present invention, specific embodiments have been described, but various modifications can be made without departing from the scope of the present invention. For example, the physical feature extraction method may be added as well as illustrated in the present specification, and the equation for calculating the difference value according to each physical feature extraction method is presented, but this may vary. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims below, but also by those equivalent to the claims.

According to the present invention, a scene change can be detected by applying an appropriate physical feature according to each category by using information according to a category obtained from an EPG table.

In addition, when a scene change is detected in the broadcast program, the representative frame can be indexed by an optimal method suitable for each category.

Claims (8)

A scene change detection method for a broadcast program broadcasted on digital television, (a) pre-databaseting the physical feature extraction methods applied differently according to the category of the broadcast program and each threshold value according to the physical feature extraction method; (b) extracting information on a category for a broadcast program from an EPG; (c) extracting from said database at least one physical feature extraction method applied to said category; (d) calculating a difference value for a previous frame and a current frame using each of at least one physical feature extraction method, and if the difference is one or more, obtaining a total difference value that is a sum; (e) extracting a threshold determined for each category from the database and comparing the total difference with the threshold; (f) if the total difference is greater than the threshold, determining that the scene has been switched Adaptive scene change detection method comprising a. The method of claim 1, wherein the database further includes information on weights for respective physical feature extraction methods according to a category, and step (d) includes obtaining a difference value according to each physical feature extraction method. Extracting a corresponding weight from the step, multiplying the difference value according to each physical feature extraction method and the corresponding weight to obtain a weighted difference value, and adding a weighted difference value according to each physical feature extraction method to obtain a total difference value. Adaptive scene change detection method characterized in that. The method of claim 1, wherein the physical feature extraction method comprises at least one of local variance, histogram, entropy coding, and camera movement. The method of claim 1, wherein the physical feature extraction method comprises a camera movement, and the database includes a motion vector threshold value for the camera movement. The adaptive scene change detection method may include obtaining a motion vector for the camera motion, comparing the motion vector with the motion vector threshold value, and when the motion vector exceeds the motion vector threshold value, from the scene change detection. Adaptive scene change detection method further comprising the step of excluding. In the representative frame indexing method for a broadcast program, (a) pre-databaseting a category-specific feature table including an optimal physical feature extraction method for each category for the program and feature information according to each physical feature extraction method; (b) detecting a representative frame between a previous detection frame and a current frame section when a scene change is detected in the broadcast program; (c) extracting feature information according to a predetermined physical feature extraction method from the feature table for each category; (d) detecting a feature from a representative frame according to the feature information; (e) comparing the representative feature of the previously stored query image with the feature of the representative frame; And (f) indexing the representative frame according to the representative feature of the query image when the query image and the representative frame have similarities Representative frame indexing method comprising a. 6. The representative frame according to claim 5, wherein the representative lepaim indexing method further comprises a database of a summary video table in advance, and the summary video table includes a query image having a representative feature for each category. Indexing method. The method of claim 5, wherein the representative frame indexing method further comprises: inputting and storing a description of a user in the indexed representative frame. The method of claim 7, wherein the inputting and storing of the user's description comprises: determining whether the user's description already exists in the query image, and if the user's description already exists, the user's description of the query image. Extracting, and storing the extracted user description in the representative frame.