KR100441963B1 - Scene Change Detector Algorithm in Image Sequence - Google Patents

Abstract

The present invention relates to a scene change detection method for accurately and stably recognizing a scene change by applying a two-stage detection process. And a second step of classifying, and a second step of reconfirming the scene change by re-confirming whether the classified frames are scene change.

Description

Scene Change Detector Algorithm in Image Sequence

The present invention relates to a method for detecting a scene change of a digital image, and more particularly, to a method for detecting a scene change and a key frame extraction of a digital image using a two-step detection process.

Recently, various multimedia service systems have been developed, including video retrieval by video index. In general, digital video has a very large amount of data, but similar images are contiguous within a scene, so indexing the video scene by scene enables efficient video retrieval. At this time, the technique of finding the time point at which the scene is changed and extracting the key frame, which is an image representing the scene, becomes an essential element in constructing the video indexing and searching system.

The scene change detection method aims to detect the following kinds of scene change.

① Cut: The screen is changed suddenly.

② Fade: The screen is gradually darkened or brightened and switched.

③ Dissolve: two screens overlapping and switching

④ Wipe: Switching to the next screen as if the previous screen is wiped off

Cut transitions by cuts can be easily detected by simple algorithms by detecting only the differences between frames.However, transitions made by other fades, dissolves, or wipes can be detected by the movement of a person, object, or camera because the transition is a gradual process. It is confused with the gradual change in the scene by which it is difficult to detect accurately.

There are two main approaches to scene change detection.

The first is an approach that detects scene transitions by extracting only some information such as motion vectors and discrete cosine transform (DCT) coefficients without completely decoding the compressed video data. This approach has the advantage that the processing speed is relatively high because the compressed video is processed without being completely decoded, but has the following disadvantages.

Since only a portion of the video is decoded and used, the accuracy decreases due to lack of information, and the scene change detection method depends on the video compression method. Recently, since the video compression method is very diverse, the detection method must be changed according to the compression method. In addition, since the motion vector, macroblock type, etc., which are mainly used in this approach, values may vary considerably according to encoding algorithms, scene transition detection results may vary depending on the type of encoder and the encoding method even for video having the same content. have.

The second approach is to fully decode the compressed video and then detect the transition in the image domain. This method has higher accuracy of scene change detection than the former method, but has a disadvantage in that the processing speed is reduced by the time required to decode the compressed video. However, in recent years, computer performance has been rapidly improved, hardware can be used to decode video, and software optimization techniques, such as MMX 3DNow technology, can be used for decoding since the computational cost of decoding does not matter. It is more important to increase the accuracy of scene change detection than to save time.

The present invention follows the latter approach.

The scene change detection method by the latter approach currently under study is based on the method of using the difference of pixel value (Template matching), the method of using the histogram difference, the method of using the change of edges, the method using block matching And briefly described as follows.

Template matching uses a difference of pixel values to obtain a difference between two pixel values having the same spatial position between two frames adjacent to each other in time, and uses the same as a measure for detecting a scene change. Histogram comparison is a method of expressing a brightness component, a color component, and the like in a picture by using a histogram, and using a histogram difference between the frames. The method using the change of the edge detects the outline of the object in the image, and detects the scene change by using the change of the outline. If the transition does not occur, the position of the current contour and the position of the contour of the previous frame are similar, but when the transition occurs, the position of the current contour is different from the position of the contour of the previous frame. The method using block matching is used as a measure of detecting transitions by using block matching to find similar blocks between two adjacent frames. First, the image is divided into several blocks that do not overlap each other, and for each block, the most similar to the previous frame. Find the block. Then, the difference between the most similar blocks found is expressed as a value between 0 and 1, and these values are generated through the nonlinear filter to generate the difference between the frames, and then use the values to determine whether to change scenes. It is a way.

However, the conventional scene change detection method has the following problems.

Conventional scene change detection method does not cognitively understand the contents of each scene to find the point of time when the contents change, but observes the change of basic primitive features such as the color or brightness of the pixel. How to recognize. Thus, there is a disadvantage in that a gradual change in a scene due to movement of a person, an object or a camera and a change in a gradual scene such as a fade, dissolve, or wipe cannot be distinguished.

The present invention has been made to solve the above problems, the present invention also recognizes the scene change by detecting the change in the basic image characteristics, but by applying a two-step detection process any type of scene change accurately and stably Its purpose is to provide a way to recognize it.

1 illustrates image differences between adjacent frames along a time axis.

2 is a flowchart of a scene change detection method according to the present invention;

3 illustrates quantum conversion from YCbCr space to HSV space.

4 is a flow chart of a second step of FIG.

5 is a diagram illustrating a method of dividing frames stored in IS and VS into segments.

6 is a flowchart for determining whether each segment needs to be separated into independent scenes.

In order to achieve the above object, the present invention provides a method for detecting a scene change by detecting a change in a characteristic of an image frame, and determining whether or not there is a change between adjacent frames so that a transition state with a change between adjacent frames and a stay without change between adjacent frames are provided. The first step of classifying into a state of state and the presence or absence of the segmentation of the divided segments of the frames classified as the transition state among the classified frames as independent scenes are reconfirmed whether the scene transition is performed through the key frame of the corresponding segment. The second step is to determine.

The first step includes initializing a mode and a stack, decoding a current frame, storing an image in an image stack, and extracting feature vectors from an image of the current frame. Storing in the vector stack (VS), storing the difference between the feature vectors of the last two frames stored in the VS in a difference queue, and whether the difference between the feature vectors stored in the DQ satisfies a mode conversion condition. And an algorithm comprising determining, determining whether the IS and VS are full, and checking whether the frame is the last frame.

The second step is an algorithm applied when the difference between the feature vectors stored in the DQ in the first step satisfies the mode switching condition, or when the IS and VS are full or the frame is the last frame. Setting all stored frames in one segment; dividing and setting the stored frames into multiple segments in transition mode; checking whether there are segments to be processed according to each mode; and If there is a segment, the algorithm consists of determining whether each segment needs to be separated into an independent scene.

Hereinafter, the scene change detection method according to the present invention will be described in detail with reference to the drawings.

1 illustrates an image difference between adjacent frames along a time axis.

As shown in FIG. 1, scenes having a plurality of frames are listed along the time axis, and the frames of each scene have image feature vectors calculated based on image characteristics such as color and boundary strength. The change between adjacent frames calculated using the plot is shown.

Frames of the scenes may be classified into frames having no change and frames having no change between adjacent frames due to the difference of the image feature vectors. [0037] Thresholds T1 and T2 (T1 <T2) shown in the drawing may be classified. As a reference, frames having a value greater than or equal to T2 may be classified as a frame having a sudden change, and frames larger than T1 and smaller than T2 may be classified as frames having a gradual change. Also, frames having a value smaller than the threshold T1 are defined as frames without change.

The frames are largely divided into transition frames and stationary frames in the scene change detection method of the present invention. That is, if more than N frames exceeding the threshold T2 or not exceeding the threshold T2 as shown in Fig. 1, but exceeding the threshold T1 appear consecutively, N or more consecutive frames that do not exceed the threshold T1 continue from the starting point. The start point of the case is classified as a transition frame, and subsequent frames are classified as stationary frames.

That is, the first step of the present invention is to classify a frame having no change between adjacent frames and a frame having change between adjacent frames as described above.

In FIG. 1, a portion above the threshold T2 corresponds to a cut in which the scene changes abruptly, and a portion in which more than N consecutive frames exceeding T1 are faded or dissolved is not exceeded the threshold T2. , Equivalent to a gradual transition such as a wipe. In other words, the scene transition may occur suddenly between frames, but may also occur through a gradual process over several frames. If it is assumed that a new scene starts immediately after the scene transition process is completely finished, as shown in Fig. 1, the change can be regarded as a scene by binding the start point of the stationary state to the end point of the transition state. .

In the second step of the present invention, if the scene boundary is incorrectly recognized by re-checking whether the scene is changed according to the state change recognized in the first step, or the scene determined not to be divided into an independent scene is integrated with the previous scene. .

For example, if the lighting changes suddenly due to lightning or flash in the video, or if a part of the screen is damaged due to a transmission error or the like, a sudden change between the frames occurs and the scene is converted, but both boundaries are divided. Since the same scene appears in this case, the two scenes must be merged. Also, in the case of scenes fading out to white or black, the scene is divided into frames with a gradual change, but since the scene after fading out has only white or black scenes, it can be classified as an independent scene. It is not worth it, so it integrates with the previous scene. By performing this correction in the second step, it is possible to detect the scene change more accurately.

As described above, the scene change detection method of the present invention comprises a first step of determining whether there is a change between adjacent frames, and a second step of reconfirming the scene change of the classified frames to determine the scene change. do.

The configuration of the present invention will now be described in detail with reference to the drawings.

The first step is to initialize the mode and stack, to decode the current frame and store the image in the IS, to extract the feature vector from the image of the current frame and store it in the VS, and to store the last two frames stored in the VS. Storing the difference between the feature vectors in the DQ, determining whether the difference between the feature vectors stored in the DQ satisfies a mode switching condition, confirming that the IS and VS are full, and that the frame is last It consists of an algorithm comprising the step of checking whether the frame.

2 is a flowchart of the first step.

As shown in Fig. 2, in the initialization step 201, the state variable mode indicating whether the current frame is in the stationary state or the transition state is initialized to the stationary state, and the IS, VS, and DQ are initialized. IS is a stack for storing an image of each frame, and VS is a stack for storing a feature vector extracted from an image of a frame. IS and VS can each store M items. In the present invention, the M is effectively set to about 180. In addition, the DQ may store N items as a circular queue for storing changes between adjacent frames, and N value is appropriately set to about 3.

In the initialized state, the video decoder decodes one frame of video and stores it in the IS (202). Most video is compressed and stored in YCbCr format, so IS stores images in YCbCr format. Next, the feature vector is extracted from the current frame stored in the IS and stored in VS (203).

The feature vector uses an edge histogram and a color histogram. The border histogram and the color histogram are complementary image features. The border histogram mainly reflects the change in the brightness (Y) component, and the color histogram mainly indicates the change in the color (CbCr) component.

The border histogram divides the Y component image into W horizontally and vertically H non-overlapping blocks, and calculates the strength of the border component in four directions (horizontal, vertical, 45 ° and 135 ° directions) in each block. Thus, the border histogram will have W x H x 4 items. When calculating the boundary histogram, the absolute difference between adjacent pixels is accumulated in the above-described four directions, and this calculation can be performed at high speed using a single instruction multiple data (SIMD) structure such as MMX technology.

Meanwhile, the color histogram is performed on a Hue Saturation Value (HSV) space. The YCbCr color model is very effective for compressing video data, but it is a color model far from human cognitive characteristics. Therefore, the pixel value of each frame expressed in YCbCr space is mapped to HSV space and then quantized. Find the histogram.

The conversion from the YCbCr space to the HSV space is performed by the following equation.

Equation 1 V = Y, 0≤V≤255

Equation 2 S = √ {(Cr-128) ² + (Cb-128) ² }, 0≤S≤128

Equation 3 H = tan ^-1 {(Cr-128) / (Cb-128) x (180 / π)}-108, 0 ≦ H ≦ 360

Quantization is performed in the same manner as in FIG. 3. In other words, pixels with saturation less than or equal to 5 are regarded as gray scale, ignoring hue, quantizing the intensity by dividing intensity from 4 levels in 64 levels, and saturation. Is greater than 5 and less than or equal to 30, and the color is quantized by dividing the color into six steps of 60 degrees for Hue and two levels of 128 levels of intensity. Colors with saturation (Saturation) greater than 30 are quantized by dividing the intensity into six levels in 60-degree increments, ignoring the intensity. The reason why the saturation is greater than 30 and the saturation is less than 30 is used to reflect the probability distribution that the saturation value is relatively small in a general video image. This creates a histogram with a total of 22 items.

As described above, when the feature vector is extracted, it is stored in VS (203), and the difference between the frames is calculated using the difference between the feature vector extracted in the previous frame and stored in VS and the feature vector extracted in the current frame. Store the result in the circular queue DQ. The difference between the feature vectors is calculated by the following equation.

Equation 4 D = W _e D _e + W _c D _c

Where D _e and D _c are the difference values of the feature vectors using the boundary histogram and the color histogram, respectively, and W _e and W _c are constants representing the weights, respectively.

D _e and D _c are calculated by accumulating the histogram difference between the current frame and the previous frame, respectively.

[Equation 5] D _e = ∑∥EH _n [i]-EH _n-1 [i] ∥

Equation 6 D _c = ∑∥CH _n [i]-CH _n-1 [i] ∥

Where EH [i] and CH [i] represent the I-th item of the border histogram and the color histogram, respectively, and the subscripts n and n-1 represent indexes representing the current frame and the previous frame, respectively.

When the change between two adjacent frames is calculated and stored in the circular queue DQ (204), it is determined whether or not to switch the value of the state variable mode by using it (205). As already mentioned, the mode is a state variable indicating whether the current frame is in a stationary state or a transition state.

The mode switching condition is as follows.

When the current mode is the stationary mode, the mode must be switched to the transition mode when the most recent value stored in the DQ exceeds the threshold T2 or the most recent N values exceed the T1.

On the other hand, when the current mode is the transition mode, if the values of the last N items stored in the DQ are all smaller than the threshold T1, the mode must be switched to the stationary mode.

Every time the mode is switched, the second step 206 is checked. The process of the second step will be described later in detail.

After the second step 207, the IS and VS are emptied and the value of the state variable mode is switched. In this case, when transitioning from the stationary state to the transition state, all values stored in the IS and VS may be deleted and restarted. However, when the transition state is changed from the transition state to the stationary state, the last N items in the stack IS and VS are changed. Note that you do not delete it.

The reason for this is that in order to transition from the transition state to the stationary state, since the last N frames must be frames that do not change between adjacent frames, the mode is switched only after N frames have passed after the actual state transition. Because it can. Thus, work in the next stationary state must start back N frames. However, this effect can be achieved by leaving the last N items in the stack without deleting them.

If the mode change does not occur (205), the current mode is continued while continuing. Since the image and the feature vector are stored in the stack every frame, it is checked whether the stack is full (208). Both IS and VS are stacks that can hold M defined items, limiting the maximum length of scene that can be processed at one time. If a scene runs longer than this without changing the mode, the stack will be full and move on to the second stage.

Generally, even if there is little change between adjacent frames in a scene, if the camera moves very slowly or if a person or object in the screen moves little by little and accumulates for a long time, a significant change will be made. Therefore, it is necessary to confirm whether to divide the scene at regular intervals. . The size of the stack IS and VS is this time interval, and if the stack is full beyond this time interval, it is checked to see if it is better to split the scene at this point. In this case, when the second step is finished, the stack is emptied for the next scene processing (209). In this case, the stack is completely empty regardless of the mode.

When all of these processes are completed, it is determined whether the current frame is the last frame (210), if it is not the last frame, the next frame is decoded and processed (211), and if the last frame, the last scene is processed. The final scene processing method also follows the process of the second step 206, and confirms whether to process a series of frames remaining at the end of the video as one independent scene even if there is no mode change. After processing the last frame, all work ends (212).

4 is a flow chart of a second step.

The second step is an algorithm applied when the difference between the feature vectors stored in the DQ in the first step satisfies the mode switching condition, or when the IS and VS are full or the frame is the last frame. Setting the entire stored frames into one segment; dividing and setting the stored frames into multiple segments in the transition mode; checking whether there is a segment to be processed according to each mode; and If present, it consists of an algorithm comprising determining whether each segment needs to be separated into an independent scene.

As shown in FIG. 4, in the stationary state according to the state variable mode 401, all frames stored in the stack IS and VS are treated as one segment (402), and in the transition state, the stack is processed. The frames stored in the IS and the VS are divided and processed in units of segments (403).

The segment is divided as follows.

As shown in FIG. 5, the frames in the transition state as shown in FIG. 5 may be bundled together with the frames in the stationary state as one scene, but the sudden change over the threshold T2 as shown in FIG. It is desirable to separate the frames intervening into separate scenes. Therefore, frames in the transition state are considered to be divided into segments based on frames exceeding the threshold T2. That is, if there are K frames that exceed the threshold T2 among the frames in the transition state, and if there are K-1 segments, it is checked whether the segments need to be separated into independent scenes for each segment (405).

6 is a flowchart of this operation.

Determining whether each segment needs to be separated into an independent scene when the segment exists, extracting a key frame, confirming that the key frame matches a key frame already stored, does not match Otherwise, determining whether the key frame has information, storing the key frame in the key frame list if the information is present, and outputting scene change information based on the information of the stored key frame list. It is characterized by consisting of an algorithm comprising a.

As shown in Fig. 6, a key frame list is used to perform this operation. The key frame list is a memory space that stores an image of a frame representing the scene and a feature vector extracted from the image, for a scene recognized as an independent scene. First, the center frame of the current segment is selected as the key frame (601). If there is an item stored in the key frame list, it is checked whether the current segment is similar to the recently detected scene by comparing the latest L key frames with the key frame extracted from the current segment (602). There are two reasons for examining similarities with the recent L key frames:

First, in the case of sudden lighting changes or fast moving objects passing through the screen, the difference between frames temporarily increases, but the scene is divided, but the scene may be divided, and similarity with the previously detected scene is examined. In this case, it is possible to correct the division of the scene by mistake. Second, when the camera alternates between two or three characters, the same scene is repeated once every two or three scenes, and such unnecessary repeated scene division can be corrected by examining similarities with adjacent two or three scenes.

In order to determine the similarity with the recent L key frames, the method of determining the similarity of the images using the feature vectors extracted from each key frame and the correlation coefficient between each key frame image is obtained to see if it exceeds a certain threshold. Let's see how it works.

If the key frame of the current segment does not have similarity to the recently detected L key frames, it is checked whether the scene has enough information to be separated into an independent scene (603). To do this, we calculate the variance of the current key frame to see if it exceeds a certain threshold. If the current key frame distribution does not exceed a certain threshold, the screen is black or white due to a fade-out effect, or if the information is not meaningful even when divided into independent scenes, the scene is not divided.

Since the segment that passed all of these verification processes is qualified to be recognized as an independent scene, the keyframe extracted from the current segment and the specific vector are stored in the key frame list (604), and the scene transition information such as the start point of the segment is output. (605).

As described above, the scene change detection method of the present invention has the following effects.

According to the present invention, any type of scene change can be accurately detected, and the detection speed is very fast, so that the scene change recognition process is performed at a speed corresponding to about 4% of the speed of playing the video without performing the scene change recognition process. can do.

Claims (15)

In the method for detecting a scene change by detecting a change in the characteristics of the image frame, A first step of judging whether there is a change between adjacent frames and classifying it into a transition state with a change between adjacent frames and a stationary state with no change between adjacent frames; When classified as a stationary state among the classified frames, all frames are set to one segment, and when classified as a transition state, whether or not the classified frames are divided into independent scenes of the divided segments into key frames of the corresponding segments. And a second step of reconfirming the scene change and confirming the scene change. The method of claim 1 wherein the first step is Initializing the mode and stack; Decoding the current frame and storing the image in the IS; Extracting a feature vector from an image of a current frame and storing the feature vector in VS; Storing the difference between the feature vectors of the last two frames stored in the VS in the DQ; Determining whether the difference between the feature vectors stored in the DQ satisfies mode conversion; Checking whether the IS and VS are full; And an algorithm comprising determining whether the frame is the last frame. The method of claim 1, wherein the second step Setting all frames to one segment in the stationary mode; Partitioning and setting frames into a plurality of segments in the transition mode; Confirming the presence or absence of the segment for each mode; And determining whether each segment needs to be separated into an independent scene when the segment is present. 3. The method of claim 2, wherein the difference between the feature vectors stored in the DQ satisfies a mode switching condition, or when the IS and VS are full, or the frame is the last frame. Scene change detection method characterized in that. The method of claim 3, wherein the determining of whether the segments need to be separated into independent scenes when the processable segments exist includes: Extracting a key frame from the current segment; Comparing the key frame with a similarity with key frames of a recently detected preset scene; Confirming whether there is independent information necessary for scene change if the key frame does not match as a result of the comparison; Storing the key frame in a key frame list when there is information; And a scene change information is output based on the information of the stored key frame list. 5. The method of claim 4, wherein if the difference between the feature vectors stored in the DQ is satisfied to convert the mode to determine whether there is a segment to be processed in the second step, IS, VS if the segment does not exist. Scene change detection method, characterized in that for emptying and switching modes. 7. The scene change detection method according to claim 6, wherein when switching from the transition mode to the stationary mode, the predetermined number of items recently stored in the IS and VS are not deleted. The scene change detection according to claim 4, wherein when the IS and VS are full and a step of checking whether a segment to be processed in the second step exists exists, the IS and VS are emptied if the segment does not exist. Way. The method according to claim 4, wherein when the frame to be processed is the last frame and the step of checking whether the segment to be processed in the second step exists, if the segment does not exist, the algorithm of the scene change detection method of the present invention ends. Scene change detection method, characterized in that. The scene change detection method according to claim 1, wherein the difference between adjacent frames is classified according to the time axis by applying threshold values T1 and T2 (T1 < T2). 11. The method of claim 10, wherein if a difference between adjacent frames exceeds a threshold T2 or exceeds a threshold T2 but more than a predetermined number of frames continuously exceed the threshold T1, a predetermined number of frames consecutively do not exceed the threshold T1 from the start point. The transition point detection method is characterized by classifying the transition point up to the start point in the case of continuous succession, and classifying the subsequent frame as a stationary frame. 3. The scene change detection method according to claim 2, wherein the IS and the VS store a predetermined number of items. 13. The method of claim 12, wherein the predetermined number is about 180. 3. The method of claim 2, wherein the DQ stores a predetermined number of items. 15. The method of claim 14, wherein the predetermined number is about three.