KR20020017216A - Method of video scene segmentation based on color and motion features - Google Patents

Abstract

PURPOSE: A method of dividing a video scene on the basis of color and motion characteristics is provided to accurately reflect the meaning of video and reduce scene's excessive division effect. CONSTITUTION: When an MPEG video signal is inputted(S100), the boundary of a shot is detected by using the boundary between a sharp distribution and a gradual distribution of the shot(S200). Color characteristic of the shot is extracted and color similarity of the shot is measured(S300A), and the motion of the shot is extracted and motion similarity is measured(S300B) in order to extract the characteristics of the detected shot. A weighted value is adaptively applied to the similarity between shots with respect to the color and motion(S400A). A scene is divided using the adaptive weighted value and an improved overlapping links technique(S400B). Then, the scene's excessive division effect is compensated(S500).

Description

Video scene segmentation based on color and motion features {METHOD OF VIDEO SCENE SEGMENTATION BASED ON COLOR AND MOTION FEATURES}

The present invention relates to a video scene segmentation method based on color and motion features, and more particularly, to a video scene segmentation method using not only color features but also motion features during video scene segmentation.

Along with the rapid increase in video data, many researches have been done on video structuring and feature extraction algorithms. Conventional video structuring mainly detects the edge of a shot first for a video sequence and extracts a representative frame within each shot. At this time, the contents of each shot are represented by a representative frame. In addition, by using the representative frame of each shot, similar shots are grouped based on characteristics such as color histogram, and shot group extraction is performed so that the user can know the contents of the video.

However, such shot or shot group level video structuring has a disadvantage in that it does not sufficiently reflect an event or meaning in the video, and since the existing scene segmentation uses only color features, the screen progress in the video is particularly dynamic. In the interval, it is not good to over-divide the scene in which one scene is actually divided into several scenes, and also, since the similarity is measured for all shot sequences, a large amount of time is required for application.

Disclosure of Invention The present invention, which is devised to solve the above-mentioned problems, has a purpose to more accurately reflect the meaning in video and to reduce scene over-split by applying color features as well as motion features in the video scene segmentation. have.

Another object of the present invention is to reduce the computation time by using IMPROVED OVERLAPPING LINK (IOL) technique in video scene segmentation.

Still another object of the present invention is to extract a hierarchical structure of shots and scenes and use them for video browsing and searching.

The present invention for achieving the above object, hierarchically browsing the content of the video by representing the video as a scene, shot, key frame using the color and motion information of the video A video scene segmentation method based on color and motion features, the method comprising: receiving an MPEG video signal, detecting a shot boundary of the input video signal, extracting color features of the shot, and Measuring color similarity, and further, measuring motion feature extraction and motion similarity; clustering the shots by adaptive weighting; and post-processing the clustered shots.

The present invention also provides a method for extracting color features and measuring color similarity of a shot including: a sub-shot extraction process consisting of successive frames having similar color distributions over a predetermined frame in one shot; The method may further include extracting a representative frame located in the middle of time, and measuring color similarity using the representative frames by a formula.

The present invention also includes the step of measuring the motion feature extraction and motion similarity, extracting the vector component of the macroblock in the P frame in the shot, normalizing the extracted vector component, and the normalized vector component The method may further include obtaining a normalized motion size of the macroblock, calculating a mean size of the normalized motion size, and extracting a motion similarity between two shots.

The present invention also includes the step of clustering shots, further comprising searching for similar shots using IOL techniques and dividing the boundary of the scene.

The present invention is further characterized by obtaining an average motion of the similar shot search section and applying color and motion weights using this value.

In the post-processing process, the present invention also determines whether the divided scene is located at a time interval less than τ, and if the neighboring scene and the motion similarity are greater than the threshold value, the two scenes are determined. It further comprises the step of merging.

The invention further includes increasing the spacing between the reference frame and the comparison frame to divide into sub-shots, and splitting from this point into a new sub-shot if the dissimilarity between the two frames is greater than a predetermined threshold. Characterized in that.

The present invention may further include limiting the number of representative frames that can be extracted in one shot by limiting the sub-shot extraction time to a predetermined time in the sub-shot extraction process.

Figure 1 shows the overall configuration of the present invention in one embodiment.

Figure 2a shows in detail the color feature extraction and color similarity measurement process in FIG.

FIG. 2B illustrates the motion feature extraction and motion similarity measurement process in FIG. 1 in detail.

3A illustrates in detail the process of obtaining the adaptive weights in FIG. 1.

3B illustrates the shot clustering process in FIG. 1 in detail.

4 illustrates the post-processing process of FIG. 1 in detail.

Figure 5 shows the sub-shot extraction process in the present invention.

6a and 6b show a similar shot search process, 6a shows a conventional OL (OVERLAPPING LINKS) technique, 6b shows an IPR (IMPROVER OVERLAPPING LINKS) technique in the present invention.

FIG. 7 is a diagram illustrating a scene segmentation method using the IOL technique of FIG. 6B.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

Figure 1 shows the overall configuration of the present invention in one embodiment. As shown, when the MPEG video signal is input (S100), the boundary of the shot is detected based on the sharp distribution and the gradual distribution of the shot (S200). In order to extract the characteristic of the detected shot, color feature extraction and color similarity of the shot are measured (S300A), and motion feature extraction and motion similarity of the shot are extracted (S300B). The weight is similarly applied to the color and motion to the similarity between the shots obtained through the extracted color and motion features (S400A), and the scene is divided using the adaptive weighting linking (IOL) technique. Through the shot clustering process (S400B), the post-processing process (S500) that compensates for the excessive division of the scene has been shown.

Figure 2a shows in detail the process of measuring the color characteristics and color similarity in FIG. In order to measure the color similarity between shots, a method of measuring the similarity between them by extracting a representative frame of each shot is used instead of measuring the similarity for all frames in each shot. First, a sub-shot, which is a group having a similar color distribution composed of consecutive frames of a predetermined frame or more within one shot, is extracted (T1). The process of extracting sub-shots is shown in FIG. 5. As shown, when the color distribution changes a lot in one shot, frames having similar color similarity are divided into one or more sub-shots while increasing the interval of each frame.

That is, the present invention proposes a method of dividing one shot into several sub-shots in consideration of the case where the color distribution is changed in one shot. Even if a camera motion or an object's motion increases even in a single shot, the color distribution in the shot changes accordingly. In the conventional method, a method of comparing color similarity between neighboring frames is used to detect shot boundaries, but it cannot reflect that the color distribution is changed within the shot due to the above situation.

The present invention proposes a method of increasing and comparing an interval between a reference frame and a comparison frame for dividing into sub-shots. If the dissimilarity between the two frames is larger than the predetermined threshold, a new sub shot is started from this point. The advantage of this algorithm is that it can fully reflect the color distribution in the shot with a small amount of computation.

The frame located in the middle in time in each sub-shot divided by the above method is extracted as a representative frame (T2). In addition, the number of representative frames that can be detected in one shot may be limited in consideration of the minimum duration T of each sub-shot.

The color similarity of the representative frame of each shot is obtained by the following equations (1) and (2) (T3).

p = 1,2, ......., n : n is the number of sub-shots of shot i

q = 1,2, ......., m : m is the number of sub-shots of shot j

r _ip is the representative frame index of the pth sub-shot of shot i

r _iq is the representative frame index of the qth sub-shot of shot j

r _jp is the representative frame index of the pth sub-shot of shot i

r _jq is the representative frame index of the qth sub-shot of shot j

N _Bin : The number of color bins in the histogram used to measure color similarity

FIG. 2B illustrates the motion extraction and motion similarity measurement in FIG. 1 in detail. The motion feature in each shot is extracted from the P frame in which the motion feature with the I or P frame is stored in the vector form in macroblock units among the frames of the MPEG-1 video. Therefore, the motion vector of the 16 x 16 macroblock in the P frame is extracted to find the displacement in the x and y directions (R1). The displacement is divided by 16 to normalize (NORMALIZATION) to have a value between -1 and 1 (R2). Now, the magnitude of the normalized motion of each macroblock is obtained by using Equation 3 below (R3).

Of the P frames in the shotkThe normalized absolute displacement of the first forward macroblock in the x and y directions.

After the block motion size is obtained, the normalized average motion size in the shot is obtained by using Equation 4 below (R4).

N _MB means the total number of forward macroblocks in the shot.

The motion similarity between two shots of the i and j th shots is obtained using Equation 5 below using the average motion size obtained above (R5).

In order to calculate the similarity between shots, weights of the similarity between the shots obtained through the color and the motion feature are respectively weighted. As shown in FIG. 7, since similar shots are found in the search section, weights are assigned to colors and motions according to the characteristics of the search section. The description of FIG. 7 will be described later with reference to FIG. 3B.

If the average motion in the search section is large, this section is a dynamic section in which the screen is fast-moving like the action scene, so it gives more weight to motion than color, and conversely, if the average motion in the search range is small, this section is static. Because it is an in interval, you should apply a greater weight to the color than to the motion.

The process is shown in detail in FIG. 3a. FIG. 3A illustrates the adaptive weighting process in FIG. 1 in detail. As shown, in order to find the current shot and similar shots, the average motion is obtained using the following equation (6) for the shots in the search section (M1).

In this case, I _CS means an index of a shot, which is a reference for searching in a shot similarity search interval, and SR (SEARCH RANGE) means the number of shots to be searched from the I _CS .

Similarity between shots is obtained using weights W _M for motion similarity and weights W _C for color similarity. Here, the weight (W _M ) for the motion similarity uses the average motion within the shot search range (M2), and the weight W _C for the color similarity can be automatically obtained by subtracting the motion application weight from 1 (M3).

3B illustrates the shot clustering process in FIG. 1 in detail. As shown, starting with the first shot of a certain search section as the scene boundary (L1) and searching for similar shots in the section (L2), this process will be described with reference to FIGS. 6A and 6B. do.

6A illustrates an OL (OVERLAPPING LINKS) technique, which is a conventional shot detection method. Since this technique searches in the forward direction when searching for similar shots, the similarity needs to be measured for all shots in the search section.

6B illustrates an IOL (IMPROVED OVERLAPPING LINKS) technique, which is a similar shot detection method according to the present invention. Compared with the conventional OL method, since the last shot in the similar shot search section is searched backwards, for example, when the last shot of the search section is a similar shot, only one similarity comparison process is required. Can be reduced.

Using the IOL technique, if a similar shot that exceeds a reference threshold and a given threshold is first found through a reverse search process from a search section (L3), the two shots are linked (LINK) to each other (L4), and a new search from the similar shot is performed. The section is set to search for similar shots again (L5). If a similar shot is not found, the scene boundary is defined and the scene is divided (L3 ').

The above method is shown in detail in FIG. FIG. 7 is a diagram illustrating a scene segmentation method using the IOL technique of FIG. 6B. As shown in the figure, when the search starts from the last shot of the search section from the last shot of the search section and a similar shot a 'is searched for a, the a and a' shots are connected, The boundary of the scene is extended by searching for a similar shot for a 'shot in the search section. Since a ″, a similar shot of a ', is found, the a' and a ″ shots are concatenated, so a, a 'and a ″ shots are connected to each other, and the boundary of the scene is extended ②. When a similar shot for a ″ no longer appears, the previous shot is searched for by moving to the previous shots.

b, the previous shot of a ″, is connected to each other because it is similar to b ', and the video scene extends to b'. Since b 'has no similar shot in the search section and no similar shot is detected for the shot before b', the scene is split at b '. As such, if no similar shot is detected for the previous shots within the search section, a new scene starts from the next shot, and the above algorithm is applied.

Applying the above scene segmentation algorithm, many scenes with a time interval less than τ are detected. These scenes are over-segmented because the similarity of color features is low in the similar shot search section, and these scenes are difficult to deliver the correct contents to the user.

In order to compensate for such scene over-division, the post-processing process shown in FIG. 4 is performed so that the shots in one scene have similar motion characteristics to compare two motion scenes with neighboring scenes. Allow to merge

4 illustrates the post-processing process of FIG. 1 in detail. The process of FIG. 4 is performed for each obtained scene. First, if the time interval of the scene is less than τ (N1), if it is greater than or equal to τ, it is not necessary to go through the post-processing process (N1 '), and if it is less than τ, the motion characteristic of the scene is obtained ( N2). The motion characteristics of each scene are obtained using the following equation.

I _SS and I _ES indicate the index of the first and last shot of each scene.

The motion similarity between two scenes adjacent to each other as a motion characteristic of the scene obtained as shown in Equation 7 is calculated by Equation 8 and compared with a threshold value (N3). If it is smaller, it ends without merging the scenes (N3 '), and if it is larger than the threshold, the two scenes are merged with each other (N4).

By merging these short scenes, you can reduce excessive splitting of the scenes, and the merged scenes can more faithfully reflect their meaning.

The present invention has the effect of extracting representative frames proportional to the degree of change in color distribution in a shot, and faithfully reflecting various color distributions in a shot when comparing color similarity compared to conventional methods using the first and last frames as representative frames. .

The present invention also has the effect of reducing oversplitting of the scene through adaptive weighting and post-processing according to the motion processing of the search section.

The present invention also has an effect of significantly reducing the computation time using the IMPROVED OVERLAPPING LINK (IOL) technique, compared to the conventional method, and extracting hierarchical structures of shots and scenes can be used for video searching or browsing. .

Claims (15)

In the MPEG video scene segmentation method, Expressing the video in scenes, shots, and keyframes using the color and motion information of the video; A video scene segmentation method based on color and motion information comprising hierarchically browsing the entire content of a video. In the MPEG video scene segmentation method, Receiving an MPEG video signal; Detecting a shot boundary of the input video signal; Measuring color feature extraction and color similarity of the shot, and also measuring motion feature extraction and motion similarity; Clustering the shots by adaptive weighting; and A video scene segmentation method based on color and motion features comprising post-processing the clustered shots In the MPEG video scene segmentation method, A sub-shot extraction process consisting of successive frames having similar color distribution over a certain frame in one shot; Extracting a representative frame located in the temporal center of the sub-shot; And extracting color feature and color similarity of a shot, which is a search unit of a signal, in a frame including measuring color similarity of the representative frames. 4. The method of claim 3, further comprising increasing the spacing between the reference frame and the comparison frame to divide into sub-shots, and further dividing into new sub-shots from this point if the dissimilarity between the two frames is greater than a predetermined threshold. And a video scene segmentation method based on color and motion features. The color and the method of claim 3, further comprising limiting the number of representative frames that can be extracted in one shot by limiting the sub-shot extraction time to a predetermined time in the sub-shot extraction process. Video scene segmentation method based on motion characteristics. The video scene segmentation method of claim 3, wherein the color similarity of the representative frame of each shot is obtained by the following Equations 1 and 2. p = 1,2, ......., n : n is the number of sub-shots of shot i q = 1,2, ......., m : m is the number of sub-shots of shot j r _ip is the representative frame index of the pth sub-shot of shot i r _iq is the representative frame index of the qth sub-shot of shot j r _jp is the representative frame index of the pth sub-shot of shot i r _jq is the representative frame index of the qth sub-shot of shot j N _Bin : The number of color bins in the histogram used to measure color similarity In the MPEG video scene segmentation method, Extracting a vector component of a macroblock in a P frame in a shot; Normalizing the extracted vector components; Obtaining a normalized motion size of a macroblock using the normalized vector component; Obtaining an average size of the normalized motion size; Measuring the motion feature extraction and the motion similarity in each shot from the P frame in which the motion feature with the I or P frame of the MPEG video including the process of extracting the similarity between the two shots is stored in the vector form in macroblock units. A video scene segmentation method based on the containing color and motion features. The method of claim 7, wherein The process of normalizing the extracted vector components further comprises dividing the extracted vector components by 16 to have a value between -1 and 1, and the video scene segmentation method based on color and motion features. The method of claim 7, wherein The magnitude of the normalized motion is obtained by using Equation 3, The normalized average motion size in the shot is obtained by using Equation 4, The motion similarity between the two shots is further obtained by using Equation (5). ( Of the P frames in the shotkNormalized absolute displacement of the first forward macroblock in the x and y directions) N _MB is the total number of forward macroblocks in the shot. In the MPEG video scene segmentation method, Searching for similar shots using IOL techniques; 23. A method of video scene segmentation based on color and motion features comprising clustering, weighting color and motion according to characteristics of a search interval to obtain similarity between shots comprising dividing a boundary of a scene. The method of claim 10, And calculating the average motion of the similar shot search interval and using the value to apply the color and motion weighting values to the video scene segmentation method based on the color and motion characteristics. The method of claim 11, A video scene segmentation method based on color and motion features, further comprising obtaining an average motion using Equation 6 for the shots within the search interval to find the current shot and the similar shot. (I _CS is the index of the shot which is the basis of the search in the shot similarity search interval, and SR (SEARCH RANGE) is the number of shots to be searched from the I _CS .) In the MPEG video scene segmentation method, Searching for similar shots backward from the last shot in the similar shot search section and connecting similar shots to each other; if the similar shot is not detected, the video scene is divided and a search section is newly started. A video scene segmentation method based on color and motion features, using an IOL (IMPROVED OVERLAPPING LINKS) technique, which is a similar shot detection method. In the MPEG video scene segmentation method, Determining whether or not the divided scene is located at a time interval less than τ; Based on color and motion features, including post-processing to compensate for scene over-split, including merging two scenes when the neighboring scene and the motion similarity are greater than a threshold when located at a time interval less than τ. How to split a video scene. The method of claim 14, If less than [tau] to obtain a motion feature using the equation (7), and using the equation (8) further comprising calculating a motion similarity of two adjacent scenes. (I _SS and I _ES represent the index of the first and last shot of each scene)