JPH09322171A - Dissolve detection method and device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a dissolve from a compression coded video image at a high speed. SOLUTION: A changing tendency calculation stage 3 calculates a changing tendency 4 based on coding information included in an inter-frame forward direction prediction coding image string. A slow changing direction calculation stage 5 extracts an area where a changing direction 4 is constant over a plurality of adjacent frames as a slow change area 6 having a slow change. A discrimination stage 7 discriminates it that a dissolve is in existence when the share of the slow change area 6 on the image is higher than a threshold level.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for detecting a dissolve in compression-coded video data.

[0002]

2. Description of the Related Art The amount of video data is enormous, and the only way to know the contents is to watch the video in chronological order. If the video is divided according to a certain standard, it is useful for skipping the video or roughly understanding the content.

When the images are viewed in chronological order, the point at which the shots are switched is called a cut point. Since the contents of the image change sharply before and after the cut point, the cut point can be detected by a method of "calculating the difference between adjacent images in time order and regarding the place where the difference is large as the cut point". For example, a "moving cut automatic dividing method" that detects a cut point by calculating a change in a continuous frame from the number of pixels in which physical quantities such as brightness and color at a position (x, y) of a continuous image frame have changed (special feature Kaihei 5-37853) and the like. Under such circumstances, stable detection of slowly changing scene changes such as dissolve has been a problem.

Dissolve is a type of scene change from image A to image B, in which the image contents keep changing gradually. The fade-in and fade-out in which the image gradually disappears or changes from white can be considered as a special case where the scene A or the scene B is only white or black. These scene changes are generally linear changes, with gradual changes in luminance and color components.

The characteristics of the dissolve change will be described with reference to FIG. Scene A (41) in the original image sequence (Fig. 5 (1))
To scene B (45) over a T frame 42. In the process of changing from the scene A to the scene B, the corresponding pixel (x, y) gradually changes from the component value of the scene A to the component value of the scene B (FIG. 5 (2)). . The component value of each pixel in the dissolve is
It is expressed by the following mathematical formula.

[0006]

[Equation 1] However, I (x, y) _A and I (x, y) _B are blocks I (x, y) in the images in scene A and scene B, respectively.
y) means the component, T is the total number of frames in the dissolve generation period, and t is the frame number counted from the first frame where the dissolve starts. 43 and 44 are luminosity at t ₁ th frame. In the dissolve, the above-mentioned gently changing frames continuously continue over the entire screen. When encoding is performed, such a gradual change is subject to motion prediction. Therefore, in the P picture sequence of encoded data, the difference information between the motion vector due to the motion estimation and the block on the reference frame is recorded. ing. Therefore,
Motion estimation is performed, and only the difference information between blocks on the reference frame is recorded and transmitted. Thus, for example, in the case of encoding scheme as P Bikuchiya is continuous, DC component of the difference information 46 contained in the t ₁ frame of blocks of P picture in the change dissolve (Fig. 5 (3))
Is

[0007]

[Equation 2] Becomes A feature amount that reflects such a phenomenon is calculated by the slowly changing region calculation unit. It should be noted that even in the case of a gently changing scene change, a wipe in which a part of the content is continuously replaced between the scene A and the scene B without the process of merging the images is treated separately from the dissolve.

In detecting the change in dissolve, it is difficult to distinguish whether the change in the image component such as the luminance is caused by the movement or the change caused by the gradual change such as the dissolve. There is a problem that a scene in which a subject moves is erroneously detected as a dissolve. Therefore, “Automata” that solves the above problems by obtaining a motion vector or an optical flow
tic partitioning of full-
"motion video" (Hong Jian
Zhang, Atreyi Kankanhalli, S
tephen W. Smolial, Multimedi
a Systems 1993 1: 10-28) and the like.

In order to detect a dissolve in a compression-encoded image such as the MPEG system, there is no choice but to decode the data once to restore an uncompressed digital image and detect it using a conventional technique. However, the decoding process has a problem that it requires a long calculation time. Further, the above-mentioned dissolve detection technique has a problem that it takes a very long time to perform a calculation for obtaining a motion vector. It is inefficient to require a huge amount of time for both the decoding process and the motion vector calculation process. Therefore, a technique for detecting a dissolve without requiring a decoding process is required, but such a technique does not currently exist.

[0010]

In the case of detecting a dissolve from encoded compressed data, after performing a decoding process,
Although it is necessary to perform the dissolve detection process by the conventional method, there is a problem that both the decoding process and the dissolve detection process take time.

An object of the present invention is to solve the problems of the prior art, to solve the dissolve without performing the decoding process from the compression-encoded video, and without requiring the time-consuming process such as calculating the motion vector. It is an object of the present invention to provide a dissolve detection method and device for detecting a high speed.

[0012]

According to the dissolve detection method of the present invention, an encoding included in an image data example is performed from an image data string compressed by a compression encoding method including an interframe or interfield predictive encoding method. A change tendency calculation step of calculating a change tendency based on information, and a slowly changing area extracting step of extracting an area in which the changing tendency is constant over a plurality of adjacent frames as a slowly changing area having a gentle change. And a determination step of determining that a dissolve exists when the ratio of the slowly changing region to the screen is larger than a threshold value.

Further, the dissolve detecting apparatus of the present invention is based on the coding information contained in the image data sequence compressed from the image data sequence compressed by the compression encoding system including the inter-frame or inter-field predictive encoding system. A change tendency calculation unit that calculates a change tendency, a slow change region extraction unit that extracts a region in which the change tendency is constant over a plurality of adjacent frames as a slow change region in which a gentle change has occurred, and the slow change When the ratio of the area to the screen is larger than the threshold value, the determination unit determines that the dissolve exists.

According to the present invention, a change tendency, which indicates how the image sequence is changing, is calculated from the encoding information included in the image data sequence compressed by the inter-frame or inter-field predictive encoding method, and a plurality of changes are calculated. It is determined that a dissolve exists when the ratio of the area exhibiting the same change tendency over the frame to the entire screen is larger than the threshold value. As described above, according to the present invention, since the feature information is directly extracted from the encoded data without decoding the data, the dissolve can be detected at high speed.

According to the embodiment of the present invention, the change tendency calculating step calculates the change tendency on the basis of the sign of the difference information in the area where the motion prediction is correct in the encoded information included in the image data string.

According to another embodiment of the present invention, in the change tendency calculating step, the change tendency is calculated based on the sign of the difference information in the area where the motion prediction is effective in the coded information included in the image data string. At the time of calculation, the sign of the difference information is changed in consideration of whether the motion prediction is predicted from the past or the future direction, and the change tendency is calculated.

According to another embodiment of the present invention, the change tendency calculating step calculates the change tendency on the basis of the value of the difference information in the area of the coded information included in the image data string, which is subject to the motion prediction. .

In the present invention, the tendency of change is calculated by referring to the assigned difference information in the region (non-intra block in MPEG) where the motion prediction is included in the inter-frame or inter-field predictive coded image. . Interframe or interfield predictive coding scheme is
A region similar to the encoding target region is searched from images at different times, and motion vector information indicating the position of the similar region and difference information between the encoding target block and the reference region are held. Since the motion vector is calculated in advance by the motion prediction, the difference information is equal to the change in the image content after removing the motion of the camera and the motion of the subject. That is, it is possible to handle the change in the image content without decoding the encoded image and then removing the camera movement or the like.

From such difference information, it is calculated what kind of change tendency the image shows, and an area showing a certain change tendency is calculated as a slowly changing area. Since the dissolve exhibits a characteristic that it gradually changes over a plurality of frames, it is possible to detect the dissolve based on the magnitude relationship of the ratio of the gently changing region on the screen.

[0020]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a flow chart of a dissolve detection method according to an embodiment of the present invention.

The inter-frame forward predictive coded image sequence 2 is extracted from the input compressed image sequence 1 and input to the change tendency calculation stage 3. In the change tendency calculation step 3, the change tendency 4 of each image sequence is calculated. The changing tendency 4 is input to the slowly changing area extraction step 5. In the slowly changing region extracting step 5, the feature amount for measuring the change in the image content of each image sequence over a long time is extracted as the slowly changing region 6 of the image sequence. Examples of the change tendency calculating step 3 and the slowly changing area extracting step 5 will be described later. The gradual change region 6 is input to the decision step 7, and the final dissolve 8 is obtained based on the ratio of the gradual change region 6 to the screen.

In the change tendency calculation step 3, the tendency of change in the characteristic quantities such as brightness, hue, texture and edges that characterize the image is expressed as a symbol. For example, the tendency of brightness change can be quantified by the following simple method. By referring to the difference information allocated in the region (non-intra block in MPEG) included in the encoded information, which is subjected to the motion prediction, the code {+, 0,-} of the luminance difference DC component is a symbol indicating the change tendency. And The symbol “+” indicates that the brightness increases between fields, the symbol “0” indicates that the brightness is constant, and the symbol “−” indicates that the brightness decreases.
Of course, when the difference value is d, “+” may be assigned as a symbol if d> n, “0” if −m <= d <n, and “+” if d <−m. Within design. As for the hue, the sign of the color difference DC component can be used as in the case of the brightness.

In the slowly changing region calculation step 5, the changing tendency is
Count the areas that are constantly changing over several frames. For example, it is a step of calculating a region in which the brightness is gradually increased and the change is uniform over several frames characteristic of the dissolve.

FIG. 2 is a block diagram showing the structure of the dissolve detecting apparatus according to the embodiment of the present invention.

This dissolve detection device detects the dissolve 8 from the input compressed image data sequence 1. In this embodiment, the input compressed image data string 1 is an image data string {P _t-2 , P _t-1 , P _t , P _t , which is composed of P pictures compressed by the inter-frame or inter-field predictive coding method.
_{Pt + 1} , _{Pt + 2} }. The input compressed image data sequence 1 is
It may be an image file stored in a storage device such as a hard disk or a CD-ROM, or may be an image file transmitted by being connected to a network. size,
The sample rate is arbitrary, but an interframe or interfield predictive coding method must be adopted. As an example, H.264. There are formats such as H.261, MPEG1, and MPEG2.

The input compressed image data sequence 1 is analyzed by the encoding information analysis unit 10 to analyze the picture data, the serial number from the first picture, the picture type, etc., and the data sequence memory 11 for storing the data over several tens of frames. The information is also stored in. In addition to supplementary information such as the whole or part of the encoded data, the frame number from the top file of the image, and the feature amount obtained by the preprocessing, the data string memory 11 can be obtained by using an external device and added. External information such as the shooting time and the shooting location, and external data 9 that is user data separately input by the user may be stored as necessary.

Of the image sequence read out from the data sequence memory 11, the P picture sequence 2 (P _t ) is used as the change tendency calculation unit 12
And the change tendency 4 is calculated. The change tendency from the change tendency 4 to a plurality of frames is compared, and the gentle change area 6 (DL
_Pt ) is calculated. Obtained slow change region 6 (DLP _t )
Is input to the dissolve determination unit 14, and when the ratio of the slowly changing region 6 to the entire screen is larger than the threshold value,
Judge as final dissolve 8.

Next, a detailed description will be given of examples of the slowly changing region extracting section 13 and the dissolve determining section 14 in order.

Change tendency calculator 12 and slowly changing area calculator 1
3 and the first example of the dissolve determination unit 14 will be described with reference to FIG. Here, attention is paid to the +/− code included in the difference information.

In the first example, with reference to the inter-frame predictive coding frame, the region having the same sign of the difference in luminance and color difference is set as the slowly changing region for three consecutive frames. Specific processing will be described with reference to the flowchart of FIG. The variables used here will be described. Variable sign
(I, j) holds the sign (+, 0,-) of the brightness difference d (i, j) (i, j is a variable indicating the position of the block) of the block (i, j) in the previous frame. Also, ite
r (i, j) is a variable for measuring how many times the same code appears in block (i, j). Also, c
nt is a variable for counting the number of blocks having the same code over three consecutive frames.

First, in step 21, i, j are set to 0
Initialize to The step 22 determines whether or not the motion prediction block having the brightness difference is a hit block, and the step 23 determines whether the sign of the current brightness difference and the sign of the past brightness difference are both positive. It is determined whether or not the condition is satisfied, and in step 24, it is determined whether or not the condition that both the sign of the current brightness difference and the sign of the past brightness difference are negative. If steps 23 to 24 are satisfied, these areas are determined to be slowly changing areas, and the slowly changing areas are counted in step 25. On the other hand, if the conditions in steps 23 to 24 are not satisfied, the past code and the current code are different, so the process jumps to step 29 and the code information used for the presence determination process of the slowly changing region in the next frame and subsequent frames is obtained. Reset. Also, since the sign of the brightness difference has changed, i is the accumulation of the sign from the past.
The value of ter (i, j) is also reset in step 30.

On the other hand, when it is determined in step 22 that the block has no luminance difference, the block (i, j) is generally an intra-block coded block. The intra-block coded block appears when the pixel changes greatly and the motion estimation is deviated. In the encoding of the dissolve change, it is assumed that an intra-block coded block may be formed in the process of the change.
The process is continued according to the conditions of 3 to step 24 being satisfied.

Furthermore, depending on the encoding process, an exceptional block may appear that is neither the block on which motion prediction has been hit, nor the intra-block encoded block. In such a case, it is difficult to guess what value the coded information was, and it is ignored. The code information of the brightness difference used for the presence determination process of the slowly changing area after the next frame stores the past code information, and the number of consecutive codes is reset. Since these exceptional blocks appear only infrequently, it is assumed that all blocks for which motion estimation has been missed are intra-coded blocks,
The processing may be continued. That is, in step 26, "N
The path from "o" to step 30 may be eliminated.

Through the above processing, the number of times the same code appears consecutively is stored in iter (i, j). Next, in step 27, it is determined whether or not iter (i, j) is 3 or more, and if it is 3 or more, 1 is added to cnt in step 28. Through steps 31 to 34, the above processing is performed for all blocks.

As a result of the above processing, the variable cnt
Indicates the area of a region having the same sign of luminance difference over three consecutive frames. If the area of the slowly changing region indicated by this variable cnt is larger than a threshold value,
Determine that a dissolve exists.

In the above example, in step 22, the condition of using the predicted block is used.
It is also possible to change to the condition that | d (i, j) | is larger than a certain positive value. Also, instead of performing the processing of blocks encoded in the frame (step 26), i
It can be modified in various ways, such as not changing the value of ter (i, j).

Change tendency calculation unit 12, slowly changing region extraction unit 1
3 and the second example of the dissolve determination unit 14 will be described with reference to FIG. Here, attention is paid to the size of the difference information.

In the second example, with reference to the inter-frame predictive coding frame, the region having the same difference value in luminance between two consecutive frames is set as the slowly changing region.
Specific processing will be described with reference to the flowchart of FIG.
The variables used here will be described. The variable is based on the first example (FIG. 3), but d _t (i, j) holds the value of the luminance difference of the block (i, j) in the t frame. Instead of steps 23 and 24 in the first example, the process of step 35 is performed, and d _t (i, j) and d _t-1 (i,
If the absolute value of the difference in j) is smaller than the threshold value σ, the difference value is said to be the same value. In step 35, the threshold value σ is set to a fixed value, but σ may be changed adaptively. It may also include the step of determining the value of σ based on the quantization step included in the encoded information.

In this example, the input video is only the image sequence (P picture sequence) by the inter-frame or inter-field forward predictive coding system, but in the MPEG system, other than this, the frame is also used. Image (B picture) by the inter-field or inter-field bidirectional predictive coding method, or an image by the intra-frame or intra-field coding method (I
(Picture) appears. In this example, I pictures and B pictures are ignored. However, since the B picture holds the difference information like the P picture, the change tendency is calculated from the B picture, and the slowly changing area is extracted.
Processing such as application to dissolve detection is also possible.
However, in this case, since the B picture refers to both the past and future images, it is necessary to consider whether the difference information is the difference information between one of the past and future images or both of the images. It is necessary to include processing such as reversing the sign.

[0041]

As described above, according to the present invention, a dissolve is detected in a compression-coded image data string by using various kinds of information included in an inter-frame or inter-field predictive coded image. This allows the dissolve to be detected at high speed without decoding the data.

[Brief description of drawings]

FIG. 1 is a flowchart showing a dissolve detection method according to an embodiment of the present invention.

FIG. 2 is a configuration diagram of a dissolve detection device according to an embodiment of the present invention.

FIG. 3 is a flowchart showing a first example of a change tendency calculation unit 12, a slowly changing region calculation unit 13, and a dissolve determination unit 14.

FIG. 4 is a flowchart showing a second example of a change tendency calculation unit 12, a slowly changing region calculation unit 13, and a dissolve determination unit 14.

FIG. 5 is a schematic diagram of a change in dissolve.

[Explanation of symbols]

1 Input compressed image data 2 Forward prediction coded image sequence between frames 3 Change tendency calculation stage 4 Change tendency 5 Slow change region extraction stage 6 Slow change region 7 Judgment stage 8 Dissolve 9 External data 10 Coding information analysis unit 11 Data sequence Memory 12 Change tendency calculator 13 Slow change area extractor 14 Dissolve determiner 21-35 Step 41 Scene A 42 Number of frames T 43 t ₁ 44 I (x, y) _A + (I (x, y) _A- I ( x,
y) _B ) / Tt ₁ 45 Scene B 46 Difference information for each frame

Claims (5)

[Claims] 1. A change tendency calculation for calculating a change tendency from an image data string compressed by a compression coding method including an inter-frame or inter-field predictive coding method based on the coding information included in the image data string. A step, a slow change area extraction step of extracting an area whose change tendency is constant over a plurality of adjacent frames as a slow change area having a gentle change, and a ratio of the slow change area to the screen. A dissolve detection method comprising a determination step of determining that a dissolve is present when it is greater than a threshold value. 2. The dissolve according to claim 1, wherein the change tendency calculating step calculates the change tendency based on a sign of difference information in a region of the coded information included in the image data string that has been subjected to motion prediction. Detection method. 3. In the change tendency calculating step, when the change tendency is calculated based on the sign of the difference information in the area where the motion prediction is effective, out of the encoded information included in the image data string, the motion prediction is performed in the past. 3. The dissolve detection method according to claim 2, wherein the sign of the difference information is changed and the change tendency is calculated in consideration of which direction, ie, the future direction, is predicted. 4. The dissolve according to claim 1, wherein the change tendency calculating step calculates the change tendency based on a value of difference information in a region of the encoded information included in the image data string, the area being subjected to motion prediction. Detection method. 5. A change tendency calculation for calculating a change tendency from an image data string compressed by a compression coding method including an inter-frame or inter-field predictive coding method based on coding information included in the image data string. Section, a slowly changing area extracting section that extracts an area whose change tendency is constant over a plurality of adjacent frames as a slowly changing area where a gentle change has occurred, and a ratio of the slowly changing area to the screen. A dissolve detection apparatus having a determination unit that determines that a dissolve is present when the value is larger than a threshold value.