JP2005237002A - Method for detecting cut point in moving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a cut detecting method with which a special cut, such as dissolve or wipe can be detected using a method simpler than the conventional method. <P>SOLUTION: Images for one frame are inputted in an image input 10, and an instantaneous cut is detected in instantaneous cut detection 11. It is detected whether or not an image determined as an instantaneous cut is a flash image in flash detection 12 and when it is not the flash image, an instantaneous cut is registered/displayed in instantaneous cut registration/display 13. As regards an image that is not determined as an instantaneous cut, a dissolve image is detected in basic dissolve detection 14. In this case, when the image is determined as dissolve, it is determined in pan/motion detection 15 whether the image is panning or motion. In the case of a negative result of the determination, a dissolve cut is registered/displayed 16 or in the case of a positive result, processing moves to wipe cut detection 18. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a cut point detection method for moving image data, and more particularly to a cut point detection method that can perform cut point detection for classifying a scene in moving image search with very high efficiency and high accuracy.

  There are special cuts such as dissolve, fade-in, fade-out, and wipe in addition to instantaneous cuts at the cut points of moving images. Regarding instantaneous cut, for example, in Reference 1 (BLYeo and B. Liu, “Rapid scene analysis on compressed video”, IEEE Tr. Circuits and Systems for Video Technology, Dec. 1995), the inter-frame difference value has a peak point. This is detected to determine the cut point. However, in this case, a peak point is generated even in the flash, resulting in excessive detection. Therefore, a method for distinguishing instantaneous cut from flash is proposed (first conventional method). In the above-mentioned document, flash detection is performed from the maximum value and the average value of the inter-frame differences for several seconds.

  For special cuts when the scene changes gradually over several seconds from several frames, such as dissolves and fades, refer to Reference 2 (A. Hampapur, R. Jain and T. Weymouth, "Digital Video Segmentation", Proc. ACM In Multimedia 94, pp.357-364, 1994), it is determined to be a dissolve when the inter-frame luminance difference value becomes a substantially constant value (second conventional method).

Regarding the detection of a wipe in which the spatial position of the screen gradually changes, for example, in the above-mentioned document 2, a wipe model is used, and when the partial differential value of the luminance component in the horizontal direction is constant, it is determined that the wipe is in the horizontal direction ( Third conventional method).
BLYeo and B. Liu, "Rapid scene analysis on compressed video", IEEE Tr. Circuits and Systems for Video Technology, Dec. 1995 A. Hampapur, R. Jain and T. Weymouth, "Digital Video Segmentation", Proc. ACM Multimedia 94, pp.357-364, 1994

  In the first conventional method, it is possible to detect a sharp peak value in a flash portion, but there is a problem that it is difficult to detect with a continuous flash or a weak flash. Further, in the second conventional method, the change in the inter-frame luminance difference value is very difficult to distinguish from the case of motion, and the difference value does not always take a constant value in a dissolve with motion. . Furthermore, it can be said that the third conventional method is very sensitive to the movement of the camera and the object. Furthermore, in order to cope with various directions, it is necessary to obtain a differential value for each direction, and there is a problem that processing is complicated.

  The object of the present invention is to solve the above-mentioned problems of the prior art and provide a cut point detection method capable of detecting special cut points such as dissolves and wipes using a simpler method than the conventional method. There is to do.

  In order to achieve the above object, according to the present invention, in the moving image cut point detection method, the cumulative sum of a section having a moving average difference of the in-screen activity of each frame is calculated, and the cumulative sum is calculated from a certain threshold value. The first feature is that when a section where the cumulative sum is larger than a certain threshold value appears after a small section, it is determined that the flat portion is dissolved.

  In addition, the present invention is characterized in that it is determined to be dissolved when the number of frames in which the normalized prediction error of the DC component of the prediction error is larger than a threshold value in a certain section is larger than the threshold value. There are two features.

  The present invention also provides a motion scene when the number of blocks in a frame n is greater than a certain threshold value and the difference between frames is large. When the number of blocks larger than a certain threshold is larger than a certain threshold and the screen average of the motion vector in the horizontal or vertical direction is larger than a certain threshold, the panning scene is distinguished from the cut point. The point has the third feature.

  Furthermore, the present invention provides a frame in which a frame difference value in a certain section is larger than a certain threshold value and a frame number is smaller than a certain threshold value in a section before and after the section. A fourth feature is that a wipe is determined when the number is larger than a certain threshold.

  According to the present invention, it is possible to improve the dissolve detection accuracy at the flat portion for the detection of the cut point, to distinguish between a motion scene and a dissolve scene, and to detect a wipe scene with high accuracy. Cut detection is possible with high accuracy.

  According to the present invention, there is an effect that the accuracy of detecting a dissolve such as a flat portion can be improved and the erroneous detection of a motion scene as a dissolve can be reduced. In addition, there is an effect that it is possible to detect a wipe scene change that has been difficult to detect.

  As an example, when a cut point detection for one hour of a TV program compressed with MPEG1 is performed using the present invention, the detection rate is 90% when the method disclosed in Japanese Patent Application No. 8-252333 is used. These scene changes could be detected with an accuracy of about 95%.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 shows a flowchart relating to the first embodiment of the present invention. First, one frame of an image is input by the image input 10, and an instantaneous cut is detected by the instantaneous cut detection 11. Here, the flash detection 12 detects whether or not the image is determined to be an instantaneous cut. If it is not a flash image, the instantaneous cut registration and display processing is performed on the instantaneous cut registration display 13 and the remaining frame processing 20 is performed. When the instantaneous cut is not determined by the instantaneous cut detection 11 or when the flash detection 12 is the flash detection, the basic dissolve detection 14 detects the dissolve image.

  If it is determined to be a dissolve, pan / motion detection 15 determines whether it is panning or a motion image. If not, the dissolve cut registration display 16 performs dissolve cut registration and display processing for the frame as a dissolve, and the remaining frame processing 20 is performed. If the pan / motion detection 15 determines that the image is panning or a motion image, the process proceeds to the wipe cut detection 18.

  For frames that are not detected by the basic dissolve detection 14, it is determined in the flat area dissolve detection 17 whether or not they are dissolved in the flat area. If it is determined to be a dissolve, the display shifts to a dissolve cut registration display 16. If it is not determined as a dissolve cut, the process proceeds to wipe cut detection 18. In the wipe cut detection 18, it is determined whether or not it is a wipe cut. If it is determined to be a wipe cut, the frame is registered as a wipe cut or displayed on the wipe cut registration display 19, and the process proceeds to the remaining frame processing 20. If the wipe cut is not determined, the remaining frame processing 20 is performed.

  Hereinafter, the process of detecting cut points from MPEG data will be described in detail with reference to FIG. Although FIG. 2 integrally constitutes the cut detection device, in order to make the explanation easy to understand, the cut detection device is divided into (a) to (e).

  The encoded data shown in FIG. 5A is input to the variable length decoding unit 21, and the variable length decoding unit 21 decodes the quantized two-dimensional DCT coefficient of each block, and the two-dimensional DCT coefficient is an average value. Input to the component extraction unit 22. As the average value component extraction method of the average value component extraction unit 22, for example, “Detection of cut points from compressed video data by simple decoding process” by Ushihara and Nakajima et al., Information Processing Society of Japan 51st National Convention, 6S-9 (1995), Japanese Patent Application No. 7-263681, and the like can be used.

  The average value component obtained by the average value component extraction unit 22 is input to the instantaneous cut point detection unit 23, and an instantaneous cut is detected. The method described in the above document can also be used for the instantaneous cut point detection method.

  In the flash, the screen changes greatly in units of one frame, so it is often erroneously detected as a scene change. Therefore, when the instantaneous cut detection unit 23 determines that the instantaneous cut has occurred, it is input to the flash scene determination unit 2i in FIG.

  The luminance and color difference distribution in the frame where the flash occurs is completely different from that in the previous frame, but unlike the scene change, these distributions return to the original state in one to two frames after the flash scene. The flash scene model is shown in FIGS. FIG. 3 shows a single flash scene model, and FIG. 4 shows a continuous flash scene model.

  For example, when only frame n is a flash scene as shown in FIG. 3 (a), the correlation between frames n and n-1 is low (L in the figure), but the correlation between frames n + 1 and n-1 is high ( H) shown.

  Therefore, in the color difference histogram calculation unit 2g in FIG. 5B, a frame color difference histogram is calculated from the average value component data of the color difference signal input from the average value component extraction unit 22, and the calculation result is used as the color difference correlation calculation unit 2h. And input to the fifth memory 2j. In the color difference correlation calculation unit 2h, the color difference histogram correlation ρ (n, n-1) is calculated from the color difference histogram of the frame n + 1 inputted and the color difference histogram of the frame n and the frame n-1 previously stored in the fifth memory 2j. ), Ρ (n + 1, n−1) are obtained and input to the flash scene determination unit 2i.

  In addition, as a calculation method of the color difference histogram and the color difference correlation, for example, “Cut detection from compressed image data by inter-frame luminance difference and color difference correlation” by Nakajima, the IEICE Autumn Meeting D-501 (1994), The method disclosed in Japanese Patent Application No. 5-216895 or Japanese Patent Application No. 6-46561 can be used.

As shown in FIG. 3A, when the frame n is a flash scene, the flash scene determination unit 2i has a low correlation ρ (n, n-1) between the frames n and n-1, but the frames n + 1 and Since the correlation ρ (n + 1, n−1) of the frame n−1 is high, the frame n can be determined as a flash scene when the following expression (1) is satisfied.
ρ (n, n−1) <Th_fl, ρ (n + 1, n−1)> Th_fh (1)
Here, Th_fl and Th_fh indicate the first and second threshold values, respectively, and Th_fl <Th_fh. The first and second threshold values are low values (for example, 0.7) that can be determined as cut points and high values (for example, 0.9) that can be determined as non-cut points, respectively.

  As shown in FIG. 3B, when the frame n-1 is a flash scene, the correlation between the frame n and the frame n-1 is low, but the correlation between the frame n and the frame n-2 is high. In addition, when flash scenes are continuously formed as shown in FIG. 4, it is possible to determine from the correlation with adjacent frames in the same manner.

  If it is determined that the scene is a flash scene (Yes in the flash scene determination unit 2i), the cut detection process is terminated, the process returns to the encoded data input shown in FIG. 5A, and the next frame is input. If it is determined that the scene is not a flash scene, the cut point image is registered and held in the cut point image holding unit 2d, and the cut point image is displayed in the cut point image display unit 2k. As for the display of the cut point image, the method disclosed in “Special Cut Point Detection Device for Moving Image” of Japanese Patent Application No. 8-252333 can be used.

  For an image that has not been determined to be an instantaneous cut by the instantaneous cut point detection unit 23, the basic dissolve detection unit 24 determines a dissolve image. As a method for determining a dissolve image, for example, a method disclosed in “Special image cut point detection device for moving image” of Japanese Patent Application No. 8-252333 can be used.

  Even when the basic dissolve detection unit 24 determines that a dissolve has occurred, a motion scene or panning scene may be detected in some cases. For example, as for the activity disclosed in Japanese Patent Application No. 8-252333, “Special Cut Point Detection Device for Moving Images”, both dissolve and motion scene show similar changes. However, in the motion scene, almost all the coding blocks are motion compensation blocks, and the interframe difference is also large. On the other hand, in the dissolve, the number of motion compensation blocks is small and the difference between frames is also small.

  Therefore, when the basic dissolve detection unit 24 determines that the motion is a dissolve, the motion scene determination unit 2a determines whether the scene is a motion scene. The motion scene determination unit 2a receives the motion vector MV of each block from the variable length decoding unit 21, and calculates the interframe difference Dn from the interframe difference calculation unit 2b. For Dn, the block average value of the luminance signal of the frame n obtained from the method disclosed in Japanese Patent Application No. 5-216895 and the average value component extracting unit 22 is stored in the third memory 2j, and the third The cumulative sum of the absolute differences between frames of the block average value of the luminance signal can be calculated using the block average value of the frame n−1 obtained from the memory 2j.

The motion scene determination unit 2 a determines a motion scene based on the interframe difference Dn obtained from the interframe difference calculation unit 2 b and the motion vector MV obtained from the variable length decoding unit 21. Many blocks are motion compensation blocks in motion scenes and panning scenes. For this, the following judgment formula (2) can be used.
MVC, PMVC> Th_mvc (2)
The Th_mvc is a threshold value, for example, 0.45MN.

Here, MVC and PMVC are the number of motion vectors of the newest frame and the number of motion vectors of the second newest frame, respectively. For example, in the case of MPEG, it is possible to use the P picture closest to the input frame and the P picture closest to the next. However, the number of motion vectors counts the number of vectors whose motion vector size exceeds a threshold Th_mv (for example, 4 pel / frame). In addition, a motion scene has a large inter-frame difference. In the panning scene, many blocks have the same motion vector. Therefore, a moving scene and a panning scene can be discriminated using the following equations (3) and (4), respectively. Dn, Dn-1> Th_bm (3)
Dn> Th_mm, | <mvx> | or | <mvy>|> Th_am (4)

  Here, Dn is an interframe difference between the frame n and the frame n-1, and Dn-1 is an interframe difference U10 between the frame n-1 and the frame n-2. Further, Th_bm, Th_mm, and Th_am are threshold values, and for example, Th_bm = 7.5MN, Th_mm = 1.5MN, Th_am = 5 pel / frame can be used. <mvx> and <mvy> indicate the screen average of the horizontal motion vector and the vertical motion vector.

  Therefore, when either (3) or (4) and (2) are satisfied, it can be determined that the scene is a moving scene or a panning scene.

  If the motion scene determination unit 2a determines that the scene is a motion scene, the wipe scene determination unit 2e in FIG. If it is determined that the scene is not a moving scene, it can be determined as a dissolve scene, and the process proceeds to the cut-point image holding unit 2d shown in FIG.

  When the basic dissolve detection unit 24 determines that it is not a dissolve, the process proceeds to the first flat portion determination unit 25. For example, when using the moving average difference of activities disclosed in Japanese Patent Application No. 8-252333 "Special image cut detection device for moving images", there is a frame in which the absolute value of the difference value exceeds a threshold value Judge as a dissolve. When a dissolve occurs in a flat part, the change in activity is very small, and the absolute value of the moving average difference value of the activity may not exceed the threshold value. However, in the dissolve section, the difference value changes while maintaining a value close to the threshold value. Accordingly, when the difference value has a certain size in a certain section and the integral value is large, the flat portion can be dissolved. In the case of a dissolve, as described in Japanese Patent Application No. 8-252333, a moving average difference of activities has a section having a positive value after a section having a negative value. Therefore, these sections can be determined using the following equations (5) to (7).

In Equation (7), dh and dm are positive integers, and are constants that specify the size of the dissolve interval. Th_sa and Th_ea are threshold values for determination. For example, Th_sa = 70 and Th_ea = 1 can be used. DMV (t) is a moving average difference of activities in the screen, and the method disclosed in Japanese Patent Application No. 8-252333 can be used. Specifically, as shown in (d) of the figure, the in-screen activity calculation unit 2l calculates the in-screen activity and puts it in the moving average calculation unit 2m to move the activity in the screen. Find the average difference DMV (t). The in-screen activity is stored in the memory 2n.

  When the difference DMV (t) satisfies the expressions (5) and (6), the first flat part determining unit 25 can determine that the flat part is dissolved.

  When the first flat part determining unit 25 determines that the flat part is dissolved, the first flat part determining unit 25 inputs the result to the second flat part determining unit 27. On the other hand, if it is not determined to be a dissolve, the moving average difference DMV (t) of the activity in the screen during slow panning has a change similar to the dissolve in the flat part. May be detected. In slow panning, the amount of motion is small, the motion compensation prediction efficiency is high, and the prediction error is small. On the other hand, since two screens are synthesized in the dissolve, the motion compensation prediction efficiency is not so high. Therefore, it is possible to distinguish dissolve and panning according to the amount of prediction error.

  The DC component DCn (i, j) of the DCT coefficient of the coding prediction error in the block (i, j) of the frame n from the variable length decoding unit 21 is input to the normalized prediction error calculation unit 29, and the normalized prediction error NPEn is Desired. The normalized prediction error amount NPEn can be calculated based on the screen average value of the DC component as follows.

Here, MN is the total number of blocks in the screen, and k is the number of inter-frame encoded blocks.
The normalized prediction error amount NPEn is input to the second memory 28 and also input to the second flat portion determination unit 27. Since NPEn has a large value in the dissolve section, the number of frames having a large prediction error can be determined as a dissolve section by comparing with the threshold value. Therefore, it can be determined using the following formula.

Here, PEfl is a flag indicating whether or not there is a frame having a large prediction error. PEfl is set to 1 when the normalized prediction error NPEl of the frame l is larger than a threshold Th_pe (for example, 3MN). In addition, a section before bd_p and dd_p dissolve and a section during dissolve are shown. As a result, when the expression (9) is satisfied, it can be determined as a dissolve in a flat portion. If it is determined to be a dissolve, the process proceeds to the cut point image holding unit 2d shown in FIG. 5B, and the cut point image is held and recorded. If it is not determined to be a dissolve, the process proceeds to the wipe scene determination unit 2e shown in FIG.

  The wipe scene change model is shown in FIG. FIG. 4A shows an example in which a new shot B appears on the current shot A while moving in the right direction. FIG. 6B shows an example in which the shot A gradually decreases in the horizontal direction while the shot B appears while expanding in the modified example of FIG. (C) and (d) are modified examples of FIG. (B), in which the scene shifts while being enlarged or reduced in the vertical direction. FIG. 4E shows an example in which the scene shifts by turning the page. The page of the current shot A is turned over, and the next shot B appears.

  In any wipe, the position of two shots changes while the scene changes, and before and after the change, each shot is stationary and stable unless there is a large movement in the shot. Furthermore, the position and shape of each shot during the wipe scene change slowly and constantly move. Accordingly, any wipe scene change between frames can be represented by a simple model as shown in FIG. FIG. 6 shows a wipe model using inter-frame differences.

  The inter-frame difference Dn obtained by the inter-frame difference calculation unit 2b in FIG. 2 (a) is input to the fourth memory 2f and the wipe scene determination unit 2e in FIG. 2 (c). The wipe scene determination unit 2e determines the wipe scene by comparing the inter-frame difference with the wipe model. Since the inter-frame difference is stable and large in the wipe section, and the inter-frame difference is small in the preceding and following sections, the following determination formula (11) can be used.

BW> Th_bw, DW> Th_dw, AW> Th_aw (11)
Here, BW, DW, and AW are the numbers of frames recognized as before, during, and after wiping, respectively, and Th_bw, Th_dw, and Th_aw are threshold values. As the threshold values Th_bw, Th_dw, Th_aw, for example, Th_bw = 27, Th_dw = 24, Th_aw = 5 can be used. BW, DW, and AW can be obtained using the following equation (12).

DL (k) and DH (k) are flags indicating whether the inter-frame difference Dk in frame k is high or low, and can be determined using the following equation (13).
if Dk> Th_wp then DL (k) = 0, DH (k) = 1, else DL (k) = 1, DH (k) = 0 (13)
Here, for example, Th_wp = 1.36MN can be used as the threshold Th_wp.

  When the wipe scene determination unit 2e determines that a wipe scene is detected, the cut point image holding unit 2d and the cut point image display unit 2k in FIG. Move on to frame input processing. If it is not determined to be a wipe, the process directly proceeds to the next frame input process.

  In the above-described embodiment, the cut point detection method when compressed by MPEG has been described. However, the present invention is not limited to this, and cut points can be obtained by the same process even in uncompressed moving image data and other compression methods. Can be detected.

  In addition, the present invention can be partially combined, and it is also possible to combine with other cut point detection methods using only the dissolve detection part, or to combine only wipe scene detection with instantaneous cut detection.

It is a processing flowchart which shows the outline | summary of operation | movement of this invention. It is a block diagram which shows the structure of one Embodiment of this invention. It is explanatory drawing of a single flash scene model. It is explanatory drawing of a continuous flash scene model. It is explanatory drawing of a wipe model. It is explanatory drawing of the wipe model at the time of using the difference between frames.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 21 ... Variable length decoding part, 22 ... Average value component detection part, 23 ... Instantaneous cut point detection part, 24 ... Basic dissolve detection part, 25 ... Flat part determination part 1,26,28, 2c, 2f, 2j, 2h ... 1st to 6th memory, 27 ... flat part determination unit 2, 2d ... cut point image holding unit, 2e ... wipe scene determination unit, 2g ... color difference histogram calculation unit, 2h ... color difference correlation calculation unit, 2i ... flash scene determination Part, 2k ... cut point image display part, 2l ... activity calculation part in the screen, 2m ... moving average calculation part.

Claims (8)

In the moving image cut point detection method,
Calculate the cumulative sum of sections with moving average differences of in-screen activities for each frame,
A moving image cut point detection method, characterized in that when a section in which the cumulative sum is larger than a certain threshold appears after a section in which the cumulative sum is smaller than a certain threshold, it is determined that the flat portion is dissolved. The moving image cut detection method according to claim 1,
A moving image cut point detection method characterized by distinguishing a slowly moving scene from a dissolve by using an encoding prediction error of each frame. In the moving image cut detection method of claim 2,
The moving picture cut point detection method using the normalized prediction error of the entire screen using the DC component of the prediction error as the encoding prediction error. In the moving image cut detection method of claim 3,
A cut point detection method for moving images, characterized in that, when the number of frames with a normalized prediction error of the encoded prediction error is larger than a threshold value is larger than the threshold value in a certain section, it is determined as dissolve. In the moving image cut point detection method,
A moving image cut point detection method characterized in that, for an input frame, a motion vector size and an inter-frame difference value are calculated, and the cut point and the motion scene are distinguished by evaluating these values. In the moving image cut point detection method of claim 5,
When the number of blocks in a frame n is larger than a certain threshold and the difference between frames is large,
Also, if the number of blocks in frame n is greater than a certain threshold and the screen average of horizontal or vertical motion vectors is greater than a certain threshold, the panning scene is cut. A moving image cut point detection method characterized by distinguishing from a point. In the moving image cut point detection method,
A moving image cut point detection method characterized in that, in a certain section, when there is a section having a small inter-frame difference before and after a section having a large inter-frame difference, the section having the large inter-frame difference is determined as a wipe section. . In the moving image cut point detection method of claim 7,
Threshold value with a frame number greater than a certain threshold value in a certain section and larger than a certain threshold value and with a frame number smaller than a threshold value with a difference value between frames in the preceding and following sections. A moving image cut point detection method, characterized in that if it is larger, it is determined as a wipe.

Images