JP4261603B2 - Moving image cut point detection device - Google Patents

Description

  The present invention relates to a cut point detection method for moving image data, and more particularly to a cut point detection method that can perform cut point detection for classifying a scene in moving image search with very high efficiency and high accuracy.

  There are special cuts such as dissolve, fade-in, fade-out, and wipe in addition to instantaneous cuts at the cut points of moving images. Regarding instantaneous cut, for example, in Reference 1 (BLYeo and B. Liu, “Rapid scene analysis on compressed video”, IEEE Tr. Circuits and Systems for Video Technology, Dec. 1995), the inter-frame difference value has a peak point. This is detected to determine the cut point. However, in this case, a peak point is generated even in the flash, resulting in excessive detection. Therefore, a method for distinguishing instantaneous cut from flash is proposed (first conventional method). In the above-mentioned document, flash detection is performed from the maximum value and the average value of the inter-frame differences for several seconds.

  For special cuts when the scene changes gradually over several seconds from several frames, such as dissolves and fades, refer to Reference 2 (A. Hampapur, R. Jain and T. Weymouth, "Digital Video Segmentation", Proc. ACM In Multimedia 94, pp.357-364, 1994), it is determined to be a dissolve when the inter-frame luminance difference value becomes a substantially constant value (second conventional method).

Regarding the detection of a wipe in which the spatial position of the screen gradually changes, for example, in the above-mentioned document 2, a wipe model is used, and when the partial differential value of the luminance component in the horizontal direction is constant, it is determined that the wipe is in the horizontal direction ( Third conventional method).
BLYeo and B. Liu, "Rapid scene analysis on compressed video", IEEE Tr. Circuits and Systems for Video Technology, Dec. 1995 A. Hampapur, R. Jain and T. Weymouth, "Digital Video Segmentation", Proc. ACM Multimedia 94, pp.357-364, 1994

  In the first conventional method, it is possible to detect a sharp peak value in a flash portion, but there is a problem that it is difficult to detect with a continuous flash or a weak flash. Further, in the second conventional method, the change in the inter-frame luminance difference value is very difficult to distinguish from the case of motion, and the difference value does not always take a constant value in a dissolve with motion. . Furthermore, it can be said that the third conventional method is very sensitive to the movement of the camera and the object. Furthermore, in order to cope with various directions, it is necessary to obtain a differential value for each direction, and there is a problem that processing is complicated.

The object of the present invention is to solve the above-mentioned problems of the prior art, distinguish a cut point from a moving scene and detect a special cut point such as a wipe by using a method simpler than the conventional method. An object of the present invention is to provide a cut point detection device that can perform the above-described process.

To achieve the above object, the present invention comprises a moving image data input unit, a motion vector calculation unit, an inter-frame difference calculation unit, an instantaneous cut point detection unit, a dissolve detection unit, and a motion scene determination unit, and the instantaneous cut inspection When an instantaneous cut is detected by the output unit and the instantaneous cut is not determined, a dissolve candidate is detected by the dissolve detection unit, and the motion vector calculation unit and the inter-frame difference calculation unit Obtaining information and inter-frame difference values, the motion scene determination unit performs panning scene and motion scene detection on the dissolve candidate images using the motion vector information and inter-frame difference values, and the panning scene and If it is determined that it does not correspond to a moving scene, it is determined as a dissolve cut, and the panning scene If it is determined that the scene corresponds to a moving scene or if the dissolve detection unit does not determine a candidate for a dissolve, wipe cut detection is performed using a difference value between frames in a certain section and the sections before and after the section. The first feature is in the point of determination .

Further, the present invention, the motion vector calculation unit obtains the motion vector information of the blocks of an image frame, said motion scene determination unit, in the frame n, a first tooth size of the motion vector is predetermined greater than a second threshold value which is determined larger number of blocks than threshold in advance, and a scene motion when the interframe difference is large, and in the frame n, the motion vector magnitude is the first A panning scene is set when the screen average of a horizontal or vertical motion vector is greater than a predetermined third threshold, and the motion scene and the panning scene are distinguished from cut points. There is a second feature in this point.

  According to the present invention, a motion scene and a cut point can be discriminated with high accuracy, and a wipe scene can be detected with high accuracy. Therefore, cut detection can be performed with very high accuracy as compared with the prior art. For this reason, there is an effect that it is possible to reduce erroneous detection of a motion scene as a cut point. In addition, there is an effect that it is possible to detect a wipe scene change that has been difficult to detect.

  As an example, when a cut point detection for one hour of a TV program compressed with MPEG1 is performed using the present invention, the detection rate is 90% when the method disclosed in Japanese Patent Application No. 8-252333 is used. These scene changes could be detected with an accuracy of about 95%.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 shows a flowchart relating to the first embodiment of the present invention. First, one frame of an image is input by the image input 10, and an instantaneous cut is detected by the instantaneous cut detection 11. Here, the flash detection 12 detects whether or not the image is determined to be an instantaneous cut. If it is not a flash image, the instantaneous cut registration and display processing is performed on the instantaneous cut registration display 13 and the remaining frame processing 20 is performed. When the instantaneous cut is not determined by the instantaneous cut detection 11 or when the flash detection 12 is the flash detection, the basic dissolve detection 14 detects the dissolve image.

  If it is determined to be a dissolve, pan / motion detection 15 determines whether it is panning or a motion image. If not, the dissolve cut registration display 16 performs dissolve cut registration and display processing for the frame as a dissolve, and the remaining frame processing 20 is performed. If the pan / motion detection 15 determines that the image is panning or a motion image, the process proceeds to the wipe cut detection 18.

  For frames that are not detected by the basic dissolve detection 14, it is determined in the flat area dissolve detection 17 whether or not they are dissolved in the flat area. If it is determined to be a dissolve, the display shifts to a dissolve cut registration display 16. If it is not determined as a dissolve cut, the process proceeds to wipe cut detection 18. In the wipe cut detection 18, it is determined whether or not it is a wipe cut. If it is determined to be a wipe cut, the frame is registered as a wipe cut or displayed on the wipe cut registration display 19, and the process proceeds to the remaining frame processing 20. If the wipe cut is not determined, the remaining frame processing 20 is performed.

  Hereinafter, the process of detecting cut points from MPEG data will be described in detail with reference to FIG. Although FIG. 2 integrally forms the cut detection device, it is divided into FIGS. (A) to (d) for ease of explanation.

  The encoded data shown in FIG. 5A is input to the variable length decoding unit 21, and the variable length decoding unit 21 decodes the quantized two-dimensional DCT coefficient of each block, and the two-dimensional DCT coefficient is an average value. Input to the component extraction unit 22. As the average value component extraction method of the average value component extraction unit 22, for example, “Detection of cut points from compressed video data by simple decoding process” by Ushihara and Nakajima et al., Information Processing Society of Japan 51st National Convention, 6S-9 (1995), Japanese Patent Application No. 7-263681, and the like can be used.

  The average value component obtained by the average value component extraction unit 22 is input to the instantaneous cut point detection unit 23, and an instantaneous cut is detected. The method described in the above document can also be used for the instantaneous cut point detection method.

  In the flash, the screen changes greatly in units of one frame, so it is often erroneously detected as a scene change. Therefore, when the instantaneous cut point detection unit 23 determines that the cut is an instantaneous cut, it is input to the flash scene determination unit 2i shown in FIG.

  The luminance and color difference distribution in the frame where the flash occurs is completely different from that in the previous frame, but unlike the scene change, these distributions return to the original state in one to two frames after the flash scene. The flash scene model is shown in FIGS. FIG. 3 shows a single flash scene model, and FIG. 4 shows a continuous flash scene model.

  For example, when only frame n is a flash scene as shown in FIG. 3 (a), the correlation between frames n and n-1 is low (L in the figure), but the correlation between frames n + 1 and n-1 is high ( H) shown.

  Therefore, in the color difference histogram calculation unit 2g in FIG. 5B, a frame color difference histogram is calculated from the average value component data of the color difference signal input from the average value component extraction unit 22, and the calculation result is used as the color difference correlation calculation unit 2h. And input to the fifth memory 2j. In the color difference correlation calculation unit 2h, the color difference histogram correlation ρ (n, n-1) is calculated from the color difference histogram of the frame n + 1 inputted and the color difference histogram of the frame n and the frame n-1 previously stored in the fifth memory 2j. ), Ρ (n + 1, n−1) are obtained and input to the flash scene determination unit 2i.

  In addition, as the calculation method of the color difference histogram and the color difference correlation, for example, “Detection of cut from compressed image data by inter-frame luminance difference and color difference correlation” by Nakajima et al., IEICE Autumn Meeting D-501 (1994), The method disclosed in Japanese Patent Application No. 5-216895 or Japanese Patent Application No. 6-46561 can be used.

  As shown in FIG. 3A, when the frame n is a flash scene, the flash scene determination unit 2i has a low correlation ρ (n, n-1) between the frames n and n-1, but the frames n + 1 and Since the correlation ρ (n + 1, n−1) of the frame n−1 is high, the frame n can be determined as a flash scene when the following expression (1) is satisfied.

  ρ (n, n−1) <Th_fl, ρ (n + 1, n−1)> Th_fh (1)

  Here, Th_fl and Th_fh indicate the first and second threshold values, respectively, and Th_fl <Th_fh. The first and second threshold values are low values (for example, 0.7) that can be determined as cut points and high values (for example, 0.9) that can be determined as non-cut points, respectively.

  As shown in FIG. 3B, when the frame n-1 is a flash scene, the correlation between the frame n and the frame n-1 is low, but the correlation between the frame n and the frame n-2 is high. In addition, when flash scenes are continuously formed as shown in FIG. 4, it is possible to determine from the correlation with adjacent frames in the same manner.

  If it is determined that the scene is a flash scene (Yes in the flash scene determination unit 2i), the cut detection process is terminated, the process returns to the encoded data input shown in FIG. 5A, and the next frame is input. If it is determined that the scene is not a flash scene, the cut point image is registered and held in the cut point image holding unit 2d, and the cut point image is displayed in the cut point image display unit 2k. As for the display of the cut point image, the method disclosed in “Special Cut Point Detection Device for Moving Image” of Japanese Patent Application No. 8-252333 can be used.

  For an image that has not been determined to be an instantaneous cut by the instantaneous cut point detection unit 23, the basic dissolve detection unit 24 determines a dissolve image. As a method for determining a dissolve image, for example, a method disclosed in “Special image cut point detection device for moving image” of Japanese Patent Application No. 8-252333 can be used.

  Even when the basic dissolve detection unit 24 determines that a dissolve has occurred, a motion scene or panning scene may be detected in some cases. For example, as for the activity disclosed in Japanese Patent Application No. 8-252333, “Special Cut Point Detection Device for Moving Images”, both dissolve and motion scene show similar changes. However, in the motion scene, almost all the coding blocks are motion compensation blocks, and the interframe difference is also large. On the other hand, in the dissolve, the number of motion compensation blocks is small and the difference between frames is also small.

  Therefore, when the basic dissolve detection unit 24 determines that the motion is a dissolve, the motion scene determination unit 2a determines whether the scene is a motion scene. The motion scene determination unit 2a receives the motion vector MV of each block from the variable length decoding unit 21, and calculates the interframe difference Dn from the interframe difference calculation unit 2b. As for Dn, the block average value of the luminance signal of the frame n obtained from the method disclosed in Japanese Patent Application No. 5-216895 and the average value component extraction unit 22 is stored in the third memory 2c, and the third The cumulative sum of the absolute differences between frames of the block average value of the luminance signal can be calculated using the block average value of the frame n−1 obtained from the memory 2j.

  The motion scene determination unit 2 a determines a motion scene based on the interframe difference Dn obtained from the interframe difference calculation unit 2 b and the motion vector MV obtained from the variable length decoding unit 21. Many blocks are motion compensation blocks in motion scenes and panning scenes. For this, the following judgment formula (2) can be used.

  MVC, PMVC> Th_mvc (2)

  The Th_mvc is a threshold value, for example, 0.45MN.

  Here, MVC and PMVC are the number of motion vectors of the newest frame and the number of motion vectors of the second newest frame, respectively. For example, in the case of MPEG, it is possible to use the P picture closest to the input frame and the P picture closest to the next. However, the number of motion vectors counts the number of vectors whose motion vector size exceeds a threshold Th_mv (for example, 4 pel / frame). In addition, a motion scene has a large inter-frame difference. In the panning scene, many blocks have the same motion vector. Therefore, moving scenes and panning scenes can be discriminated using equations (3) and (4) below.

  Dn, Dn-1> Th_bm (3)

  Dn> Th_mm, | <mvx> | or | <mvy> |> Th_am (4)

  Here, Dn is an interframe difference between the frame n and the frame n-1, and Dn-1 is an interframe difference U10 between the frame n-1 and the frame n-2. Further, Th_bm, Th_mm, and Th_am are threshold values, and for example, Th_bm = 7.5MN, Th_mm = 1.5MN, Th_am = 5 pel / frame can be used. <mvx> and <mvy> indicate the screen average of the horizontal motion vector and the vertical motion vector.

  Therefore, when either (3) or (4) and (2) are satisfied, it can be determined that the scene is a moving scene or a panning scene.

  If the motion scene determination unit 2a determines that the scene is a motion scene, the wipe scene determination unit 2e in FIG. If it is determined that the scene is not a moving scene, it can be determined as a dissolve scene, and the process proceeds to the cut-point image holding unit 2d shown in FIG.

  When the basic dissolve detection unit 24 determines that it is not a dissolve, the process proceeds to the first flat portion determination unit 25. For example, when using the moving average difference of activities disclosed in Japanese Patent Application No. 8-252333 "Special image cut detection device for moving images", there is a frame in which the absolute value of the difference value exceeds a threshold value Judge as a dissolve. When a dissolve occurs in a flat part, the change in activity is very small, and the absolute value of the moving average difference value of the activity may not exceed the threshold value. However, in the dissolve section, the difference value changes while maintaining a value close to the threshold value. Accordingly, when the difference value has a certain size in a certain section and the integral value is large, the flat portion can be dissolved. In the case of a dissolve, as described in Japanese Patent Application No. 8-252333, a moving average difference of activities has a section having a positive value after a section having a negative value. Therefore, these sections can be determined using the following equations (5) to (7).

  In Equation (7), dh and dm are positive integers, and are constants that specify the size of the dissolve interval. Th_sa and Th_ea are threshold values for determination. For example, Th_sa = 70 and Th_ea = 1 can be used. DMV (t) is a moving average difference of activities in the screen, and the method disclosed in Japanese Patent Application No. 8-252333 can be used. Specifically, as shown in (d) of the figure, the in-screen activity calculation unit 2l calculates the in-screen activity and puts it in the moving average calculation unit 2m to move the activity in the screen. Find the average difference DMV (t). The in-screen activity is stored in the memory 2n.

  When the difference DMV (t) satisfies the expressions (5) and (6), the first flat part determining unit 25 can determine that the flat part is dissolved.

  When the first flat part determining unit 25 determines that the flat part is dissolved, the first flat part determining unit 25 inputs the result to the second flat part determining unit 27. On the other hand, if it is not determined to be a dissolve, the moving average difference DMV (t) of the activity in the screen during slow panning has a change similar to the dissolve in the flat part. May be detected. In slow panning, the amount of motion is small, the motion compensation prediction efficiency is high, and the prediction error is small. On the other hand, since two screens are synthesized in the dissolve, the motion compensation prediction efficiency is not so high. Therefore, it is possible to distinguish dissolve and panning according to the amount of prediction error.

  The DC component DCn (i, j) of the DCT coefficient of the coding prediction error in the block (i, j) of the frame n from the variable length decoding unit 21 is input to the normalized prediction error calculation unit 29, and the normalized prediction error NPEn is Desired. The normalized prediction error amount NPEn can be calculated based on the screen average value of the DC component as follows.

  Here, MN is the total number of blocks in the screen, and k is the number of inter-frame encoded blocks.

  The normalized prediction error amount NPEn is input to the second memory 28 and also input to the second flat portion determination unit 27. Since NPEn has a large value in the dissolve interval, the number of frames having a large prediction error can be determined as a dissolve interval by comparing with the threshold value. Therefore, it can be determined using the following formula.

  Here, PEfl is a flag indicating whether or not there is a frame having a large prediction error. PEfl is set to 1 when the normalized prediction error NPEl of the frame l is larger than a threshold Th_pe (for example, 3MN). In addition, a section before bd_p and dd_p dissolve and a section during dissolve are shown. As a result, when the expression (9) is satisfied, it can be determined as a dissolve in a flat portion. If it is determined to be a dissolve, the process proceeds to the cut point image holding unit 2d shown in FIG. 5B, and the cut point image is held and recorded. If it is not determined to be a dissolve, the process proceeds to the wipe scene determination unit 2e shown in FIG.

  The wipe scene change model is shown in FIG. FIG. 4A shows an example in which a new shot B appears on the current shot A while moving in the right direction. FIG. 6B shows an example in which the shot A gradually decreases in the horizontal direction while the shot B appears while expanding in the modified example of FIG. (C) and (d) are modified examples of FIG. (B), in which the scene shifts while being enlarged or reduced in the vertical direction. FIG. 4E shows an example in which the scene shifts by turning the page. The page of the current shot A is turned over, and the next shot B appears.

  In any wipe, the position of two shots changes while the scene changes, and before and after the change, each shot is stationary and stable unless there is a large movement in the shot. Furthermore, the position and shape of each shot during the wipe scene change slowly and constantly move. Accordingly, any wipe scene change between frames can be represented by a simple model as shown in FIG. FIG. 6 shows a wipe model using inter-frame differences.

  The inter-frame difference Dn obtained by the inter-frame difference calculation unit 2b in FIG. 2 (a) is input to the fourth memory 2f and the wipe scene determination unit 2e in FIG. 2 (c). The wipe scene determination unit 2e determines the wipe scene by comparing the inter-frame difference with the wipe model. Since the inter-frame difference is stable and large in the wipe section, and the inter-frame difference is small in the preceding and following sections, the following determination formula (11) can be used.

BW> Th_bw, DW> Th_dw, AW> Th_aw (11)
Here, BW, DW, and AW are the numbers of frames recognized as before, during, and after wiping, respectively, and Th_bw, Th_dw, and Th_aw are threshold values. As the threshold values Th_bw, Th_dw, Th_aw, for example, Th_bw = 27, Th_dw = 24, Th_aw = 5 can be used. BW, DW, and AW can be obtained using the following equation (12).

  DL (k) and DH (k) are flags indicating whether the inter-frame difference Dk in frame k is high or low, and can be determined using the following equation (13).

  if Dk> Th_wp then DL (k) = 0, DH (k) = 1, else DL (k) = 1, DH (k) = 0 (13)

  Here, for example, Th_wp = 1.36MN can be used as the threshold Th_wp.

  When the wipe scene determination unit 2e determines that a wipe scene is detected, the cut point image holding unit 2d and the cut point image display unit 2k in FIG. Move on to frame input processing. If it is not determined to be a wipe, the process directly proceeds to the next frame input process.

  In the above-described embodiment, the cut point detection method when compressed by MPEG has been described. However, the present invention is not limited to this, and cut points can be obtained by the same process even in uncompressed moving image data and other compression methods. Can be detected.

  In addition, the present invention can be partially combined, and it is also possible to combine with other cut point detection methods using only the dissolve detection part, or to combine only wipe scene detection with instantaneous cut detection.

It is a processing flowchart which shows the outline | summary of operation | movement of this invention. It is a block diagram which shows the structure of one Embodiment of this invention. It is explanatory drawing of a single flash scene model. It is explanatory drawing of a continuous flash scene model. It is explanatory drawing of a wipe model. It is explanatory drawing of the wipe model at the time of using the difference between frames.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 21 ... Variable length decoding part, 22 ... Average value component detection part, 23 ... Instantaneous cut point detection part, 24 ... Basic dissolve detection part, 25 ... Flat part determination part 1,26,28, 2c, 2f, 2j, 2h ... 1st to 6th memory, 27 ... flat part determination unit, 2a ... motion scene determination unit, 2d ... cut point image holding unit, 2e ... wipe scene determination unit, 2g ... color difference histogram calculation unit, 2h ... color difference correlation Arithmetic unit, 2i... Flash scene determining unit, 2k... Cut point image display unit, 2l.

Claims (3)

A moving image data input unit, a motion vector calculation unit, an inter-frame difference calculation unit, an instantaneous cut point detection unit, a dissolve detection unit, and a motion scene determination unit,
When the instantaneous cut point detection unit detects an instantaneous cut and is not determined to be an instantaneous cut, the dissolve detection unit detects a dissolve candidate,
The motion vector calculation unit and the inter-frame difference calculation unit obtain motion vector information and inter-frame difference values for moving image data,
The motion scene determination unit performs panning scene and motion scene detection on the dissolve candidate image using the motion vector information and inter-frame difference values, and is determined not to correspond to the panning scene and motion scene. If it is determined as a dissolve cut and it is determined that the scene corresponds to the panning scene and the motion scene, or if it is not determined as a dissolve candidate by the dissolve detection unit, the difference between frames in a certain section and the sections before and after the section. An apparatus for detecting a cut point of a moving image, wherein wipe cut detection is performed using a value to determine wipe cut . The moving image cut point detection device according to claim 1,
The motion vector calculation unit obtains motion vector information of a block of an image frame,
The motion scene determination unit has a predetermined number of blocks having a motion vector magnitude greater than a predetermined first threshold value in frame n for a dissolve candidate image detected by the dissolve detection unit. A motion scene is set when the difference is larger than a predetermined second threshold value and the inter-frame difference is large, and the magnitude of the motion vector is larger than the first threshold value and horizontal in the frame n. Alternatively, when the screen average of the motion vector in the vertical direction is larger than a predetermined third threshold value, a panning scene is set, and the motion scene and the panning scene are distinguished from cut points, and cut inspection of a moving image is characterized. Out device. In the moving image cut point detection device according to claim 1 or 2 ,
The number of frames having an inter-frame difference value larger than the fourth threshold value in a certain section is greater than a predetermined fifth threshold value, and the inter-frame difference value is at least in the fourth section before and after the section. An apparatus for detecting a cut point of a moving image, wherein a wipe is determined when the number of frames smaller than a predetermined threshold is greater than a predetermined sixth threshold.

Images