JP4895315B2 - Moving image cut point detection device - Google Patents

Description

  The present invention relates to a moving image cut point detection device, and more particularly, to a moving image cut point detection device capable of detecting cut points in a moving image with high accuracy.

  Many techniques have been developed for detecting cut points from moving image data, that is, cut point detection. In particular, cut points are detected for moving image data compressed by moving image encoding such as MPEG. There are many techniques. Here, a shot means a series of image sequences taken by one camera operation.

  For example, Patent Document 1 discloses a method of sampling a time-series frame image in a video, obtaining an edge intensity coefficient for each frame image unit, and detecting a video change point using its temporal change characteristics. ing.

  Patent Document 2 discloses a method for detecting an instantaneous cut point or a special cut point in a DC image obtained from variable length decoding and simple motion compensation by using a luminance difference, a color difference histogram correlation, a first-order differential coefficient of in-screen activity, and the like. Has been.

  Further, in Patent Document 3, in order to detect candidate sections for dissolve and fade, a difference in color distribution based on a color histogram, a temporal and spatial distribution of macroblocks, a temporal and spatial distribution of motion vectors, and an edge A method of selecting and using either a temporal and spatial distribution or a change characteristic is disclosed.

JP-A-7-288840 Japanese Patent Laid-Open No. 10-224741 JP 2001-285712 A

  Up to now, many cut point detection techniques have been developed that use compressed moving image data such as MPEG and use parameters obtained in a compressed data area such as a motion vector and macroblock type without decoding moving image data. However, those techniques have a problem that it is difficult to accurately detect a special cut point indicating a dissolve or fade section, and further an instantaneous cut point.

  An object of the present invention is to provide a moving image cut point detection apparatus capable of accurately detecting a special cut point indicating a dissolve or fade section and further an instantaneous cut point in order to solve the above-described problems.

In order to solve the above problems, the present invention provides moving image data input means for inputting moving image data, color arrangement information extraction means for extracting color arrangement information in a screen from the moving image data, and the color arrangement information extraction. The change of the color arrangement information extracted by the means is observed for a certain time or more, and the section where the difference from the adjacent frame of the color arrangement information changes in a convex shape is defined as a dissolve or fade section, and a special cut point indicating the section is detected. A special cut point detecting means is provided.

  In addition, the present invention further includes an instantaneous cut point detection unit, wherein the instantaneous cut point detection unit detects a point at which color arrangement information between adjacent frames changes to a predetermined threshold or more as a cut point. .

  In addition, the present invention is characterized in that the color arrangement information is composed of a conversion coefficient obtained by converting a degenerate image into a frequency domain.

  Further, the present invention is characterized in that the conversion to the frequency domain is a discrete cosine transform.

  Furthermore, the present invention is characterized in that the moving image data is compression-encoded data, and the color arrangement information extracting means extracts color arrangement information from only a DC component of the moving image data.

  In the present invention, a change in color arrangement information is observed for a certain time or more, and a special cut point indicating a dissolve or fade section is detected from the nature of the difference between the color arrangement information and an adjacent frame. A special cut point indicating a section can be detected with high accuracy. Also, the instantaneous cut point can be detected from the change in the color arrangement information. Here, the color arrangement information can be extracted from the DC component of the moving image data at high speed without decoding the compressed moving image data to the pixel level.

It is a functional block diagram showing cut point detection processing using an edge feature. It is a flowchart which shows operation | movement of the special cut point detection process part of FIG. It is a functional block diagram which shows the cut point detection process in this invention. It is a functional block diagram showing cut point detection processing using both edge features and color arrangement information. It is a flowchart which shows operation | movement of the special cut point detection process part 15 of FIG. It is a functional block diagram which shows the instantaneous cut point detection process between fields. 7 is a flowchart illustrating an example of an inter-field instantaneous cut point detection operation in FIG. 6. 7 is a flowchart showing another example of the inter-field instantaneous cut point detection operation in FIG. 6. It is a functional block diagram which shows a flash area detection process. It is explanatory drawing of operation | movement (end determination) of a flash removal process part. It is explanatory drawing of operation | movement (continuation determination) of a flash removal process part. FIG. 10 is a flowchart showing an instantaneous cut point detection and flash section detection operation in FIG. 9. FIG. It is a functional block diagram which shows the cut point detection process which combined FIG.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing cut point detection processing using edge features. In the following, description will be made on the premise of moving image data compressed by MPEG, but the present invention is not limited to this, and can also be applied to moving image data by a similar compression method.

  First, MPEG moving image data (hereinafter simply referred to as moving image data) is input to the moving image data input unit 10. The moving image data input unit 10 includes a variable length decoding unit 11 and a simple motion compensation unit 12, and outputs a DC component of moving image data by variable length decoding and simple motion compensation of the input moving image data.

  An image composed of DC components of moving image data corresponds to 1/8 of the original image in the horizontal and vertical directions, and can be obtained without decoding up to the pixel level. By using this DC component as a processing target, the processing load can be reduced and the processing speed can be increased. Note that variable-length decoding of moving image data and simple motion compensation are described in Patent Document 2 and Japanese Patent Application Laid-Open No. 10-79957, and the description thereof is omitted here because the technique can be used.

  The luminance / color difference feature extraction unit 13 based on the DC component of the moving image data, the luminance difference D between frames, the color difference histogram correlation ρ, the in-screen activity FA and its first derivative DA, the normalized motion compensation prediction error NPE, etc. Ask for.

  The edge feature extraction unit 14 extracts an edge feature in the screen from the DC component of the moving image data. The instantaneous cut point detection processing unit 15 detects an instantaneous cut point. This instantaneous cut point can be detected based on various types of information extracted by the luminance / color difference feature extraction unit 13, changes in edge features extracted by the edge feature extraction unit 14 between adjacent frames, and the like.

  The special cut point detection processing unit 16 detects a special cut point including a dissolve and a fade by using the temporal change of the edge feature extracted by the edge feature extraction unit 14. The detection of the special cut point here uses the property that the edge component in the screen becomes weak in the dissolve and fade sections.

  The edge features extracted by the edge feature extraction unit 14 include an edge power EP distributed in the screen, an edge histogram distributed in the screen, an edge histogram correlation ε between the input screen and a screen positioned before it in time, etc. There is one of them.

  An edge histogram as an edge feature can be calculated by applying filters in any number of directions to the entire screen. Further, the edge histogram for each divided region can be calculated by dividing the screen into a plurality of small regions and applying filters in an arbitrary number of directions to each divided region. That is, the unit region for calculating the edge histogram may be the entire screen or a divided region obtained by dividing the entire screen by an arbitrary number. Furthermore, the size of the small area to be divided may be arbitrarily set or may be dynamically changed according to the position and characteristics in the screen.

  Specifically, the edge histogram descriptor defined in MPEG-7 can be used, and the entire screen is divided into 4 × 4 areas, and each divided area is filtered in five directions. An edge histogram can be obtained.

  The instantaneous cut point detection processing unit 15 and the special cut point detection processing unit 16 output information on the instantaneous cut point and the special cut point as a time code, and the key frame generation unit 17 starts from each shot based on the time code. Generate keyframes. As the key frame generation unit 17, a technique described in Japanese Patent Laid-Open No. 2001-258038 can be used. The key frame display unit 18 displays the key frame generated by the key frame generation unit 17. This display allows the user to browse the shot list.

  FIG. 2 is a flowchart showing the operation of the special cut point detection processing unit 16 of FIG. In the figure, first, i = 0 is set by initialization processing (S21), and moving image data of frame n + i is input (S22). Next, i is sequentially incremented by 1 (S25), and successive frames are sequentially input. At each frame input, the primary differential coefficient DA of the in-screen activity FA is equal to or greater than a threshold TH_DA, and the primary differential coefficient DA. Is convex in time, and the normalized motion compensation prediction error NPE is greater than or equal to a certain threshold TH_NPE, and it is determined whether or not the edge power EP is concave (S23).

  If the determination result in S23 is affirmative (Y), the frame n + i is set as a dissolve candidate (S24). If the determination result in S23 is negative (N), it is determined whether or not the value of i is greater than or equal to a certain threshold N_dis1 (S26). S26 is provided to determine that the section is a dissolve section when the condition of S23 is satisfied over a predetermined frame section, thereby preventing erroneous detection.

  If the value of i is less than N_dis1, the process is terminated without determining that it is a dissolve. If the value of i is N_dis1 or more, the section from frame n to frame n + i−1 is determined as a dissolve section (S27). .

  FIG. 3 is a functional block diagram showing cut point detection processing in the present invention. In the present invention, the special cut point is detected by using the color arrangement information, thereby enabling highly accurate detection. In FIG. 3, the same or equivalent parts as in FIG.

  First, as in FIG. 1, moving image data is input to the moving image data input unit 10 and a DC component of the moving image data is output. The luminance / color difference feature extraction unit 13 based on the DC component of the moving image data, the luminance difference D between frames, the color difference histogram correlation ρ, the in-screen activity FA and its first derivative DA, the normalized motion compensation prediction error NPE, etc. Ask for.

  The color arrangement information extraction unit 31 extracts color arrangement information in the screen from the DC component of the moving image data. The instantaneous cut point detection processing unit 32 detects an instantaneous cut point using a temporal change in the color arrangement information extracted by the color arrangement information extraction unit 31. The special cut point detection processing unit 15 detects a special cut point by observing a temporal change in the color arrangement information extracted by the color arrangement information extraction unit 31 for a predetermined time or more.

  The detection of the instantaneous cut point here uses the property that the color arrangement information between adjacent frames changes greatly at the cut point. Further, the detection of the special cut point uses the property that the difference between the color arrangement information and the adjacent frame changes to a convex shape in the case of the special cut point. The instantaneous cut point detection processing unit 15 and the special cut point detection processing unit 16 may detect the instantaneous cut point and the special cut point by using various information extracted by the luminance / color difference feature extraction unit 13 together.

  As the color arrangement information, a color layout descriptor defined in MPEG-7 can be used. In the color layout descriptor, the original image is reduced to 8 × 8, the reduced image is converted into the frequency domain by discrete cosine transformation, and a set of the conversion coefficients can be used as color arrangement information. As described above, by extracting the color layout descriptor by further reducing the DC component of the moving image data instead of the original image, the decoding processing of the compressed moving image data such as MPEG can be omitted.

  FIG. 4 is a functional block diagram showing cut point detection processing using both edge features and color arrangement information. For this purpose, both the edge feature extraction unit 14 and the color arrangement information extraction unit 31 are provided.

  FIG. 5 is a flowchart showing the operation of the special cut point detection processing unit 15 of FIG. This flowchart differs from FIG. 2 in that S51 is used instead of S23. In S51, i is sequentially incremented by 1 (S25), and successive frames are sequentially input. In these frames, the edge power EP is below a certain threshold TH_EP, and the normalized motion compensation error NPE or color arrangement information Whether or not the difference CL with the immediately preceding frame is greater than or equal to a threshold TH_NPE or TH_CL, the first derivative DA of the intra-screen activity FA is less than or equal to a threshold TH_DA, and whether or not the edge histogram correlation ε is greater than or equal to a threshold TH_ε. judge. Others are the same as in FIG.

  Next, inter-field instantaneous cut point detection will be described. Since a television image has a field structure that is displayed every 1/60 seconds, if this is encoded with a progressive structure, instantaneous cut points may occur between the fields. MPEG-2 or the like has an encoding algorithm corresponding to the field structure, but MPEG-1 or the like only supports a frame structure (progressive). Inter-field instantaneous cut points cannot be detected by ordinary instantaneous cut point detection means in many cases, so that specialized detection means are required. In the fourth embodiment, an instantaneous cut point between fields in a progressive moving image can be detected using edge features such as an edge histogram correlation ε.

  FIG. 6 is a functional block diagram showing the inter-field instantaneous cut point detection process. Here, the instantaneous cut point detection processing unit 15 detects an inter-field instantaneous cut point in accordance with the following FIG. 7 or FIG.

  FIG. 7 is a flowchart showing an example of the inter-field instantaneous cut point detection operation in FIG. First, i = 0 is set by the initialization process (S21), and moving image data of frame n is input (S22). In S71, it is determined whether or not the luminance difference D is equal to or greater than a threshold TH_D2 and the edge histogram correlation ε is equal to or less than a threshold TH_ε in adjacent frames.

  If this determination result is negative (N), the process ends as it is. If the determination is affirmative (Y), i is incremented by 1 (S72), and then it is determined whether i = 2 (S73). If i = 2 is not satisfied, n is incremented by 1 (S74), and the process returns to S22. If i = 2, it is determined that there is an inter-field instantaneous cut point in frame n (S75). The inter-field instantaneous cut point is detected when two frames that satisfy the condition in S71 appear in succession by the above operation.

  FIG. 8 is a flowchart showing another example of the inter-field instantaneous cut point detection operation in FIG. First, moving images of three consecutive frames n-2, n-1, and n are input (S81), and these frames are processed. Next, in S82, the color difference histogram correlation ρ between the currently focused frame n and the frame n-2 two frames before is less than a certain threshold TH_ρ2, and the frame n and the immediately preceding frame n−. It is determined whether or not the luminance difference D from 1 is greater than or equal to a threshold TH_D (S82). If the determination result is affirmative (Y), it is determined that there is an inter-field instantaneous cut point in frame n (S83), and if the determination is negative, the process ends.

  Next, flash section detection will be described. In the flash section detection, a flash section over a plurality of frames where the flash is emitted is detected using the detected luminance difference D, color difference histogram correlation ρ, and edge feature of successive frames.

  Means for detecting a flash included in moving image data is disclosed in Patent Document 2 and the like. However, in Patent Document 2, since only the color difference histogram correlation is used, the entire image becomes whitish or particularly over a plurality of frames. It is difficult to detect flashes that appear continuously.

  FIG. 9 is a functional block diagram showing flash section detection processing. Here, the flash section that continuously appears over a plurality of frames is detected, and a flash removal processing section 91 that removes the flash section is provided. The special cut point detection processing section 16 stores the data from which the flash section has been removed. A special cut point is detected as a processing target.

  10 and 11 are explanatory diagrams of the operation of the flash removal processing unit 91. FIG. A flash interval over a plurality of frames is detected as follows. Here, it is assumed that frames n−1 to n + 5 are sequentially input and the flash is emitted over frames n to n + 4.

  FIG. 10 is an explanatory diagram of determination of the end of the flash section over a plurality of frames. The frame n is detected by the instantaneous cut point detection processing unit 15 when the luminance difference D and the color difference histogram correlation ρ reach a peak, and becomes a flash section start candidate. The flash removal processing unit 91 observes the edge histogram correlation ε and the color difference histogram correlation ρ between the frame n + 3 and the frame n−1 after this frame n.

  The edge histogram correlation ε and the color difference histogram correlation ρ are equal to or greater than a certain threshold in the frame n + 5. The flash removal processing unit 91 determines that frames n to n + 4 are flash sections. Generally, in the frame n + F (F = 3, 4, 5,...), When both the edge histogram correlation ε and the chrominance histogram correlation ρ are equal to or greater than a certain threshold, the section from the frame n to n + F−1 is determined. Judge as a flash interval.

  FIG. 11 is an explanatory diagram of determination of continuation of the flash section over a plurality of frames. Using frame n as a flash section start candidate, edge histogram correlation ε and color difference histogram correlation ρ with frame n−1 are calculated after frame n + 3. Next, when any one of these values is equal to or less than a certain threshold value, the color difference histogram correlation ρ between the current frame and the previous frame and the previous frame is calculated. If both of the values of the color difference histogram correlation ρ are equal to or greater than a certain threshold value, the flash section is regarded as a continuous flash section. If any of these values falls below a certain threshold value, the frame is regarded as a frame immediately after the end of the flash section. To do.

  In general, frame n is set as a flash section start candidate, and edge histogram correlation ε and color difference histogram correlation ρ are calculated in frame n−1 and frame n + C (C = 1, 2, 3,...). If any of these values is below a certain threshold value, the color difference histogram correlation ρ is calculated between frame n + C and frame n + C−1 and between frames n + C and n + C−2. If both the values of the color difference histogram correlation ρ are equal to or greater than a certain threshold value, the flash section is regarded as a continuous flash section. If this value falls below a certain threshold value, the flash section continuation is terminated. This is performed in the specified frame Fmax.

  FIG. 12 is a flowchart showing the instantaneous cut point detection and flash section detection operations in FIG. First, n = 2 and C = 3 (S121), and moving image data of frame n-1 and frame n is input (S122). Next, it is determined whether or not the luminance difference D and the color difference histogram correlation ρ peak in the frame n (S123).

  If the determination result in S123 is negative (N), n is incremented by 1 (S124) and the process returns to S122. If affirmative (Y), frame n is set as a flash section start candidate (S125). The above is the processing in the instantaneous cut point detection processing unit 15.

  Next, the continuation determination of the flash section is performed. First, the frame n + C is input (S126). Next, in S127, edge histogram correlation ε and color difference histogram correlation ρ are calculated in frame n-1 and frame n + C, and it is determined whether or not these values are both greater than a certain threshold TH_ε2, TH_ρ3 (S127).

  If the determination result in S127 is negative (N), the color difference histogram correlation ρ is further calculated between the frame n + C-1 and the frame n + C and between the frame n + C-2 and the frame n + C. It is determined whether or not both are above a certain threshold value and TH_ρ4 or more (S128).

  If the determination result is affirmative (Y), it is regarded as a continuous flash period, and n is incremented by 1 (S129), and the process returns to S126. If the determination result is negative (N), and if the determination result in S127 is affirmative (Y), frames n to n + C-1 are determined to be continuous flash sections (S130).

  FIG. 13 is a functional block diagram illustrating cut point detection processing in which all of FIGS. 1 to 12 are combined. Since the operation of each processing unit is the same as described above, description thereof is omitted. Thus, the cut point can be detected by appropriately combining one or more other cut point detection processes with the cut point detection process of FIG.

  According to the present invention, cut point detection for enabling effective search and editing in moving image data compressed by MPEG or the like can be achieved with high accuracy. For example, when compared with the means disclosed in Patent Document 2 in an experiment for several hours of US news video, an average of 10% or more is obtained by using the cut point detection device according to the present invention. The improvement in recall was seen up to about 20%.

  DESCRIPTION OF SYMBOLS 10 ... Moving image data input part, 11 ... Variable length decoding part, 12 ... Simple motion compensation part, 13 ... Luminance / color difference feature extraction part, 14 ... Edge feature extraction part, 15 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ··· ..Flash removal processing section

Claims (5)

Moving image data input means for inputting moving image data;
Color arrangement information extracting means for extracting color arrangement information in the screen from the moving image data;
A change in the color arrangement information extracted by the color arrangement information extraction means is observed for a certain time or more, and a section where the difference from the adjacent frame of the color arrangement information changes in a convex shape is defined as a dissolve or fade section, and a special section indicating the section A moving image cut point detecting device comprising a special cut point detecting means for detecting a cut point. Furthermore, an instantaneous cut point detection means is provided,
2. The moving image cut point detection apparatus according to claim 1, wherein the instantaneous cut point detection unit detects a point at which color arrangement information between adjacent frames changes to a predetermined threshold value or more as an instantaneous cut point.   3. The moving image cut point detection device according to claim 1, wherein the color arrangement information includes a conversion coefficient obtained by converting a reduced image into a frequency domain. 4.   4. The moving image cut point detection apparatus according to claim 3, wherein the transformation into the frequency domain is a discrete cosine transformation.   5. The moving image data according to claim 1, wherein the moving image data is compression-coded data, and the color arrangement information extracting unit extracts color arrangement information from only a DC component of the moving image data. The moving image cut point detection device according to claim 1.

Images