JP3377678B2 - Encoded video cut detection method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for detecting a cut from coded video data, and more particularly to a cut detection method suitable for constructing a user interface for a coded video playback device, an editing device, or the like. .

[0002]

2. Description of the Related Art Various methods have been proposed for detecting a cut, which is a change of a scene from a video (a change of cameras or a connection between edits). When a cut can be detected, one or more representative screens are extracted from the shots (one scene) separated by the cuts and displayed in one column so that the video is not viewed in chronological order from the beginning to the end. Can understand the outline of the video and can configure a user interface that allows quick access to the required scene.

The conventional cut detection method mainly targets unencoded video data, calculates the correlation between adjacent frames, and regards a portion where the correlation is small as a cut. To calculate the correlation between frames in the same manner for encoded video data,
The problem is that a time-consuming decoding process is necessary.

In view of these problems, some methods have been proposed for detecting a cut directly from encoded data without going through a decoding process.

Scene change detection device (Japanese Patent Laid-Open No. 6-22
No. 304), a feature amount (cumulative value of prediction error, data amount of encoded data, number of pixels encoded in a frame, etc.) that can be calculated at relatively high speed from encoded data of an inter-coded frame is disclosed. A method for detecting a cut based on the above is disclosed.

[0006]

However, the conventional method for detecting a cut based on the feature amount that can be calculated at a relatively high speed from the above coded data is (1) an intra-coded frame (correlation within the screen is The problem that a cut cannot be correctly detected for video data in which mixed frames (compressed by using) and inter-coded frames (frames compressed by using correlation between screens) are mixed,
(2) There is a problem in that it takes a lot of calculation time to calculate the above feature amount for each frame.

The problems (1) and (2) will be described below in order.

First, the problem (1) will be described by taking MPEG encoded video as an example. In MPEG, the correlation between an intra-screen coded frame (I picture) to be encoded and the past reference frame and the target frame is used by using only the intra-screen correlation (without using information other than the target frame). Inter-picture forward-direction coded frame (P picture) coded by utilizing the above, and inter-screen bidirectional code encoded by utilizing the correlation between the past and future two reference frames and the target frame Video is coded by combining coded frames (B pictures). For example, I, P, B pictures alternately appear, such as I, B, B, P, B, B, P, B, B, I, B, .... M
In the PEG standard, the order of I, P, and B and the spacing can be freely set under certain restrictions. Considering encoded video data P ₁ , P ₂ , P ₃ , P ₄ , P ₅ , P ₆ , ... Consisting of only P pictures, if there is a cut at the time of frame P ₃ , the frame P ₂ and the frame Since the correlation between P ₃ becomes small,
The feature amount (the number of pixels encoded in a frame, the amount of encoded data, etc.) disclosed in the above publication increases. Therefore, the cut can be correctly detected by thresholding the feature amount. However, for images I ₁ , P ₁ , P ₂ , P ₃ , P ₄ , P ₅ , I ₂ , ... Compressed by a combination of I and P pictures, P ₁ , P ₂ , P ₃ , P ₄ and P ₅ , if there is a cut, the cut can be detected correctly, but if there is a cut between P ₅ and I ₂ , it cannot be detected.
Because I ₂ is not coded by utilizing the correlation between frames, the feature value such as the cumulative value of prediction errors and the number of pixels coded in a frame has no meaning, and the data of coded data Since the amount always takes a larger value than that of the P picture, the cut cannot be correctly detected based on such a feature amount.

Next, the problem (2) will be described. In order to calculate the above feature amount (excluding the case where the encoded data amount is the feature amount) from the encoded data, the variable length coding method (value that appears at high frequency) for all inter-coded frames Since it is necessary to expand the data compressed by the method of allocating a short code to (2), it takes a lot of calculation time (especially when the above method is realized by software) is a problem.

As described above, the conventional technique has a problem that it takes a long time for the decoding process and the variable length code expansion process. Among the conventional techniques, the method of using the data amount of encoded data is fast because it does not require variable-length code expansion processing, but for encoded video in which intra-frame encoded frames and inter-frame encoded frames are mixed. Had a problem that it could not detect the cut correctly.

The present invention has been made to solve the above problems, and can detect a cut at high speed and accurately even in a coded video in which intra-frame coded frames and inter-frame coded frames are mixed. It is an object to provide a cut detection method.

[0012]

In order to achieve the above object, the present invention is an intra-frame coded frame that is coded using intra-frame correlation and is coded using inter-frame correlation. A method for detecting a cut from a coded video including an inter-screen coded frame, wherein a section S in which the cut may exist is estimated based on a feature amount obtained from the coded data of the intra-coded frame. And a step of determining the presence / absence of a cut in the inter-coded frame included in the section S based on the feature amount obtained from the coded data.

In the step of determining the presence or absence of the cut, the amount of coded data of the inter-frame coded frame included in the section S is used as a feature amount, and the screen before or after the inter-frame coded frame or either one of the frames is displayed. A process of estimating a section S ′ included in a section S in which a cut may exist by comparing with a feature amount of the inter-coded frame, and coded data for the inter-coded frame included in the section S ′ And a step of determining the presence or absence of a cut based on the feature amount obtained from

Further, in the process of determining the presence / absence of a cut on the inter-coded frame included in the section S ′ based on the feature amount obtained from the coded data, the coded frame included in the section S ′ is displayed on the screen. In the case of an inter-coded frame and an intra-frame coded frame, the presence or absence of a cut is determined based on a feature amount obtained by decoding the coded frame.

In the present invention, a section S in which a cut may exist is obtained based on the feature amount obtained from the coded data of the intra-frame coded frame, and the inter-frame coded frame included in the section S is calculated. Only by calculating the feature amount, the time required for decoding and variable length code expansion processing is reduced. At the same time, the cut is detected based on the feature amount calculated from the inter-frame coded frame, so that the cut can be correctly detected even when the intra-frame coded frame and the inter-frame coded frame are mixed.

Further, in the above description, a section in which a cut exists is narrowed down by using the "coded data amount" constituting the inter-screen encoded frame included in the section S as a feature amount, and the inter-screen code included in the section S'is selected. The variable-length code expansion process is performed only on the encoded frames, and the cut detection process based on the feature amount of the inter-coded frames is performed, thereby reducing the calculation time associated with the variable-length code expansion process and further cutting. Make detection faster.

Further, in the above, when the coded frames included in the section S'are the inter-coded frames and the intra-frame coded frames, only these coded frames are decoded to detect the cut. By performing the cut detection at higher speed, the cut detection can be made more accurate when the intra-frame coded frame and the inter-frame coded frame are mixed.

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

1, 2, 3 and 4 are process flow charts of an embodiment of the present invention. The processing starts from the main routine shown in FIG. 1, and the cut is detected while calling the subroutines PORC (b), (c) and (d) shown in FIGS. Here, as an example of the frame sequence, I
Consider that pictures and P-pictures appear in the following order:

I _t-1 , P _t , ₁ , P _t , ₂ , P _t , ₃ , ...
P _t , ₁₄ , I _t , P _{t + 1} , ₁ , P _{t + 1} , ₂ , ..., P _{t + 1} , ₁₄ ,
I _{t + 1} , ... (T = 0, 1, 2, ...) That is, it is a picture sequence in which 14 P pictures follow an I picture.

<Main Routine> The main routine shown in FIG. 1 will be described.

First, for each block from the I picture I _t , D
The DC component of the CT coefficient is extracted and set as i _t (x, y) (step 100). Here, x and y are variables that represent the spatial position of the block in the frame, and x = 0,
1, ..., NX-1; y = 0, 1, 2, ..., NY-1 (N
X and NY represent the numbers of blocks in the horizontal and vertical directions, respectively. The DC component of the DCT coefficient is the average value of the luminance and color difference within the square block. Therefore, i _t
(X, y) is an image obtained by reducing the original image.

Next, the previously calculated I picture I
The difference d _t (x, y) from the direct _- current component it _-1 (x, y) of _{t-1.
= I t (x, y)} -i t-1 (x, y) (- 255 ≦ d
_t (x, y) ≤ 255) is calculated (step 101),
The number of pixels D _t (0 ≦ D _t ≦ NX · NY) whose absolute value | d _t (x, y) | of the difference is larger than the threshold value T ₁ is counted (step 102). D _t is an amount for evaluating the difference in pattern between two I pictures. That is, the two frames I _t and I _t-1
When the pattern is significantly different, D _t has a large value.
Other feature amounts may be used for the purpose of evaluating the design. For example, the color histogram difference and other statistics may be used.

Next, the number of pixels D _t is compared with the threshold value T ₂ (step 103), and when it is larger than the threshold value, it is determined that there is a high possibility that there is a cut between I _t and I _t-1 . Call the subroutine Proc (b) for more detailed examination (step 104); otherwise, determine "no cut", add 1 to the variable t (step 105), and reach the final frame of the video. If (step 106),
The processing moves to step 100.

<Subroutine Proc (b)> Subsequently,
Subroutine Proc (b) using the flowchart of FIG.
The process will be described. In this subroutine, I _t-1
Based on the data amount of the coded data of the 14 P-pictures between I and I _t , the section in which the cut may exist is further narrowed down.

First, in step 110, i = 1, 2, ...,
Respectively calculating a P-picture P _t, the data amount L _i of _i of coded data to 14. For convenience, L ₀ = 0 is set. i
In order to perform the processing of steps 112 to 116 for = 1, 2, ..., 14, i = 1 is initialized (step 11
1).

Next, at step 112, the condition L _i -max
It is checked whether (L _i−1 , L _{t + 1} )> T ₃ is satisfied, and if the condition is satisfied, it is determined that there is a high possibility that a cut exists in P _t , _i . Because, if there is a cut between P _t , _i-1 and P _t , _i , the pattern changes greatly between them, so the correlation between frames becomes small, and efficient compression is achieved by the method using the correlation between frames. This is because the amount of data L _i tends to be larger than that of the other frames (L _i−1 , L _{t + 1} ). If the above condition is satisfied in step 112, the subroutine Proc (c) is called to detect the cut in detail (step 113).
As a result, when Proc (c) returns the result “with cut”, the process is terminated and the process returns to the main routine (step 114). Otherwise, i is incremented by 1 (step 115), and i is If it is 14 or less, the process returns to step 112 (step 116).

When i reaches 15, it means that no cut was detected in the P picture, but P _t , ₁₄
There is still the possibility that there is a cut between I _t and I _t . To detect it, the subroutine Proc (d) is called (step 117) and the process returns to the main routine.

<Subroutine Proc (c)> The processing of the subroutine Proc (c) will be described with reference to the flowchart of FIG. This subroutine is for detecting a cut from a section where a cut may exist.

First, the variable length code of the P picture P _t , _i of interest is developed and the information called macroblock type is read out to count the intra-coded blocks, and B _i (0 ≦ B _i ≦ NB)
(NB is the number of blocks) is calculated (step 120).
When B _i is larger than a predetermined threshold value T ₄ (step 121), “with cut” is output (step 1).
22), otherwise output "no cut" (step 123). This is because if there is a cut between P _t , _i−1 and P _t , _i , the correlation between frames becomes small, and therefore the number of intra-coded blocks B _i tends to increase.

<Subroutine Proc (d)> The processing of the subroutine Proc (d) will be described with reference to the flowchart of FIG. This subroutine is for determining whether or not there is a cut between P _t , ₁₄ and I _t .

First, the frame images of I _t , P _t and ₁₄ are decoded (step 130). In order to decode P _t , ₁₄ , _d is traced back to It _-1 and P _t , ₁ , P _t , ₂ , P _t , ₃ , ..., P _t ,
_Although it takes calculation time because the decoding process needs to be performed in the order of _14, the number of times the subroutine Proc (d) is called is smaller than that of Proc (b) and Proc (c), so the calculation time consumption rate is large. Absent. Next, I _t and P _{t, 14} frame difference d _t (x, y) between = i
Calculate _t (x, y) -i _t-1 (x, y) (step 13
1) The absolute value of the difference | dt (x, y) | counts the number of pixels larger than a certain threshold T ₅ (step 13
2) Compare the count value D _t with the threshold value T ₆ (step 13
3) If the count value is greater than the threshold value, it is determined that there is a cut between I _t and P _t , ₁₄ (step 134), otherwise it is determined that there is no cut (step 135), and the subroutine ends. Instead of this subroutine, when exceeds a certain threshold D _t determined in the main routine (takes a value greater than the threshold T _2), so as to determine that a cut between I _t and P _{t, 14} Good.

FIG. 5 shows how the calculation amount is reduced in the above embodiment. Reference numeral 200 is a schematic representation of the original frame sequence, and it is assumed that the black wide line represents an I picture and the white wide line represents a P picture, in which cuts are present at two positions.

A section (210, 211, 212) in which the condition of step 103 of the main routine is satisfied is determined as a section S having a high possibility that a cut exists. Section 21
Cuts are included in 0 and 211, but no cut is included in the section 212. The shaded section in 213 is
It is the section that was judged to have no cut in the main routine and was eliminated. Most of the sequence of frames is screened out at this stage, a time-consuming subroutine Pro
It is no longer necessary to perform the processes of c (b), (c), and (d) (cuts appear at a rate of one to several tens of seconds on average, so many sections are eliminated at this stage).
Further, the section in which the cut exists is further narrowed down by the subroutine Proc (b), and the section S '(220, 221, 2)
22) is obtained. The frame in which the condition in step 112 of the subroutine Proc (b) is satisfied is 220, and the frames in which the condition in step 116 is not satisfied are 221 and 22.
It is 2. At this stage, the section is further narrowed down. The subroutine Proc (c) is called for the frame 220, and the presence or absence of a cut is finally determined. Subroutine Proc for frames 221 and 222
(D) is called, and the presence or absence of a cut is finally determined. In this way, after the processing in which the calculation time is not required (main routine and Proc (b)) is used to narrow down the section in which the cuts exist in stages, the calculation processing takes time but is accurate (Proc (c) and Proc (d)). )) Is used to finally determine the cut. That is, it is possible to reduce the processing time and accurately detect the cut. The conventional method is Proc (c) or Pro for all frames.
Since time-consuming processing corresponding to c (d) is performed, the processing takes time.

[0035]

According to the present invention, a section in which a cut may exist is determined based on a feature amount obtained from the coded data of an intra-coded frame, and the inter-frame coding included in the section is determined. The feature amount is calculated from the data and the presence or absence of the cut is determined. It is possible to detect the cut at high speed because the section in which the cut may exist is obtained and the time-consuming processing such as the variable length code expansion processing is performed only on the inter-screen coded data in the section. Is obtained. In addition, by using the feature amount calculated from the intra-frame coded frame, it is possible to obtain the effect of correctly detecting the cut even when the intra-frame coded frame and the inter-frame coded frame are mixed.

In addition, in the above case, when the section in which there is a possibility of further cuts is narrowed down using the data amount of the encoded data as the feature amount of the inter-coded frame, the variable length is particularly long. The processing time associated with code expansion can be greatly reduced, and the cutting point can be detected even faster.

Further, in the above description, when the section in which the narrowed cut may exist includes the inter-frame coded frame and the intra-frame coded frame, only those coded frames are decoded to detect the cut. When this is done, in particular, it is possible to further speed up the cut detection, and it is possible to obtain the effect that the cut detection becomes more accurate when the intra-frame coded frame and the inter-frame coded frame are mixed.

[Brief description of drawings]

FIG. 1 is a process flow diagram of an embodiment of the present invention, which is a process flow diagram of a main routine.

FIG. 2 is a process flow diagram of an embodiment of the present invention, which is a process flow diagram of a subroutine Proc (b).

FIG. 3 is a process flow diagram of an embodiment of the present invention, which is a process flow diagram of a subroutine Proc (c).

FIG. 4 is a process flow diagram of an embodiment of the present invention, which is a process flow diagram of a subroutine Proc (d).

FIG. 5 is a diagram for explaining the effect of reducing the calculation amount according to the embodiment.

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Yayoi Arakura 3-19-2 Nishishinjuku, Shinjuku-ku, Tokyo Nihon Telegraph and Telephone Corporation (56) Reference JP-A-6-22304 (JP, A) JP-A-7-236153 (JP, A) JP-A-7-284017 (JP, A) JP-A-9-322174 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/76-5/956 H04N 5/262-5/278 H04N 7/24-7/68

Claims (3)

(57) [Claims] 1. A cut is detected from a coded video including an intra-frame coded frame coded using intra-frame correlation and an inter-frame coded frame coded using inter-frame correlation. A method of estimating a section S in which a cut may exist based on a feature amount obtained from the coded data of the intra-frame coded frame; and an inter-frame coded frame included in the section S. On the other hand, a coded video cut detection method comprising: a step of determining the presence or absence of a cut based on a feature amount obtained from the encoded data. 2. The step of determining the presence or absence of the cut is characterized in that the amount of coded data of the inter-frame coded frame included in the section S is used as a feature amount, and before or after the inter-frame coded frame or one of the frames. A process of estimating a section S ′ included in a section S in which a cut may exist by comparing with a feature amount of the inter-coded frame; and coded data for the inter-coded frame included in the section S ′. The coded video cut detection method according to claim 1, further comprising: a step of determining the presence or absence of a cut based on the feature amount obtained from 3. In the process of determining the presence / absence of a cut on the inter-coded frames included in the section S ′ based on the feature amount obtained from the coded data, the coded frames included in the section S ′ are displayed on the screen. The coded video according to claim 2, wherein in the case of an inter-coded frame and an intra-frame coded frame, the presence / absence of a cut is determined based on a feature amount obtained by decoding the coded frame. Cut detection method.