JP2008005536A - Method and apparatus for detecting scene cuts in block-based video coding system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and an apparatus for detecting scene cuts. <P>SOLUTION: A scene change detector compares predicted macroblocks from an anchor image 140 to input macroblocks from an input image 120 on a macroblock-by-macroblock basis to generate a residual macroblock 142 representing the difference between each predicted macroblock and each input macroblock. A variance 146 for each residual macroblock and a variance 148 for each input macroblock are computed after each comparison. The residual variance is compared to the input macroblock variance. Whenever the variance of the residual macroblock exceeds the variance of the input macroblock, a counter 152 is incremented. A scene cut detector repeats this process until each macroblock in the predicted image is compared to each input macroblock. If the count value ever exceeds a threshold level 154 while an input image is being processed, the scene cut detector sets a scene cut indicator flag. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to block-based video coding techniques, and more particularly, the present invention relates to a method and apparatus for detecting scene cuts in a video sequence within a block-based video coding system.

  Block-based video coding systems typically use coding techniques that take advantage of both spatial and temporal redundancy within an image (within a picture) and between images (within a picture) (inter-picture). This block-based image coding system is known as the Moving Picture Expert Group (MPEG) (standard for video coding), that is, ISO / IEC international standard 11172-2 (1994) (generally referred to as MPEG-1), 13818-2 (Draft January 20, 1995) (generally referred to as MPEG-2). In order to efficiently code the video sequence into a bitstream that can be transmitted using the redundancy in the input video sequence, block-based coding techniques provide information that the sequential pictures in the input video sequence are substantially similar. That is, it is assumed that the image scene has almost no change from picture to picture. Scene cuts that occur in picture sequences violate basic assumptions for efficient coding. Thus, after a scene change (scene cut), block-based coding techniques must use a large number of bits to code the first picture following the scene change. Since the number of bits available for coding any one image is usually limited, scene cuts can cause substantial errors in coding, resulting in substantial distortion of the decoded picture.

  Accordingly, there is a need in the art for a method and apparatus for detecting the occurrence of a scene cut prior to coding a picture so that the coding system can take certain steps to avoid substantial coding errors.

  The disadvantages of the prior art are overcome by an embodiment of the present invention, a scene cut detector that can be incorporated into a conventional block-based video coding system. The scene cut detector compares the predicted macroblock from the predicted image with the input macroblock from the input image on a per-macroblock basis and represents the difference between each predicted macroblock and each input macroblock Is generated. The variance of each residual macroblock and the variance of each input macroblock are calculated after each comparison. The variance of the residual macroblock and the input macroblock is compared with a decision function. The counter is incremented by the result of the decision function comparison. The scene cut detector repeats this process until each macroblock of the predicted image is compared to each input macroblock. If the count value exceeds the threshold during the processing of the input image, the scene cut detector confirms that the input image is in a new scene and sets the scene cut display flag accordingly. .

  The teachings of embodiments of the present invention can be readily understood by considering the following description in conjunction with the accompanying drawings, in which:

    FIG. 1 shows a block diagram of a block-based coding system 100 (specifically an MPEG encoder) incorporating an embodiment of the present invention. The input signal to the system at port 102 is a pre-processed image that is divided into a plurality of blocks, where the blocks are sequentially provided as inputs to the system. In the MPEG standard, these blocks of pixels are commonly known as macroblocks, eg 16 × 16 pixel blocks. The following disclosure uses MPEG standard terminology, but the term macroblock is, of course, intended to describe a block of arbitrarily sized pixels used in motion compensation standards. Yes.

  The system calculates a series of predicted macroblocks (P) from the system output signal. Each prediction macroblock is generated diagrammatically by decoding the output signal just as a receiver of the transmitted output signal decodes the received signal. A subtractor 106 subtracts the predicted macroblock from the input macroblock to generate a residual signal on the path 107 (also referred to in the art as a residual or residual macroblock).

  If the predicted macroblock is substantially similar to the input macroblock, the residual is relatively small and is easily coded using few bits. In such a scenario, the input macroblock is said to be motion compensated. However, if the difference between the predicted macroblock and the input macroblock is large, it is difficult to code the residual. Thus, the system is easier to code the input macroblock directly instead of coding the motion compensated residual macroblock. This selection is known as coding mode selection. The coding of the input macroblock I is called intra coding, and the coding of the residual is called inter coding. The choice between these two modes is known as Intra Inter Decision (IID).

  The IID is performed by the IID circuit 110, which sets the coding mode switch 108. The IID is calculated by first calculating the variance of the residual macroblock (Var R) and the variance of the input macroblock (Var I). Coding decisions are based on these values. There are several functions that can be used to make this determination. For example, if VarR is less than Var I, IID selects inter mode. Conversely, if Var I is less than Var R, IID selects intra mode.

  The selected block is processed in a discrete cosine transform (DCT) block 112. The DCT generates coefficients that represent the input signal to the DCT. A quantizer 114 quantizes the coefficients and generates an output block at port 104. The speed control block 116 controls the quantization measure (step size) used for coefficient quantization.

  In order to generate the correct prediction block and achieve efficient half-PEL motion vector generation, the encoder needs to access the decoded image. To achieve this access, the output of quantizer 114 is passed through both inverse quantizer 118 and inverse DCT 120. Ideally, the output of the inverse DCT is the same as the input to the DCT 112. In inter mode, a decoded macroblock is generated by summing the output of the inverse DCT and the predicted macroblock. During intra mode, the decoded macroblock is simply the output of the inverse DCT. The decoded macroblock is then stored in the frame store 124. The frame store stores a plurality of these “reconstructed” macroblocks that make up all reconstructed frames of image information. The reconstructed frame is used by a motion vector predictor 126 to produce a motion vector that is used to generate a prediction macroblock for the next appearing input image.

  In order to generate motion vectors, motion vector predictor 126 comprises three components. A full PEL motion estimator 128, a half PEL motion estimator 130, and a motion mode block 132. Full PEL motion estimator 128 is a “coarse” motion vector generator that seeks a coarse match between the macroblock of the previous image and the current input macroblock. The preceding image is called an anchor image. In the MPEG standard, anchor images are what are known as I or P frames in an image sequence known as a group of pictures (GOP). The motion vector is a vector representing a relative position where a coarse match is found between two macroblocks. The coarse motion vector generator generates an accurate motion vector for one picture element (PEL).

  The accuracy of a full PEL motion estimator is improved with a half PEL motion estimator. The half PEL estimator uses the full PEL motion vector and the reconstructed macroblock from the frame store 124 to calculate a motion vector that matches the half PEL accuracy. The half PEL motion vector is then sent to motion mode block 132. Usually there are multiple motion vectors for each macroblock. The mode block 132 selects the best motion vector that represents the motion for each input macroblock.

  A full PEL estimator is a task that is computationally more concentrated than a half PEL estimator. For this reason, in some embodiments it is calculated independently on dedicated hardware. All full PEL motion vectors are often calculated before half PEL processing begins.

  The above MPEG encoder system is a conventional system available as a model L64120 as a set of integrated circuits from LSILogic, Inc. of Milpitas, California. Importantly, this MPEG encoder stores the entire frame of the full PEL motion vector before the half PEL estimator begins operation.

  The concept of motion estimation and motion compensation is based on the basic assumption that the current picture is not very different from the previously generated picture (anchor image). However, when a scene change (also called a scene cut) occurs, the anchor picture is substantially different from the current picture. Therefore, the predicted macroblock is very inaccurate and the residual is large. Therefore, for most input macroblocks of a picture, IID selects an input macroblock (intra mode) for coding instead of coding a residual (intermode). It should be emphasized that this coding decision occurs even in the absence of scene changes, and that a correctly coded picture may contain a mix of intra and inter code macroblocks. However, the percentage of intra code macroblocks increases greatly when a scene cut occurs. The scene cut detector of an embodiment of the present invention analyzes all macroblocks in a picture and then determines whether a scene cut is occurring. This is accomplished by counting the number of intracode macroblocks and comparing the count to a threshold value. Specifically, if the proportion of I code macroblocks in any given frame exceeds a threshold, the frame is considered to follow a scene cut.

  In a typical MPEG encoder, the actual IID determination is made after the half PEL motion vector is generated and the best motion vector is selected. Since the full PEL estimator 128 generates motion vectors for all frames before the first macroblock is coded by the encoder, the inventive scene cut detector device 134 that monitors these full PEL results is , An IID estimate for all macroblocks, i.e. an estimate that the IID would make when actually analyzing the residual, can be generated. The scene cut detector includes an IID estimator 136 connected in series to an intra code macroblock counter 138. Counter 138 generates a scene cut decision (flag) indicating that the scene cut detector has determined that a scene cut has occurred.

  FIG. 2 shows a detailed block diagram of the inventive scene cut detector 134 of the MPEG encoder 100. The full PEL motion estimator 128 is provided with an I macroblock along with an appropriate I or P anchor image where the predicted macroblock is found. The anchor image is stored in the frame memory 140. Full PEL motion vector generator 141 is disclosed in US Pat. No. 5,351,095 issued September 27, 1994, which is incorporated herein by reference, and US patent filed on September 2, 1994, which is incorporated herein by reference. One of many well-known methods, including those disclosed in application 08 / 300,023, is used to generate a motion vector for each input macroblock. Using the full PEL motion vector and the anchor image from anchor image store 140, motion compensator 145 generates a predicted macroblock (P) for each input macroblock (I).

  The input macroblock (path 120) and the predicted image macroblock (P) form the input to the scene cut detector 134. The IID estimator 136 calculates the full PEL residual by subtracting the predicted macroblock from the input macroblock (subtractor 142). The IID estimator then uses variance blocks 146, 148 to calculate the variance of the input macroblock (Var I) and the variance of the full PEL residual (Var R). The IID circuit 150 then performs its IID estimation based on these variances. The counter 152 counts the number of intra mode decisions and compares the number of counts with a count threshold at block 154. If the count number exceeds a threshold during processing of a given picture, the intra MB counter generates a scene cut decision flag.

  FIG. 3 shows a graph 300 of a typical decision function used by the IID estimator when comparing Var R and Var I. The simplest decision function is a linear function 302. This function is performed using a comparator so that the IID estimation is in intra mode (region 310) when VarI is less than VarR. Conversely, when Var R is less than Var I, IID estimation is in inter mode (region 312).

  However, although simple, linear functions do not help provide the best results. Thus, the non-linear function 304 represents a more typical function. This function is vertical at a specific value 306 for Var R and then linear. In operation, macroblocks with relatively small Var R values are coded using inter mode. All values of VarR greater than value 306 are compared using direct comparison of function 302.

  Since the half PEL motion estimator defines the motion vector more accurately, a macroblock with a VarR value that is only slightly larger than the Var I value, for example, a point close to but directly below the curve 302 is a half PEL estimate. If a more accurate estimate is obtained using the instrument, it can be shifted above the curve. Thus, the IID of the MPEG encoder will use inter-mode coding, but the CID estimation of the scene cut detector nevertheless assumes that intra-mode coding is used. A function 308 is generally used to compensate for this anomaly. The form of function 308 is similar to function 304, but function 308 is slightly shifted below function 304. Thus, inaccurate IID estimation is avoided, i.e. values that would be near the curve are now in the inter-mode region.

  Of course, the functions shown in FIG. 3 are merely illustrative. Other linear and non-linear functions are used to obtain specific results. Embodiments of the invention include arbitrary functions within the IID estimator.

  During a normal sequence of macroblocks (no scene cuts), predictive macroblocks will largely lead to inter-mode decisions, even at full PEL accuracy. Therefore, the scene cut flag will not be set. Note that the rate of intra mode determination made after half-pel refinement is reduced by the additional accuracy of the motion vectors.

  When a scene cut occurs, it does not matter whether the system generates a motion vector that matches the half PEL or full PEL accuracy. Motion estimation is inaccurate in both cases. A significant number of intra-mode macroblocks are selected in both full PEL and half PEL stages. Thus, a coarse count of intra mode determination using a full PEL accuracy IID estimator is sufficient for scene cut detection.

  When 33% (1/3) of the estimated decision is an intra mode decision, it is generally set to exceed the threshold. Of course, the threshold can be set to any ratio required by the expected image sequence content.

  In addition to detecting scene cuts, embodiments of the invention are also useful for detecting pictures that are not well coded. For example, if an image scene is changing unpredictably, such as occurs when an object enters and leaves behind another object in the scene, the two successive pictures of the sequence representing the scene will be substantially different. In such an image, the encoder could exceed the coding bit budget for the picture sequence, i.e. it could code the image using more bits than can be transmitted over the communication channel. An inventive detector is used to detect pictures that are difficult to predict and code before the start of coding. Thus, the encoder can change the coding strategy to avoid exceeding the bit budget.

  One special device that uses a scene cut detector flag is described in US patent application Ser. No. 08 / 606,622, filed at the same time as this specification on Feb. 26, 1996, incorporated herein by reference. Yes. Depending on the flag, the device changes the quantization measure so that the encoder coarsely codes the picture to maintain the bit budget.

  While a single embodiment incorporating the teachings of the embodiments of the present invention has been illustrated and described in detail, those skilled in the art can readily devise many other various embodiments incorporating the present teachings.

FIG. 3 shows a block diagram of a block-based coding system incorporating a scene cut detector of an embodiment of the present invention. FIG. 2 shows a detailed block diagram of a scene cut detector according to an embodiment of the present invention. Figure 3 shows a graph of a decision function used by an IID estimator of an embodiment of the present invention.

Claims (4)

Each input image of the sequence of input images is divided into a plurality of macroblocks, and at least one motion vector is calculated for each of the macroblocks, and a sequence of predicted images is generated. A method for detecting a scene cut between a first image and a second image in a sequence of input images in a block-based video encoder comprising a plurality of predictive macroblocks derived from the input image and the motion vector Because
(A) generating a residual macroblock based on a difference between a predicted macroblock from the first image and an input macroblock from the second image;
(B) calculating a variance of the input macroblock and a variance of the residual macroblock;
(C) comparing the variance of the input macroblock and the variance of the residual macroblock, the variance of the residual macroblock exceeds a first threshold, and the variance of the residual macroblock is the input Incrementing the counter value if greater than the sum of the variance of the macroblock and the shift value;
(D) repeating steps (a), (b), and (c) for all macroblocks of the second image;
(E) setting a flag indicating the occurrence of the scene cut when the count value exceeds a second threshold;
Having a method.   The method of claim 1, wherein the second threshold is set to 33% of the total number of macroblocks in the input image. Each input image of the sequence of input images is divided into a plurality of macroblocks, and at least one motion vector is calculated for each of the macroblocks, and a sequence of predicted images is generated. In a block-based video encoder comprising a plurality of prediction macroblocks derived from the input image and the motion vector, a scene cut between a first image and a second image in the sequence of input images is performed. A device for detecting,
A subtractor configured to generate a residual macroblock based on a difference between a predicted macroblock from the first image and an input macroblock from the second image;
A residual variance generator configured to calculate a variance of the residual macroblock;
An input variance generator configured to calculate a variance of the input macroblock;
Generating a first output if the variance of the residual macroblock exceeds a first threshold and the variance of the residual macroblock is greater than the sum of the variance of the input macroblock and the shift value; A decision circuit configured to generate a second output in the case of
A counter configured to count the first output;
Threshold means configured to set a flag indicating the occurrence of the scene cut when the count value counted by the counter exceeds a second threshold;
A device comprising: 4. The apparatus of claim 3, wherein the second threshold is set to 33% of the total number of macroblocks in the input image.

Images