JP2007518303A - Processing method and apparatus using scene change detection - Google Patents

Abstract

The present invention relates to a method of processing digitally encoded video data that is available in the form of a video stream having consecutive frames. Those frames that are divided into macroblocks are at least I frames (intra) and P frames (prediction), and at least two frames that are temporarily placed between and between them And B frames predicted bidirectionally. In accordance with the present invention, the processing method determines, if any, each successive macroblock of the current frame associated with the encoding parameter that characterizes the weighted prediction, and provides statistics associated with the parameter. Collecting the parameters for all the continuous macroblocks of the current frame, analyzing the statistics for determining a change in preference for a prediction direction, and each time a change in preference is determined. Detecting the occurrence of gradual scene changes in a series of frames.

Description

  The present invention relates to H.264. The present invention relates to a method that makes it possible to automatically detect gradual scene transitions in H.264 / AVC video streams. The method is very efficient and cost effective. H.264 is based on using the new encoding parameters introduced.

  In recent years, international video coding standards have played an important role in facilitating the adoption of digital video in various business and general purpose applications. The most influential standards have sometimes been jointly developed by two organizations, namely ITU-T and ISO / IEC MPEG (eg MPEG-2 / H.262). The newest joint standard is H.264. H.264 / AVC, which is ISO / IEC and recommended H.264 as international standard 14496-10 (MPEG-4 Part 10) Advanced Video Coding (AVC). It was expected to be officially approved by ITU-T in 2003 as H.264 / AVC. H. The main purpose of the H.264 / AVC standard is to obtain significant gains in compression performance and “network friendly” video display addressing “conversational” (telephone) and “non-conversational” (memory, broadcast, streaming) applications. To achieve and to give. H. H.264 / AVC is widely recognized as providing significantly improved rate-distortion efficiency over existing standards. H.264 / AVC-based solutions are also being considered by other standards organizations such as DVB and DVD Forum. H. For implementation of H.264 / AVC encoder / decoder, see, for example, “New H.264 Standard: Overview and Implementation of TMS320C64x Digital Media Platform—Official Government Report (http://www.ubvideo.com The number of sites that provide information on H.264 / AVC on the Internet has become very large, as shown in / public. , ITU-T / MPEG JVT [Joint Video Team] (official H.264 document and JVT software (ftp://ftp.imc-files.org/jvt-experts/reference)) official database H.264 / AVC status and progress, including updates It provides free access to a document that reflects the situation.

  H. H.264 / AVC syntax and encoding tools can now be reproduced. First, H.C. H.264 / AVC employs the same principles of block-based motion compensated transform coding as understood by established standards such as MPEG-2. H. H.264 syntax is therefore organized as a normal hierarchy of headers (eg, picture block headers, slice block headers, macroblock headers, etc.) and data (eg, motion vectors, block transform coefficients, quantization scales, etc.). . While most of the known concepts regarding data structuring (eg, I, P or B pictures, intra and inter macroblocks) are maintained, some new concepts are also introduced at both the header level and the data level. Has been. Mainly H. H.264 / AVC is a video coding layer defined to efficiently represent the content of video data and headers in a manner suitable for formatting the data and for transmission by high level (coarse transport) systems. Separates the network abstraction layer that supplies the information.

  H. at the data level. One of the main features of H.264 / AVC is also to use more elaborate partitioning and manipulation of 16x16 macroblocks (macroblock MB includes both 16x16 luminance blocks and corresponding 8x8 chrominance blocks However, many operations, such as motion estimation, actually take only luminance and reflect the result in chrominance). Therefore, the motion compensation process uses a motion vector accuracy of a quarter or less of the sample grid to subdivide the MB as small as 4 × 4. In addition, the selection process for the motion compensated prediction of the sample block can include a plurality of stored decoded pictures instead of only adjacent pictures. Even with intra coding, it is possible to perform block prediction using samples previously decoded from neighboring blocks (the prediction rule based on such a space is described in so-called intra prediction mode). ) After either motion compensated prediction or spatial based prediction, the resulting prediction error is usually transformed and quantized based on a 4x4 block size instead of the traditional 8x8 size. This feature is particularly relevant to the invention as defined in the description below and will be highlighted later in the description. H. H.264 / AVC further uses other specific recognitions (eg, entropy coding), most of which are fixed or changed only at or beyond the picture level.

Regarding motion compensation, H.C. The general concepts and features of H.264 / AVC also need to be reproduced. For example, most conventional video coding standards, such as MPEG-2, essentially use block-based motion compensation as a practical method that utilizes correction between subsequent pictures in the video. This method attempts to predict each macroblock in a given picture by “optimal matching” in adjacent pre-decoded reference pictures. If the pixel difference between the macroblock and its prediction is sufficiently small, this difference (or residue) is encoded, not the difference of the macroblock itself. The relative movement of the prediction block relative to the actual MB grid position is represented by a motion vector, which is encoded separately. FIG. 1 shows this for the case of bi-directional prediction, where two reference diagrams P _i and P _{i + 1} are used, one in the past and one in the future (in display order). Picture to be predicted this manner (such as B _i in FIG. 1) is referred to as a B-picture. Instead, a picture that is predicted by referring only to the past is called a P picture.

  H. When using H.264 / AVC, these basic concepts are further refined. First, H.C. Motion compensation in H.264 / AVC is based on multiple reference picture predictions. Suitability for a given block can be searched for in distant past or future pictures instead of neighboring pictures. Second, H.M. H.264 / AVC allows the MB to be divided into smaller blocks and each of those blocks to be predicted separately. This means that the prediction for a given MB is basically composed of different blocks using different motion vectors and taken from different reference pictures. The number, size, and orientation of the prediction block are uniquely determined by selecting an inter mode. Some such modes are specified such that block sizes such as 16x8, 8x8, etc. are reduced to 4x4.

  Other H.C. The innovation in H.264 / AVC allows motion compensated prediction signals to be weighted and offset by an amount specified by the encoder. This is the case for bi-directional prediction for frames B (i) predicted from previous frames P (i−n) and P (i−1) and subsequent frames P (i + j) and P (i + m). The means that the prediction block from the past and the prediction block from the future can select a unique amount that contributes in the overall prediction. This feature makes it possible to greatly improve the coding efficiency for scenes containing fades.

  The problem, however, is as follows. Recent developments in computing, communications and digital data storage have led to the enormous growth of large digital archives characterized by steadily increasing capacity and content richness. An efficient search method that instantly searches stored information of interest is therefore very important. Searching manually for terabyte orders of data that is not stored even in a system is tedious and time consuming, and there is an increasing demand for information retrieval and transfer of retrieval tasks to automated systems. Searches and searches in large archives of unstructured video content are typically performed after the content has been indexed using content analysis techniques. These techniques are based on, for example, algorithms such as image processing, pattern recognition and artificial intelligence, which aim to automatically generate annotation descriptions of video material in terms of the video content. (Such annotations vary from low level signal related features such as color and texture to high level information such as face presence and location, for example).

  One of the most important content descriptions is a shot boundary indicator, as described, for example, in the patent literature, WO 01/03429. A shot is a video segment taken using a single camera in succession, and a shot is generally regarded as a basic unit having a video. Detecting shot boundaries therefore means playing those basic video units, which also provides the basis for almost all video abstraction and high-level video segmentation algorithms (eg, (Ref. "Video Abstracting", by R. Lienhart, et al., Communications of the ACM, 40 (12), 1997, pp 55-62).

  During video editing, shots are connected using shot transitions that can be classified into at least two classes: sudden transitions and gradual transitions. Sudden transitions, also called hard cuts, are obtained without any modification of the two shots and are very easy to detect, they account for the majority of all types of video playback . For example, gradual transitions such as fades, dissolves and wipes can be obtained by applying specific transformations of two contained shots. During video playback, each transition type is carefully selected to support the content and context of the video sequence. Therefore, automatically playing back all of their positions and types helps the device deduct high-level semantics. For example, in feature films, dissolves are often used to convey the passage of time. Dissolves also occur more often in feature films, documentaries, biographies and scientific video material than newscasts, sports, comedies and shows. The converse is true for wipes. Therefore, automatic detection of transitions and transition types is used for automatic recognition of video genres.

The following H. Because of the large application area for the H.264 / AVC standard, There is an increasing demand for efficient solutions for H.264 / AVC video content analysis. In recent years, several efficient content analysis algorithms and methods have been shown for MPEG-2 video that operate almost exclusively in the compressed domain. As mentioned above, H.M. H.264 / AVC, in a sense, specifies a superset of MPEG-2 syntax, so most of these methods are H.264. H.264 / AVC. However, due to MPEG-2 limitations, those existing methods cannot provide adequate or reliable performance, which is an additional and often cost-effective operation in the pixel or audio domain. Have the disadvantage of being typically addressed by having a low method.
“Video Abstracting”, by R.D. Lienhart, et al. , Communications of the ACM, 40 (12), 1997, pp 55-62.

  Therefore, an object of the present invention is to provide a method that makes it possible to avoid the disadvantages in all situations where weighted prediction of a frame is performed with a unique prediction amount from the future and past of the frame to be predicted. That is.

To this end, the present invention is a method for processing digitally encoded video data that can be used in the form of a video stream having consecutive frames divided into macroblocks, wherein the frames are at least independently encoded. Between the I frame and the P frame temporarily arranged between the I frames, the P frame predicted from at least the previous I frame or the P frame, and between the I frame and the P frame or 2 B frames that are temporarily placed between two P frames and that are bi-directionally predicted from at least two frames placed between them, the prediction being a unique amount of prediction from the past and the future A method performed by weighted prediction with:
-Determining, if any, each successive macroblock of the current frame associated with an encoding parameter characterizing the weighted prediction;
-Collecting said parameters for all successive macroblocks of the current frame to provide statistics related to said parameters;
Analyzing the statistics to determine a preference change for the direction of prediction; and- detecting the presence of gradual scene changes in the sequence of frames each time a preference change is determined;
Relates to a method comprising:

  More particularly, the analyzing step according to the present invention determines the number of macroblocks having the same direction preference and similar weighting for a predetermined threshold derived in relation to the total number of macroblocks in the current frame. It is provided for comparison. Preferably, information regarding the location and duration of each scene change is generated and stored in a file.

  Another object of the present invention is to provide a processing device which makes it possible to carry out the above method.

To this end, the present invention is an apparatus for processing digitally encoded video data available in the form of a video stream having continuous frames divided into macroblocks, wherein the frames are at least independently encoded. Between the I frame and the P frame temporarily arranged between the I frames, the P frame predicted from at least the previous I frame or the P frame, and between the I frame and the P frame or 2 B frames that are temporarily placed between two P frames and that are bi-directionally predicted from at least two frames placed between them, the prediction being a unique amount of prediction from the past and the future A device that is made by weighted prediction with:
Determining means, if any, provided for determining each successive macroblock of the current frame associated with the coding parameters characterizing the weighted prediction;
A collecting means provided for collecting said parameters for all successive macroblocks of the current frame in order to supply statistics related to said parameters;
-An analysis means provided for analyzing said statistics to determine a preference change for the direction of prediction; and-the presence of a gradual scene change in the sequence of frames each time a preference change is determined Detection means provided for detecting
The present invention relates to a device having

  As described above, H. When recalling the general concepts and features of H.264 / AVC, the motion compensated prediction signal can be weighted by the amount specified by the encoder. Weighted prediction is used to obtain bi-directional predictions (B pictures) where prediction blocks from the past and from the future exist in all predictions by a unique amount (when using MPEG-2, this is 1/2 Only limited to one possibility of weighting both prediction signals).

  The principle of the present invention is that due to such inadequacy, the presence of gradual shot transitions is represented by gradual changes in preferences for predictions from one direction to the other. Such a change in prediction for the direction of prediction is detected by analyzing statistics relating the coding parameters that characterize the weighted prediction. For example, the analysis can include comparing the number of macroblocks having the same direction preference and similar weighting against a predetermined threshold. In addition, the (local) uniformity of such macroblock distributions confirms that changes in the directional scene transition direction preference for prediction are actually the result of gradual scene transitions. Be examined. Some additional analysis also considers the effective use of sub-macroblock motion prediction and in weighted prediction, for example H.264. It can be implemented as is possible in H.264 / AVC.

For example, H.3 indicates prediction of picture B _i from previous pictures P _i-n , P _i−1 and subsequent pictures P _{i + j} , P _{i + m} . An example of bidirectional prediction in H.264 / AVC is shown in FIG. It is called MB _Pred and is equal to B1 “+” B2 “+” B3, and B1 = alpha1. b ₁ + alpha2. The prediction for a macroblock MB that is b ₂ (where alpha1 and alpha2 are coefficients) has three prediction blocks, so the lower half of the macroblock MB _Pred is two 8 × 8 blocks B ₂ and predicted by B _3, top half of the macroblock is predicted by one 8x16 block _{B 1.} Each of these prediction blocks is H.264. As is possible in H.264, it has distinct motion vectors associated with different reference pictures. Unlike B ₂ and B ₃ , block B ₁ is obtained using a weighted prediction, ie that block B ₁ exists in the sum of unique quantities controlled by the corresponding weighting parameters alpha 1 and alpha 2 2. It is obtained by adding two blocks b1 and b2. Statistics of those weighting parameters (absolute value and sign) are collected for all macroblocks, and the statistical value distribution over multiple macroblocks is analyzed to achieve gradual scene transition detection.

The implementation of the processing method according to the invention is illustrated in the block diagram of FIG. The above concept in the case of H.264 / AVC bitstream is shown and the above embodiment, however, does not limit the scope of the invention. In the decoding apparatus shown in the figure, the demultiplexer 21 receives the transport stream TS and generates a demultiplexed audio stream AS and video stream VS. The video stream is H.264 to provide a decoded video stream DVS as usual. Received by the H.264 / AVC decoder 22. The decoder 22 mainly includes an inverse quantization circuit 221 (Q ⁻¹ ), an inverse transform circuit 222 (T−1) that is an inverse DCT circuit in this case, and a motion compensation circuit 223. . It is also the received coding parameters that characterize the weighted prediction to be performed (eg, some relevant coding parameters are “luma_weight”, “luma_offset”, “luma_log2_weight_denom”, etc., which are prediction samples A so-called Network Abstraction Layer Unit (NALU) 224 that is provided for collecting The output signal of the unit 224 is weighted with the prediction parameter statistic WPPS received by the analysis circuit 23 for appropriate processing. The processing operations performed in that circuit 23 then generate information about the location and duration of gradual scene changes in the initially received stream, which information is then used, for example, as commonly used CPI (Characteristic). It is stored in file 24 in the form of a Point Information) table. This output information is now available for applications such as video summarization, automatic chaptering, etc.

  Hardware or software (the method of the invention is therefore a computer program product for a processing unit having a set of instructions that, when loaded into the processing unit, causes the processing unit to perform the method as described above. Note that there are multiple ways of performing functions with both items). In this respect, the above figures are schematic and show only one possible embodiment of the invention. Therefore, although the figure (in this case, FIG. 3) shows different functions as different blocks, this by no means excludes that a single item of hardware or software performs multiple functions. Absent. Nor does it exclude that an assembly of items of hardware or software or both carry out a function. The remarks are made in connection with the drawings to evoke that the above detailed description does not limit the invention and that many variations are possible which fall within the scope of the claims. Yes. The word “comprising” does not exclude the presence of other elements or steps than those listed in a claim. The singular representation of an element or stage does not exclude the presence of a plurality of such elements or stages.

It is a figure which shows the conventional Example of bidirectional | two-way prediction. H. It is a figure which shows the basic principle of the weighted prediction about B frame in the case of H.264 / AVC standard. FIG. 4 is a block diagram for the implementation of a processing method according to the present invention.

Claims (6)

A method of processing digitally encoded video data that is available in the form of a video stream having consecutive frames divided into macroblocks, the frames comprising at least independently encoded I frames; P frames temporarily placed between I frames and predicted from at least a previous I or P frame, and at least those two temporarily placed and placed between I frames and P frames or between two P frames B and B frames predicted bi-directionally from the frame, and the prediction of the P and B frames is performed by weighted prediction having a unique prediction amount from the past and the future, and the processing method is:
Determining, if any, each successive macroblock of the current frame associated with an encoding parameter characterizing the weighted prediction;
Collecting the parameters for all the consecutive macroblocks of the current frame to provide statistics associated with the parameters;
Analyzing the statistics to determine a change in preference for the prediction direction; and detecting the occurrence of gradual scene changes in a series of frames each time a change in preference is determined;
A processing method characterized by comprising:   2. The processing method according to claim 1, wherein the analyzing step has a preference in the same direction and a similar weighting for a predetermined threshold derived in relation to the total number of macroblocks in the current frame. A processing method, characterized in that it is given to compare the number of blocks.   3. A processing method according to claim 2, wherein information about the position and duration of each scene change is generated and stored in a file.   4. The processing method according to claim 1, wherein the syntax and semantics of the processed video stream are H.264. H.264 / AVC standard syntax and semantics. An apparatus for processing digitally encoded video data that is available in the form of a video stream having consecutive frames divided into macroblocks, the frames comprising at least independently encoded I frames; and P frames temporarily placed between I frames and predicted from at least a previous I or P frame, and at least those two temporarily placed and placed between I frames and P frames or between two P frames B and B frames predicted bi-directionally from the frame, and the prediction of the P and B frames is performed by weighted prediction with a unique prediction amount from the past and the future, the device:
Determining means for determining, if any, each successive macroblock of the current frame associated with an encoding parameter characterizing the weighted prediction;
Collecting means for collecting the parameters for all the continuous macroblocks of the current frame to provide statistics associated with the parameters;
Analysis means for analyzing the statistics for determining a change in preference for the prediction direction; and a detection means for detecting the occurrence of gradual scene changes in a series of frames each time a change in preference is determined;
A device characterized by comprising:   A computer program for a digital video data decoding device, wherein when loaded into the decoding device, the decoding device executes the steps of the processing method according to any one of claims 1 to 4 A computer program characterized by having a set of instructions to do so.

Images