JP5222958B2 - Moving picture coding apparatus, moving picture coding method, moving picture decoding apparatus, and moving picture decoding method - Google Patents

Description

  The present invention relates to a moving image encoding / decoding technique for encoding and decoding a moving image, and more particularly to a reduction in code amount in inter-screen prediction.

  In recent video coding standards, various pixel value prediction methods using temporal and spatial correlation of images are used, and the coding is performed in units of variable length blocks obtained by dividing the screen into small small areas. The method to do is adopted. H. was formulated in 2003. In the H.264 / AVC (Advanced Video Coding) standard, an encoding mode is defined as a combination of such a pixel value prediction method and a block size, and an optimum mode is selected from a number of encoding modes according to the property of the image. By selecting, the coding efficiency is increased. H. In H.264 / AVC, an encoding tool group called a profile is defined in order to limit the encoding techniques that can be used depending on the type of application that is assumed.

  H. In H.264 / AVC, an encoding mode is determined for each macroblock having a 16 × 16 pixel size. Here, one of the prediction methods of intra prediction (intra prediction) that compresses using the pixel correlation of the block in the screen and inter prediction (inter prediction) that uses the pixel correlation of the block between the screens is applied. To do. Predictive prediction that specifies one reference image and bi-directional predictive prediction that can specify two reference images are specified as pixel value prediction methods between screens. (When using the Baseline profile, only Predictive prediction is available).

  Further H. In the H.264 / AVC format, the block size used for prediction is divided into 16 × 16 pixel size into small blocks of 8 × 8, 8 × 4, 4 × 8, and 4 × 4, thereby further improving the encoding efficiency. ing. Regarding block division technology, for example, Patent Document 1 discloses that an input image is divided into first blocks of M × N size, and further, the first block is divided into second blocks of m × n size, The division shape of the first block is determined based on the feature information extracted from the image of the block.

JP 2008-17305 A

  H. In the H.264 / AVC encoding scheme, when inter-screen prediction is used, prediction is performed while switching a plurality of pixel value prediction methods and block sizes for each macroblock. For this reason, it is necessary to encode the pixel value prediction method and block size information as inter-screen prediction mode information for each macroblock and add the result to the stream. Specifically, a code of 1 to 8 bits per macroblock is allocated as the inter-screen prediction mode information. Considering that the average code amount per macroblock is several bits, the prediction mode information is not a small amount, and if it can be reduced, the code amount can be greatly reduced.

  An object of the present invention is to realize a further reduction in the code amount by reducing the code amount of the inter prediction mode information.

  A moving image encoding apparatus according to the present invention includes a block dividing unit that divides an input image into blocks of a predetermined size among a plurality of sizes, and an inter-screen that generates a predicted image by inter-screen prediction from a reference image for the block. A prediction unit; a subtraction unit that calculates a prediction difference between the prediction image and the input image; and encoding that generates a coded stream by performing frequency conversion processing, quantization processing, and variable length coding processing on the prediction difference. A stream generation unit, and a frame memory that holds a prediction mode when a prediction image is generated for each block of the encoded image, and the inter-screen prediction unit is configured to store the encoded image stored in the frame memory. Based on the prediction mode of the block at the same position, a prediction mode skip operation for determining the prediction mode of the block is executed.

  When the prediction mode skip operation is executed in the inter-screen prediction unit, the encoded stream generation unit deletes prediction mode information in the encoded stream and indicates that the prediction mode skip operation has been executed. Give a flag.

  The moving image encoding method of the present invention includes a block dividing step of dividing an input image into blocks of a predetermined size among a plurality of sizes, and a screen for generating a predicted image by inter-screen prediction from a reference image for the block A code for generating an encoded stream by performing an inter prediction step, a subtraction step for calculating a prediction difference between the prediction image and the input image, and performing a frequency conversion process, a quantization process, and a variable-length encoding process on the prediction difference The inter-picture prediction step includes a prediction mode skip operation for determining the prediction mode of the block based on the prediction mode of the block at the same position of the encoded image.

  The moving picture decoding apparatus of the present invention includes a differential image decoding unit that generates a prediction difference by performing a variable length decoding process, an inverse quantization process, and an inverse frequency transform process on an encoded stream, and the encoded stream An inter-screen prediction unit that generates a prediction image by inter-screen prediction from a reference image by dividing into blocks to be decoded, an addition unit that generates a decoded image by adding the prediction image and the prediction difference, and has been decoded A frame memory that holds a prediction mode when a prediction image is generated for each block of the image, and the inter-screen prediction unit sets a prediction mode of a block at the same position of the decoded image held in the frame memory. Based on this, a prediction mode skip operation for determining the prediction mode of the block is executed.

  The moving image decoding method of the present invention includes a difference image decoding step for generating a prediction difference by performing a variable length decoding process, an inverse quantization process, and an inverse frequency transform process on an encoded stream; An inter-screen prediction step of generating a predicted image by inter-screen prediction from a reference image divided into blocks to be decoded, and an adding step of generating a decoded image by adding the predicted image and the prediction difference, The inter-screen prediction step includes a prediction mode skip operation for determining the prediction mode of the block based on the prediction mode of the block at the same position of the decoded image.

  According to the present invention, there is provided a moving image encoding technique and a decoding technique that can further reduce the amount of code without degrading the image quality.

1 is a configuration diagram (Example 1) showing an embodiment of a moving picture encoding apparatus according to the present invention. The figure which showed notionally the operation | movement of the prediction process between screens. The figure which shows the kind of encoding mode which can be utilized with the inter-screen prediction method. The conceptual diagram which shows the mode determination by prediction mode skip. The figure which shows the data area which can be reduced by prediction mode skip. The schematic diagram which shows the structure of the encoding stream which provided the skip flag. The block diagram which shows one Example of the moving image decoding apparatus by this invention (Example 2). An example of switching a prediction mode skip with reference to a motion vector (Example 3). The example which switches prediction mode skip with reference to the prediction mode of an adjacent block. FIG. 10 is a diagram illustrating a reference picture when the target image is a B picture (Example 4).

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing an embodiment of a moving picture coding apparatus according to the present invention. The moving image encoding apparatus includes an input image memory 102 that holds an input original image 101 and a block dividing unit 103 that divides the input image into small regions. In addition, an intra-screen prediction unit 105 that performs intra-screen prediction in divided block units, an inter-screen prediction unit 106 that performs inter-screen prediction in block units based on the amount of motion detected by the motion search unit 104, A mode selection unit 107 that determines predictive encoding means (prediction method and block size) that matches the properties is provided. Then, a subtraction unit 108 that calculates a prediction difference between the input image and the prediction image, a frequency conversion unit 109 and a quantization unit 110 that perform encoding on the prediction difference, and encoding according to the symbol occurrence probability. A variable length encoding unit 111 is included. In addition, an inverse quantization processing unit 112 and an inverse frequency transform unit 113 that decode a prediction difference that has been encoded once, an addition unit 114 that generates a decoded image using the decoded prediction difference, and a decoded image A reference image memory 115 is stored and used for later prediction. Furthermore, it has a frame memory 116 that holds the inter picture prediction mode of the previous picture.

  First, the operation of the entire apparatus will be described. The input image memory 102 holds one image as an encoding target image from the original image 101, and divides it into fine blocks by the block dividing unit 103, and a motion search unit 104, an in-screen prediction unit 105, And to the inter-screen prediction unit 107. The motion search unit 104 calculates the amount of motion of the corresponding block using the reference image stored in the reference image memory 115 and passes the motion vector to the inter-screen prediction unit 106. The intra-screen prediction unit 105 and the inter-screen prediction unit 106 execute the intra-screen prediction process and the inter-screen prediction process in units of several blocks to generate a predicted image. The mode selection unit 107 selects either optimal prediction image. Subsequently, the subtraction unit 108 calculates a prediction difference between the input image and the prediction image and passes it to the frequency conversion unit 109. The frequency conversion unit 109 and the quantization processing unit 110 perform frequency conversion and quantization processing such as discrete cosine transformation (DCT) for each block of a size specified for the transmitted prediction difference. And pass to the variable length coding processing unit 111 and the inverse quantization processing unit 112. Further, the variable length coding processing unit 111 decodes the prediction difference information represented by the frequency conversion coefficient, the prediction direction used when performing intra prediction, for example, the motion vector used when performing inter prediction, and the like. The encoded stream 117 is generated by performing variable-length coding on the information necessary for the generation based on the occurrence probability of the symbols. In addition, the inverse quantization processing unit 112 and the inverse frequency transform unit 113 perform the inverse frequency transform such as inverse quantization and inverse DCT (Inverse DCT) on the quantized frequency transform coefficient, respectively, and obtain a prediction difference. To the adder 114. Subsequently, the adder 114 generates a decoded image and stores it in the reference image memory 115.

Next, the operation of the inter-screen prediction unit 106 will be described.
FIG. 2 is a diagram conceptually illustrating the operation of the inter-screen prediction process. H. In H.264 / AVC, the encoding target image is encoded in block units according to the raster scan order. When performing inter-screen prediction, a decoded image of an encoded image included in the same image sequence 301 as the encoding target image 303 is used as a reference image 302, and a block having a high correlation with the target block 304 in the target image (prediction Image) 305 is searched from the reference image. At this time, in addition to the prediction difference calculated as the difference between both blocks, the difference between the coordinate values of both blocks is encoded as a motion vector (MV) 306. On the other hand, in the decoding process, the reverse procedure may be performed, and the decoded image is generated by adding the decoded prediction difference to the block (predicted image) 305 in the reference image.

  FIG. 2 is a diagram illustrating types of encoding modes that can be used in a Baseline profile, for example, with respect to an inter-screen prediction method defined by H.264 / AVC. H. In H.264 / AVC, as a pixel value prediction method between screens, forward prediction that designates one reference image (Predictive prediction) and bi-directional prediction that can designate two reference images (Bi-directional predictive) Prediction), but when using the Baseline profile, only Predictive prediction is available. In each frame, encoding is sequentially performed in the order of raster scanning from the upper left macroblock to the lower right macroblock. The macroblock can be divided into smaller blocks (sub-blocks), and encoding is performed by selecting an optimum one from several sizes determined in advance for each type of prediction method.

  As block sizes for inter-screen prediction, 16 × 16 pixels (P16 × 16 mode), 16 × 8 pixels (P16 × 8 mode), 8 × 16 pixels (P8 × 16 mode), 8 × 8 pixels (P8 × 8) Mode) is prepared, and in the case of 8 × 8 pixel size, it can be further divided into sub-blocks of 8 × 8 pixel, 8 × 4 pixel, 4 × 8 pixel, and 4 × 4 pixel size. . Further, a PSskip mode that does not encode motion vector information is prepared for a block size of 16 × 16 pixels, and a P8 × 8ref0 mode that does not encode reference frame numbers is prepared for an 8 × 8 pixel size.

  In the case of performing inter-picture prediction coding, a region having a high correlation with the corresponding block is searched in the reference frame, the amount of motion is detected, and motion compensation is performed to create a reference image. In this case, motion vector information and a reference frame number are written in the header portion of the encoded stream. For each macroblock, a prediction image is generated in units of blocks of any one of the sizes described above, and encoding is performed by performing frequency conversion and quantization processing on the prediction difference from the original image.

  In this way, in inter-screen prediction, prediction is performed while switching between a plurality of pixel value prediction methods and block sizes for each macroblock, so pixel value prediction method and block size information are assigned to each macro block as inter-screen prediction mode information. Therefore, it was necessary to encode.

  Therefore, in this embodiment, as a new prediction method that omits this prediction mode information, a method for determining the prediction mode of the current macroblock with reference to the prediction mode of the macroblock at the same position of the previous coded picture. Is introduced. Hereinafter, this method is referred to as “prediction mode skip”. In contrast, a conventional method for selecting and determining an optimal block size from the viewpoint of image quality and code amount for the current macroblock regardless of the prediction mode of the previous picture is referred to as “non-skip”. In this embodiment, in order to execute “prediction mode skip”, a frame memory 116 that holds the prediction mode of the previous picture is provided. In the case of “prediction mode skip”, the inter prediction unit 106 refers to the prediction mode of the macroblock at the same position of the previous picture stored in the frame memory 116 and predicts the current macroblock in the same prediction mode. Generate an image. Then, a “skip flag” indicating “prediction mode skip” is added to the data encoded by “prediction mode skip”. However, since the code amount required for the “skip flag” is very small (1 bit is sufficient), the effect of reducing the code amount by omitting the conventional prediction mode information is great, and as a result, the code amount of the output stream can be reduced. it can.

  FIG. 4 is a conceptual diagram showing mode determination by inter-screen prediction mode skip. Here, as a division mode of forward prediction (Predictive prediction), it is assumed that each mode of P16 × 16, P16 × 8, P8 × 16, and P8 × 8 and an intra-screen prediction mode (Intra) are selected.

  As shown in (a), in the “prediction mode skip” method, the prediction mode of the current macroblock is determined based on the prediction mode of the same-position macroblock of the previous picture. Here, since the prediction mode of the macroblock MB1 of the previous picture at the same position as the encoding target macroblock MB2 of the current picture is the P8 × 16 mode, the P8 × 16 mode is adopted as the prediction mode of the encoding target macroblock MB2. To do. It determines similarly about other adjacent macroblocks.

  As shown in (b), when the prediction mode of the same-position macroblock MB1 of the previous picture is the intra prediction mode (Intra), the intra prediction mode is set as the prediction mode of the encoding target macroblock MB2. Since it cannot be used as it is, it is changed to the inter-screen prediction mode (for example, P16 × 16) and set.

  Here, whether or not to adopt inter-screen prediction mode skip for a macroblock to be encoded is determined from the viewpoint of image quality and code amount of the target block (skip adoption rule). That is, when the correlation with the previous picture is strong, there is a high possibility that the same mode as the prediction mode of the previous picture is optimal. Therefore, even if the prediction mode skip is adopted, the image quality is hardly deteriorated. Specifically, it is determined by a mechanism similar to mode selection that determines a coding method such as RD-optimization, which is one of elemental technologies for performing moving image coding, for each macroblock. Alternatively, the degree of similarity of mode information between pictures is calculated, and when the degree of similarity is larger than the threshold, prediction mode skip may be employed.

By employing inter-screen prediction mode skip, the amount of codes used for inter-screen prediction mode information can be reduced. This will be described below.
FIG. 5 is a diagram showing data areas in the stream that can be reduced by adopting prediction mode skip. (A) is a case where the block size is larger than 8 × 8, and the area of the macroblock type indicating the macroblock prediction mode is reduced. (B) is a case where the block size is 8 × 8 or less, and in addition to the case of (a), the sub macroblock type area indicating the prediction mode of the sub macroblock is also reduced. The reduction of the code amount in each region is 1 to 8 bits, and these regions exist for each macroblock, so that the amount of reduction is large when viewed in the entire stream.

  FIG. 6 is a schematic diagram showing the structure of an encoded stream to which a skip flag is assigned. One skip flag (SF) may be assigned in units of the data range for which the inter-screen prediction mode skip is performed. Since the code amount of the skip flag is 1 bit, the increase in the code amount is extremely small (for example, skip = “1”, non-skip = “0”).

  The range of data to which the skip flag is assigned is specified not only in units of macroblocks in (d) but also in units of (a) GOP, (b) pictures, and (c) slices in units of a plurality of consecutive macroblocks. Make it possible. Furthermore, when performing the inter-screen prediction mode skip for the entire stream, a skip flag may be added to the stream header. The decoding device (decoder) determines the range of data for which the prediction mode skip is performed based on the skip flag assignment position in the stream, and executes decoding of the stream. As described above, since only one skip flag can be provided not only in units of macroblocks but also over a plurality of macroblocks, it is possible to further suppress an increase in code amount.

  FIG. 7 is a block diagram showing an embodiment of the moving picture decoding apparatus according to the present invention. For example, the moving picture decoding apparatus decodes a prediction difference with a variable length decoding unit 202 that performs the reverse procedure of variable length coding on the encoded stream 201 generated by the moving picture encoding apparatus shown in FIG. A dequantization processing unit 203 and an inverse frequency conversion unit 204 for converting to a normal frequency. In addition, an inter-screen prediction unit 205 that performs inter-screen prediction to generate a predicted image and an intra-screen prediction unit 206 that performs intra-screen prediction are included. And it has the addition part 207 which adds a prediction difference and a prediction image, and produces | generates the decoded image 210, and the reference image memory 208 which memorize | stores the decoded image 210 temporarily. In addition, it has a frame memory 209 that holds prediction modes for all macroblocks of the previous picture.

  Here, the operation of the inter-screen prediction unit 205 will be described. The input encoded stream 201 is provided with inter-screen prediction mode information (pixel value prediction method and block size information) at the time of encoding. Alternatively, when the inter-screen prediction mode information is omitted, a “skip flag” indicating inter-screen prediction mode skip is assigned. When inter-screen prediction mode information is given, a prediction image is generated according to the specified prediction method and block size. When the “skip flag” is assigned, the frame memory 209 is referred to, the prediction mode of the current macroblock is determined based on the prediction mode of the same-position macroblock of the previous picture, and a prediction image is generated.

  In the above, the decoding apparatus determines the prediction mode according to the inter-screen prediction mode skip adoption result (that is, the presence or absence of the skip flag) determined by the encoding apparatus. On the other hand, the decoding apparatus can determine whether to adopt the inter-screen prediction mode skip by itself. In this case, the skip acceptance rule of the encoding device can be used as it is as the skip acceptance rule. If both rules (judgment criteria) are made common, the skip acceptance / rejection result will be well reproduced, and the skip flag added to the encoded stream to be transmitted can be omitted.

In the present embodiment, a specific example of how to determine whether to adopt inter-screen prediction mode skip (adoption rule) will be described.
FIG. 8 is an example of switching between adoption and non-use of the inter-screen prediction mode skip with reference to the magnitude of the motion vector. As shown in FIG. 2, the motion vector MV is a positional deviation amount between the target block 304 in the encoding target image and the highly correlated block 305 in the reference image. For determination, a threshold value is set in advance for the motion vector MV.

  (A) is a case where the motion vector MV of the previous picture is smaller than a predetermined value (threshold). When the MV is small, the change in the image between the screens is small, and the prediction mode for each macroblock is often the same in the inter-screen prediction. Therefore, inter-screen prediction mode skip is adopted, and the prediction mode of the same-position macroblock MB1 of the previous picture is used as the prediction mode of the macroblock MB2.

  (B) is a case where the motion vector MV of the previous picture is larger than a predetermined value. When the MV is large, the image change between the screens is large, and the prediction mode for each macroblock is often different in the inter-screen prediction. Therefore, the inter-screen prediction mode skip is not adopted, and a “non-skip mode” for uniquely determining the optimal prediction mode is set for the macroblock MB2.

  FIG. 9 is an example of switching between adoption and non-use of the inter-screen prediction mode skip with reference to the inter-screen prediction mode of the adjacent macroblock. That is, if the inter-screen prediction mode of the adjacent macroblock of the current picture is the prediction mode skip, the prediction mode skip is also adopted for the macroblock.

  (A) is a case where the adjacent macroblock MB3 adopts the inter-picture prediction mode skip, and the prediction mode skip is also adopted for the macroblock MB2 and the prediction mode of the same-position macroblock MB1 of the previous picture is used. .

  (B) is a case where the adjacent macroblock MB3 does not employ the inter-picture prediction mode skip (non-skip mode), and the non-skip mode is set for the macroblock MB2 and an optimal prediction mode is uniquely obtained.

  In the above embodiment, the inter-screen prediction mode skip between the forward prediction images (between P pictures) has been described, but the same method can be applied to the inter-screen prediction mode skip between the bi-prediction images (between B pictures). . However, in the case of a B picture, since the number of pictures that can be referred to increases, the degree of freedom of selection increases. Hereinafter, the picture to be referred to will be described in detail.

FIG. 10 is a diagram illustrating a picture to be referred for skipping the inter-picture prediction mode when the encoding target image is a B picture. Here, three methods are shown.
In the case of (a), an image of a B picture having the same picture type is referred to. Here, the previous picture # 3, which is the same B picture, is referred to the encoding target picture # 4. Then, based on the prediction mode of the same-position macroblock in picture # 3, the prediction mode of the target macroblock in picture # 4 is determined.

  In the case of (b), it is possible to refer to forward prediction images (P pictures) that are different picture types. Here, with respect to the encoding target picture # 4, in addition to the previous picture # 3 of the same type (B picture), a different type of P picture # 6 can be referred to. Then, based on the prediction mode of the same-position macroblock in picture # 6, the prediction mode of the target macroblock in picture # 4 is determined. In this case, which of the picture # 3 and the picture # 6 is used may be operated according to the same rule on the encoding side and the decoding side.

  In the case of (c), a P picture that is temporally closer is referred to between the two forward prediction images (P pictures). That is, two P pictures # 6 and # 11 can be referred to the encoding target picture # 8. Here, reference is made to picture # 6 which is closer in time. Then, based on the prediction mode of the same-position macroblock in picture # 6, the prediction mode of the target macroblock in picture # 8 is determined.

  The skip acceptance rule in the case of the inter-screen prediction mode skip between the bi-predictive images described above (between B pictures) is also the same as that in the case of the inter-predictive image (p-picture) described in the above embodiments. Can be determined. Since the picture to be referred to may be other than the previous picture, the frame memories 116 and 209 of the encoding device and the decoding device hold not only the previous picture but also a prediction mode of a picture necessary for reference. To do.

  DESCRIPTION OF SYMBOLS 101 ... Original image, 102 ... Input image memory, 103 ... Block division part, 104 ... Motion search part, 105 ... In-screen prediction part, 106 ... Inter-screen prediction part, 107 ... Mode selection part, 108 ... Subtraction part, 109 ... Frequency conversion unit, 110 ... quantization unit, 111 ... variable length coding unit, 112 ... inverse quantization processing unit, 113 ... inverse frequency conversion unit, 114 ... addition unit, 115 ... reference image memory, 116 ... frame memory, 201 ... encoded stream, 202 ... variable length decoding unit, 203 ... inverse quantization processing unit, 204 ... inverse frequency transform unit, 205 ... inter-screen prediction unit, 206 ... intra-screen prediction unit, 207 ... addition unit, 208 ... see Image memory, 209... Frame memory, 210.

Claims (4)

In the moving image encoding device that encodes the prediction difference between the input image and the prediction image by inter-screen prediction,
A block dividing unit that divides an input image into blocks of a predetermined size among a plurality of sizes;
For the block, an inter-screen prediction unit that generates a prediction image by inter-screen prediction from a reference image;
A subtraction unit that calculates a prediction difference between the predicted image and the input image;
An encoded stream generation unit that generates an encoded stream by performing a frequency conversion process, a quantization process, and a variable-length encoding process on the prediction difference;
A frame memory that holds a prediction mode when a prediction image is generated for each block of the encoded image;
The inter-screen prediction unit performs a prediction mode skip operation for determining the prediction mode of the block based on the prediction mode of the block at the same position of the encoded image held in the frame memory ,
When the prediction mode skip operation is executed by the inter-screen prediction unit, the encoded stream generation unit deletes the prediction mode information in the encoded stream and indicates that the prediction mode skip operation has been executed. A moving picture encoding apparatus characterized by providing a flag . The moving image encoding device according to claim 1,
The video encoding apparatus according to claim 1, wherein the skip flag is assigned in units of a block unit or a stream unit including a plurality of continuous blocks, a GOP unit, a picture unit, or a slice unit. In a video encoding method for encoding a prediction difference between a prediction image by inter-screen prediction and an input image,
A block dividing step for dividing the input image into blocks of a predetermined size among a plurality of sizes;
An inter-screen prediction step for generating a predicted image by inter-screen prediction from a reference image for the block;
A subtraction step of calculating a prediction difference between the predicted image and the input image;
A coded stream generation step of generating a coded stream by performing a frequency conversion process, a quantization process, and a variable length coding process on the prediction difference;
The inter-screen prediction step includes a prediction mode skip operation for determining the prediction mode of the block based on the prediction mode of the block at the same position of the encoded image,
When the prediction mode skip is executed in the inter-screen prediction step, the prediction mode information in the encoded stream is deleted, and a skip flag indicating that the prediction mode skip operation is executed is added. A moving image encoding method. In the moving image encoding method according to claim 3,
The moving image encoding method according to claim 1, wherein the skip flag is assigned in units of a block unit or a stream unit composed of a plurality of continuous blocks, a GOP unit, a picture unit, or a slice unit.

Images