JP7242811B2 - Image encoding method and image decoding method - Google Patents

Description

The present invention relates to encoding and decoding methods for moving and still images.

In recent years, ITU-T Rec. 264). In H.264, prediction processing, transform processing, and entropy coding processing are performed in rectangular block units (eg, 16×16 pixel block units, 8×8 pixel block units, etc.). In the prediction process, a rectangular block to be coded (block to be coded) is subjected to motion compensation for prediction in the temporal direction by referring to an already coded frame (reference frame). In such motion compensation, it is necessary to encode motion information including a motion vector as spatial shift information between a block to be coded and a block referenced in a reference frame, and send the encoded information to the decoding side. Furthermore, when performing motion compensation using a plurality of reference frames, it is necessary to encode the reference frame number together with the motion information. Therefore, the amount of code related to motion information and reference frame numbers may increase.

One example of a method of obtaining a motion vector in motion-compensated prediction is to derive a motion vector to be assigned to a block to be encoded from the motion vectors already assigned to already encoded blocks, and perform prediction based on the derived motion vector. There is a direct mode for generating images (see Patent Documents 1 and 2). Since the direct mode does not encode motion vectors, it is possible to reduce the code amount of motion information. Direct mode is adopted for H.264/AVC, for example.

Patent No. 4020789 U.S. Pat. No. 7,233,621

In the direct mode, the motion vector of the block to be coded is predicted and generated by a fixed method of calculating the motion vector from the median value of the motion vectors of the coded blocks adjacent to the block to be coded. Therefore, the degree of freedom of motion vector calculation is low.

In order to increase the degree of freedom in motion vector calculation, a method has been proposed in which one of a plurality of encoded blocks is selected and a motion vector is assigned to the encoding target block. In this method, selection information identifying the selected block must always be sent so that the decoder can identify the selected encoded block. Therefore, when one of a plurality of encoded blocks is selected and a motion vector to be assigned to the encoding target block is determined, there is a problem that the amount of code related to selection information increases.

SUMMARY OF THE INVENTION It is an object of the present invention to solve the above problems and to provide an image encoding and image decoding method with high encoding efficiency.

An image coding method according to an embodiment of the present invention comprises a first step of selecting at least one motion reference block from coded pixel blocks having motion information; a second step of selecting, from said motion reference blocks, at least one available block of pixel blocks having candidate information, said block having different motion information; a third step of selecting a block; a fourth step of generating a predicted image of the block to be encoded using the motion information of the selected block; and encoding a prediction error between the predicted image and the original image. and a sixth step of encoding selection information specifying the selected block with reference to a code table predetermined according to the number of available blocks.

An image decoding method according to another embodiment of the present invention provides a first step of selecting at least one motion reference block from among decoded pixel blocks having motion information, and applying a second step of selecting at least one available pixel block having motion information candidates and having different motion information from the motion reference blocks; a third step of obtaining selection information for specifying a selection block by decoding the input encoded data with reference to a predetermined code table; a fourth step of selecting one selected block from the selected block; a fifth step of generating a prediction image of the block to be decoded using the motion information of the selected block; and a seventh step of obtaining a decoded image from the predicted image and the predicted residual.

According to the present invention, coding efficiency can be improved.

1 is a block diagram schematically showing the configuration of an image encoding device according to a first embodiment; FIG. FIG. 2 is a diagram showing an example of the size of a macroblock, which is a processing unit for encoding by the image decoding unit shown in FIG. 1; FIG. 8 is a diagram showing another example of the size of a macroblock, which is the unit of encoding processing of the image decoding unit shown in FIG. 1; 2 is a diagram showing the order in which the image encoding unit shown in FIG. 1 encodes pixel blocks in a frame to be encoded; FIG. 2 is a diagram showing an example of a motion information frame held by a motion information memory shown in FIG. 1; FIG. 2 is a flow chart showing an example of a procedure for processing the input image signal of FIG. 1; FIG. 2 is a diagram showing an example of inter prediction processing executed by the motion compensation unit in FIG. 1; FIG. FIG. 4 is a diagram showing another example of inter prediction processing executed by the motion compensation unit in FIG. 1; FIG. 4 is a diagram illustrating an example of the size of motion compensation blocks used for inter prediction processing; FIG. 10 is a diagram showing another example of the size of motion compensation blocks used for inter prediction processing; FIG. 10 is a diagram showing still another example of the size of motion compensation blocks used for inter prediction processing; FIG. 10 is a diagram showing another example of the size of motion compensation blocks used for inter prediction processing; FIG. 4 is a diagram showing an example of arrangement of spatial direction and temporal direction motion reference blocks; FIG. 10 is a diagram showing another example of arrangement of spatial direction motion reference blocks; FIG. 8C is a diagram showing relative positions of spatial direction motion reference blocks with respect to the encoding target block shown in FIG. 8B; FIG. 10 is a diagram showing another example of arrangement of temporal motion reference blocks; FIG. 10 is a diagram showing still another example of the arrangement of temporal motion reference blocks; FIG. 10 is a diagram showing yet another example of the arrangement of temporal motion reference blocks; 2 is a flow chart showing an example of a method of selecting a usable block from among motion reference blocks by the usable block obtaining unit of FIG. 1; Fig. 10 is a diagram showing an example of available blocks selected according to the method of Fig. 9 from among the motion reference blocks shown in Fig. 8; 2 is a diagram showing an example of available block information output by the available block acquisition unit in FIG. 1; FIG. 2 is a diagram showing an example of identity determination of motion information between blocks by the available block obtaining unit in FIG. 1; FIG. FIG. 8 is a diagram showing another example of determination of identity of motion information between blocks by the available block acquisition unit in FIG. 1 ; FIG. 10 is a diagram showing still another example of determination of identity of motion information between blocks by the available block acquisition unit in FIG. 1 ; FIG. 8 is a diagram showing another example of determination of identity of motion information between blocks by the available block acquisition unit in FIG. 1 ; FIG. 10 is a diagram showing still another example of determination of identity of motion information between blocks by the available block acquisition unit in FIG. 1 ; FIG. 8 is a diagram showing another example of determination of identity of motion information between blocks by the available block acquisition unit in FIG. 1 ; 2 is a block diagram schematically showing the configuration of a prediction unit in FIG. 1; FIG. FIG. 14 is a diagram showing a group of motion information output by the temporal direction motion information acquisition unit in FIG. 13; FIG. 14 is an explanatory diagram for explaining interpolation processing with a few pixel precision that can be used in the motion compensation processing by the motion compensation unit in FIG. 13; FIG. 14 is a flow chart showing an example of the operation of the prediction unit in FIG. 13; FIG. 14 is a diagram showing how the motion compensation unit in FIG. 13 copies the motion information of the temporal motion reference block to the encoding target block; FIG. 2 is a block diagram schematically showing the configuration of a variable-length coding unit in FIG. 1; FIG. FIG. 10 is a diagram showing an example of generating syntax according to available block information; FIG. 10 is a diagram showing an example of binarization of selected block information syntax corresponding to available block information; FIG. 4 is an explanatory diagram for explaining scaling of motion information; FIG. 4 shows a syntax structure according to an embodiment; FIG. 4 is a diagram showing an example of macroblock layer syntax according to the first embodiment; FIG. 4 is a diagram showing another example of macroblock layer syntax according to the first embodiment; H. 2 is a diagram showing a code table corresponding to mb_type and mb_type in B slice in H.264. FIG. It is a figure which shows an example of the code table which concerns on embodiment. H. 2 is a diagram showing a code table corresponding to mb_type and mb_type in P-slice in H.264. FIG. It is a figure which shows the other example of the code table which concerns on embodiment. FIG. 10 illustrates an example codebook corresponding to mb_type and mb_type in B slices, according to an embodiment; FIG. 11 shows another example of codebooks corresponding to mb_type and mb_type in P slices according to an embodiment; FIG. 10 is a block diagram schematically showing the configuration of an image encoding device according to a second embodiment; FIG. FIG. 27 is a block diagram schematically showing the configuration of a prediction unit in FIG. 26; FIG. 28 is a block diagram schematically showing the configuration of a second predictor in FIG. 27; FIG. 27 is a block diagram schematically showing the configuration of a variable-length coding unit in FIG. 26; FIG. 10 is a diagram showing an example of macroblock layer syntax according to the second embodiment; FIG. 10 is a diagram showing another example of macroblock layer syntax according to the second embodiment; FIG. 12 is a block diagram schematically showing an image decoding device according to a third embodiment; FIG. 32 is a block diagram showing in more detail the coded sequence decoding unit shown in FIG. 31; FIG. 32 is a block diagram showing in more detail the predictor shown in FIG. 31; FIG. FIG. 11 is a block diagram schematically showing an image decoding device according to a fourth embodiment; FIG. 34 is a block diagram showing in more detail the coded sequence decoding unit shown in FIG. 33; FIG. 34 is a block diagram showing in more detail the predictor shown in FIG. 33; FIG.

Hereinafter, image encoding and image decoding methods and apparatuses according to embodiments of the present invention will be described with reference to the drawings as necessary. It should be noted that, in the following embodiments, the same numbered parts are assumed to perform the same operation, and redundant description will be omitted.

(First embodiment)
FIG. 1 schematically shows the configuration of an image coding apparatus according to the first embodiment of the invention. This image encoding apparatus comprises an image encoding section 100, an encoding control section 150 and an output buffer 120, as shown in FIG. This image encoding device may be implemented by hardware such as an LSI chip, or may be implemented by causing a computer to execute an image encoding program.

An original image (input image signal) 10, which is a moving image or a still image, is input to the image encoding unit 100, for example, in units of pixel blocks obtained by dividing the original image. The image encoding unit 100 compresses and encodes the input image signal 10 to generate encoded data 14, as will be described later in detail. The generated encoded data 14 is temporarily stored in the output buffer 120 and sent to a storage system (storage media) or a transmission system (communication line) (not shown) at output timings managed by the encoding control unit 150. .

The encoding control unit 150 controls overall encoding processing of the image encoding unit 100, such as feedback control of generated code amount, quantization control, prediction mode control, and entropy encoding control. Specifically, the encoding control unit 150 provides encoding control information 50 to the image encoding unit 100 and receives feedback information 51 from the image encoding unit 100 as appropriate. The coding control information 50 includes prediction information, motion information 18, quantization parameter information, and the like. The prediction information includes prediction mode information and block size information. The motion information 18 includes motion vectors, reference frame numbers and prediction directions (unidirectional prediction, bidirectional prediction). The quantization parameter information includes quantization parameters such as a quantization width (quantization step size) and a quantization matrix. The feedback information 51 includes the amount of code generated by the image encoding unit 100, and is used, for example, to determine the quantization parameter.

The image encoding unit 100 encodes the input image signal 10 in units of pixel blocks (for example, macroblocks, subblocks, one pixel, etc.) into which the original image is divided. Therefore, the input image signal 10 is sequentially input to the image encoding unit 100 in units of pixel blocks obtained by dividing the original image. In the present embodiment, the encoding processing unit is a macroblock, and the encoding target pixel block (macroblock) corresponding to the input image signal 10 is simply referred to as the encoding target block. An image frame including an encoding target block, that is, an encoding target image frame is referred to as an encoding target frame.

Such an encoding target block may be, for example, a 16×16 pixel block as shown in FIG. 2A or a 64×64 pixel block as shown in FIG. 2B. Also, the encoding target block may be a 32×32 pixel block, an 8×8 pixel block, or the like. Moreover, the shape of the macroblock is not limited to the square shape shown in FIGS. 2A and 2B, and may be set to an arbitrary shape such as a rectangular shape. Furthermore, the processing unit is not limited to pixel blocks such as macroblocks, but may be frames or fields.

Note that the encoding process for each pixel block in the encoding target frame may be performed in any order. In this embodiment, in order to simplify the explanation, as shown in FIG. 3, the pixel blocks are arranged row by row from the upper left pixel block to the lower right pixel block of the frame to be encoded, that is, in raster scan order. shall be subjected to encoding processing.

The image coding unit 100 shown in FIG. , a motion information memory 108 and a usable block acquisition unit 109 .

In the image encoding unit 100 , an input image signal 10 is input to a prediction unit 101 and a subtractor 102 . The subtractor 102 receives the input image signal 10 and also receives the predicted image signal 11 from the prediction unit 101, which will be described later. A subtractor 102 calculates the difference between the input image signal 10 and the predicted image signal 11 to generate a predicted error image signal 12 .

The transform/quantization unit 103 receives the prediction error image signal 12 from the subtractor 102, performs transform processing on the received prediction error image signal 12, and generates transform coefficients. The transform processing is, for example, an orthogonal transform such as a discrete cosine transform (DCT). In another embodiment, the transform/quantization unit 103 may generate transform coefficients using methods such as wavelet transform and independent component analysis instead of discrete cosine transform. Further, the transform/quantization unit 103 quantizes the generated transform coefficients based on the quantization parameter given by the encoding control unit 150 . The quantized transform coefficients (transform coefficient information) 13 are output to the variable length coding unit 104 and the inverse quantization/inverse transform unit 105 .

The inverse quantization/inverse transform unit 105 inversely quantizes the quantized transform coefficients 13 according to the quantization parameter given by the encoding control unit 150, that is, the same quantization parameter as the transform/quantization unit 103. . Subsequently, an inverse quantization/inverse transform unit 105 performs inverse transform on the inversely quantized transform coefficients to generate a decoded prediction error signal 15 . The inverse transform processing by the inverse quantization/inverse transform unit 105 matches the inverse transform processing of the transform processing by the transform/quantization unit 103 . For example, the inverse transform processing is inverse discrete cosine transform (IDCT) or inverse wavelet transform.

The adder 106 receives the decoded prediction error signal 15 from the inverse quantization/inverse transform unit 105 and further receives the predicted image signal 11 from the prediction unit 101 . Adder 106 adds decoded prediction error signal 15 and predicted image signal 11 to generate locally decoded image signal 16 . The generated locally decoded image signal 16 is stored in the frame memory 107 as the reference image signal 17 . The reference image signal 17 stored in the frame memory 107 is read out and referred to by the prediction unit 101 when encoding the subsequent encoding target block.

The prediction unit 101 receives the reference image signal 17 from the frame memory 107, and also receives the available block information 30 from the available block acquisition unit 109, which will be described later. Furthermore, the prediction unit 101 receives reference motion information 19 from a motion information memory 108, which will be described later. The prediction unit 101 generates a prediction image signal 11 of the encoding target block, motion information 18 and selected block information 31 based on the reference image signal 17 , reference motion information 19 and available block information 30 . Specifically, the prediction unit 101 generates the motion information 18 and the selected block information 31 based on the available block information 30 and the reference motion information 19, and based on the motion information 18, the motion information selection unit 118 generates: A motion compensator 113 for generating the predicted image signal 11 is provided. Predicted image signal 11 is sent to subtractor 102 and adder 106 . The motion information 18 is stored in the motion information memory 108 for subsequent prediction processing for the coding target block. Also, the selected block information 31 is sent to the variable length coding section 104 . The prediction unit 101 will be described later in detail.

The motion information memory 108 temporarily stores the motion information 18 as the reference motion information 19 . FIG. 4 shows an example of the configuration of the motion information memory 108. As shown in FIG. As shown in FIG. 4, the motion information memory 108 holds the reference motion information 19 in frame units, and the reference motion information 19 forms the motion information frame 25 . The motion information memory 108 is sequentially input with motion information 18 relating to encoded blocks, and as a result, the motion information memory 108 holds a plurality of motion information frames 25 with different encoding times.

The reference motion information 19 is held in the motion information frame 25 in predetermined block units (eg, 4×4 pixel block units). The motion vector block 28 shown in FIG. 4 represents a block of pixels of the same size as the block to be coded, the available block and the selected block, for example a 16×16 pixel block. A motion vector is assigned to the motion vector block 28 for each 4×4 pixel block, for example. Inter prediction processing using motion vector blocks is called motion vector block prediction processing. The reference motion information 19 held by the motion information memory 108 is read by the prediction unit 101 when the motion information 18 is generated. The motion information 18 possessed by the available block, which will be described later, refers to the reference motion information 19 held in the area in the motion information memory 108 where the available block is located.

Note that the motion information memory 108 is not limited to holding the reference motion information 19 in units of 4×4 pixel blocks, and may hold the reference motion information 19 in units of other pixel blocks. For example, the pixel block unit for the reference motion information 19 may be one pixel, or may be a 2×2 pixel block. Also, the shape of the pixel block related to the reference motion information 19 is not limited to the square shape, and may be any shape.

The available block acquisition unit 109 of FIG. Select available blocks that can be used for the prediction process of . The selected available block is sent as available block information 30 to the prediction section 101 and the variable length coding section 104 . A coded block that is a candidate for selecting an available block is called a motion reference block. The method of selecting motion reference blocks and available blocks will be described in detail later.

In addition to the transform coefficient information 13, the variable-length coding unit 104 receives the selected block information 31 from the prediction unit 101, prediction information and coding parameters such as quantization parameters from the coding control unit 150, and receives the available block acquisition unit. It receives available block information 30 from 109 . The variable-length coding unit 104 performs entropy coding (e.g., equal-length coding, Huffman coding, arithmetic coding, etc.) on the quantized transform coefficients 13, selected block information 31, available block information 30, and coding parameters. ) to generate encoded data 14 . The coding parameters include selected block information 31 and prediction information as well as all parameters necessary for decoding, such as information on transform coefficients and information on quantization. The generated encoded data 14 is temporarily stored in the output buffer 120 and sent to a storage system or transmission system (not shown).

FIG. 5 shows the processing procedure of the input image signal 10. As shown in FIG. As shown in FIG. 5, first, a predicted image signal 11 is generated by the prediction unit 101 (step S501). In generating the predicted image signal 11 in step S501, one of the available blocks, which will be described later, is selected as a selected block, and the selected block information 31, the motion information of the selected block, and the reference image signal 17 are used to generate the predicted image. A signal 11 is created. A difference between the predicted image signal 11 and the input image signal 10 is calculated by the subtractor 102 to generate the predicted error image signal 12 (step S502).

Subsequently, the transform/quantization unit 103 performs orthogonal transform and quantization on the prediction error image signal 12 to generate transform coefficient information 13 (step S503). The transform coefficient information 13 and the selected block information 31 are sent to the variable length coding unit 104 and subjected to variable length coding to generate the coded data 14 (step S504). Further, in step S504, the code table is switched according to the selected block information 31 so that the code table has the same number of entries as the number of usable blocks, and the selected block information 31 is variable-length encoded. A bit stream 20 of coded data is sent to a storage system or transmission line (not shown).

The transform coefficient information 13 generated in step S503 is inversely quantized by the inverse quantization/inverse transforming unit 105, subjected to inverse transform processing, and becomes a decoded prediction error signal 15 (step S505). The decoded prediction error signal 15 is added to the reference image signal 17 used in step S501 to become the local decoded image signal 16 (step S506), which is stored in the frame memory 107 as a reference image signal (step S507).

Next, each configuration of the image encoding unit 100 described above will be described in more detail.
A plurality of prediction modes are prepared in the image encoding unit 100 of FIG. 1, and each prediction mode differs in the method of generating the predicted image signal 11 and the motion compensation block size. Specifically, the method by which the prediction unit 101 generates the predicted image signal 11 can be broadly divided into intra prediction (intra-frame prediction), and inter prediction (inter-frame prediction) in which a prediction image is generated using a reference image signal 17 relating to one or more encoded reference frames (reference fields). The prediction unit 101 selectively switches between intra prediction and inter prediction to generate a prediction image signal 11 of the encoding target block.

FIG. 6A shows an example of inter prediction by the motion compensation unit 113. FIG. In inter-prediction, as shown in FIG. 6A , a block in a reference frame one frame previous to which encoding has already been completed and at the same position as the encoding target block (also referred to as a prediction block) 23 from , the predicted image signal 11 is generated using the reference image signal 17 for the block 24 at the position spatially shifted according to the motion vector 18 a contained in the motion information 18 . That is, in generating the predicted image signal 11, the reference image signal 17 related to the block 24 in the reference frame, which is specified by the position (coordinates) of the encoding target block and the motion vector 18a included in the motion information 18, is used. . In inter-prediction, motion compensation with a small number of pixels accuracy (for example, 1/2 pixel accuracy or 1/4 pixel accuracy) is possible, and interpolation pixel values are generated by performing filtering processing on the reference image signal 17. be done. For example, H. H.264 allows interpolation processing up to 1/4 pixel accuracy for a luminance signal. When performing motion compensation with 1/4 pixel precision, the information amount of the motion information 18 is four times the integer pixel precision.

Note that inter prediction is not limited to the example of using the reference frame one frame before as shown in FIG. 6A, and any encoded reference frame may be used as shown in FIG. 6B. . When reference image signals 17 relating to a plurality of reference frames at different temporal positions are held, information indicating from which temporal position the reference image signal 17 the predicted image signal 11 was generated is represented by a reference frame number. A reference frame number is included in the motion information 18 . The reference frame number can be changed for each area (picture, block, etc.). That is, different reference frames can be used for each pixel block. As an example, when the encoded reference frame one frame before is used for prediction, the reference frame number of this region is set to 0, and when the encoded reference frame two frames before is used for prediction, The reference frame number for this region is set to one. As another example, when the reference image signal 17 for only one frame is held in the frame memory 107 (the number of reference frames is 1), the reference frame number is always set to 0.

Furthermore, in inter prediction, a block size suitable for the encoding target block can be selected from among a plurality of motion compensation blocks. That is, the encoding target block may be divided into a plurality of small pixel blocks, and motion compensation may be performed for each small pixel block. 7A to 7C show the motion compensation block size in units of macroblocks, and FIG. 7D shows the motion compensation block size in units of subblocks (pixel blocks of 8×8 pixels or less). As shown in FIG. 7A, when the encoding target block is 64×64 pixels, the motion compensation block may be a 64×64 pixel block, a 64×32 pixel block, a 32×64 pixel block, a 32×32 pixel block, or the like. can be selected. Also, as shown in FIG. 7B, when the encoding target block is 32×32 pixels, the motion compensation block is a 32×32 pixel block, a 32×16 pixel block, a 16×32 pixel block, or a 16×16 pixel block. Blocks and the like can be selected. Furthermore, as shown in FIG. 7C, when the encoding target block is 16×16 pixels, the motion compensation block is a 16×16 pixel block, a 16×8 pixel block, an 8×16 pixel block or an 8×8 pixel block. It can be set as a block or the like. Furthermore, as shown in FIG. 7D, if the block to be coded is 8×8 pixels, the motion compensation block can be an 8×8 pixel block, an 8×4 pixel block, a 4×8 pixel block or a 4×4 pixel block. A pixel block or the like can be selected.

As described above, since a small pixel block (for example, a 4×4 pixel block) in a reference frame used for inter prediction has motion information 18, according to the local properties of the input image signal 10, the optimal Motion compensated block shapes and motion vectors can be used. Also, the macroblocks and sub-macroblocks in FIGS. 7A to 7D can be arbitrarily combined. When the encoding target block is a 64×64 pixel block as shown in FIG. 7A, each block size shown in FIG. 7B is set for each of four 32×32 pixel blocks obtained by dividing the 64×64 pixel block. By selecting, blocks of 64×64 to 16×16 pixels can be used hierarchically. Similarly, when block sizes up to the block size shown in FIG. 7D can be selected, block sizes from 64×64 to 4×4 can be hierarchically used.

Next, motion reference blocks will be described with reference to FIGS. 8A to 8F.
The motion reference block is selected from encoded regions (blocks) in the encoding target frame and the reference frame according to a method determined by both the image encoding device in FIG. 1 and the image decoding device to be described later. be. FIG. 8A shows an example of placement of motion reference blocks selected according to the position of the encoding target block. In the example of FIG. 8A, nine motion reference blocks A to D and TA to TE are selected from coded regions in the target frame and the reference frame. Specifically, four blocks A, B, C, and D adjacent to the left, upper, upper right, and upper left of the target block for encoding are selected as motion reference blocks from the target frame for encoding. A block TA at the same position as the block to be encoded and four pixel blocks TB, TC, TD and TE adjacent to the right, bottom, left and top of the block TA are selected as motion reference blocks. In this embodiment, the motion reference block selected from the encoding target frame is called a spatial motion reference block, and the motion reference block selected from the reference frame is called a temporal motion reference block. The symbol p attached to each motion reference block in FIG. 8A indicates the index of the motion reference block. The indices are numbered in the order of motion reference blocks in the temporal direction and the spatial direction, but this is not a limitation, and the order may not necessarily be as long as the indices do not overlap. For example, motion reference blocks in the temporal and spatial directions may be numbered in random order.

Note that the spatial direction motion reference block is not limited to the example shown in FIG. 8A. As shown in FIG. macroblocks, etc.). In this case, the relative position (dx, dy) from the upper left pixel e to each pixel a, b, c, d in the encoding target block is set as shown in FIG. 8C. Here, in the examples shown in FIGS. 8A and 8B, macroblocks are shown as being N×N pixel blocks.

Also, as shown in FIG. 8D, all blocks A1 to A4, B1, B2, C, and D adjacent to the encoding target block may be selected as spatial direction motion reference blocks. In the example of FIG. 8D, the number of spatial motion reference blocks is eight.

Furthermore, the temporal motion reference blocks may overlap each block TA-TE partially as shown in FIG. It doesn't matter if In FIG. 8E, the portion where the temporal motion reference blocks TA and TB overlap is indicated by diagonal lines. Furthermore, the temporal motion reference block is not necessarily limited to the block at the position (Collocate position) corresponding to the target block to be encoded and the blocks positioned therearound, and may be arranged at any position within the reference frame. It doesn't matter if it's a block with For example, a block in a reference frame identified by the position of the reference block and the motion information 18 of any encoded block adjacent to the target block to be encoded is defined as a center block (eg, block TA), and this The central block and its surrounding blocks may be selected as temporal motion reference blocks. Furthermore, the temporal reference blocks do not have to be arranged at equal intervals from the center block.

In any of the above cases, if the number and positions of spatial and temporal motion reference blocks are determined in advance by the encoding device and the decoding device, how are the number and positions of the motion reference blocks set? I don't mind. Also, the size of the motion reference block does not necessarily have to be the same size as the encoding target block. For example, as shown in FIG. 8D, the size of the motion reference block may be larger or smaller than the encoding target block. Furthermore, the motion reference block is not limited to a square shape, and may be set to an arbitrary shape such as a rectangular shape. Also, the motion reference block may be set to any size.

Also, motion reference blocks and usable blocks may be arranged in only one of the temporal direction and the spatial direction. In addition, temporal motion reference blocks and usable blocks may be arranged according to slice types such as P slices and B slices, or spatial motion reference blocks and usable blocks may be arranged.

FIG. 9 shows how the available block acquisition unit 109 selects available blocks from motion reference blocks. The usable blocks are blocks to which motion information can be applied to the encoding target block, and have motion information different from each other. The available block acquisition unit 109 refers to the reference motion information 19, determines whether each motion reference block is a available block, and outputs available block information 30 according to the method shown in FIG.

As shown in FIG. 9, first, a motion reference block whose index p is zero is selected (S800). In the description of FIG. 9, it is assumed that motion reference blocks are processed in order from index p 0 to M−1 (M indicates the number of motion reference blocks). In addition, it is assumed that the usability determination processing for motion reference blocks with indices p from 0 to p−1 is completed, and the index of the motion reference block whose usability is to be determined is p. .

The available block acquisition unit 109 determines whether or not the motion reference block p has motion information 18, that is, whether or not at least one motion vector is assigned (S801). If the motion reference block p does not have a motion vector, that is, the temporal motion reference block p is a block in an I slice that does not have motion information, or If all the small pixel blocks have been intra-prediction coded, the process proceeds to step S805. In step S805, the motion reference block p is determined as an unavailable block.

If the motion reference block p has motion information in step S801, the process proceeds to step S802. The available block acquisition unit 109 selects a motion reference block q (available block q) that has already been selected as a available block. Here, q is a value smaller than p. Subsequently, the available block obtaining unit 109 compares the motion information 18 of the motion reference block p with the motion information 18 of the available block q, and determines whether or not they have the same motion information (S803). . If it is determined that the motion information 18 of the motion reference block p is the same as the motion information 18 of the motion reference block q selected as a usable block, the process proceeds to step S805, where the motion reference block p is an unusable block. is determined.

If it is determined in step S803 that the motion information 18 of the motion reference block p and the motion information 18 of the available block q are not the same for all available blocks q that satisfy q<p, the process proceeds to step S804. . In step S804, the usable block acquisition unit 109 determines the motion reference block p as a usable block.

When it is determined that the motion reference block p is a usable block or an unusable block, the usable block obtaining unit 109 determines whether or not all the motion reference blocks have been subjected to the usable determination ( S806). If there is a motion reference block for which availability determination has not been performed, for example, if p<M−1, the process proceeds to step S807. Subsequently, the available block acquisition unit 109 increments the index p by 1 (step S807), and executes steps S801 to S806 again. When the usability determination is executed for all motion reference blocks in step S806, the usability determination processing ends.

By executing the above-described usability determination process, it is determined whether each motion reference block is a usable block or an unusable block. The available block acquisition unit 109 generates available block information 30 including information on available blocks. By selecting the available blocks from among the motion reference blocks in this way, the amount of information regarding the available block information 30 is reduced, and as a result, the amount of encoded data 14 can be reduced.

FIG. 10 shows an example of the result of performing the usability determination process on the motion reference block shown in FIG. 8A. In FIG. 10, two spatial motion reference blocks (p=0,1) and two temporal motion reference blocks (p=5,8) are determined to be available blocks. An example of available block information 30 for the example of FIG. 10 is shown in FIG. As shown in FIG. 11, the available block information 30 includes motion reference block indices, availability and motion reference block names. In the example of FIG. 11, indexes p=0, 1, 5, 8 are available blocks, and the number of available blocks is four. The prediction unit 101 selects one optimal available block from these available blocks as a selected block, and outputs information (selected block information) 31 on the selected block. The selected block information 31 includes the number of available blocks and the index value of the selected available block. For example, if the number of available blocks is four, the corresponding selected block information 31 is encoded by the variable length encoding unit 104 using a code table with four entries at maximum.

Note that in step S801 of FIG. 9, if at least one of the blocks in the temporal direction motion reference block p is an intra-prediction-encoded block, the available block obtaining unit 109 uses the motion reference block p. It may be judged as an impossible block. That is, only when all blocks in the temporal motion reference block p have been coded by inter prediction, the process may proceed to step S802.

FIGS. 12A to 12E show examples in which the comparison of the motion information 18 in step S803 determines that the motion information 18 of the motion reference block p and the motion information 18 of the available block q are the same. 12A-12E each show a plurality of hatched blocks and two white blocks. 12A to 12E assume that the shaded blocks are not considered and the motion information 18 of these two white blocks are compared for the sake of simplicity. Let one of the two white blocks be the motion reference block p and the other be the motion reference block q already determined to be available (available block q). Unless otherwise specified, either of the two white blocks may be the motion reference block p.

FIG. 12A shows an example where both the motion reference block p and the available block q are spatial blocks. In the example of FIG. 12A, if the motion information 18 of blocks A and B are the same, it is determined that the motion information 18 is the same. At this time, the sizes of blocks A and B need not be the same.

FIG. 12B shows an example where one of motion reference block p and available block q is block A in the spatial direction and the other is block TB in the temporal direction. In FIG. 12B, there is one block with motion information within block TB in the temporal direction. If the motion information 18 of the block TB in the temporal direction and the motion information 18 of the block A in the spatial direction are the same, it is determined that the motion information 18 are the same. At this time, the sizes of blocks A and TB do not have to be the same.

FIG. 12C shows another example where one of motion reference block p and available block q is block A in spatial direction and the other is block TB in temporal direction. FIG. 12C shows a case where the block TB in the time direction is divided into a plurality of small blocks, and there are a plurality of small blocks having motion information 18 . In the example of FIG. 12C, the motion information 18 is identical if all blocks with motion information 18 have the same motion information 18 and that motion information 18 is identical to the motion information 18 of block A in the spatial direction. is determined. At this time, the sizes of blocks A and TB do not have to be the same.

FIG. 12D shows an example where the motion reference block p and available block q are both temporal blocks. In this case, if the motion information 18 of the blocks TB and TE are the same, it is determined that the motion information 18 is the same.

FIG. 12E shows another example where motion reference block p and available block q are both temporal blocks. FIG. 12E shows a case where each of blocks TB and TE in the time direction is divided into a plurality of small blocks, and there are a plurality of small blocks each having motion information 18 . In this case, the motion information 18 is compared for each small block in the block, and if the motion information 18 is the same for all the small blocks, the motion information 18 for the block TB and the motion information 18 for the block TE are the same. It is determined that there is

FIG. 12F shows yet another example where motion reference block p and available block q are both temporal blocks. FIG. 12F shows a case where the block TE in the time direction is divided into a plurality of small blocks, and there are a plurality of small blocks having motion information 18 in the block TE. If all the motion information 18 of the block TE is the same motion information 18 and is the same as the motion information 18 that the block TD has, it is determined that the motion information 18 of the blocks TD and TE are the same.

Thus, in step S803, it is determined whether or not the motion information 18 of the motion reference block p and the motion information 18 of the available block q are the same. In the examples of FIGS. 12A to 12F, the number of available blocks q compared with the motion reference block p is 1, but when the number of available blocks q is 2 or more, the motion information of the motion reference block p 18 with the motion information 18 of each available block q. Further, when scaling, which will be described later, is applied, the motion information 18 after scaling becomes the motion information 18 described above.

Note that the determination that the motion information of the motion reference block p and the motion information of the available block q are the same is not limited to the case where the motion vectors included in the motion information completely match. For example, if the norm of the difference between two motion vectors is within a predetermined range, it may be considered that the motion information of the motion reference block p and the motion information of the available block q are substantially the same.

FIG. 13 shows a more detailed configuration of the prediction section 101. As shown in FIG. As described above, the prediction unit 101 receives the available block information 30, the reference motion information 19 and the reference image signal 17, and outputs the predicted image signal 11, the motion information 18 and the selected block information 31. FIG. The motion information selection unit 118 includes a spatial direction motion information acquisition unit 110, a temporal direction motion information acquisition unit 111, and a motion information changeover switch 112, as shown in FIG.

The spatial motion information acquisition unit 110 receives the available block information 30 and the reference motion information 19 regarding the spatial motion reference block. The spatial direction motion information acquisition unit 110 outputs motion information 18A including motion information of each available block located in the spatial direction and index values of the available blocks. When the information shown in FIG. 11 is input as the available block information 30, the spatial direction motion information acquisition unit 110 generates two motion information outputs 18A, each motion information output 18A representing the available block and its utilization. It contains the motion information 19 that the possible blocks have.

The temporal motion information acquisition unit 111 receives the available block information 30 and the reference motion information 19 regarding the temporal motion reference block. The temporal motion information acquisition unit 111 outputs the motion information 19 of the available temporal motion reference block identified by the available block information 30 and the index value of the available block as the motion information 18B. A temporal motion reference block is divided into a plurality of small pixel blocks, and each small pixel block has motion information 19 . The motion information 18B output by the temporal motion information acquisition unit 111 includes a group of motion information 19 possessed by each small pixel block in the usable block, as shown in FIG. When the motion information 18B includes a group of motion information 19, motion compensation prediction can be performed on the block to be encoded in units of small pixel blocks obtained by dividing the block to be encoded. When the information shown in FIG. 11 is input as the available block information 30, the temporal motion information acquisition unit 111 generates two motion information outputs 18B, each motion information output representing the available block and its available block. It contains a group of motion information 19 that the block has.

Note that the temporal motion information acquisition unit 111 obtains the average value or representative value of the motion vectors included in the motion information 19 of each pixel small block, and outputs the average value or representative value of the motion vectors as the motion information 18B. I don't mind.

The motion information changeover switch 112 in FIG. 13 selects one appropriate usable block as a selection block based on the motion information 18A and 18B output from the spatial direction motion information acquisition section 110 and the temporal direction motion information acquisition section 111. and outputs the motion information 18 (or a group of motion information 18) corresponding to the selected block to the motion compensator 113 . Also, the motion information changeover switch 112 outputs selected block information 31 regarding the selected block. The selected block information 31 includes the index p or the name of the motion reference block, and is also simply referred to as selection information. The selected block information 31 is not limited to the index p and the name of the motion reference block, and may be any information as long as the position of the selected block can be specified.

The motion information changeover switch 112 selects, as a selected block, an available block that minimizes the coding cost derived from, for example, the cost formula shown in Equation 1 below.

where J denotes the coding cost and D denotes the coding distortion representing the sum of squared errors between the input image signal 10 and the reference image signal 17 . Also, R indicates the code amount estimated by provisional encoding, and λ. Indicates a Lagrangian undetermined coefficient determined by a quantization width or the like. Instead of Equation 1, the encoding cost J may be calculated using only the code amount R or the encoding distortion D, and the value approximated to the code amount R or the encoding distortion D is used to calculate the You can even create a cost function. Furthermore, the coding distortion D is not limited to the sum of squared errors, and may be the sum of absolute values (SAD: sums of absolute difference) of prediction errors. For the code amount R, only the code amount related to the motion information 18 may be used. In addition, it is not limited to the example in which the available block with the lowest coding cost is selected as the selected block. It does not matter if it is selected.

The motion compensation unit 113 derives the position of the pixel block from which the reference image signal 17 is extracted as the predicted image signal 11, based on the motion information (or motion information group) of the selected block selected by the motion information selection unit 118. do. When a group of motion information is input to the motion compensation unit 113, the motion compensation unit 113 divides the pixel block from which the reference image signal 17 is extracted as the predicted image signal 11 into small pixel blocks (for example, 4×4 pixel blocks). and by applying the corresponding motion information 18 to each of these small pixel blocks, the predicted image signal 11 is obtained from the reference image signal 17 . The position of the block from which the predicted image signal 11 is obtained is, for example, a position shifted in the spatial direction from the small pixel block according to the motion vector 18a included in the motion information 18, as shown in FIG. 4A.

Motion compensation processing for a block to be coded is described in H.264. A motion compensation process similar to H.264 can be used. Here, as an example, an interpolation method with quarter-pixel precision will be specifically described. In quarter-pixel precision interpolation, if each component of the motion vector is a multiple of 4, the motion vector points to integer pixel locations. Otherwise, the motion vector points to the predicted position corresponding to the fractional precision interpolated position.

Here, x and y indicate vertical and horizontal indices indicating the starting position (for example, upper left vertex) of the block to be predicted, and x_pos and y_pos indicate corresponding prediction positions of the reference image signal 17 . (mv_x, mv_y) denotes a motion vector with quarter-pixel accuracy. Next, for the determined pixel positions, the pixel positions corresponding to the reference image signal 17 are supplemented or interpolated to generate predicted pixels. In FIG. An example of H.264 predicted pixel generation is shown. In FIG. 15, squares indicated by capital letters (diagonal squares) indicate pixels at integer positions, and hatched squares indicate interpolation pixels at 1/2 pixel positions. there is A white square indicates an interpolated pixel corresponding to a 1/4 pixel position. For example, in FIG. 15, interpolation processing of 1/2 pixels corresponding to positions of alphabets b and h is calculated by Equation 3 below.

Here, alphabets (eg, b, h, C1, etc.) shown in Equation 3 and Equation 4 below indicate pixel values of pixels given the same alphabet in FIG. Also, ">>" indicates a right shift operation, and ">> 5" corresponds to division by 32. That is, the interpolated pixel at the 1/2 pixel position is calculated using a 6-tap FIR (Finite Impulse Response) filter (tap coefficients: (1, -5, 20, 20, -5, 1)/32).

Further, the interpolation processing of 1/4 pixels corresponding to the positions of the letters a and d in FIG. 15 is calculated by the following formula 4.

Thus, the interpolated pixel at the 1/4 pixel position is calculated using a 2-tap mean value filter (tap coefficients: (1/2, 1/2)). A half-pixel interpolation process corresponding to an alphabet j lying between four integer-pixel positions is generated using both vertical 6-tap and horizontal 6-tap directions. Interpolated pixel values are generated in a similar manner for pixel positions other than those described above.

Note that the interpolation processing is not limited to the examples of Equations 3 and 4, and may be generated using other interpolation coefficients. Further, the interpolation coefficient may be a fixed value given from the encoding control unit 150, or the interpolation coefficient is optimized for each frame based on the encoding cost described above, and the optimized interpolation coefficient is used. may be generated using

Further, in the present embodiment, the motion reference block is a motion vector block prediction process in units of macroblocks (for example, 16×16 pixel blocks). Prediction processing may be performed in 8×16 pixel block units, 8×8 pixel block units, 8×4 pixel block units, 4×8 pixel block units, or 4×4 pixel block units. In this case, information about motion vector blocks is derived on a pixel block basis. Also, the prediction process may be performed in units larger than 16×16 pixel blocks, such as 32×32 pixel blocks, 32×16 pixel blocks, and 64×64 pixel blocks.

When substituting the reference motion vector in the motion vector block as the motion vector of the small pixel block in the block to be encoded, (A) a negative value (inverted vector) of the reference motion vector may be substituted, or (B) A reference motion vector corresponding to a small block and a weighted average value or median value, maximum value, or minimum value using reference motion vectors adjacent to this reference motion vector may be substituted.

FIG. 16 schematically shows the operation of the prediction section 101. As shown in FIG. As shown in FIG. 16, first, a reference frame (motion reference frame) including a temporal reference motion block is acquired (step S1501). A motion reference frame is typically a reference frame having the shortest temporal distance from an encoding target frame, and is a temporally past reference frame. For example, the motion reference frame is the frame encoded immediately before the encoding target frame. In another example, any reference frame for which motion information 18 is stored in motion information memory 108 may be obtained as a motion reference frame. Next, the spatial direction motion information acquisition unit 110 and the temporal direction motion information acquisition unit 111 each acquire the available block information 30 output from the available block acquisition unit 109 (step S1502). Next, the motion information changeover switch 112 selects one of the available blocks as the selected block, for example, according to Equation 1 (step S1503). Subsequently, the motion compensation unit 113 copies the motion information of the selected selected block to the encoding target block (step S1504). At this time, if the selected block is a spatial direction reference block, the motion information 18 of this selected block is copied to the coding reference block, as shown in FIG. Also, when the selected block is a temporal direction reference block, the group of motion information 18 possessed by this selected block is copied to the encoding target block together with the position information. Next, the motion information 18 or the group of motion information 18 copied by the motion compensation unit 113 is used to perform motion compensation, and the predicted image signal 11 and the motion information 18 used for motion compensation prediction are output.

FIG. 18 shows a more detailed configuration of variable length coding section 104 . The variable-length coding unit 104 includes a parameter coding unit 114, a transform coefficient coding unit 115, a selected block coding unit 116 and a multiplexing unit 117, as shown in FIG. The parameter encoding unit 114 encodes parameters required for decoding such as prediction mode information, block size information, quantization parameter information, etc., excluding transform coefficient information 13 and selected block information 31, to generate encoded data 14A. . The transform coefficient encoding unit 115 encodes the transform coefficient information 13 to generate encoded data 14B. Also, the selected block encoding unit 116 refers to the available block information 30, encodes the selected block information 31, and generates encoded data 14C.

When the available block information 30 includes an index and the availability of a motion reference block corresponding to the index, as shown in FIG. Exclude the reference blocks and convert only the motion reference blocks that are available to syntax (stds_idx). In FIG. 19, 5 motion reference blocks out of 9 motion reference blocks are unavailable, so the syntax stds_idx is sequentially assigned from 0 to the 4 motion reference blocks excluding them. In this example, the selected block information to be encoded is selected from four available blocks instead of from nine, so the code amount (number of bins) to be assigned can be reduced on average.

FIG. 20 shows an example of a code table indicating syntax stds_idx and binary information (bin) of syntax stds_idx. As shown in FIG. 18, the smaller the number of motion reference blocks available, the smaller the average number of bins required to encode the syntax stds_idx. For example, if the number of available blocks is 4, the syntax stds_idx can be represented in 3 bits or less. The binary information (bin) of the syntax stds_idx may be binarized so that all stds_idx have the same number of bins for each number of usable blocks, and is binarized according to the binarization method determined by pre-learning. I don't mind. Alternatively, a plurality of binarization methods may be prepared and switched adaptively for each block to be encoded.

Entropy coding (e.g., equal length coding, Huffman coding, arithmetic coding, etc.) can be applied to these coding units 114, 115, 116, and the generated coded data 14A, 14B, 14C is multiplexed by multiplexing section 117 and output.

In the present embodiment, an example is described in which a frame encoded one frame before the encoding target frame is referred to as a reference frame. The frame number may be used to scale (or normalize) the motion vector and the reference motion information 19 may be applied to the block to be coded.

This scaling processing will be specifically described with reference to FIG. tc shown in FIG. 21 indicates the temporal distance (POC (display order number) distance) between the encoding target frame and the motion reference frame, and is calculated by Equation 5 below. tr[i] shown in FIG. 21 indicates the time distance between the motion reference frame and the frame i referred to by the selected block, and is calculated by Equation 6 below.

Here, curPOC indicates the POC (Picture Order Count) of the encoding target frame, colPOC indicates the POC of the motion reference frame, and refPOC indicates the POC of the frame i referenced by the selected block. Also, Clip(min,max,target) outputs min if target is less than min, max if it is greater than max, and target otherwise. clip function. Also, DiffPicOrderCnt(x,y) is a function that calculates the difference between two POCs.

Assuming that the motion vector of the selected block is MVr=(MVr_x, MVr_y) and the motion vector to be applied to the encoding target block is MV=(MV_x, MV_y), the motion vector MV is calculated by Equation 7 below.

Here, Abs(x) denotes a function that extracts the absolute value of x. In this way, in the motion vector scaling, the motion vector MVr assigned to the selected block is transformed into the motion vector MV between the encoding target frame and the first motion reference frame.

Other examples of motion vector scaling are also described below.
First, for each slice or frame, a scaling factor (DistScaleFactor[i]) is obtained for all possible temporal distances tr of the motion reference frames according to Equation 8 below. The number of scaling factors is equal to the number of frames referenced by the selected block, ie the number of reference frames.

The calculation of tx shown in Expression 8 may be tabulated in advance.

At the time of scaling for each block to be encoded, the motion vector MV can be calculated only by multiplication, addition, and shift operations by using Equation 9 below.

When such scaling processing is performed, the post-scaling motion information 18 is applied to the processing of the available block acquisition unit 109 together with the prediction unit 101 . When the scaling process is performed, the reference frame referred to by the encoding target block becomes a motion reference frame.

FIG. 22 shows the syntax structure in the image encoding unit 100. As shown in FIG. As shown in FIG. 22, the syntax mainly includes three parts: high level syntax 901 , slice level syntax 904 and macroblock level syntax 907 . The high-level syntax 901 holds syntax information of upper layers above slices. The slice level syntax 904 holds the information needed for each slice, and the macroblock level syntax 907 holds the data needed for each macroblock shown in Figures 7A-7D.

Each part contains more detailed syntax. High level syntax 901 includes sequence and picture level syntax such as sequence parameter set syntax 902 and picture parameter set syntax 903 . Slice level syntax 904 includes slice header syntax 905, slice data syntax 906, and the like. Furthermore, macroblock level syntax 907 includes macroblock layer syntax 908, macroblock prediction syntax 909, and the like.

Figures 23A and 23B show examples of macroblock layer syntax. available_block_num shown in FIGS. 23A and 23B indicates the number of available blocks, and if this is a value greater than 1, encoding of selected block information is required. Further, stds_idx indicates selected block information, and stds_idx is encoded using the code table corresponding to the number of usable blocks described above.

FIG. 23A shows the syntax for encoding the selected block information after mb_type. If the mode indicated by mb_type is a specified size or a specified mode (TARGET_MODE) and if available_block_num is a value greater than 1, encode stds_idx. For example, the motion information of the selected block can be used when the block size is 64×64 pixels, 32×32 pixels, or 16×16 pixels, or when the stds_idx is encoded in the direct mode.

FIG. 23B shows the syntax for encoding the selected block information before mb_type. If available_block_num is greater than 1, encode stds_idx. Also, if available_block_num is 0, H. Since conventional motion compensation represented by H.264 is performed, mb_type is coded.

Note that syntax elements not specified in the present invention can be inserted between the rows of the tables shown in FIGS. 23A and 23B, and other descriptions related to conditional branching may be included. . Alternatively, it is possible to divide and integrate the syntax table into multiple tables. Moreover, it is not always necessary to use the same terminology, and the terminology may be arbitrarily changed depending on the mode of use. Furthermore, each syntax element described in the macroblock layer syntax may be changed as specified in the macroblock data syntax described below.

Also, by using stds_idx information, mb_type information can be reduced. Figure 24A shows the H. 2 is a code table corresponding to mb_type and mb_type at the time of B slice in H.264. N shown in FIG. 24A is a value, such as 16, 32, 64, representing the size of the block to be encoded, and M is half the value of N. Therefore, when mb_type is 4 to 21, it indicates that the encoding target block is a rectangular block. In addition, L0, L1, and Bi in FIG. 24A indicate unidirectional prediction (List0 direction only), unidirectional prediction (List1 direction only), and bidirectional prediction, respectively. When the encoding target block is a rectangular block, mb_type includes information indicating which of L0, L1, and Bi prediction is performed for each of the two rectangular blocks in the encoding target block. B_Sub means that the above processing is performed on each pixel block obtained by dividing a macroblock into four. For example, when the encoding target block is a 64×64 pixel macroblock, the encoding target block is further assigned mb_type for each of four 32×32 pixel blocks obtained by dividing this macroblock into four. become.

Here, if the selected block indicated by stds_idx is Spatial Left (the pixel block adjacent to the left side of the target block), the motion information of the pixel block adjacent to the left side of the target block is Since it is motion information, stds_idx is predicted for the encoding target block using horizontally long rectangular blocks indicated by mb_type=4, 6, 8, 10, 12, 14, 16, 18, and 20 in FIG. 24A. is equivalent to executing Also, when the selected block indicated by stds_idx is Spatial Up, the motion information adjacent to the upper side of the block to be encoded is used as the motion information of the block to be encoded. Equivalent to performing prediction on tall rectangular blocks indicated by 9, 11, 13, 15, 17, 19, 21. Therefore, by using stds_idx, it is possible to create a code table in which the columns of mb_type=4 to 21 in FIG. 24A are reduced as shown in FIG. 24B. Similarly, the H.264 shown in FIG. 24C. As for the code table corresponding to mb_type and mb_type at the time of P-slicing in H.264, it is possible to create a code table in which the number of mb_types is reduced as shown in FIG. 24D.

Also, stds_idx information may be included in mb_type information for encoding. FIG. 25A is a code table when stds_idx information is included in mb_type information, and shows an example of a code table corresponding to mb_type and mb_type in a B slice. B_STDS_X (X=0, 1, 2) in FIG. 25A indicates the mode corresponding to stds_idx, and B_STDS_X is added by the number of available blocks (the number of available blocks is 3 in FIG. 25A). Similarly, another example of mb_type for P slices is shown in FIG. 25B. Description of FIG. 25B is omitted because it is the same as the B slice.

The order and binarization method (binning) of mb_type are not limited to the examples shown in FIGS. 25A and 25B, and mb_type may be encoded according to other orders and binarization methods. B_STDS_X and P_STDS_X do not have to be consecutive and may be placed between each mb_type. Also, the binarization method (binning) may be designed based on pre-learned selection frequencies.

In this embodiment, the present invention can also be applied to extended macroblocks in which a plurality of macroblocks are collectively subjected to motion compensation prediction. Moreover, in the present embodiment, any scan order for encoding may be used. For example, the present invention can also be applied to line scanning or Z scanning.

As described above, the image coding apparatus according to the present embodiment selects usable blocks from a plurality of motion reference blocks, and selects motion reference blocks to be applied to the coding target block according to the number of selected usable blocks. is generated and the information is encoded. Therefore, according to the image coding apparatus according to the present embodiment, while reducing the amount of code related to motion vector information, motion compensation can be performed in units of small pixel blocks that are finer than the coding target block, resulting in high coding efficiency. can be realized.

(Second embodiment)
FIG. 26 shows an image encoding device according to the second embodiment of the present invention. In the second embodiment, mainly the parts and operations different from those in the first embodiment will be described. As shown in FIG. 26, the image coding unit 200 according to this embodiment differs from the first embodiment in the configurations of the prediction unit 201 and the variable length coding unit 204. FIG. The prediction unit 201 includes a first prediction unit 101 and a second prediction unit 202 as shown in FIG. . The first prediction unit 101 has the same configuration as the prediction unit 101 (FIG. 1) according to the first embodiment, and uses the motion information 18 of the selected block to perform motion compensation according to a prediction method (first prediction method). , to generate the predicted image signal 11 . The second prediction unit 202 performs motion compensation on the encoding target block using one motion vector. A prediction image signal 11 is generated according to a prediction method (second prediction method) such as H.264. The second prediction section 202 uses the input image signal 10 and the reference image signal 17 from the frame memory to generate the predicted image signal 11B.

FIG. 28 schematically shows the configuration of the second prediction section 202. As shown in FIG. As shown in FIG. 28, the second prediction unit 202 includes a motion information acquisition unit 205 that generates motion information 21 using the input image signal 10 and the reference image signal 17, a motion information acquisition unit 205 that is provided with a motion compensator 113 (FIG. 1) that generates a predicted image signal 11A using . The motion information acquisition unit 205 obtains a motion vector to be assigned to the encoding target block based on the input image signal 10 and the reference image signal 17, for example, by block matching. As an evaluation criterion for matching, a value obtained by accumulating the difference between the input image signal 10 and the interpolated image after matching for each pixel is used.

Note that the motion information acquisition unit 205 may use a value obtained by converting the difference between the predicted image signal 11 and the input image signal 10 to determine the optimum motion vector. Also, the optimum motion vector may be determined by considering the size of the motion vector and the code amount of the motion vector and the reference frame number, or by using Equation 1. The matching method may be executed based on search range information provided from the outside of the image coding apparatus, or may be executed hierarchically for each pixel precision. Alternatively, the motion information given by the encoding control unit 150 may be used as the output 21 of the motion information acquisition unit 205 without performing search processing.

The prediction unit 101 in FIG. 27 further includes a prediction method switch 203 that selects and outputs either the predicted image signal 11A from the first prediction unit 101 or the predicted image signal 11B from the second prediction unit 202. there is For example, the prediction method changeover switch 203 uses the input image signal 10 for each of the predicted image signals 11A and 11B to obtain the coding cost according to, for example, Equation 1, and selects the predicted image so that the coding cost becomes smaller. Either one of the signals 11A and 11B is selected and output as the predicted image signal 11. FIG. Furthermore, the prediction method changeover switch 203 is a prediction changeover switch that indicates whether the output prediction image signal 11 is output from either the first prediction unit 101 or the second prediction unit 202 together with the motion information 18 and the selected block information 31. Information 32 is also output. The output motion information 18 is multiplexed with the encoded data 14 after being encoded by the variable length encoding unit 204 .

FIG. 29 schematically shows the configuration of the variable length coding section 204. As shown in FIG. The variable-length coding unit 204 shown in FIG. 29 has a motion information coding unit 217 in addition to the configuration of the variable-length coding unit 104 shown in FIG. Also, unlike the selected block coding unit 116 in FIG. 18, the selected block coding unit 216 in FIG. 29 codes the prediction switching information 32 to generate the coded data 14D. When the first prediction unit 101 executes the prediction process, the selected block coding unit 216 further codes the available block information 30 and the selected block information 31 . The coded available block information 30 and selected block information 31 are included in the coded data 14D. When the second prediction unit 202 executes the prediction process, the motion information encoding unit 217 encodes the motion information 18 to generate the encoded data 14E. The selected block encoding unit 216 and the motion information encoding unit 217 each perform prediction processing based on the prediction switching information 32 indicating whether or not the predicted image has been generated by motion compensation prediction using the motion information of the selected block. It is determined which one of the first prediction unit 101 and the second prediction unit 202 has been executed.

The multiplexing unit 117 receives the encoded data 14A, B, D, and E from the parameter encoding unit 114, the transform coefficient encoding unit 115, the selected block encoding unit 216, and the motion information encoding unit, and converts the received encoded data 14A, B, D, E are multiplexed.

30A and 30B each show an example of macroblock layer syntax according to this embodiment. available_block_num shown in FIG. 30A indicates the number of available blocks, and when this is a value greater than 1, selected block coding section 216 codes selected block information 31 . In addition, stds_flag is a flag indicating whether or not the motion information of the selected block is used as the motion information of the target block in motion compensation prediction. 202 is selected. If the number of available blocks is greater than 1 and stds_flag is 1, it indicates that motion information possessed by the selected block was used for motion compensated prediction. Also, if stds_flag is 0, H.264 is processed without using the motion information of the selected block. As with H.264, the information of the motion information 18 is coded directly or a predicted difference value. Furthermore, stds_idx indicates selected block information, and the code table corresponding to the number of usable blocks is as described above.

FIG. 30A shows the syntax for encoding the selected block information after mb_type. The stds_flag and stds_idx are coded only when the mode indicated by mb_type is a defined size or a defined mode. For example, stds_flag and stds_idx are encoded when the motion information of the selected block is available when the block size is 64×64, 32×32, 16×16, or when in direct mode.

FIG. 30B shows the syntax for encoding the selected block information before mb_type. For example, if stds_flag is 1, mb_type need not be encoded. If stds_flag is 0, mb_type is encoded.

As described above, the image coding apparatus according to the second embodiment uses the first prediction unit 101 according to the first embodiment and a prediction method such as H.264 so as to reduce the coding cost. The second prediction unit 202 is selectively switched to compress and encode the input image signal. Therefore, in the image coding apparatus according to the second embodiment, the coding efficiency is improved as compared with the image coding apparatus according to the first embodiment.

(Third embodiment)
FIG. 31 schematically shows an image decoding device according to the third embodiment. This image decoding apparatus comprises an image decoding section 300, a decoding control section 350 and an output buffer 308, as shown in FIG. The image decoding section 300 is controlled by a decoding control section 350 . An image decoding device according to the third embodiment corresponds to the image encoding device according to the first embodiment. That is, the decoding process by the image decoding apparatus of FIG. 31 has a complementary relationship with the encoding process by the image encoding process of FIG. The image decoding device in FIG. 31 may be implemented by hardware such as an LSI chip, or may be implemented by causing a computer to execute an image decoding program.

The image decoding device in FIG. I have. In the image decoding unit 300 , encoded data 80 from a storage system or a transmission system (not shown) is input to a coded sequence decoding unit 301 . This coded data 80 corresponds to, for example, the coded data 14 sent in a multiplexed state from the image coding apparatus of FIG.

In this embodiment, a pixel block (for example, a macroblock) to be decoded is simply referred to as a decoding target block. An image frame including a decoding target block is called a decoding target frame.

In the encoded string decoding unit 301, decoding by parsing is performed based on the syntax for each frame or field. Specifically, the coded string decoding unit 301 sequentially performs variable-length decoding on the coded string of each syntax, and transform coefficient information 33, selected block information 61, and prediction information such as block size information and prediction mode information. Decode the encoding parameters related to the block to be decoded, including the encoding parameters.

In this embodiment, the decoding parameters include transform coefficients 33, selected block information 61 and prediction information, and include all parameters necessary for decoding such as transform coefficient information and quantization information. The prediction information, the information about transform coefficients, and the information about quantization are input to decoding control section 350 as control information 71 . The decoding control unit 350 provides each unit of the image decoding unit 300 with decoding control information 70 including parameters necessary for decoding such as prediction information and quantization parameters.

Further, the encoded string decoding unit 301 simultaneously decodes the encoded data 80 to obtain prediction information and selected block information 61, as will be described later. Motion information 38, including motion vectors and reference frame numbers, need not be decoded.

The transform coefficients 33 decoded by the coded sequence decoding unit 301 are sent to the inverse quantization/inverse transform unit 302 . Various information related to quantization decoded by the coded sequence decoding unit 301, that is, the quantization parameter and the quantization matrix are given to the decoding control unit 350, and are subjected to inverse quantization/inverse transform when performing inverse quantization. is loaded into section 302 . The inverse quantization/inverse transform unit 302 performs inverse quantization on the transform coefficients 33 according to the loaded quantization information, and then performs inverse transform processing (for example, inverse discrete cosine transform) to obtain a prediction error signal 34. obtain. The inverse transform processing by the inverse quantization/inverse transform unit 302 in FIG. 31 is the inverse transform of the transform processing by the transform/quantization unit in FIG. For example, when wavelet transform is performed by the image coding apparatus (FIG. 1), the inverse quantization/inverse transform unit 302 executes corresponding inverse quantization and inverse wavelet transform.

The prediction error signal 34 restored by the inverse quantization/inverse transform unit 302 is input to the adder 303 . The adder 303 adds the prediction error signal 34 and the prediction image signal 35 generated by the prediction unit 305 (to be described later) to generate the decoded image signal 36 . The generated decoded image signal 36 is output from the image decoding unit 300 , temporarily stored in the output buffer 308 , and then output according to the output timing managed by the decoding control unit 350 . Also, this decoded image signal 36 is stored in the frame memory 304 as the reference image signal 37 . The reference image signal 37 is sequentially read from the frame memory 304 frame by frame or field by field, and input to the prediction section 305 .

The available block acquisition unit 307 receives reference motion information 39 from a motion information memory 306, which will be described later, and outputs available block information 60. FIG. The operation of the available block acquisition unit 307 is the same as that of the available block acquisition unit 109 (FIG. 1) described in the first embodiment.

The motion information memory 306 receives the motion information 38 from the prediction unit 305 and temporarily stores it as reference motion information 39 . The motion information memory 306 temporarily stores the motion information 38 output from the prediction unit 305 as reference motion information 39 . FIG. 4 shows an example of the motion information memory 306. As shown in FIG. The motion information memory 306 holds a plurality of motion information frames 26 with different encoding times. The motion information 38 or group of motion information 38 whose decoding has been completed is stored as reference motion information 39 in the motion information frame 26 corresponding to the decoding time. In the motion information frame 26, the reference motion information 39 is stored, for example, in units of 4×4 pixel blocks. The reference motion information 39 held by the motion information memory 306 is read and referred to by the prediction unit 305 when generating the motion information 38 of the block to be decoded.

Next, motion reference blocks and usable blocks according to this embodiment will be described. A motion reference block is a candidate block that is selected according to a predetermined method by the above-described image coding apparatus and image decoding apparatus from areas that have already been decoded. FIG. 8A shows an example of available blocks. In FIG. 8A, a total of nine motion reference blocks are arranged, including four motion reference blocks in the decoding target frame and five motion reference blocks in the reference frame. Motion reference blocks A, B, C, and D in the decoding target frame in FIG. 8A are blocks adjacent to the left, upper, upper right, and upper left of the decoding target block. In this embodiment, a motion reference block selected from a decoding target frame that includes the decoding target block is referred to as a spatial direction motion reference block. A motion reference block TA in a reference frame is a pixel block at the same position as a block to be decoded in the reference frame. Selected as a reference block. A motion reference block selected from pixel blocks in a reference frame is called a temporal motion reference block. A frame in which a temporal motion reference block is located is called a motion reference frame.

The spatial motion reference block is not limited to the example shown in FIG. 8A. As shown in FIG. It does not matter if it is selected. In this case, the relative positions (dx, dy) of pixels a, b, c, and d with respect to the upper left pixel in the block to be decoded are shown in FIG. 8C.

Also, as shown in FIG. 8D, all pixel blocks A1 to A4, B1, B2, C, and D adjacent to the decoding target block may be selected as spatial direction motion reference blocks. In FIG. 8D, the number of spatial motion reference blocks is eight.

Also, the temporal motion reference blocks TA to TE may partially overlap each other as shown in FIG. 8E, or may be separated from each other as shown in FIG. 8F. Also, the temporal motion reference block does not necessarily have to be positioned at or around the block at the Collocate position, and may be a pixel block at any position within the motion reference frame. For example, using the motion information of already-decoded blocks adjacent to the decoding target block, the reference block indicated by the motion vector included in the motion information is selected as the center of the motion reference block (for example, block TA). I don't mind. Furthermore, the reference blocks in the time direction need not be arranged at regular intervals.

In the method of selecting motion reference blocks as described above, if both the image decoding device and the image decoding device share information about the number and positions of spatial and temporal motion reference blocks, motion reference blocks may be selected from any number and position. Also, the size of the motion reference block does not necessarily have to be the same size as the decoding target block. For example, as shown in FIG. 8D, the size of the motion reference block may be larger or smaller than the size of the block to be decoded, or may be any size. Also, the shape of the motion reference block is not limited to square, and may be rectangular.

Next, available blocks will be described. A usable block is a pixel block selected from among the motion reference blocks and for which motion information can be applied to the block to be decoded. Available blocks have different motion information. The available blocks are selected by performing the available block determination process shown in FIG. 9 for a total of nine motion reference blocks in the decoding target frame and the reference frame shown in FIG. 8A, for example. . FIG. 10 shows the result of executing the available block determination process shown in FIG. In FIG. 10, shaded pixel blocks indicate unavailable blocks, and white blocks indicate available blocks. That is, two of the spatial motion reference blocks and two of the temporal motion reference blocks, a total of four, are determined as usable blocks. The motion information selection unit 314 in the prediction unit 305 selects the most suitable one from among these available blocks arranged in the time and space directions according to the selected block information 61 received from the selected block decoding unit 323. Select a block as the selected block.

Next, the available block acquisition unit 307 will be described. The available block acquisition unit 307 has the same function as the available block acquisition unit 109 of the first embodiment, acquires the reference motion information 39 from the motion information memory 306, and determines the available blocks or available blocks for each motion reference block. It outputs available block information 60, which is information indicating an impossible block.

The operation of the available block acquisition unit 307 will be described with reference to the flowchart of FIG. First, the available block acquisition unit 307 determines whether or not the motion reference block (index p) has motion information (step S801). That is, in step S801, it is determined whether or not at least one small pixel block in the motion reference block p has motion information. If it is determined that the motion reference block p does not have motion information, that is, the temporal motion reference block is a block in an I-slice without motion information, or all small blocks in the temporal motion reference block are If the pixel block has been intra-prediction decoded, the process proceeds to step S805. In step S805, this motion reference block p is determined as an unavailable block.

If it is determined in step S801 that the motion reference block p has motion information, the available block acquisition unit 307 selects a motion reference block q (referred to as available block q) that has already been determined to be a available block. (step S802). Here, q is a value smaller than p. Subsequently, the available block acquisition unit 307 compares the motion information of this motion reference block p with the motion information of the available block q for all q, and determines that the motion reference block p is a available block q. It is determined whether or not it has the same motion information as (S803). If the motion reference block p has the same motion vector as the available block q, the process proceeds to step S805. be judged. If the motion reference block p has motion information different from all available blocks q, the available block acquisition unit 307 determines that this motion reference block p is a available block in step S804.

By executing the above-described usable block determination processing for all motion reference blocks, it is determined whether each motion reference block is a usable block or an unusable block, and usable block information 60 is generated. An example of available block information 60 is shown in FIG. The available block information 60 includes indices p and availability of motion reference blocks, as shown in FIG. In FIG. 11, the available block information 60 indicates that motion reference blocks with indices p of 0, 1, 5 and 8 have been selected as available blocks, and the number of available blocks is four.

Note that in step S801 of FIG. 9, if at least one of the blocks in the temporal direction motion reference block p is a block subjected to intra-prediction coding, the available block obtaining unit 307 uses the motion reference block p. It may be judged as an impossible block. That is, only when all blocks in the temporal motion reference block p have been coded by inter prediction, the process may proceed to step S802.

FIGS. 12A to 12E show examples in which the comparison of the motion information 38 in step S803 determines that the motion information 38 of the motion reference block p and the motion information 38 of the available block q are the same. 12A-12E each show a plurality of hatched blocks and two white blocks. FIGS. 12A-12E assume, for simplicity of explanation, that the motion information 38 of these two white blocks are compared without considering the shaded blocks. Let one of the two white blocks be the motion reference block p and the other be the motion reference block q already determined to be available (available block q). Unless otherwise specified, either of the two white blocks may be the motion reference block p.

FIG. 12A shows an example where both the motion reference block p and the available block q are spatial blocks. In the example of FIG. 12A, if the motion information 38 of blocks A and B are the same, it is determined that the motion information 38 is the same. At this time, the sizes of blocks A and B need not be the same.

FIG. 12B shows an example where one of motion reference block p and available block q is block A in the spatial direction and the other is block TB in the temporal direction. In FIG. 12B, there is one block with motion information within block TB in the temporal direction. If the motion information 38 of the block TB in the temporal direction and the motion information 38 of the block A in the spatial direction are the same, it is determined that the motion information 38 are the same. At this time, the sizes of blocks A and TB do not have to be the same.

FIG. 12C shows another example where one of motion reference block p and available block q is block A in spatial direction and the other is block TB in temporal direction. FIG. 12C shows a case where the block TB in the time direction is divided into a plurality of small blocks, and there are a plurality of small blocks having motion information 38 . In the example of FIG. 12C, the motion information 38 is identical if all blocks with motion information 38 have the same motion information 38 and that motion information 38 is identical to the motion information 38 of block A in the spatial direction. is determined. At this time, the sizes of blocks A and TB do not have to be the same.

FIG. 12D shows an example where the motion reference block p and available block q are both temporal blocks. In this case, if the motion information 38 of the blocks TB and TE are the same, it is determined that the motion information 38 is the same.

FIG. 12E shows another example where motion reference block p and available block q are both temporal blocks. FIG. 12E shows a case where each of blocks TB and TE in the time direction is divided into a plurality of small blocks, each of which has a plurality of small blocks having motion information 38 . In this case, the motion information 38 is compared for each small block in the block, and if the motion information 38 is the same for all the small blocks, the motion information 38 for the block TB and the motion information 38 for the block TE are the same. It is determined that there is

FIG. 12F shows yet another example where motion reference block p and available block q are both temporal blocks. FIG. 12F shows a case where the block TE in the time direction is divided into a plurality of small blocks, and there are a plurality of small blocks having motion information 38 in the block TE. If all the motion information 38 of the block TE is the same motion information 38 and is the same as the motion information 38 that the block TD has, it is determined that the motion information 38 of the blocks TD and TE are the same.

Thus, in step S803, it is determined whether or not the motion information 38 of the motion reference block p and the motion information 38 of the available block q are the same. In the examples of FIGS. 12A to 12F, the number of available blocks q compared with the motion reference block p is 1, but when the number of available blocks q is 2 or more, the motion information of the motion reference block p 38 with motion information 38 for each available block q. Further, when scaling, which will be described later, is applied, the motion information 38 after scaling becomes the motion information 38 described above.

Note that the determination that the motion information of the motion reference block p and the motion information of the available block q are the same is not limited to the case where the motion vectors included in the motion information completely match. For example, if the norm of the difference between two motion vectors is within a predetermined range, it may be considered that the motion information of the motion reference block p and the motion information of the available block q are substantially the same.

FIG. 32 is a block diagram showing the coded sequence decoding unit 301 in more detail. As shown in FIG. 32, the coded sequence decoding unit 301 includes a separation unit 320 that separates the encoded data 80 into syntax units, a transform coefficient decoding unit 322 that decodes transform coefficients, and a selected block information decoder. and a parameter decoding unit 321 for decoding parameters related to predicted block size and quantization.

The parameter decoding unit 321 receives the encoded data 80A including parameters related to the predicted block size and quantization from the separation unit, decodes the encoded data 80A, and generates the control information 71. FIG. The transform coefficient decoding unit 322 receives the encoded transform coefficients 80B from the separation unit 320, decodes the encoded transform coefficients 80B, and obtains the transform coefficient information 33. FIG. The selected block decoding unit 323 receives the encoded data 80C and the available block information 60 regarding the selected block and outputs the selected block information 61 . The input available block information 60 indicates the availability of each motion reference block, as shown in FIG.

Next, the prediction unit 305 will be described in detail with reference to FIG.
The prediction unit 305 includes a motion information selection unit 314 and a motion compensation unit 313, as shown in FIG. A selector switch 312 is provided. The prediction unit 305 basically has the same configuration and functions as the prediction unit 101 described in the first embodiment.

The prediction unit 305 receives the available block information 60, the selected block information 61, the reference motion information 39 and the reference image signal 37, and outputs the predicted image signal 35 and the motion information 38. FIG. A spatial direction motion information acquisition unit 310 and a temporal direction motion information acquisition unit 311 have the same functions as the spatial direction motion information acquisition unit 110 and the temporal direction motion information acquisition unit 111 described in the first embodiment, respectively. The spatial direction motion information acquisition unit 310 uses the available block information 60 and the reference motion information 39 to generate motion information 38A including motion information and indices of each available block located in the spatial direction. The time direction motion information acquisition unit 311 uses the available block information 60 and the reference motion information 39 to obtain motion information (or a group of motion information) including motion information and indices of each available block located in the time direction. 38B.

In the motion information changeover switch 312, according to the selected block information 61, one of the motion information 38A from the spatial direction motion information acquisition unit 310 and the motion information (or motion information group) 38B from the temporal direction motion information acquisition unit 311 is selected. Select one to obtain motion information 38 . The selected motion information 38 is sent to the motion compensator 313 and motion information memory 306 . The motion compensation unit 313 performs motion compensation prediction according to the selected motion information 38 in the same manner as the motion compensation unit 113 described in the first embodiment, and generates the predicted image signal 35 .

The motion vector scaling function of the motion compensator 313 is the same as that described in the first embodiment, so description thereof will be omitted.

FIG. 22 shows the syntax structure in the image decoding unit 300. As shown in FIG. As shown in FIG. 22, the syntax mainly includes three parts: high level syntax 901 , slice level syntax 904 and macroblock level syntax 907 . The high-level syntax 901 holds syntax information of upper layers above slices. The slice level syntax 904 holds the information needed for each slice, and the macroblock level syntax 907 holds the data needed for each macroblock shown in Figures 7A-7D.

Each part contains more detailed syntax. High level syntax 901 includes sequence and picture level syntax such as sequence parameter set syntax 902 and picture parameter set syntax 903 . Slice level syntax 904 includes slice header syntax 905, slice data syntax 906, and the like. Furthermore, macroblock level syntax 907 includes macroblock layer syntax 908, macroblock prediction syntax 909, and the like.

Figures 23A and 23B show examples of macroblock layer syntax. available_block_num shown in FIGS. 23A and 23B indicates the number of available blocks, and if this is a value greater than 1, decoding of selected block information is required. Further, stds_idx indicates selected block information, and stds_idx is encoded using the code table corresponding to the number of usable blocks described above.

FIG. 23A shows the syntax for decoding the selected block information after mb_type. If the prediction mode indicated by mb_type is a specified size or a specified mode (TARGET_MODE), and if available_block_num is a value greater than 1, stds_idx is decoded. For example, the motion information of the selected block can be used when the block size is 64×64 pixels, 32×32 pixels, or 16×16 pixels, or when the stds_idx is encoded in the direct mode.

FIG. 23B shows the syntax for decoding the selected block information before mb_type. If available_block_num is greater than 1, decode stds_idx. Also, if available_block_num is 0, H. Since conventional motion compensation represented by H.264 is performed, mb_type is coded.

Syntax elements not defined in the present invention can be inserted between the rows of the tables shown in FIGS. 23A and 23B, and descriptions related to other conditional branches may be included. Alternatively, it is possible to divide and integrate the syntax table into multiple tables. Moreover, it is not always necessary to use the same terminology, and the terminology may be arbitrarily changed depending on the mode of use. Furthermore, each syntax element described in the macroblock layer syntax may be changed as specified in the macroblock data syntax described below.

As described above, the image decoding apparatus according to this embodiment decodes the image encoded by the image encoding apparatus according to the first embodiment. Therefore, the image decoding according to this embodiment can reproduce a high-quality decoded image from relatively small encoded data.

(Fourth embodiment)
FIG. 34 schematically shows an image decoding device according to the fourth embodiment. The image decoding device comprises an image decoding section 400, a decoding control section 350 and an output buffer 308, as shown in FIG. An image decoding device according to the fourth embodiment corresponds to the image encoding device according to the second embodiment. In the fourth embodiment, mainly the parts and operations different from those in the third embodiment will be described. As shown in FIG. 34, the image decoding unit 400 according to this embodiment differs from the third embodiment in a coded sequence decoding unit 401 and a prediction unit 405 .

The prediction unit 405 of this embodiment uses a prediction method (first prediction method) for performing motion compensation using motion information of a selected block, and H.264. A prediction image signal 35 is generated by selectively switching between a prediction method (second prediction method) such as H.264 that performs motion compensation using one motion vector for a block to be decoded.

FIG. 35 is a block diagram showing the coded sequence decoding unit 401 in more detail. The coded sequence decoding unit 401 shown in FIG. 35 includes a motion information decoding unit 424 in addition to the configuration of the coded sequence decoding unit 301 shown in FIG. Also, unlike the selected block decoding unit 323 shown in FIG. 32, the selected block decoding unit 423 shown in FIG. The prediction switching information 62 indicates which of the first and second prediction schemes the prediction unit 101 in the image coding apparatus of FIG. 1 used. When the prediction switching information 62 indicates that the prediction unit 101 uses the first prediction scheme, that is, when the decoding target block is encoded by the first prediction scheme, the selected block decoding unit 423 performs encoding Selected block information 61 is obtained by decoding selected block information in data 80C. When the prediction switching information 62 indicates that the prediction unit 101 uses the second prediction method, that is, when the decoding target block is encoded by the second prediction method, the selected block decoding unit 423 converts the selected block information , the motion information decoding unit 424 decodes the encoded motion information 80D to obtain the motion information 40 .

FIG. 36 is a block diagram showing the predictor 405 in more detail. The prediction unit 405 shown in FIG. 34 includes a first prediction unit 305 , a second prediction unit 410 and a prediction method switch 411 . The second prediction unit 410 performs motion compensation prediction similar to the motion compensation unit 313 in FIG. 35B. The first prediction section 305 is the same as the prediction section 305 described in the third embodiment, and generates a prediction image signal 35B. Also, the prediction method switch 411 selects either one of the predicted image signal 35B from the second prediction unit 410 and the predicted image signal 35A from the first prediction unit 305 based on the prediction switching information 62. It is output as the predicted image signal 35 of the prediction unit 405 . At the same time, the prediction method switch 411 sends the motion information used by the selected first prediction unit 305 or second prediction unit 410 to the motion information memory 306 as the motion information 38 .

Next, the syntax structure according to this embodiment will be mainly described with respect to the differences from the third embodiment.

30A and 30B each show an example of macroblock layer syntax according to this embodiment. available_block_num shown in FIG. 30A indicates the number of available blocks, and when this is a value greater than 1, selected block decoding section 423 decodes selected block information in encoded data 80C. Also, stds_flag is a flag indicating whether or not the motion information of the selected block is used as the motion information of the block to be decoded in motion compensated prediction. 410 is selected. If the number of available blocks is greater than 1 and stds_flag is 1, it indicates that motion information possessed by the selected block was used for motion compensated prediction. Also, if stds_flag is 0, H.264 is processed without using the motion information of the selected block. In the same way as H.264, information of motion information is coded directly or a difference value obtained by prediction. Furthermore, stds_idx indicates selected block information, and the code table corresponding to the number of usable blocks is as described above.

FIG. 30A shows the syntax for decoding the selected block information after mb_type. The stds_flag and stds_idx are decoded only when the prediction mode indicated by mb_type is a defined block size or a defined mode. For example, if the block size is 64x64, 32x32, 16x16, or in direct mode, decode stds_flag and stds_idx.

FIG. 30B shows the syntax for decoding the selected block information before mb_type. For example, if stds_flag is 1, mb_type need not be decoded. If stds_flag is 0, mb_type is decoded.

As described above, the image decoding apparatus according to this embodiment decodes the image encoded by the image encoding apparatus according to the second embodiment described above. Therefore, the image decoding according to this embodiment can reproduce a high-quality decoded image from relatively small encoded data.

It should be noted that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying constituent elements without departing from the scope of the present invention at the implementation stage. Further, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all components shown in the embodiments. Furthermore, components of different embodiments may be combined as appropriate.

As an example of this, similar effects can be obtained by modifying the first to fourth embodiments described above as follows.

(1) In the first to fourth embodiments, the frame to be processed is divided into rectangular blocks such as 16×16 pixel blocks, and the pixel block moves from the upper left pixel block to the lower right pixel block as shown in FIG. Although the case of encoding or decoding in order is described as an example, the encoding or decoding order is not limited to this example. For example, the encoding or decoding order may be the order from the lower right to the upper left of the screen, or may be the order from the upper right to the lower left. Also, the encoding or decoding order may be a spiral order from the center of the screen to the periphery, or may be an order from the periphery to the center of the screen.

(2) In the first to fourth embodiments, an example is explained in which the luminance signal and the color difference signal are not divided, but are limited to one of the color signal components. However, different prediction processes may be used, or the same prediction process may be used for luminance and color difference signals. If a different prediction process is used, the selected prediction method for the chrominance signal is encoded/decoded in the same way as for the luma signal.

In addition, it goes without saying that the present invention can be implemented in the same manner even if various prejudices are applied without departing from the gist of the present invention.

INDUSTRIAL APPLICABILITY The image encoding/decoding method according to the present invention can improve encoding efficiency, and thus has industrial applicability.

10 Input image signal 11 Predicted image signal 12 Prediction error image signal 13 Quantized transform coefficient 14 Encoded data 15 Decoded prediction error signal 16 Locally decoded image signal 17 Reference image Signal 18 Motion information 20 Bit stream 21 Motion information 25, 26 Information frame 30 Available block information 31 Selected block information 32 Prediction switching information 33 Transform coefficient information 34 ... prediction error signal, 35 ... prediction image signal, 36 ... decoded image signal, 37 ... reference image signal, 38 ... motion information, 39 ... reference motion information, 40 ... motion information, 50 ... coding control information, 51 ... feedback information , 60... available block information, 61... selected block information, 62... prediction switching information, 70... decoding control information, 71... control information, 80... encoded data, 100... image encoding unit, 101... prediction unit, 102 Subtractor 103 Transform/quantization unit 104 Variable-length coding unit 105 Inverse quantization/inverse transformation unit 106 Adder 107 Frame memory 108 Information memory 109 Available Block acquisition unit 110 Spatial motion information acquisition unit 111 Temporal motion information acquisition unit 112 Information changeover switch 113 Motion compensation unit 114 Parameter coding unit 115 Transform coefficient coding unit 116 117 Multiplexer 118 Motion information selector 120 Output buffer 150 Encoding controller 200 Image encoder 201 Predictor 202 Second predictor , 203 ... prediction method changeover switch, 204 ... variable length coding unit, 205 ... motion information acquisition unit, 216 ... selected block coding unit, 217 ... motion information coding unit, 300 ... image decoding unit, 301 ... coding Column decoding unit 301 Code sequence decoding unit 302 Inverse quantization/inverse transform unit 303 Adder 304 Frame memory 305 Prediction unit 306 Information memory 307 Available block acquisition unit 308... Output buffer 310... Spatial direction motion information acquisition unit 311... Time direction motion information acquisition unit 312... Motion information changeover switch 313... Motion compensation unit 314... Information selection unit 320... Separation unit 321... Parameter Decoding unit 322 Transform coefficient decoding unit 323 Selected block decoding unit 350 Decoding control unit 400 Image decoding unit 401 Encoded sequence decoding unit 405 Prediction unit 410 Second prediction section 411 ... prediction method changeover switch 423 ... selected block decoding section 424 ... information decoding section , 901 ... High level syntax, 902 ... Sequence parameter set syntax, 903 ... Picture parameter set syntax, 904 ... Slice level syntax, 905 ... Slice header syntax, 906 ... Slice data syntax, 907 ... Macroblock level syntax, 908 ... Macroblock Layer syntax, 909...Macroblock prediction syntax.

Claims (3)

a determination unit that determines whether or not a plurality of candidate blocks having a predetermined positional relationship with respect to a target block are available in accordance with a predetermined order according to the positional relationship;
a selection unit that selects a selection block from among the candidate blocks determined to be available;
when the number of candidate blocks determined to be available exceeds 1, encoding identification information specifying the selected block based on a coding rule according to the number of candidate blocks determined to be available; an encoding unit that does not encode the identification information when the number of candidate blocks determined to be usable is 1;
a prediction unit that generates a predicted image of the target block based on the motion information corresponding to the selected block;
with
The determining unit selects, among the plurality of candidate blocks, candidate blocks to be determined second and subsequent ones as candidate blocks that have motion information and that the motion information is already available. An image encoding device that determines a usable candidate block when it does not match corresponding motion information. Determining whether or not a plurality of candidate blocks having a predetermined positional relationship with respect to the target block are available according to a predetermined order according to the positional relationship;
selecting a selection block from among the determined available candidate blocks;
when the number of candidate blocks determined to be available exceeds 1, encoding identification information specifying the selected block based on a coding rule according to the number of candidate blocks determined to be available; not encoding the identification information if the number of candidate blocks determined to be available is one;
generating a predicted image of the target block based on motion information corresponding to the selected block;
with
The step of determining includes determining that the candidate block has motion information and that the motion information is already available for the second and subsequent candidate blocks to be determined among the plurality of candidate blocks. A method of coding an image, comprising: determining an available candidate block if it does not match motion information corresponding to . determination means for determining whether or not a plurality of candidate blocks having a predetermined positional relationship with respect to the target block are available in accordance with a predetermined order according to the positional relationship;
selecting means for selecting a selected block from among the candidate blocks determined to be available;
when the number of candidate blocks determined to be available exceeds 1, encoding identification information specifying the selected block based on a coding rule according to the number of candidate blocks determined to be available; An encoding unit that does not encode the identification information when the number of candidate blocks determined to be usable is 1, and generating a predicted image of the target block based on motion information corresponding to the selected block. A program for causing a computer to function as a means of predicting
The determination means selects a candidate block for which the second and subsequent candidate blocks among the plurality of candidate blocks are determined to have motion information and for which the motion information is already available. A program that determines an available candidate block if it does not match the corresponding motion information.

Images