JP6871447B2 - Moving image coding method and moving image decoding method - Google Patents

Description

An embodiment of the present invention relates to a motion information compression method, a moving image coding method, and a moving image decoding method in coding and decoding a moving image.

In recent years, image coding methods with significantly improved coding efficiency have been developed in collaboration with ITU-T and ISO / IEC, such as ITU-T Rec. H.264 and ISO / IEC 14496-10 (hereinafter, H.264). It is recommended as). In H.264, prediction processing, conversion processing, and entropy coding processing are performed in rectangular block units (for example, 16 × 16 pixel block units, 8 × 8 pixel block units, etc.). In the prediction process, motion compensation is performed on the rectangular block to be encoded (block to be encoded) by referring to the already encoded frame (reference frame) to make a prediction in the time direction. In such motion compensation, it is necessary to encode the motion information including the motion vector as the spatial shift information between the coded block and the block referenced in the reference frame and send it to the decoding side. Further, when motion compensation is performed using a plurality of reference frames, it is necessary to encode the reference frame number together with the motion information. Therefore, the amount of code related to the motion information and the reference frame number may increase. Further, there is a motion information prediction method for deriving the predicted motion information of the coded block by referring to the motion information stored in the motion information memory of the reference frame (Patent Document 1 and Non-Patent Document 2). The capacity of the motion information memory that stores the data may increase.

As an example of a method of reducing the capacity of the motion information memory, (Non-Patent Document 2) derives representative motion information in a predetermined block and stores only the representative motion information in the motion information memory.

Patent No. 4020789 J. Jung et al, “Temporal MV predictor modification for MV-Comp, Skip, Direct and Merge schemes”, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 Document, JCTVC-D164, January 20110. Yeping Su et al, “CE9: Reduced resolution storage of motion vector data”, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11 Document, JCTVC-D072, January 20110.

However, when the method for deriving the predicted motion information shown in Non-Patent Document 1 and the method for deriving the representative motion information shown in Non-Patent Document 2 are different, the time correlation of the predicted motion information is reduced. There is a problem that the amount is increased.

The problem to be solved by the present invention is to solve the above-mentioned problems, and to provide a moving image coding device and a moving image decoding device including a motion information compression device capable of improving coding efficiency. To provide.

According to the embodiment, the moving image coding method is a method of dividing an input image signal into pixel blocks and performing inter-prediction for these divided pixel blocks. This method includes selecting predicted motion information from a motion information buffer that holds motion information in a coded region, and using the predicted motion information to predict motion information of a block to be encoded. Further, in this method, representative motion information is acquired from a plurality of motion information in the encoded region according to the first information indicating the method of selecting the predicted motion information, and only the representative motion information is obtained. Including.

The block diagram which shows schematic the structure of the image coding apparatus which concerns on 1st Embodiment. Explanatory drawing of the predicted coding order of a pixel block. Explanatory drawing of an example of a pixel block size. Explanatory drawing of another example of pixel block size. Explanatory drawing of another example of pixel block size. Explanatory drawing of an example of a pixel block in a coding tree unit. Explanatory drawing of another example of a pixel block in a coding tree unit. Explanatory drawing of another example of a pixel block in a coding tree unit. Explanatory drawing of another example of a pixel block in a coding tree unit. The block diagram which shows the structure of the entropy coding part of FIG. 1 schematicly. The explanatory view which shows the structure of the motion information memory of FIG. 1 schematicly. It is explanatory drawing of an example of the inter-prediction processing executed by the inter-prediction part of FIG. Explanatory drawing of another example of the inter-prediction processing executed by the inter-prediction part of FIG. Explanatory drawing of an example of a prediction unit. Explanatory drawing of another example of a prediction unit. Explanatory drawing of another example of a prediction unit. Explanatory drawing of another example of a prediction unit. Explanatory drawing of another example of a prediction unit. Explanatory drawing of another example of a prediction unit. Explanatory drawing of another example of a prediction unit. Explanatory drawing which shows skip mode, merge mode, inter mode. The block diagram which shows the structure of the motion information coding part of FIG. 4 schematicly. The explanatory view which shows the example of the position of the predicted motion information candidate with respect to the prediction unit to be encoded. Explanatory drawing which shows still another example of the position of the predicted motion information candidate with respect to the prediction unit to be encoded. The explanatory view which shows the example of the list which shows the relationship between the block position of a plurality of predicted motion information candidates, and index Mvpidx. The explanatory view which shows the example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be encoded is 32x32. The explanatory view which shows the example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 32x16. The explanatory view which shows the example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 16x32. The explanatory view which shows the example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 16x16. The explanatory view which shows the example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 16x8. The explanatory view which shows the example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 8x16. The explanatory view which shows still another example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 32x32. The explanatory view which shows still another example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 32x16. The explanatory view which shows still another example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 16x32. The explanatory view which shows still another example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 16x16. The explanatory view which shows still another example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 16x8. Explanatory drawing which shows still another example of the reference motion information acquisition position which shows the center of a prediction unit when the size of the prediction unit to be coded is 8x16. Explanatory drawing about spatial direction reference motion information memory 501 and temporal direction reference motion information memory 502. The flowchart which shows an example of the operation of the motion information compression part of FIG. The explanatory view which shows the example of the reference motion information acquisition position which shows the upper left end of the prediction unit when the size of the prediction unit to be encoded is 32x32. The explanatory view which shows the example of the reference motion information acquisition position which shows the upper left end of the prediction unit when the size of the prediction unit to be coded is 32x16. The explanatory view which shows the example of the reference motion information acquisition position which shows the upper left end of the prediction unit when the size of the prediction unit to be encoded is 16x32. The explanatory view which shows the example of the reference motion information acquisition position which shows the upper left end of the prediction unit when the size of the prediction unit to be coded is 16x16. The explanatory view which shows the example of the reference motion information acquisition position which shows the upper left end of the prediction unit when the size of the prediction unit to be coded is 16x8. The explanatory view which shows the example of the reference motion information acquisition position which shows the upper left end of the prediction unit when the size of the prediction unit to be coded is 8x16. Explanatory drawing which shows an example of a representative motion information position. Explanatory drawing which shows another example of representative motion information position. Explanatory drawing which shows an example of the center of the prediction unit in each prediction size. An explanatory diagram showing an example of a representative motion information position when the center of gravity of a plurality of reference motion information acquisition positions for each motion information compression block is set as a representative motion information position. Explanatory drawing which shows another example of the representative motion information position when the center of gravity of a plurality of reference motion information acquisition positions for each motion information compression block is set as a representative motion information position. Explanatory drawing which shows an example of a representative motion information position. Explanatory drawing which shows another example of representative motion information position. It is a figure which shows the syntax structure according to one Embodiment. It is a figure which shows an example of the sequence parameter set syntax according to one Embodiment. It is a figure which shows another example of the sequence parameter set syntax according to one Embodiment. It is a figure which shows an example of the prediction unit syntax according to one Embodiment. The block diagram which shows schematic the image decoding apparatus which concerns on 2nd Embodiment. The block diagram which shows schematic the entropy decoding part of FIG. FIG. 26 is a block diagram schematically showing a motion information decoding unit of FIG. 26.

Hereinafter, the moving image coding device and the moving image decoding device according to each embodiment will be described in detail with reference to the drawings. In the following description, the term "image" can be appropriately read as a term such as "video", "pixel", "image signal", and "image data". Further, in the following embodiments, the same operation is performed for the portions with the same number, and the description thereof will be omitted.
(First Embodiment)
The first embodiment relates to an image coding device. The moving image decoding device corresponding to the image coding device according to the present embodiment will be described in the second embodiment. This image coding device can be realized by hardware such as an LSI (Large-Scale Integration) chip, a DSP (Digital Signal Processor), and an FPGA (Field Programmable Gate Array). This image coding device can also be realized by causing a computer to execute an image coding program.

As shown in FIG. 1, the image coding apparatus 100 according to the present embodiment has a subtraction unit 101, an orthogonal transform unit 102, a quantization unit 103, an inverse quantization unit 104, an inverse orthogonal transform unit 105, and an addition unit 106. The image memory 107, the inter-prediction unit 108, the motion information compression unit 109, the motion information memory 110, and the entropy coding unit 112 are included. The coding control unit 114 and the output buffer 113 are usually installed outside the image coding device 100.

The image coding device 100 of FIG. 1 divides each frame or each field or each slice constituting the input image signal 151 into a plurality of pixel blocks, performs predictive coding on the divided pixel blocks, and encodes the divided pixel blocks. The conversion data 163 is output. In the following description, for simplification, it is assumed that the pixel block is predictively coded from the upper left to the lower right as shown in FIG. 2A. In FIG. 2A, in the frame f to be coded, the coded pixel block p is located on the left side and the upper side of the pixel block c to be coded.

Here, the pixel block refers to a unit for processing an image such as an M × N size block (N and M are natural numbers), a coding unit, a macro block, a sub block, and one pixel. In the following description, the pixel block is basically used in the meaning of the coding unit, but the pixel block can be interpreted in the above-mentioned meaning by appropriately reading the description. The coding unit is typically, for example, a 16 × 16 pixel block shown in FIG. 2B, but may be a 32 × 32 pixel block shown in FIG. 2C, a 64 × 64 pixel block shown in FIG. 2D, or is shown. It may be an 8 × 8 pixel block or a 4 × 4 pixel block. Also, the coding unit does not necessarily have to be square. Hereinafter, the coding target block or coding knit of the input image signal 151 may be referred to as a “prediction target block”. Further, the coding unit is not limited to a pixel block such as a coding unit, and a frame or field, a slice, or a combination thereof can be used.

3A to 3D are diagrams showing specific examples of coding units. FIG. 3A shows an example when the size of the coding unit is 64 × 64 (N = 32). Here, N represents the size of the reference coding unit, and the size when divided is defined as N, and the case where it is not divided is defined as 2N. The coding tree unit has a quadtree structure, and when divided, the four pixel blocks are indexed in Z-scan order. FIG. 3B shows an example in which the 64x64 pixel block of FIG. 3A is divided into quadtrees. The numbers shown in the figure indicate the order of Z scans. It is also possible to further divide the quadtree within the index of one quadtree of the coding unit. The depth of division is defined by Depth. That is, FIG. 3A shows an example of Dept = 0. FIG. 3C shows an example of a coding tree unit having a size of 32 × 32 (N = 16) when Dept = 1. The largest unit of such a coding tree unit is called a large coding tree unit or a tree block, and as shown in FIG. 2A, the input image signal is encoded in the raster scan order in this unit.

The image coding device 100 of FIG. 1 has inter-prediction (also referred to as inter-screen prediction, inter-frame prediction, motion compensation prediction, etc.) or inter-prediction for pixel blocks based on coding parameters input from the coding control unit 114. Intra-prediction (also referred to as in-screen prediction, in-frame prediction, etc.) (not shown) is performed to generate a prediction image signal 159. The image coding device 100 orthogonally transforms and quantizes the prediction error signal 152 between the pixel block (input image signal 151) and the predicted image signal 159, performs entropy coding, and generates coded data 163. Output.

The image coding device 100 of FIG. 1 selectively applies a plurality of prediction modes having different block sizes and methods for generating the predicted image signal 159 to perform coding. The prediction image signal 159 is roughly classified into two types: intra-prediction, which makes predictions within a frame to be encoded, and inter-prediction, which makes predictions using one or more reference frames that differ in time. is there.

Hereinafter, each element included in the image coding apparatus 100 of FIG. 1 will be described.
The subtraction unit 101 subtracts the corresponding prediction image signal 159 from the coded target block of the input image signal 151 to obtain the prediction error signal 152. The subtraction unit 101 inputs the prediction error signal 152 to the orthogonal transform unit 102.

The orthogonal transform unit 102 performs orthogonal transform such as discrete cosine transform (DCT) on the prediction error signal 152 from the subtraction unit 101 to obtain a conversion coefficient 153. The orthogonal transform unit 102 outputs the conversion coefficient 153 to the quantization unit 103.

The quantization unit 103 performs quantization on the conversion coefficient 153 from the orthogonal transformation unit 102 to obtain a quantization conversion coefficient 154. Specifically, the quantization unit 103 performs quantization according to quantization information such as a quantization parameter and a quantization matrix specified by the coding control unit 114. The quantization parameter indicates the fineness of quantization. The quantization matrix is used to weight the fineness of quantization for each component of the conversion coefficient, but the use / non-use of the quantization matrix is not an essential part of the embodiment of the present invention. The quantization unit 103 outputs the quantization conversion coefficient 154 to the entropy coding unit 112 and the inverse quantization unit 104.

The entropy coding unit 112 has a quantization conversion coefficient 154 from the quantization unit 103, motion information 160 from the inter-prediction unit 108, prediction information 165 designated by the coding control unit 114, and a reference from the coding control unit 114. Entropy coding (for example, Huffman coding, arithmetic coding, etc.) is performed on various coding parameters such as position information 164 and quantization information, and coded data 163 is generated. The coding parameter is a parameter required for decoding such as prediction information 165, information on conversion coefficient, and information on quantization. For example, the coding control unit 114 has an internal memory (not shown), and the coding parameters are held in this memory, and when the prediction target block is encoded, the coding parameters of the adjacent already encoded pixel blocks are set. Use.

Specifically, as shown in FIG. 4, the entropy coding unit 112 includes a parameter coding unit 401, a conversion coefficient coding unit 402, a motion information coding unit 403, and a multiplexing unit 404. The parameter coding unit 401 encodes the coding parameters such as the prediction information 165 received from the coding control unit 114 to generate the coded data 451A. The conversion coefficient coding unit 402 encodes the quantized conversion coefficient 154 received from the quantization unit 103 to generate the coded data 451B.

The motion information coding unit 403 encodes the motion information 160 received from the inter-prediction unit 108 with reference to the reference motion information 166 received from the motion information memory 110 and the reference position information 164 received from the coding control unit 114. To generate coded data 451C. The details of the motion information coding unit 403 will be described later.

The multiplexing unit 404 multiplexes the coded data 451A, 451B, and 451C to generate the coded data 163. The generated coded data 163 includes motion information 160, prediction information 165, and all parameters required for decoding such as information on conversion coefficients and information on quantization.

The coded data 163 generated by the entropy coding unit 112 is temporarily stored in the output buffer 113 through, for example, multiplexing, and is output as the coded data 163 according to an appropriate output timing managed by the coding control unit 114. To. The coded data 163 is output to, for example, a storage system (storage medium) or a transmission system (communication line) (not shown).

The inverse quantization unit 104 performs inverse quantization on the quantization conversion coefficient 154 from the quantization unit 103 to obtain a restoration conversion coefficient 155. Specifically, the inverse quantization unit 104 performs inverse quantization according to the quantization information used in the quantization unit 103. The quantization information used in the quantization unit 103 is loaded from the internal memory of the coding control unit 114. The inverse quantization unit 104 outputs the restoration conversion coefficient 155 to the inverse orthogonal transformation unit 105.

The inverse orthogonal transform unit 105 performs an inverse orthogonal transform corresponding to the orthogonal transform performed by the orthogonal transform unit 102, such as an inverse discrete cosine transform, on the restoration conversion coefficient 155 from the inverse quantization unit 104. The restoration prediction error signal 156 is obtained. The inverse orthogonal transform unit 105 outputs the restoration prediction error signal 156 to the addition unit 106.

The addition unit 106 adds the restoration prediction error signal 156 and the corresponding prediction image signal 159 to generate a local decoded image signal 157. The decoded image signal 157 is subjected to a deblocking filter, a Wiener filter, or the like (not shown), and is input to the reference image memory 107.

The reference image memory 107 stores the filtered image signal 158 after local decoding in the memory, and is referred to as the reference image signal 158 when the inter-prediction unit 108 generates a predicted image as needed.

The inter-prediction unit 108 performs inter-prediction using the reference image signal 158 stored in the reference image memory 107. Specifically, the inter-prediction unit 108 performs block matching processing between the prediction target block and the reference image signal 158 to derive the amount of motion deviation (motion vector). The inter-prediction unit 108 generates an inter-prediction image by performing motion compensation (interpolation processing in the case of decimal precision motion) based on this motion vector. H. In 264, interpolation processing up to 1/4 pixel accuracy is possible. The derived motion vector is entropy-encoded as part of the motion information 160.

The motion information memory 110 has a motion information compression unit 109, appropriately compresses the motion information 160 to reduce the amount of information, and temporarily stores the motion information 160 as reference motion information 166. As shown in FIG. 5, the motion information memory 110 is held in frame (or slice) units, and the spatial direction reference motion information memory 501 that stores the motion information 160 on the same frame as the reference motion information 166 and the already Further, it has a time direction reference motion information memory 502 that stores the motion information 160 of the frame whose coding has been completed as the reference motion information 166. The time direction reference motion information memory 502 may have a plurality of reference frames depending on the number of reference frames used for prediction by the coded target frame.

Further, the spatial direction reference motion information memory 501 and the temporal direction reference motion information memory 502 may logically divide physically the same memory. Further, the spatial reference motion information memory 501 holds only the spatial motion information required by the frame currently being encoded, and sequentially compresses the spatial motion information that is no longer needed for reference, and the temporal reference motion. It may be stored in the information memory 502.

The reference motion information 166 is held in the spatial direction reference motion information memory 501 and the time direction reference motion information memory 502 in predetermined area units (for example, 4 × 4 pixel block units). The reference motion information 166 further has information indicating whether the region is encoded by the inter-prediction described later or the intra-prediction described later. In addition, the coding unit (or prediction unit) is H.I. As in the skip mode defined by 264, the direct mode, or the merge mode described later, the value of the motion vector in the motion information 160 is not encoded, and the motion information 160 predicted from the encoded region is used for interim. Even in the predicted case, the motion information of the coding unit (or prediction unit) is retained as the reference motion information 166.

When the coding process of the frame or slice to be encoded is completed, the spatial direction reference motion information memory 501 of the frame is changed to the time direction reference motion information memory 502 used for the frame to be encoded next. To. At this time, in order to reduce the memory capacity of the time direction reference motion information memory 502, the motion information 160 compressed by the motion information compression unit 109 described later is stored in the time direction reference motion information memory 502.

The prediction information 165 follows a prediction mode controlled by the coding control unit 114, and as described above, inter-prediction or intra-prediction or inter-prediction (not shown) can be selected for generating the predicted image signal 159, but intra-prediction. A plurality of modes can be further selected for each of the inter-prediction and the inter-prediction. The coding control unit 114 determines one of the plurality of prediction modes of the intra prediction and the inter prediction as the optimum prediction mode, and sets the prediction information 165.

For example, the coding control unit 114 determines the optimum prediction mode using the cost function shown in the following mathematical expression (1).

In formula (1) (hereinafter referred to as simple coding cost), OH indicates the code amount related to the prediction information 160 (for example, motion vector information, prediction block size information), and SAD indicates the prediction target block and the prediction image signal 159. The sum of the absolute values of the differences between the two (that is, the cumulative sum of the absolute values of the prediction error signal 152) is shown. Further, λ indicates a Lagrange undetermined multiplier determined based on the value of the quantization information (quantization parameter), and K indicates the coding cost. When the mathematical formula (1) is used, the prediction mode that minimizes the coding cost K is determined as the optimum prediction mode from the viewpoint of the generated code amount and the prediction error. As a modification of the equation (1), the coding cost may be estimated from only OH or only SAD, or the coding cost may be estimated using the value obtained by Hadamard transforming SAD or an approximate value thereof.

It is also possible to determine the optimum prediction mode by using a provisional coding unit (not shown). For example, the coding control unit 114 determines the optimum prediction mode using the cost function shown in the following mathematical expression (2).

In formula (2), D indicates the sum of squared errors (ie, coding distortion) between the predicted block and the locally decoded image, and R is the prediction between the predicted block and the predicted image signal 159 in the predicted mode. The amount of code estimated by tentative coding for the error is shown, and J indicates the coding cost. When deriving the coding cost J (hereinafter referred to as the detailed coding cost) of the equation (2), provisional coding processing and local decoding processing are required for each prediction mode, so that the circuit scale or the amount of calculation increases. .. On the other hand, since the coding cost J is derived based on the more accurate coding distortion and the coding amount, it is easy to determine the optimum prediction mode with high accuracy and maintain high coding efficiency. As a modification of the mathematical formula (2), the coding cost may be estimated only from R or only D, or the coding cost may be estimated using the approximate value of R or D. Moreover, these costs may be used hierarchically. The coding control unit 114 makes a determination using the mathematical formula (1) or the mathematical formula (2) based on the information obtained in advance regarding the prediction target block (prediction mode of surrounding pixel blocks, result of image analysis, etc.). The number of candidates for the prediction mode may be narrowed down in advance.

As a modification of this embodiment, it is possible to further reduce the number of candidates for the prediction mode while maintaining the coding performance by performing the two-step mode determination by combining the formula (1) and the formula (2). It becomes. Here, unlike the mathematical formula (2), the simple coding cost represented by the mathematical formula (1) does not require a local decoding process, and thus can be calculated at high speed. In the moving image coding apparatus of this embodiment, H. Since the number of prediction modes is larger than that of 264, mode determination using the detailed coding cost is not realistic. Therefore, as a first step, a mode determination using the simple coding cost is performed for the prediction mode available in the pixel block, and a prediction mode candidate is derived.

Here, the number of prediction mode candidates is changed by utilizing the property that the correlation between the simple coding cost and the detailed coding cost increases as the value of the quantization parameter that defines the roughness of the quantization increases.

Next, the prediction process of the image coding apparatus 100 will be described.
Although not shown, the image coding apparatus 100 of FIG. 1 is provided with a plurality of prediction modes, and in each prediction mode, the method of generating the prediction image signal 159 and the motion compensation block size are different from each other. The method by which the prediction unit 108 generates the prediction image signal 159 is roughly divided into an intra prediction (inside the frame) in which the prediction image is generated by using the reference image signal 158 of the coded frame (or field). Prediction) and inter-prediction (inter-frame prediction) that generates a prediction image using the reference image signal 158 of one or more encoded reference frames (or reference fields). The prediction unit 108 selectively switches between the intra prediction and the inter prediction to generate the prediction image signal 159 of the coded block.

FIG. 6A shows an example of inter-prediction. Inter-prediction is typically performed in units of prediction units, and it is possible for each prediction unit to have different motion information 160. In the inter-prediction, as shown in FIG. 6A, it is a pixel block in a reference frame (for example, a coded frame one frame before) that has already been coded, and is a prediction unit to be coded. A predicted image signal 159 is generated from the block 601 at the same position by using the reference image signal 158 of the block 602 at the position spatially shifted according to the motion vector included in the motion information 160. That is, in the generation of the predicted image signal 159, the reference image signal 158 of the block 602 in the reference frame specified by the position (coordinates) of the coded block and the motion vector included in the motion information 160 is used.

In the inter-prediction, motion compensation with a small number of pixel accuracy (for example, 1/2 pixel accuracy or 1/4 pixel accuracy) is possible, and the value of the interpolated pixel is generated by performing filtering processing on the reference image signal 158. Will be done. For example, H. In 264, interpolation processing up to 1/4 pixel accuracy is possible for the luminance signal. The interpolation process is performed by H. It can be performed by using arbitrary filtering in addition to the filtering specified in 264.

Note that the inter-prediction is not limited to the example of using the reference frame one frame before as shown in FIG. 6A, and any encoded reference frame may be used as shown in FIG. 6B. When the reference image signals 158 of a plurality of reference frames having different time positions are held, the information indicating from which time position the reference image signal 158 the predicted image signal 159 is generated is represented by the reference frame number. The reference frame number is included in the motion information 160. The reference frame number can be changed in units of areas (pictures, slices, blocks, etc.). That is, different reference frames can be used for each prediction unit. As an example, if the encoded reference frame one frame before is used for prediction, the reference frame number in this area is set to 0, and if the encoded reference frame two frames before is used for prediction, The reference frame number in this area is set to 1. As another example, when the reference image signal 158 for only one frame is held in the reference image memory 107 (the number of reference frames held is only one), the reference frame number is always set to 0. Set.

Further, in the inter-prediction, a size suitable for the coded block can be selected and used from the sizes of a plurality of prediction units prepared in advance. For example, it is possible to perform motion compensation for each prediction unit obtained by dividing the coding tree unit as shown in FIGS. 7A to 7G. Further, it is possible to perform motion compensation for each prediction unit obtained by dividing into a rectangle other than the rectangle as shown in FIGS. 7F and 7G.

As described above, since the motion information 160 of the coded pixel block (for example, 4 × 4 pixel block) in the coded target frame used for the inter-prediction is held as the reference motion information 166, the input image signal. According to the local properties of 151, the optimum motion compensation block shape, motion vector, and reference frame number can be utilized. Further, the coding unit and the prediction unit can be arbitrarily combined. When the coding tree unit is a 64 × 64 pixel block, the coding tree unit is further divided into four for each of the four coding tree units (32 × 32 pixel blocks) in which the 64 × 64 pixel block is divided. It is possible to use a block of 64 × 64 pixels to a block of 16 × 16 pixels hierarchically. Similarly, 64 × 64 pixel blocks to 8 × 8 pixel blocks can be used hierarchically. Here, assuming that the prediction unit divides the coding tree unit into four, it is possible to execute hierarchical motion compensation processing from a 64 × 64 pixel block to a 4 × 4 pixel block. ..

Further, in the inter-prediction, bidirectional prediction using two types of motion compensation can be executed for the pixel block to be coded. H. In 264, two types of motion compensation are performed on the pixel block to be coded, and a new predicted image signal is obtained by weighted averaging the two types of predicted image signals (not shown). In bidirectional prediction, the two types of motion compensation are referred to as list 0 prediction and list 1 prediction, respectively.

<Explanation of skip mode, merge mode, and intermode>
The image coding apparatus 100 according to the present embodiment uses a plurality of prediction modes having different coding processes shown in FIG. The skip mode in the figure is a mode in which only the syntax related to the predicted motion information position 954 described later is encoded, and the other syntax is not encoded. The merge mode is a mode in which only the syntax regarding the predicted motion information position 954 and the conversion coefficient information 153 are encoded, and the other syntaxes are not encoded. The intermode is a mode for encoding the syntax regarding the predicted motion information position 954, the difference motion information 953 described later, and the conversion coefficient information 153. These modes are switched by the prediction information 165 controlled by the coding control unit 114.

<Motion information coding unit 403>
Hereinafter, the motion information coding unit 403 will be described with reference to FIG.

The motion information coding unit 403 includes a reference motion vector acquisition unit 901, a predicted motion vector selection switch (also referred to as a predicted motion information selection switch) 902, a subtraction unit 903, a differential motion information coding unit 904, and a predicted motion information position coding. It has a unit 905 and a multiplexing unit 906.

The reference motion vector acquisition unit 901 receives at least one predicted motion information candidate (also referred to as a predicted motion vector candidate) 951 (951A, 951B, ...) By inputting the reference motion information 166 and the reference position information 164. .. 10 and 11 show an example of the position of the predicted motion information candidate 951 with respect to the target prediction unit. FIG. 10 shows the position of the prediction unit spatially adjacent to the target prediction unit. AX (X = 0 to nA-1) is a prediction unit adjacent to the left side of the target prediction unit, and BY (Y = 0 to nB-1) is a prediction unit adjacent to the target prediction unit. The action units, C, D, and E indicate the prediction units adjacent to the upper right, upper left, and lower left with respect to the target prediction unit, respectively. Further, FIG. 11 shows the position of the prediction unit in the already encoded reference frame with respect to the prediction unit to be encoded. Col in FIG. 11 indicates a prediction unit in the reference frame and at the same position as the prediction unit to be encoded. FIG. 12 shows an example of a list showing the relationship between the block position of the plurality of predicted motion information candidates 951 and the index Mvpidx. Mvpidx 0 to 2 indicates a predicted motion vector candidate 951 located in the spatial direction, and Mvpidx 3 indicates a predicted motion vector candidate 951 located in the time direction. The prediction unit position A is an inter-prediction of the AX shown in FIG. 10, that is, the prediction unit having the reference motion information 166, and the position where the value of X is the smallest is the prediction unit position A. Further, the prediction unit position B is an inter-prediction among the BYs shown in FIG. 10, that is, the prediction unit having the reference motion information 166, and the position where the value of Y is the smallest is referred to as the prediction unit position A. To do. When the prediction unit position C is not an inter-prediction, the reference motion information 166 of the prediction unit position D is replaced with the reference motion information 166 of the prediction unit position C. When the prediction unit positions C and D are not inter-prediction, the reference motion information 166 of the prediction unit position E is replaced with the reference motion information 166 of the prediction unit position C.

When the size of the prediction unit to be encoded is larger than the minimum prediction unit, the prediction unit position Col may hold a plurality of reference motion information 166s in the time direction reference motion information memory 502. In this case, the reference motion information 166 in the prediction unit of the position Col is acquired according to the reference position information 164. Hereinafter, the acquisition position of the reference motion information 166 in the prediction unit of the position Col is referred to as a reference motion information acquisition position. 13A to 13F show an example of the reference motion information acquisition position when the reference position information 164 indicates the center of the prediction unit at the position Col for each size (32x32 to 16x16) of the prediction unit to be encoded. The blocks in the figure each indicate a 4x4 prediction unit, and the circles indicate the positions of the 4x4 prediction units to be acquired as the predicted motion information candidate 951. Another example of the reference motion information acquisition position is shown in FIGS. 14A to 14F. In FIGS. 14A to 14F, since the 4x4 prediction unit does not exist at the position of the circle, a predetermined method such as the average value or the median value of the reference motion information 166 in the four 4x4 prediction units adjacent to the circle is used. Predictive motion information candidate 951 is generated. As yet another example of the reference motion information acquisition position, the reference motion information 166 of the 4x4 prediction unit located at the upper left end of the prediction unit at the position Col may be used as the predicted motion information candidate 951. In addition to the above examples, any position and method may be used to generate the predicted motion information candidate 951 as long as it is a predetermined method.

When the reference motion information 166 does not exist, the motion information 160 having a zero vector is output as the predicted motion information candidate 951.

As described above, at least one or more predicted motion information candidates 951 are output from the reference motion block. When the reference frame number of the predicted motion information candidate 951 and the reference frame number of the coding target prediction unit are different, the predicted motion information candidate 951 is divided into the reference frame number of the predicted motion information candidate 951 and the coding target prediction. It may be scaled according to the reference frame number of the unit.

The predicted motion information selection switch 902 selects one from a plurality of predicted motion information candidates 951 in response to a command from the coding control unit 114, and outputs the predicted motion information 952. Further, the predicted motion information selection switch 902 may output the predicted motion information position information 954 described later. The above selection may be made using an evaluation function such as mathematical formulas (1) and (2). The subtracting unit 903 subtracts the predicted motion vector information 952 from the motion information 160, and outputs the differential motion information 953 to the differential motion information coding unit 904. The difference motion information coding unit 904 encodes the difference motion information 953 and outputs the coded data 960A. In the skip mode and the merge mode, the differential motion information coding unit 904 does not need to encode the differential motion information 953.

The predicted motion information position coding unit 905 encodes the predicted motion information position information 954 (Mvpidx) indicating which predicted motion information candidate 951 is selected from the list shown in FIG. 12, and outputs the encoded data 960B. To do. The predicted motion information position information 954 is encoded using equal-length coding or variable-length coding generated from the total number of predicted motion information candidates 951. Variable length coding may be performed using the correlation with the adjacent block. Further, when a plurality of predicted motion information candidates 951 have duplicate information, a code table is created from the total number of predicted motion information candidates 951 in which the duplicated predicted motion information candidates 951 are deleted, and the predicted motion information position information 954 is encoded. It doesn't matter. Further, when the total number of the predicted motion information candidates 951 is one type, the predicted motion information candidate 951 is determined to be the predicted motion information 952, so that it is not necessary to encode the predicted motion information position information 954.

Further, the derivation method of the predicted motion information candidate 951 does not have to be the same in each of the skip mode, the merge mode, and the inter mode, and the derivation method of the predicted motion information candidate 951 may be set independently for each. In the present embodiment, the method for deriving the predicted motion information candidate 951 in the skip mode and the intermode is the same, and the method for deriving the predicted motion information candidate 951 in the merge mode is different.

<Details of motion information compression unit 109>
First, the motion information compression process will be described with reference to FIG. In FIG. 15, the reference motion information 166 of the spatial reference motion information memory 501 is compressed and stored in the temporal reference motion information memory 502. In the spatial direction reference motion information memory 501, the reference motion information 166 held at the representative motion information position is stored in the time direction reference motion information memory 502 for each motion information compression block (16x16 pixel block in the figure). When the above-mentioned motion information coding process is performed, the reference motion information 166 held at the above-mentioned reference motion information acquisition position is set as the predicted motion information candidate 951. At this time, assuming that the motion information compression block virtually has the same reference motion information 166, the reference motion information 166 held at the above-mentioned reference motion information acquisition position may be set as the predicted motion information candidate 951. None (The same predicted motion information candidate 951 is derived.)
Next, the motion information compression unit 109 will be described with reference to the flowchart shown in FIG. The motion information compression unit 109 compresses the motion information 160 and stores the motion information 160 in the time direction reference motion information memory 502 when the coding process of the frame (or an arbitrary unit such as a slice or a coding unit) is completed. ..

First, the reference position information 164 is acquired from the coding control unit 114 (step S1601), and the frame is divided into motion information compression blocks, which are compression units of the motion information 160 (step S1602). The motion information compression block is a pixel block larger than a unit (typically a 4x4 pixel block) in which motion information 160 is held by motion compensation processing, and is typically a 16x16 pixel block. The motion information compression block may be a 64x64 pixel block, a 32x32 pixel block, an 8x8 pixel block, a rectangular pixel block, or a pixel area having an arbitrary shape.

Next, the representative motion information position is generated according to the reference position information 164 (step S1603). As an example of generating the representative motion information position, when the motion information compression block is a 16x16 pixel block, the reference motion information acquisition position when the size of the prediction unit shown in FIGS. 13D, 14D, and 17D is 16x16 is representative. It is the motion information position. Next, the reference motion information 166 of the generated representative motion information position is set as the representative motion information (step S1604), and the representative motion information is stored in the time direction reference motion information memory (step S1605). The above steps S1604 to S1605 are executed for all motion information compression blocks.

Assuming that the unit in which the motion information 160 is held is the MxM block and the size of the motion information compression block is NxN (N is a multiple of M), the capacity of the reference motion information memory is increased (MxM) by executing the motion information compression process. ) / (NxN).

<Another Embodiment of Representative Movement Information Position>
As another example of generating the representative motion information position, the center position of the plurality of reference motion information acquisition positions may be set as the representative motion information position. 18A and 18B show representative motion information positions for each motion compression block of size 16x16. FIG. 18A is a representative motion information position when the reference motion information acquisition position is the position shown in FIG. 13D, and FIG. 18B is similarly representative motion information when the reference motion information acquisition position is the position shown in FIG. 17D. Each position is shown. The circles in FIGS. 18A and 18B indicate the reference motion information acquisition positions when the prediction unit is a 16x16 block, and are located at the center positions (also referred to as the center of gravity positions) of the four reference motion information acquisition positions. The representative movement information position indicated by the cross mark is arranged.

As yet another example of generating the representative motion information position, the reference motion information acquisition position for each size of a plurality of prediction units is provided as the reference position information 164, and the representative motion information position is generated from the plurality of reference motion information acquisition positions. It doesn't matter.

As an example of generating the representative motion information position, even if the reference motion information acquisition position for each size of a plurality of prediction units is provided as the reference position information 164 and the representative motion information position is generated from the plurality of reference motion information acquisition positions. I do not care. FIG. 19 shows the center (reference motion information acquisition position) of each size of the prediction unit having a size of 16x16 or more when the tree block is a 64x64 pixel block.

As another example of generating the representative motion information position, the representative motion information position may be set by using the reference motion information acquisition position arranged for each motion information compression block. FIG. 20A shows an example in which the center of gravity of a plurality of reference motion information acquisition positions for each motion information compression block is set as the representative motion information position. If the position of the center of gravity does not match the position of the 4x4 block, the nearest 4x4 block may be used as the representative motion information position, or the reference motion vector 166 of the center of gravity position may be set by using an interpolation method such as the co-primary interpolation method. You may generate it.

Further, FIG. 20B shows an example in which one of a plurality of reference motion information acquisition positions is selected for each motion information compression block and set as a representative motion information position.

Further, FIGS. 21A and 21B further show an example in which the reference motion information acquisition position is the same in each motion information compression block in the tree block. Since the representative motion information position is the same in all the motion information compression blocks, it is not necessary to switch the representative motion information position according to the position in the tree block. In addition to FIGS. 21A and 21B, the representative motion information position may be at any position such as the upper left end or the upper right end in the motion information compression block.

As an example of generating the representative motion information position, the representative motion information position may be indicated by using BlkIdx which indicates the 4x4 block position in the motion information compression block in the Z scan order. When the size of the motion information compression block is 16x16, the representative motion information position shown in FIG. 21A corresponds to the position of BlkIdx = 12. The representative motion information position shown in FIG. 21B corresponds to the position of BlkIdx = 15.

As another example in the motion information compression process, the reference frame number may be included in the motion information compression process in order to reduce the memory capacity related to the reference frame number. In this case, the reference frame number held at the representative motion information position is stored in the memory capacity related to the reference frame number. Therefore, the spatial reference motion information memory 501 and the spatial reference motion information memory 502 shown in FIG. 5 store the reference frame number in addition to the motion vector information.

As yet another example of the motion information compression process, when the reference frame number is not included in the motion information compression process, the motion vector information in the motion information at the representative motion information position is scaled using the reference frame number. It may be stored in the motion information memory 110. A typical example of the scaling process is a linear scaling process based on the reference frame number zero. In this method, when the reference frame number is a non-zero value, the motion vector information is linearly scaled so as to refer to the reference frame corresponding to the reference frame number zero. The reference frame number may be a non-zero value as the standard for the scaling process described above. If division occurs when the above-mentioned linear scaling processing is performed, the above-mentioned division may be realized by creating a table of the division processing in advance and pulling the table each time.

When the size of the motion information compression block is other than 16x16 blocks, the representative motion information position is generated by using the same processing as described above. In one example, when the size of the motion information compression block is 64x64, the reference motion information acquisition position when the size of the prediction unit is 64x64 is set as the representative motion information position. In yet another example, the representative motion information position when the size of the motion information compression block shown in FIGS. 21A and 21B is 16x16 blocks is scaled in the horizontal and vertical directions according to the size of the motion information compression block. It may be the information position.

If the reference motion information does not exist because the representative motion information position is outside the picture or slice, the new representative motion is the position where the reference motion information can be obtained in the motion information compression block such as the upper left corner of the motion information compression block. It may be replaced as an information position. Further, even when the representative motion information position is the area to which the intra prediction is applied and the reference motion information does not exist, the same process may be executed to replace it as a new representative motion information position.

<Syntax configuration>
Hereinafter, the syntax used by the image coding apparatus 100 of FIG. 1 will be described.
The syntax shows the structure of the coded data (for example, the coded data 163 of FIG. 1) when the image coding apparatus encodes the moving image data. When decoding this coded data, the moving image decoding apparatus refers to the same syntax structure and performs syntax interpretation. The syntax 2200 used by the moving image encoding device of FIG. 1 is illustrated in FIG.

The syntax 2200 includes three parts: high-level syntax 2201, slice-level syntax 2202, and coding tree-level syntax 2203. The high-level syntax 2201 contains the syntax information of the layer above the slice. A slice is a rectangular or continuous area contained in a frame or field. The slice level syntax 2202 contains the information required to decode each slice. The coding tree level syntax 2203 contains information necessary for decoding each coding tree (that is, each coding tree unit). Each of these parts contains more detailed syntax.

High-level syntax 2201 includes sequence and picture-level syntax such as sequence parameter set syntax 2204 and picture parameter set syntax 2205. Slice level syntax 2202 includes slice header syntax 2206, slice data syntax 2207, and the like. The coding tree level syntax 2203 includes a coding tree unit syntax 2208, a transform unit syntax 2209, a prediction unit syntax 2210, and the like.

The coding tree unit Syntax 2208 can have a quadtree structure. Specifically, the coding tree unit syntax 2208 can be recursively called as a syntax element of the coding tree unit syntax 2208. That is, one coding tree unit can be subdivided into quadtrees. Further, the coding tree unit syntax 2208 includes a transform unit syntax 2209 and a predi cushion unit syntax 2210. The transform unit syntax 2209 and the predi cushion unit syntax 2210 are called in each coding tree unit syntax 2208 at the end of the quadtree. The pre-cushion unit syntax 2210 describes information related to prediction, and the transform unit syntax 2209 describes information related to inverse orthogonal transformation and quantization.

FIG. 23 illustrates the sequence parameter set syntax 2204 according to this embodiment. The motion_vector_buffer_comp_flag shown in FIGS. 23A and 23B is a syntax indicating the validity / invalidity of the motion information compression according to the present embodiment with respect to the sequence. When motion_vector_buffer_comp_flag is 0, the motion information compression according to the present embodiment is invalid for the sequence. Therefore, the processing of the motion information compression unit shown in FIG. 1 is skipped. As an example, when motion_vector_buffer_comp_flag is 1, the motion information compression related to the present mobile phone is effective for the sequence. The motion_vector_buffer_comp_ratio_log2 shown in FIGS. 23 and 23B is information indicating a unit of motion information compression processing, and is shown when the motion_vector_buffer_comp_flag is 1. The motion_vector_buffer_comp_ratio_log2 indicates, for example, the size information of the motion information compression block according to the present embodiment, and the motion_vector_buffer_comp_ratio_log2 is the minimum unit of motion compensation, and the motion compensation is 2 ^{(motion_vector_buffer_buffer)} . An example in which the minimum unit of motion compensation is a 4x4 pixel block, that is, the reference motion information memory is held in the 4x4 pixel block unit is shown below. When motion_vector_buffer_comp_ratio_log2 is 1, the size of the motion information compression block according to the present embodiment is an 8x8 pixel block. Similarly, when motion_vector_buffer_comp_ratio_log2 is 2, the size of the motion information compression block according to the present embodiment is a 16x16 pixel block. The motion_vector_buffer_comp_position shown in FIG. 23B is information indicating a representative motion information position in the motion information compression block, and is shown when the motion_vector_buffer_comp_flag is 1. The motion_vector_buffer_comp_position indicates, for example, the reference motion information position in the motion information compression block as shown in FIGS. 21A and 21B, and indicates the reference motion information position for each motion information compression block as shown in FIGS. 20A and 20B. It doesn't matter. It may also be in the center of a plurality of blocks.

As another example, the layers below the prediction_vector_buffer_comp_flag, the motion_vector_buffer_comp_ratio_log2, the motion_vector_buffer_comp_position, etc. The validity / invalidity of such prediction may be specified.

FIG. 24 shows an example of the prediction unit syntax. Skip_flag in the figure is a flag indicating whether or not the prediction mode of the coding unit to which the prediction unit syntax belongs is the skip mode. When skip_flag is 1, it means that the syntaxes other than the predicted motion information position information 954 (coding unit syntax, prediction unit syntax, transform unit syntax) are not encoded. NumMVPCand (L0) and NumMVPCand (L1) indicate the number of predicted movement information candidates 951 in the list 0 prediction and the list 1 prediction, respectively. When the predicted motion information candidate 951 exists (NumMVPCand (LX)> 0, X = 0 or 1), mvp_idx_lX indicating the predicted motion information position information 954 is encoded.

When skip_flag is 0, it indicates that the prediction mode of the coding unit to which the prediction unit syntax belongs is not the skip mode. NumberCandidates shows the number of predicted motion information candidates 951 derived in FIG. 12 and the like. When the predicted motion information candidate 951 exists (NumMergeCandedates> 0), the flagage_flag, which is a flag indicating whether or not the prediction unit is in the merge mode, is encoded. When its value is 1, it indicates that the prediction unit is in merge mode, and when its value is 0, it indicates that the prediction unit uses intermode. When the message_flag is 1 and there are two or more predicted motion information candidates 951 (NumMergeCandedates> 1), the predicted motion information 952, which is the predicted motion information 952, is encoded from the predicted motion information candidates 951. ..

When the message_flag is 1, the prediction unit syntax other than the message_flag and the message_idx does not need to be encoded.

When merge_flag is 0, it indicates that the prediction unit is in intermode. In the intermode, in the case of mvd_lX (X = 0 or 1) indicating the differential motion vector information included in the differential motion information 953, the reference frame number ref_idx_lX, and the B slice, the prediction unit is unidirectional prediction (list 0 or list 1). An inter_pred_idc indicating whether it is a bidirectional prediction is encoded. Further, when NumMVPCand (L0) and NumMVPCand (L1) are acquired in the same manner as in the skip mode and the predicted motion information candidate 951 exists (NumMVPCand (LX)> 0, X = 0 or 1), the predicted motion information position information 954 Mvp_idx_lX indicating is encoded.

The above is the syntax configuration according to this embodiment.

(Second Embodiment)
The second embodiment relates to a moving image decoding device. The moving image coding device corresponding to the moving image decoding device according to the present embodiment is as described in the first embodiment. That is, the moving image decoding device according to the present embodiment decodes the coded data generated by the moving image coding device according to the first embodiment, for example.

As shown in FIG. 25, the moving image decoding apparatus according to the present embodiment includes an entropy decoding unit 2501, an inverse quantization unit 2502, an inverse orthogonal conversion unit 2503, an addition unit 2504, a reference image memory 2505, and an inter-prediction unit 2506. , The reference motion information memory 2507, the reference motion information compression unit 2508, and the decoding control unit 2510.

The moving image decoding device of FIG. 25 decodes the coded data 2550, stores the decoded image signal 2554 in the output buffer 2511, and outputs it as an output image. The coded data 2550 is output from, for example, the moving image coding device of FIG. 1, and is input to the moving image decoding device 2500 via a storage system or a transmission system (not shown).

The entropy decoding unit 2501 decodes the coded data 2550 based on the syntax for decoding. The entropy decoding unit 2501 sequentially entropy-decodes the code string of each syntax, and reproduces the coding parameters of the coded target block such as the motion information 2559 and the quantization conversion coefficient 2551. The coding parameter is a parameter required for decoding such as prediction information, information on conversion coefficient, and information on quantization.

Specifically, as shown in FIG. 26, the entropy decoding unit 2501 includes a separation unit 2601, a parameter decoding unit 2602, a conversion coefficient decoding unit 2603, and a motion information decoding unit 2604. Separation unit 2601 separates the coded data 2550, the parameter coding data 2651A is the parameter decoding unit 2602, the conversion coefficient coding data 2651B is the conversion coefficient decoding unit 2603, and the motion information coding data 2651C is the motion information. It is output to the decoding unit 2604 respectively. The parameter decoding unit 2602 decodes the coding parameter 2570 such as prediction information, outputs the coding parameter 2570, and outputs the coding parameter 2570 to the decoding control unit 2510. The conversion coefficient decoding unit 2603 inputs the coded data 2651B, decodes the conversion coefficient information 2551 and outputs it to the inverse quantization unit 2502.

The motion information decoding unit 2604 receives the coded data 2651C from the separation unit 2601, the reference position information 2560 from the decoding control unit 2510, and the reference motion information 2558 from the reference motion information memory 2507, and outputs the motion information 2559. The output motion information 2559 is input to the inter-prediction unit 2506.

As shown in FIG. 27, the motion information decoding unit 2604 includes a separation unit 2701, a difference motion information decoding unit 2702, a predicted motion information position decoding unit 2503, a reference motion information acquisition unit 2704, a predicted motion information selection switch 2705, and the motion information decoding unit 2604. The addition unit 2706 is included.

The coded data 2651C related to the motion information is input to the separation unit 2701 and separated into the coded data 2751 related to the differential motion information and the coded data 2752 related to the predicted motion information position. The difference motion information coding unit 2702 inputs the coded data 2751 related to the difference motion information and decodes the difference motion information 2753. The differential motion information 2753 is added to the predicted motion information 2756 described later by the addition unit 2706, and the motion information 2759 is output. The predicted motion information position decoding unit 2703 inputs the coded data 2752 regarding the predicted motion information position, and decodes the predicted motion information position 2754.

The predicted motion information position 2754 is input to the predicted motion information selection switch 2705, and the predicted motion information 2756 is selected from the predicted motion information candidates 2755. The predicted motion information position information 2560 is decoded by using isometric decoding or variable length decoding generated from the number of predicted motion information candidates 2755. Variable length decoding may be performed using the correlation with the adjacent block. Further, when a plurality of predicted motion information candidates 2755 overlap, the predicted motion information position information 2560 may be decoded from the code table generated from the total number of the predicted motion information candidates 2755 with the duplication removed. Further, when the total number of the predicted motion information candidates 2755 is one type, the predicted motion information candidate 2755 is determined to be the predicted motion information 2556, so that it is not necessary to decode the predicted motion information position information 2754.

The reference motion information acquisition unit 2704 has the same configuration and processing contents as the reference motion information acquisition unit 901 described in the first embodiment.

The reference motion information acquisition unit 2704 generates at least one or more predicted motion information candidates 2755 (2755A, 2755B, ...) By inputting the reference motion information 2558 and the reference position information 2560. 10 and 11 show an example of the position of the predicted motion information candidate 2755 with respect to the prediction unit to be decoded. FIG. 10 shows the position of the prediction unit spatially adjacent to the prediction unit to be decoded. AX (X = 0 to nA-1) is a prediction unit adjacent to the left side of the target prediction unit, and BY (Y = 0 to nB-1) is a prediction unit adjacent to the target prediction unit. The action units, C, D, and E indicate the prediction units adjacent to the upper right, upper left, and lower left with respect to the prediction unit to be decoded, respectively. Further, FIG. 11 shows the position of the prediction unit in the reference frame that has already been decoded with respect to the prediction unit to be decoded. Col in the figure indicates a prediction unit in the reference frame and at the same position as the prediction unit to be decoded. FIG. 12 shows an example of a list showing the relationship between the block position of the plurality of predicted motion information candidates 2755 and the index Mvpidx. Mvpidx 0 to 2 indicates a predicted motion information candidate 2755 located in the spatial direction, and Mvpidx 3 indicates a motion vector candidate 2755 located in the time direction. The prediction unit position A is an inter-prediction of the AX shown in FIG. 10, that is, the prediction unit having the reference motion information 2558, and the position where the value of X is the smallest is the prediction unit position A. Further, the prediction unit position B is an inter-prediction among the BYs shown in FIG. 10, that is, the prediction unit having the reference motion information 2558, and the position where the value of Y is the smallest is referred to as the prediction unit position A. To do. When the prediction unit position C is not an inter-prediction, the reference motion information 2558 of the prediction unit position D is replaced with the reference motion information 2558 of the prediction unit position C. When the prediction unit positions C and D are not inter-prediction, the reference motion information 2558 of the prediction unit position E is replaced with the reference motion information 2558 of the prediction unit position C.

When the size of the prediction unit to be decoded is larger than the minimum prediction unit, the prediction unit position Col may hold a plurality of reference motion information 2558 in the time direction reference motion information memory 2507. In this case, the reference motion information 2558 in the prediction unit of the position Col is acquired according to the reference position information 2560. Hereinafter, the acquisition position of the reference motion information 2558 in the prediction unit of the position Col is referred to as a reference motion information acquisition position. 13A to 13F show an example of the reference motion information acquisition position when the reference position information 2560 indicates the center of the prediction unit at the position Col for each size (32x32 to 16x16) of the prediction unit to be decoded. The blocks in the figure each indicate a 4x4 prediction unit, and the circles indicate the positions of the 4x4 prediction units to be acquired as the predicted motion information candidate 2755. Another example of the reference motion information acquisition position is shown in FIGS. 14A to 14F. In FIGS. 14A to 14F, since the 4x4 prediction unit does not exist at the position of the circle, a predetermined method such as the average value or the median value of the reference motion information 2558 in the four x4 prediction units adjacent to the circle is used. Predictive motion information candidate 2755 is generated. As yet another example of the reference motion information acquisition position, the reference motion information 2558 of the 4x4 prediction unit located at the upper left end of the prediction unit at the position Col may be used as the predicted motion information candidate 2755. In addition to the above examples, any position and method may be used to generate the predicted motion information candidate 2755 as long as it is a predetermined method.

If the reference motion information 2558 does not exist, the motion information 2559 having a zero vector is output as the predicted motion information candidate 2755.

As described above, at least one or more predicted motion information candidates 2755 are output from the reference motion block. When the reference frame number of the predicted motion information candidate 2755 and the reference frame number of the decoding target prediction unit are different, the predicted motion information candidate 2755 is the reference frame number of the predicted motion information candidate 2755 and the decoding target prediction unit. It may be scaled according to the reference frame number of. The predicted motion information selection switch 2705 selects one from a plurality of predicted motion information candidates 2755 according to the predicted motion information position 2754, and outputs the predicted motion information 952.

The inverse quantization unit 2502 performs inverse quantization on the quantization conversion coefficient 2551 from the entropy decoding unit 2501 to obtain a restoration conversion coefficient 2552. Specifically, the inverse quantization unit 2502 performs the inverse quantization according to the information regarding the quantization decoded by the entropy decoding unit 2501. The inverse quantization unit 2502 outputs the restoration conversion coefficient 2552 to the inverse orthogonal transformation unit 2503.

The inverse orthogonal transform unit 2503 performs an inverse orthogonal transform corresponding to the orthogonal transform performed on the coding side with respect to the restore conversion coefficient 2552 from the inverse quantization unit 2502, and obtains a restore prediction error signal 2553. The inverse orthogonal transform unit 2503 inputs the restoration prediction error signal 2553 to the addition unit 2504.

The addition unit 2504 adds the restoration prediction error signal 2553 and the corresponding predicted image signal 2556 to generate the decoded image signal 2554. The decoded image signal 2554 is subjected to a deblocking filter, a Wiener filter, etc. (not shown), temporarily stored in the output buffer 2511 for the output image, and also stored in the reference image memory 2505 for the reference image signal 2555. To. The decoded image signal 2554 stored in the reference image memory 2505 is referred to as a reference image signal 2554 by the inter-prediction unit 2506 in frame units or field units as necessary. The decoded image signal 2554 temporarily stored in the output buffer 2511 is output according to the output timing managed by the decoding control unit 2510.

The inter-prediction unit 2506 uses the reference image signal 2555 stored in the reference image memory 2505 to perform inter-prediction. Specifically, the inter-prediction unit 2506 acquires motion information 2559 including the amount of motion deviation (motion vector) between the prediction target block and the reference image signal 2555 from the entropy decoding unit 2501 and uses this motion vector as the motion information. Based on this, interpolation processing (motion compensation) is performed to generate an inter-prediction image. Since the generation of the inter-predicted image is the same as that of the first embodiment, the description thereof will be omitted.

The decoding control unit 2510 controls each element of the moving image decoding device shown in FIG. 25. Specifically, the decoding control unit 2510 outputs the reference position information 2560, which will be described later, to the entropy decoding unit 2501, and performs various controls for the decoding process including the above-mentioned operation.

<Explanation of skip mode, merge mode, and intermode>
The image decoding apparatus 2500 according to the present embodiment uses a plurality of prediction modes having different decoding processes shown in FIG. The skip mode in the figure is a mode in which only the syntax related to the predicted motion information position 2754 described later is decoded, and the other syntax is not decoded. The merge mode is a mode in which only the syntax regarding the predicted motion information position 2754 and the conversion coefficient information 2551 are decoded, and the other syntaxes are not decoded. The intermode is a mode for decoding the syntax regarding the predicted motion information position 2754, the difference motion information 2753 described later, and the conversion coefficient information 2551. These modes are switched by the prediction information 2571 controlled by the decoding control unit 2510.

Further, since the moving image decoding apparatus of FIG. 25 uses the same or similar syntax as the syntax described in FIG. 28, detailed description thereof will be omitted.

<Details of motion information compression unit 2508>
Next, the motion information compression unit 2508 will be described with reference to the flowchart shown in FIG. The motion information compression unit 2508 compresses the motion information 2559 and stores the motion information 2559 in the time direction reference motion information memory 502 when the decoding process of the frame (or an arbitrary unit such as a slice or a coding unit) is completed. ..

First, the reference position information 2560 is acquired from the decoding control unit 2510 (step S1601), and the frame is divided into motion information compression blocks which are compression units of the motion information 2559 (step S1602). The motion information compression block is a pixel block larger than a unit (typically a 4x4 pixel block) in which motion information 2559 is held by motion compensation processing, and is typically a 16x16 pixel block. The motion information compression block may be a 32x32 pixel block, an 8x8 pixel block, a rectangular pixel block, or a pixel area having an arbitrary shape.

Next, the representative motion information position is generated according to the reference position information 2560 (step S1603). As an example of generating the representative motion information position, when the motion information compression block is a 16x16 pixel block, the reference motion information acquisition position when the size of the prediction unit shown in FIGS. 13D, 14D, and 17D is 16x16 is representative. It is the motion information position. Next, the generated reference motion information 2558 of the representative motion information position is set as the representative motion information (step S1605), and the representative motion information is stored in the time direction reference motion information memory (step S1606). The above steps S1604 to S1605 are executed for all motion information compression blocks.

Assuming that the unit in which the motion information 2559 is held is the MxM block and the size of the motion information compression block is NxN (N is a multiple of M), the capacity of the reference motion information memory is increased (MxM) by executing the motion information compression process. ) / (NxN).

<Another Embodiment of Representative Movement Information Position>
As another example of generating the representative motion information position, the center position of the plurality of reference motion information acquisition positions may be set as the representative motion information position. 18A and 18B show representative motion information positions for each motion compression block of size 16x16. FIG. 18A is a representative motion information position when the reference motion information acquisition position is the position shown in FIG. 13D, and FIG. 18B is similarly representative motion information when the reference motion information acquisition position is the position shown in FIG. 17D. Each position is shown. The circles in FIGS. 18A and 18B indicate the reference motion information acquisition positions when the prediction unit is 16x16, and the representative motions indicated by cross marks at the center positions of the four reference motion information acquisition positions. The information position is arranged.

As yet another example of generating the representative motion information position, the reference motion information acquisition position for each size of a plurality of prediction units is provided as the reference position information 2560, and the representative motion information position is generated from the plurality of reference motion information acquisition positions. It doesn't matter. FIG. 19 shows the center (reference motion information acquisition position) of each size of the prediction unit having a size of 16x16 or more when the tree block is a 64x64 pixel block.

As another example of generating the representative motion information position, the representative motion information position may be set by using the reference motion information acquisition position arranged for each motion information compression block. FIG. 20A shows an example in which the center of gravity of a plurality of reference motion information acquisition positions for each motion information compression block is set as the representative motion information position. If the position of the center of gravity does not match the position of the 4x4 block, the nearest 4x4 block may be used as the representative motion information position, or the reference motion vector 166 of the center of gravity position may be set by using an interpolation method such as the co-primary interpolation method. You may generate it.

Further, FIG. 20B shows an example in which one of a plurality of reference motion information acquisition positions is selected for each motion information compression block and set as a representative motion information position.

Further, FIGS. 21A and 21B further show an example in which the reference motion information acquisition position is the same in each motion information compression block in the tree block. Since the representative motion information position is the same in all the motion information compression blocks, it is not necessary to switch the representative motion information position according to the position in the tree block. Further, the representative motion information position may be at any position such as the upper left end or the upper right end in the motion information compression block other than FIGS. 21A and 21B.

As an example of generating the representative motion information position, the representative motion information position may be indicated by using BlkIdx which indicates the 4x4 block position in the motion information compression block in the Z scan order. When the size of the motion information compression block is 16x16, the representative motion information position shown in FIG. 21A corresponds to the position of BlkIdx = 12. The representative motion information position shown in FIG. 21B corresponds to the position of BlkIdx = 15.

As another example in the motion information compression process, the reference frame number may be included in the motion information compression process in order to reduce the memory capacity related to the reference frame number. In this case, the reference frame number held at the representative motion information position is stored in the memory capacity related to the reference frame number. Therefore, the spatial reference motion information memory 501 and the spatial reference motion information memory 502 shown in FIG. 5 store the reference frame number in addition to the motion vector information.

As yet another example of the motion information compression process, when the reference frame number is not included in the motion information compression process, the motion vector information in the motion information at the representative motion information position is scaled using the reference frame number. It may be stored in the motion information memory 110. A typical example of the scaling process is a linear scaling process based on the reference frame number zero. In this method, when the reference frame number is a non-zero value, the motion vector information is linearly scaled so as to refer to the reference frame corresponding to the reference frame number zero. The reference frame number may be a non-zero value as the standard for the scaling process described above. If division occurs when the above-mentioned linear scaling processing is performed, the above-mentioned division may be realized by creating a table of the division processing in advance and pulling the table each time.

When the size of the motion information compression block is other than 16x16 blocks, the representative motion information position is generated by using the same processing as described above. In one example, when the size of the motion information compression block is 64x64, the reference motion information acquisition position when the size of the prediction unit is 64x64 is set as the representative motion information position. In yet another example, the representative motion information position when the size of the motion information compression block shown in FIGS. 21A and 21B is 16x16 blocks is scaled in the horizontal and vertical directions according to the size of the motion information compression block. It may be the information position.

If the reference motion information does not exist because the representative motion information position is outside the picture or slice, the new representative motion is the position where the reference motion information can be obtained in the motion information compression block such as the upper left corner of the motion information compression block. It may be replaced as an information position. Further, even when the representative motion information position is the area to which the intra prediction is applied and the reference motion information does not exist, the same process may be executed to replace it as a new representative motion information position.

Hereinafter, modification examples of each embodiment will be listed and introduced.
In the first and second embodiments, an example is described in which a frame is divided into rectangular blocks having a size of 16 × 16 pixels, and encoding / decoding is performed in order from the block on the upper left of the screen to the lower right (). See FIG. 2A). However, the coding order and decoding order are not limited to this example. For example, coding and decoding may be performed in order from the lower right to the upper left, or coding and decoding may be performed so as to draw a spiral from the center of the screen toward the edge of the screen. Further, coding and decoding may be performed in order from the upper right to the lower left, or coding and decoding may be performed so as to draw a spiral from the edge of the screen toward the center of the screen.

In the first and second embodiments, the prediction target block sizes such as 4 × 4 pixel block, 8 × 8 pixel block, and 16 × 16 pixel block have been illustrated and described, but the prediction target block is a uniform block. It does not have to be a shape. For example, the size of the prediction target block (prediction unit) may be a 16 × 8 pixel block, an 8 × 16 pixel block, an 8 × 4 pixel block, a 4 × 8 pixel block, or the like. Further, it is not necessary to unify all block sizes in one coding tree unit, and a plurality of different block sizes may be mixed. When a plurality of different block sizes are mixed in one coding tree unit, the amount of code for encoding or decoding the division information increases as the number of divisions increases. Therefore, it is desirable to select the block size in consideration of the balance between the code amount of the divided information and the quality of the locally decoded image or the decoded image.

In the first and second embodiments, for the sake of simplicity, the luminance signal and the color difference signal are not distinguished, and a comprehensive description of the color signal component is described. However, when the prediction process differs between the luminance signal and the color difference signal, the same or different prediction methods may be used. If different prediction methods are used for the luminance signal and the color difference signal, the prediction method selected for the luminance signal can be encoded or decoded in the same manner as the luminance signal.

In the first and second embodiments, for the sake of simplicity, the luminance signal and the color difference signal are not distinguished, and a comprehensive description of the color signal component is described. However, when the orthogonal transform processing differs between the luminance signal and the color difference signal, the same or different orthogonal transform methods may be used. If different orthogonal conversion methods are used between the luminance signal and the color difference signal, the orthogonal transform method selected for the luminance signal can be encoded or decoded in the same manner as the luminance signal.

In the first and second embodiments, it is possible to insert a syntax element not specified in the embodiment between the rows of the table shown in the syntax configuration, and other descriptions regarding conditional branching are included. It doesn't matter if it is. Alternatively, the syntax table can be divided and integrated into a plurality of tables. Further, it is not always necessary to use the same term, and it may be arbitrarily changed depending on the form to be used.

As described above, each embodiment can realize highly efficient orthogonal transformation and inverse orthogonal transformation while alleviating difficulties in hardware implementation and software implementation. Therefore, according to each embodiment, the coding efficiency is improved, and the subjective image quality is also improved.

In addition, the instructions given in the processing procedure shown in the above-described embodiment can be executed based on a program that is software. By storing this program in advance and reading this program, a general-purpose computer system can obtain the same effect as the effect of the moving image coding device and the moving image decoding device of the above-described embodiment. is there. The instructions described in the above-described embodiments can be executed by a computer as a program such as a magnetic disk (flexible disk, hard disk, etc.) or an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD). It is recorded on a recording medium (± R, DVD ± RW, etc.), a semiconductor memory, or a similar recording medium. The storage format may be any form as long as it is a recording medium that can be read by a computer or an embedded system. If the computer reads the program from this recording medium and causes the CPU to execute the instructions described in the program based on this program, it is similar to the moving image coding device and the moving image decoding device of the above-described embodiment. The operation can be realized. Of course, if the computer acquires or loads the program, it may acquire or load it through the network.
In addition, the OS (operating system) running on the computer based on the instructions of the program installed on the computer or embedded system from the recording medium, database management software, MW (middleware) such as the network, etc. realize this embodiment. You may perform a part of each process for doing so.
Further, the recording medium according to the embodiment of the present invention is not limited to a medium independent of a computer or an embedded system, but also includes a recording medium in which a program transmitted via a LAN, the Internet, or the like is downloaded and stored or temporarily stored. Further, the program that realizes the processing of each of the above embodiments may be stored on a computer (server) connected to a network such as the Internet and downloaded to the computer (client) via the network.
Further, the recording medium is not limited to one, and even when the processing in the present embodiment is executed from a plurality of media, it is included in the recording medium in the embodiment of the present invention, and the structure of the medium may be any configuration. Good.

The computer or embedded system according to the embodiment of the present invention is for executing each process according to the present embodiment based on the program stored in the recording medium, and is a device including one such as a personal computer and a microcomputer. , A system in which a plurality of devices are connected to a network, or the like may be used.
Further, the computer in the embodiment of the present invention includes not only a personal computer but also an arithmetic processing unit, a microcomputer, etc. included in an information processing device, and a device capable of realizing the function in the embodiment of the present invention by a program. It is a general term for devices.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

100 ... image encoding device, 101 ... subtraction unit, 102 ... orthogonal conversion unit, 103 ... quantization unit, 104, 2502 ... inverse quantization unit, 105, 2503 ... inverse orthogonal conversion unit, 106, 2504, 2706 ... addition unit , 107, 2505 ... Reference image memory, 108, 2506 ... Inter prediction unit, 109 ... Motion information compression unit, 110 ... Motion information memory, 112 ... Entropy coding unit, 113 ... Output buffer, 114 ... Coding control unit, 401 ... Parameter coding unit, 402 ... Conversion coefficient coding unit, 403 ... Motion information coding unit, 404 ... Multiplexing unit, 901 ... Reference motion vector acquisition unit, 902 ... Predicted motion vector selection switch, 903 ... Subtraction unit, 904 ... differential motion information coding unit, 905 ... predicted motion information position coding unit, 906 ... multiplexing unit, 2500 ... moving image decoding device, 2501 ... entropy decoding unit, 2507 ... reference motion information memory, 2508 ... reference motion Information compression unit, 2510 ... Decoding control unit, 2601, 2701 ... Separation unit, 2602 ... Parameter decoding unit, 2603 ... Conversion coefficient decoding unit, 2604 ... Motion information decoding unit, 2702 ... Differential motion information decoding unit, 2503 ... Predicted motion information position decoding unit, 2704 ... Reference motion information acquisition unit, 2705 ... Predicted motion information selection switch.

Claims (2)

Acquire the data including at least the first picture including the target block,
Determine the merge flag to determine at least whether the motion vector is predicted from the merge block in inter-prediction mode,
When it is specified by the merge flag that at least the motion vector is predicted from the merge block in the inter-prediction mode, a candidate for the first motion vector is determined from at least one adjacent block of the target block.
When the merge flag specifies that at least the motion vector is predicted from the merge block in the inter-prediction mode, the second motion from the reference block included in the second picture different from the first picture. Determine vector candidates and
A merge index that specifies the merge block is determined from the at least one adjacent block and the reference block.
The first motion vector of the target block is determined from any one of the candidate of the first motion vector and the candidate of the second motion vector according to the merge index.
A reference image is derived according to the first motion vector,
A prediction image is generated by the inter-prediction using the reference image.
The prediction error is derived from the difference between the predicted image and the coded image.
It is provided that the conversion coefficient is derived by performing at least the conversion process for the prediction error.
The at least one adjacent block is (1) a block on the lower left side of the target block, (2) a block on the left side of the target block, (3) a block on the upper right side of the target block, and (4) said. It comprises at least one of a block above the target block and (5) a block on the upper left side of the target block.
The candidate for the second motion vector of the reference block is derived according to the representative motion information position .
The representative motion information position for deriving the candidate for the second motion vector of the reference block is determined based on whether the position is outside the picture.
A moving image coding method in which the merge flag, the merge index, and the conversion coefficient are encoded. Acquire the coded data including at least the first picture including the target block, and
Decrypt the merge flag to determine if at least the motion vector is predicted from the merge block in inter-prediction mode.
When it is specified by the merge flag that at least the motion vector is predicted from the merge block in the inter-prediction mode, a candidate for the first motion vector is derived from at least one adjacent block of the target block.
When the merge flag specifies that at least the motion vector is predicted from the merge block in the inter-prediction mode, the second motion from the reference block included in the second picture different from the first picture. Guide vector candidates
Decrypt the merge index that specifies the merge block from the at least one adjacent block and the reference block.
The first motion vector of the target block is derived from any one of the candidate of the first motion vector and the candidate of the second motion vector according to the merge index.
A reference image is derived according to the first motion vector,
A prediction image is generated by the inter-prediction using the reference image.
Decode the conversion factor and
Prediction error is derived by performing at least inverse transformation processing on the conversion coefficient.
It is provided that the decoded image is derived by adding at least the prediction error and the prediction image.
The at least one adjacent block is (1) a block on the lower left side of the target block, (2) a block on the left side of the target block, (3) a block on the upper right side of the target block, and (4) said. It comprises at least one of a block above the target block and (5) a block on the upper left side of the target block.
The candidate for the second motion vector of the reference block is derived according to the representative motion information position .
A moving image decoding method in which a representative motion information position for deriving a candidate for the second motion vector of the reference block is determined based on whether the position is outside the picture.

Images