JP5925855B2 - Image decoding apparatus, method and program, first program and second program, server system, and download control method - Google Patents

Description

  Embodiments described herein relate generally to orthogonal transformation and inverse orthogonal transformation in encoding and decoding of moving images.

  In recent years, an image encoding method with greatly improved encoding efficiency has been jointly developed by ITU-T and ISO / IEC. H. H.264 and ISO / IEC 14496-10 (hereinafter referred to as “H.264”). H. In H.264, regardless of the prediction method applied to the target pixel block, discrete cosine transform (DCT) and inverse discrete cosine transform (IDCT) are performed as orthogonal transform and inverse orthogonal transform for the prediction error of the target pixel block, respectively.

  H. As an extension of H.264, encoding efficiency is improved by performing orthogonal transform and inverse orthogonal transform using individual transform bases for each of nine types of prediction modes defined in intra prediction (intra prediction). Is assumed.

M. Karczewicz, “Improved intra coding”, ITU-T SG16 / Q.6, VCEG Document, VCEG-AF15, April 2007.

  However, performing orthogonal transform and inverse orthogonal transform using individual transform bases for each of a plurality of types of prediction modes involves difficulty in implementation. For example, H. In addition to the dedicated hardware for DCT and IDCT required in H.264, it is necessary to provide dedicated hardware for individual orthogonal transform and inverse orthogonal transform for each of the above-described multiple types of prediction directions. The addition of these dedicated hardware increases the circuit scale.

  Regarding software implementation, in addition to the DCT matrix, individual transformation matrices for each of a plurality of types of prediction directions can be appropriately loaded from a memory, or appropriately held in a cache memory. In this case, although desired orthogonal transformation and inverse orthogonal transformation can be realized by a general-purpose multiplier, there is a problem of cost increase due to increase in memory bandwidth or cost increase due to increase in cache memory size.

  Therefore, an object of the embodiment is to provide orthogonal transform or inverse orthogonal transform that can improve coding efficiency.

  According to the embodiment, the image decoding apparatus includes a storage unit, a decoding unit, a set unit, a conversion unit, and an addition unit. The accumulation unit accumulates encoded data output from another device via a communication line. The decoding unit decodes the transform coefficient to be decoded from the accumulated encoded data. The set unit sets a combination of a vertical transformation matrix and a horizontal transformation matrix corresponding to a prediction mode to be decoded using only two types of matrices based on a predetermined relationship according to a predicted image generation method. . The conversion unit obtains a prediction error by performing vertical conversion and horizontal conversion on the conversion coefficient using the set vertical conversion matrix and horizontal conversion matrix. The adder generates a decoded image based on the prediction error. The combination is one of a combination of first conversion matrices and a combination of second conversion matrices different from the first conversion matrix. A combination of the second transformation matrices is set to a plurality of intra prediction modes respectively corresponding to the diagonal down right, vertical right, and horizontal down directions.

1 is a block diagram illustrating an image encoding device according to a first embodiment. The block diagram which illustrates the orthogonal transformation part concerning a 1st embodiment. FIG. 3 is a block diagram illustrating an inverse orthogonal transform unit according to the first embodiment. The table figure which illustrates a response | compatibility with prediction mode, a vertical conversion index, and a horizontal conversion index based on 1st Embodiment. The table figure which illustrates a response | compatibility with the vertical conversion index and 1D conversion matrix based on 1st Embodiment. The table figure which illustrates a response | compatibility with the horizontal conversion index and 1D conversion matrix based on 1st Embodiment. The table figure which illustrates a response | compatibility with a conversion index, a vertical conversion index, and a horizontal conversion index based on 1st Embodiment. The table figure which integrated FIG. 4A and 4D. The block diagram which illustrates the coefficient order control part concerning a 1st embodiment. The block diagram which illustrates the coefficient order control part concerning a 1st embodiment. Explanatory drawing of the prediction encoding order of a pixel block. Explanatory drawing of an example of pixel block size. Explanatory drawing of another example of pixel block size. Explanatory drawing of another example of pixel block size. Explanatory drawing of intra prediction mode. Explanatory drawing of the arrangement | positioning relationship between a prediction object pixel and a reference pixel. Explanatory drawing of the intra prediction mode 1. FIG. Explanatory drawing of the intra prediction mode 4. FIG. Explanatory drawing of a zigzag scan. Explanatory drawing of a zigzag scan. The table figure which shows 2D-1D conversion using a zigzag scan. The table figure which illustrates individual 2D-1D conversion for every prediction mode. 3 is a flowchart illustrating processing performed by the image encoding device in FIG. 1 on an encoding target block. 3 is a flowchart illustrating processing performed by the image encoding device in FIG. 1 on an encoding target block. Explanatory drawing of a syntax structure. Explanatory drawing of a slice header syntax. Explanatory drawing of coding tree unit syntax. Explanatory drawing of a transform unit syntax. The block diagram which illustrates the orthogonal transformation part which performs orthogonal transformation using each conversion base about each of nine types of prediction directions. The block diagram which illustrates the orthogonal transformation part concerning a 2nd embodiment. The block diagram which illustrates the inverse orthogonal transformation part which concerns on 2nd Embodiment. The table figure which illustrates a response | compatibility with prediction mode, a vertical conversion index, and a horizontal conversion index based on 2nd Embodiment. The table figure which illustrates a response | compatibility with the vertical conversion index and 1D conversion matrix based on 2nd Embodiment. The table figure which illustrates a response | compatibility with a horizontal conversion index and 1D conversion matrix based on 2nd Embodiment. The table figure which illustrates a response | compatibility with a conversion index, a vertical conversion index, and a horizontal conversion index based on 2nd Embodiment. The table figure which integrated FIG. 18A and FIG. 18D. The block diagram which illustrates the orthogonal transformation part concerning a 3rd embodiment. The block diagram which illustrates the inverse orthogonal transformation part concerning a 3rd embodiment. The table figure which illustrates a response | compatibility with prediction mode, a vertical conversion index, and a horizontal conversion index based on 3rd Embodiment. The table figure which illustrates a response | compatibility with the vertical conversion index and 1D conversion matrix based on 3rd Embodiment. The table figure which illustrates a response | compatibility with the horizontal conversion index and 1D conversion matrix based on 3rd Embodiment. The table figure which illustrates a correspondence with a conversion index, a vertical conversion index, and a horizontal conversion index concerning a 3rd embodiment. The table figure which integrated FIG. 21A and FIG. 21D. The block diagram which illustrates the picture decoding device concerning a 4th embodiment. The block diagram which illustrates the coefficient order control part concerning a 4th embodiment. The block diagram which illustrates the coefficient order control part concerning a 4th embodiment.

Hereinafter, each embodiment will be described with reference to the drawings. In the following description, the term “image” can be appropriately replaced with terms such as “image signal” and “image data”.
(First embodiment)
The first embodiment relates to an image encoding device. An image decoding apparatus corresponding to the image encoding apparatus according to the present embodiment will be described in a fourth embodiment. This image encoding device can be realized by hardware such as an LSI (Large-Scale Integration) chip, a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array). The image encoding apparatus can also be realized by causing a computer to execute an image encoding program.

  As shown in FIG. 1, the image coding apparatus according to the present embodiment includes a subtraction unit 101, an orthogonal transformation unit 102, a quantization unit 103, an inverse quantization unit 104, an inverse orthogonal transformation unit 105, an addition unit 106, and a reference image. A memory 107, an intra prediction unit 108, an inter prediction unit 109, a prediction selection unit 110, a prediction selection switch 111, a 1D (one-dimensional) transform matrix set unit 112, a coefficient order control unit 113, an entropy encoding unit 114, an output buffer 115, An encoding control unit 116 is included.

  The image coding apparatus in FIG. 1 divides each frame or each field constituting the input image 118 into a plurality of pixel blocks, performs predictive coding on the divided pixel blocks, and outputs coded data 130. To do. In the following description, for the sake of simplicity, it is assumed that pixel blocks are predictively encoded from the upper left to the lower right as shown in FIG. 6A. In FIG. 6A, the encoded pixel block p is located on the left side and the upper side of the encoding target pixel block c in the encoding processing target frame f.

  Here, the pixel block indicates, for example, a coding tree unit, a macro block, a sub block, and one pixel. In the following description, the pixel block is basically used in the meaning of the coding tree unit, but the pixel block can be interpreted in another meaning by appropriately replacing the description. The coding tree unit is typically a 16 × 16 pixel block shown in FIG. 6B, for example, but may be a 32 × 32 pixel block shown in FIG. 6C, or a 64 × 64 pixel block shown in FIG. 6D, It may be an 8 × 8 pixel block (not shown) or a 4 × 4 pixel block. The coding tree unit need not necessarily be square. Hereinafter, the encoding target block or coding tree unit of the input image 118 may be referred to as a “prediction target block”. In addition, the encoding unit is not limited to a pixel block such as a coding tree unit, and a frame, a field, or a combination thereof can be used.

  The image encoding apparatus in FIG. 1 performs intra prediction (also referred to as intra-frame prediction, intra-frame prediction, etc.) or inter prediction (inter-screen prediction) on a pixel block based on the encoding parameter input from the encoding control unit 116. Prediction image 127 is generated by performing prediction (also referred to as prediction, inter-frame prediction). This image encoding apparatus orthogonally transforms and quantizes the prediction error 119 between the pixel block (input image 118) and the predicted image 127, performs entropy encoding, and generates and outputs encoded data 130.

  The image encoding device in FIG. 1 performs encoding by selectively applying a plurality of prediction modes having different block sizes and generation methods of the predicted image 127. The generation method of the predicted image 127 can be broadly divided into two types: intra prediction in which prediction is performed within the encoding target frame and inter prediction in which prediction is performed using one or a plurality of reference frames that are temporally different. . In the present embodiment, an orthogonal transform and an inverse orthogonal transform when generating a predicted image using intra prediction will be described in detail.

Hereinafter, each element included in the image encoding device in FIG. 1 will be described.
The subtracter 101 subtracts the corresponding predicted image 127 from the encoding target block of the input image 118 to obtain a prediction error 119. The subtractor 101 inputs the prediction error 119 to the orthogonal transform unit 102.

  The orthogonal transform unit 102 performs orthogonal transform on the prediction error 119 from the subtractor 101 to obtain a transform coefficient 120. Details of the orthogonal transform unit 102 will be described later. The orthogonal transform unit 102 inputs the transform coefficient 120 to the quantization unit 103.

  The quantization unit 103 performs quantization on the transform coefficient from the orthogonal transform unit 102 to obtain a quantized transform coefficient 121. Specifically, the quantization unit 103 performs quantization according to quantization information such as a quantization parameter and a quantization matrix specified by the encoding control unit 116. The quantization parameter indicates the fineness of quantization. The quantization matrix is used for weighting the fineness of quantization for each component of the transform coefficient. The quantization unit 103 inputs the quantized transform coefficient 121 to the coefficient order control unit 113 and the inverse quantization unit 104.

  The coefficient order control unit 113 converts the quantized transform coefficient 121 that is a two-dimensional (2D) representation into a quantized transform coefficient sequence 117 that is a one-dimensional (1D) representation, and inputs the quantized transform coefficient sequence 117 to the entropy encoding unit 114. Details of the coefficient control unit 113 will be described later.

  The entropy encoding unit 114 applies various encoding parameters such as the quantized transform coefficient sequence 117 from the coefficient control unit 113, the prediction information 126 from the prediction selection unit 110, and the quantization information specified by the encoding control unit 116. Entropy coding (for example, Huffman coding, arithmetic coding, etc.) is performed on the data to generate coded data. The encoding parameter is a parameter necessary for decoding, such as prediction information 126, information on transform coefficients, information on quantization, and the like. The encoding parameter is held in an internal memory (not shown) of the encoding control unit 116, and it is possible to use the encoding parameter of an already encoded pixel block adjacent when encoding the prediction target block. . For example, H.M. In the H.264 intra prediction, the prediction value of the prediction mode of the prediction target block can be derived from the prediction mode information of the encoded adjacent block.

  The encoded data generated by the entropy encoding unit 114 is temporarily accumulated in the output buffer 115 through multiplexing, for example, and output as encoded data 130 according to an appropriate output timing managed by the encoding control unit 116. . The encoded data 130 is output to a storage system (storage medium) or a transmission system (communication line) (not shown), for example.

  The inverse quantization unit 104 performs inverse quantization on the quantized transform coefficient 121 from the quantization unit 103 to obtain a restored transform coefficient 122. Specifically, the inverse quantization unit 104 performs inverse quantization according to the quantization information used in the quantization unit 103. The quantization information used in the quantization unit 103 is loaded from the internal memory of the encoding control unit 116. The inverse quantization unit 104 inputs the restored transform coefficient 122 to the inverse orthogonal transform unit 105.

  The inverse orthogonal transform unit 105 performs inverse orthogonal transform corresponding to the orthogonal transform performed in the orthogonal transform unit 102 on the reconstructed transform coefficient 122 from the inverse quantization unit 104 to obtain a reconstructed prediction error 123. Details of the inverse orthogonal transform unit 105 will be described later. The inverse orthogonal transform unit 105 inputs the restoration prediction error 123 to the addition unit 106.

  The adding unit 106 adds the restored prediction error 123 and the corresponding predicted image 127 to generate a local decoded image 124. The locally decoded image 124 is stored in the reference image memory 107. The locally decoded image 124 stored in the reference image memory 107 is referred to as the reference image 125 by the intra prediction unit 108 and the inter prediction unit 109 as necessary.

  The intra prediction unit 108 performs intra prediction using the reference image 125 stored in the reference image memory 107. For example, H.M. In H.264, an intra prediction image is generated by performing pixel interpolation (copying or interpolation) along a prediction direction such as a vertical direction or a horizontal direction using an encoded reference pixel value adjacent to a prediction target block. . In FIG. The prediction direction of intra prediction in H.264 is shown. In FIG. The arrangement | positioning relationship between the reference pixel and encoding object pixel in H.264 is shown. FIG. 7C shows a predicted image generation method in mode 1 (horizontal prediction), and FIG. 7D shows a predicted image generation method in mode 4 (diagonal lower right prediction; Intra_NxN_Diagonal_Down_Right in FIG. 4A).

  The intra prediction unit 108 may copy the interpolated pixel value in a predetermined prediction direction after interpolating the pixel value using a predetermined interpolation method. H. Although the prediction direction of the H.264 intra prediction is illustrated, the prediction direction can be extended to use any number of prediction modes such as 17 types and 33 types by further specifying the prediction direction. Specifically, H.C. H.264 defines a prediction angle for every 22.5 degrees. For example, if a prediction angle for every 11.25 degrees is defined, 17 types of prediction modes including DC prediction can be used. Further, if a prediction angle for every 5.625 degrees is defined, 33 types of prediction modes including DC prediction can be used. Further, instead of arranging the prediction angles at equal intervals, the angle in the prediction direction may be represented by a straight line connecting the second reference points moved horizontally and vertically from the first reference point. As described above, the prediction mode can be easily expanded, and this embodiment can be applied regardless of the number of prediction modes.

  The inter prediction unit 109 performs inter prediction using the reference image 125 stored in the reference image memory 107. Specifically, the inter prediction unit 109 performs block matching processing between the prediction target block and the reference image 125 to derive a motion shift amount (motion vector). The inter prediction unit 109 performs an interpolation process (motion compensation) based on the motion vector to generate an inter prediction image. H. With H.264, interpolation processing up to 1/4 pixel accuracy is possible. The derived motion vector is entropy encoded as part of the prediction information 126.

  The selection switch 111 selects the output terminal of the intra prediction unit 108 or the output terminal of the inter prediction unit 109 according to the prediction information 126 from the prediction selection unit 110, and uses the subtraction unit 101 and the intra prediction image or the inter prediction image as the prediction image 127. The data is input to the adding unit 106. When the prediction information 126 suggests intra prediction, the selection switch 110 captures the intra prediction image from the intra prediction unit 108 as the prediction image 127. On the other hand, when the prediction information 126 suggests inter prediction, the selection switch 110 captures the inter prediction image from the inter prediction unit 109 as the prediction image 127.

  The prediction selection unit 110 has a function of setting the prediction information 126 according to the prediction mode controlled by the encoding control unit 116. As described above, intra prediction or inter prediction can be selected for generating the predicted image 127, but a plurality of modes can be further selected for each of intra prediction and inter prediction. The encoding control unit 116 determines one of a plurality of intra prediction modes and inter prediction modes as the optimal prediction mode, and the prediction selection unit 110 sets the prediction information 126 according to the determined optimal prediction mode. .

  For example, regarding intra prediction, prediction mode information is designated by the encoding control unit 116 as the intra prediction unit 108, and the intra prediction unit 108 generates a prediction image 127 according to the prediction mode information. The encoding control unit 116 may specify a plurality of prediction mode information in order from the smallest prediction mode number, or may specify a plurality of prediction mode information in order from the largest. The encoding control unit 116 may limit the prediction mode according to the characteristics of the input image. The encoding control unit 116 need not always specify all prediction modes, but may specify at least one prediction mode information for the encoding target block.

  For example, the encoding control unit 116 determines an optimal prediction mode using a cost function expressed by the following formula (1).

  In Equation (1), OH indicates a code amount related to the prediction information 126 (for example, motion vector information, prediction block size information), and SAD is a sum of absolute differences between the prediction target block and the prediction image 127 (that is, prediction). Accumulated sum of absolute values of error 119). Further, λ represents a Lagrange undetermined multiplier determined based on the value of quantization information (quantization parameter), and K represents an encoding cost. When Expression (1) is used, the prediction mode that minimizes the coding cost K is determined as the optimum prediction mode from the viewpoint of the generated code amount and the prediction error. As a modification of Equation (1), the encoding cost may be estimated from OH alone or SAD alone, or the encoding cost may be estimated using a value obtained by subjecting SAD to Hadamard transform or an approximation thereof.

  It is also possible to determine an optimal prediction mode by using a temporary encoding unit (not shown). For example, the encoding control unit 116 determines the optimal prediction mode using the cost function shown in the following mathematical formula (2).

  In Equation (2), D represents a square error sum (ie, encoding distortion) between the prediction target block and the local decoded image, and R represents a prediction error between the prediction target block and the prediction image 127 in the prediction mode. Indicates the amount of code estimated by provisional encoding, and J indicates the encoding cost. In order to derive the encoding cost J of Equation (2), provisional encoding processing and local decoding processing are required for each prediction mode, so that the circuit scale or the amount of calculation increases. On the other hand, since the encoding cost J is derived based on more accurate encoding distortion and code amount, it is easy to determine the optimal prediction mode with high accuracy and maintain high encoding efficiency. As a modification of Equation (2), the encoding cost may be estimated from only R or D, or the encoding cost may be estimated using an approximate value of R or D. In addition, the encoding control unit 116 makes a determination using Formula (1) or Formula (2) based on information obtained in advance regarding the prediction target block (prediction mode of surrounding pixel blocks, results of image analysis, and the like). The number of prediction mode candidates for performing may be narrowed down in advance.

The encoding control unit 116 controls each element of the image encoding device in FIG. Specifically, the encoding control unit 116 performs various controls for the encoding process including the above-described operation.
The 1D transform matrix set unit 112 generates 1D transform matrix set information 129 based on the prediction mode information included in the prediction information 126 from the prediction selection unit 110 and inputs the 1D transform matrix set information 129 to the orthogonal transform unit 102 and the inverse orthogonal transform unit 105. Details of the 1D conversion matrix set information 129 will be described later.

Hereinafter, details of the orthogonal transform unit 102 according to the present embodiment will be described with reference to FIG.
The orthogonal transform unit 102 includes a selection switch 201, a vertical conversion unit 202, a transposition unit 203, a selection switch 204, and a horizontal conversion unit 205. The vertical transform unit 202 includes a 1D orthogonal transform unit A206 and a 1D orthogonal transform unit B207. The horizontal transform unit 205 includes a 1D orthogonal transform unit A208 and a 1D orthogonal transform unit B209. Note that the order of the vertical conversion unit 202 and the horizontal conversion unit 205 is an example, and these may be reversed.

  The 1D orthogonal transform unit A206 and the 1D orthogonal transform unit A208 have a common function in that the input matrix is multiplied by the 1D transform matrix A, and the 1D orthogonal transform unit B207 and the 1D orthogonal transform unit B209 are input. It has a common function in that the 1D conversion matrix B is multiplied with the matrix. Therefore, the 1D orthogonal transform unit A206 and the 1D orthogonal transform unit A208 can also be realized by using physically identical hardware in a time division manner. The same applies to the 1D orthogonal transform unit B207 and the 1D orthogonal transform unit B209.

  The selection switch 201 guides the prediction error 119 to one of the 1D orthogonal transform unit A206 and the 1D orthogonal transform unit B207 according to the vertical transform index included in the 1D transform matrix set information 129. The 1D orthogonal transform unit A206 multiplies the input prediction error (matrix) 119 by the 1D transform matrix A and outputs the result. The 1D orthogonal transform unit B207 multiplies the input prediction error 119 by the 1D transform matrix B and outputs the result. Specifically, the 1D orthogonal transform unit A206 and the 1D orthogonal transform unit B207 (that is, the vertical transform unit 202) perform a one-dimensional orthogonal transform represented by the following equation (3), and the prediction error 119 in the vertical direction Remove correlation.

  In Equation (3), X represents a matrix (N × N) of the prediction error 119, V represents a 1D conversion matrix A and a 1D conversion matrix B (both N × N), and Y represents 1D The output matrix (NxN) of orthogonal transformation part A206 and 1D orthogonal transformation part B207 is shown. Specifically, the transformation matrix V is an N × N transformation matrix in which transformation bases designed to remove the correlation in the vertical direction of the matrix X are arranged in the vertical direction as row vectors. However, as described later, the 1D conversion matrix A and the 1D conversion matrix B are designed by different methods and have different properties. As the 1D conversion matrix A and the 1D conversion matrix B, it is also possible to use an integer obtained by multiplying each designed conversion base by a scalar.

  Here, when the prediction error 119 is a rectangular block expressed by M × N, the block size for performing orthogonal transform may also be M × N.

  The transposition unit 203 transposes the output matrix (Y) of the vertical conversion unit 202 and supplies it to the selection switch 204. However, the transposition unit 203 is an example, and corresponding hardware may not necessarily be prepared. For example, the result of executing the 1D orthogonal transform by the vertical transform unit 202 (each element of the output matrix of the vertical transform unit 202) is stored and read in an appropriate order when the 1D orthogonal transform is performed by the horizontal transform unit 205. If output, the transposition of the output matrix (Y) can be performed without preparing hardware corresponding to the transposition unit 203.

  The selection switch 204 guides the input matrix from the transposition unit 203 to one of the 1D orthogonal transform unit A 208 and the 1D orthogonal transform unit B 209 according to the horizontal transform index included in the 1D transform matrix set information 129. The 1D orthogonal transform unit A208 multiplies the input matrix by the 1D transform matrix A and outputs the result. The 1D orthogonal transform unit B209 multiplies the input matrix by the 1D transform matrix B and outputs the result. Specifically, the 1D orthogonal transform unit A 208 and the 1D orthogonal transform unit B 209 (that is, the horizontal transform unit 205) perform a one-dimensional orthogonal transform represented by the following formula (4) to correlate the prediction error in the horizontal direction. Remove.

  In Equation (4), H comprehensively represents the 1D transformation matrix A and the 1D transformation matrix B (both N × N), and Z represents the output matrix (N of the 1D orthogonal transformation unit A208 and the 1D orthogonal transformation unit B209). XN), which refers to the conversion factor 120. Specifically, the transformation matrix H is an N × N transformation matrix in which transformation bases designed to remove the correlation in the horizontal direction of the matrix Y are vertically arranged as row vectors. Although overlapping with the previous description, the 1D transformation matrix A and the 1D transformation matrix B are designed in different ways and have different properties. Further, the 1D conversion matrix A and the 1D conversion matrix B may be obtained by converting each designed conversion base into an integer by scalar multiplication.

  As described above, the orthogonal transform unit 102 performs orthogonal transform on the prediction error (matrix) 119 according to the 1D transform matrix set information 129 input from the 1D transform matrix set unit 112, and converts the transform coefficient (matrix) 120 into the transform coefficient (matrix) 120. Generate. H. In consideration of H.264, the orthogonal transform unit 102 may include a DCT unit (not shown), or one of the 1D transform matrix A and the 1D transform matrix B may be replaced with a matrix for DCT. For example, the 1D transformation matrix B may be a transformation matrix for DCT. Further, the orthogonal transform unit 102 may implement various orthogonal transforms such as Hadamard transform, Karhunen-Loeve transform, and discrete sine transform, which will be described later, in addition to DCT.

  Here, the difference in properties between the 1D conversion matrix A and the 1D conversion matrix B will be described. H. In the intra prediction mode supported by H.264, the prediction pixel is generated by copying the reference pixel group on one or both adjacent lines on the left side and the upper side of the prediction target block along the prediction direction or after interpolation. There is. That is, in this intra prediction mode, at least one reference pixel in the reference pixel group is selected according to the prediction direction, and a predicted image is generated by copying the reference pixel or interpolating from the reference pixel. Since the intra prediction mode uses the spatial correlation of images, the prediction accuracy tends to decrease as the distance from the reference pixel increases. That is, the absolute value of the prediction error is likely to increase according to the distance from the reference pixel. This tendency is the same regardless of the prediction direction. More specifically, an intra prediction mode in which only the reference pixel group on the left adjacent line of the prediction target block is referred (copying of the pixel value of the reference pixel or interpolation from the reference pixel) (for example, mode 1 in FIG. 7A and For mode 8), the prediction error shows a trend in the horizontal direction. In the intra prediction mode (for example, mode 0, mode 3, and mode 7 in FIG. 7A) in which only the reference pixel group on the upper adjacent line is referred to the prediction target block, the prediction error shows a tendency in the vertical direction. Furthermore, with respect to a prediction mode in which reference pixel groups on the left adjacent line and the upper adjacent line of the prediction target block are referred to (for example, mode 4, mode 5 and mode 6 in FIG. 7A), the prediction error is in the horizontal direction and the vertical direction. This trend is shown. In summary, it can be said that the tendency is related to the direction orthogonal to the line of the reference pixel group used for generating the predicted image.

  Compared to the 1D transformation matrix B, the 1D transformation matrix A has higher coefficient density when performing 1D orthogonal transformation in the orthogonal direction (vertical direction or horizontal direction) (that is, non-zero in the quantized transformation coefficient 121). It is generated by designing a common transformation base in advance so that the ratio of the coefficients becomes small. On the other hand, the 1D transformation matrix B is generated by designing a general-purpose transformation matrix having no such property. For example, a generic conversion is DCT. If 1D orthogonal transformation is performed in the orthogonal direction using the 1D transformation matrix A, the conversion efficiency of prediction errors in intra prediction is improved, and thus the coding efficiency is improved. For example, the prediction error 119 of mode 0 (vertical prediction) shows the above tendency in the vertical direction, but does not show the above tendency in the horizontal direction. Therefore, efficient orthogonal transformation can be realized by performing 1D orthogonal transformation using the 1D transformation matrix A in the vertical transformation unit 202 and performing 1D orthogonal transformation using the 1D transformation matrix B in the horizontal transformation unit 205.

Hereinafter, the details of the inverse orthogonal transform unit 105 according to the present embodiment will be described with reference to FIG.
The inverse orthogonal transform unit 105 includes a selection switch 301, a vertical inverse transform unit 302, a transposition unit 303, a selection switch 304, and a horizontal inverse transform unit 305. The vertical inverse transform unit 302 includes a 1D inverse orthogonal transform unit A306 and a 1D inverse orthogonal transform unit B307. The horizontal inverse transform unit 305 includes a 1D inverse orthogonal transform unit A308 and a 1D orthogonal transform unit B309. The order of the vertical inverse transform unit 302 and the horizontal inverse transform unit 305 is an example, and these may be reversed.

  The 1D inverse orthogonal transform unit A306 and the 1D inverse orthogonal transform unit A308 have a common function in that the input matrix is multiplied by the transposed matrix of the 1D transform matrix A described above, and the 1D inverse orthogonal transform units B307 and 1D. The inverse orthogonal transform unit B309 has a common function in that the input matrix is multiplied by the transposed matrix of the 1D transform matrix B described above. Therefore, the 1D inverse orthogonal transform unit A306 and the 1D inverse orthogonal transform unit A308 can also be realized by using physically identical hardware in a time division manner. The same applies to the 1D inverse orthogonal transform unit B307 and the 1D inverse orthogonal transform unit B309.

  The selection switch 301 guides the restoration transform coefficient 122 to one of the 1D inverse orthogonal transform unit A306 and the 1D inverse orthogonal transform unit B307 according to the vertical transform index included in the 1D transform matrix set information 129. The 1D inverse orthogonal transform unit A306 multiplies the input transform transform coefficient 122 (matrix format) by the transposed matrix of the 1D transform matrix A and outputs the result. The 1D inverse orthogonal transform unit B307 multiplies the input restored transform coefficient 122 by the transposed matrix of the 1D transform matrix B and outputs the result. Specifically, the 1D inverse orthogonal transform unit A306 and the 1D inverse orthogonal transform unit B307 (that is, the vertical inverse transform unit 302) perform one-dimensional inverse orthogonal transform represented by the following equation (5).

In Equation (5), Z 'represents a matrix of the restored transform coefficients ^{122 (N × N), V} T is generically indicates a transposed matrix of 1D transform matrix A and 1D transformation matrix B (both N × N) Y ′ represents an output matrix (N × N) of the 1D inverse orthogonal transform unit A306 and the 1D inverse orthogonal transform unit B307.

  The transposition unit 303 transposes the output matrix (Y ′) of the vertical inverse transform unit 302 and gives the result to the selection switch 304. However, the transposition unit 303 is an example, and corresponding hardware may not necessarily be prepared. For example, when the 1D inverse orthogonal transform performed by the vertical inverse transform unit 302 is stored (each element of the output matrix of the vertical inverse transform unit 302) and the 1D inverse orthogonal transform is performed by the horizontal inverse transform unit 305. If read in an appropriate order, transposition of the output matrix (Y ′) can be executed without preparing hardware corresponding to the transposition unit 303.

  The selection switch 304 guides the input matrix from the transposition unit 303 to one of the 1D inverse orthogonal transform unit A308 and the 1D inverse orthogonal transform unit B309 according to the horizontal transform index included in the 1D transform matrix set information 129. The 1D inverse orthogonal transform unit A308 multiplies the input matrix by the transposed matrix of the 1D transform matrix A and outputs the result. The 1D inverse orthogonal transform unit B309 multiplies the input matrix by the transposed matrix of the 1D transform matrix B and outputs the result. Specifically, the 1D inverse orthogonal transform unit A308 and the 1D inverse orthogonal transform unit B309 (that is, the horizontal inverse transform unit 305) perform one-dimensional inverse orthogonal transform represented by the following equation (6).

In Equation (6), ^{H T} is 1D transform matrix are generically indicates the transposed matrix of A and 1D transformation matrix B (both N × N), X 'is 1D inverse orthogonal transform unit A308 and 1D inverse orthogonal transform The output matrix (N × N) of the part B309 is shown, which indicates the restoration prediction error 123.

  As described above, the inverse orthogonal transform unit 105 performs inverse orthogonal transform on the reconstructed transform coefficient (matrix) 122 in accordance with the 1D transform matrix set information 129 input from the 1D transform matrix set unit 112, and reconstructed prediction error ( Matrix) 123 is generated. H. In consideration of H.264, the inverse orthogonal transform unit 105 may include an IDCT unit (not shown), or one of the 1D transform matrix A and the 1D transform matrix B may be replaced with a matrix for DCT. For example, the 1D transformation matrix B may be a matrix for DCT. Further, in addition to the IDCT, the inverse orthogonal transform unit 105 realizes inverse orthogonal transforms corresponding to various orthogonal transforms such as Hadamard transform, Karhunen-Loeve transform, and discrete sine transform, which will be described later, in harmony with the orthogonal transform unit 102. May be.

Hereinafter, details of the 1D conversion matrix set information 129 according to the present embodiment generated by the 1D conversion matrix set unit 112 will be described.
The 1D transformation matrix set information 129 includes a vertical transformation index for selecting a transformation matrix used for vertical orthogonal transformation and vertical inverse orthogonal transformation, and a transformation matrix used for horizontal orthogonal transformation and horizontal inverse orthogonal transformation. The horizontal transformation index for selecting is directly or indirectly indicated. For example, the 1D transformation matrix set information 129 can be expressed by a transformation index (TransformIdx) illustrated in FIG. 4D. With reference to the table of FIG. 4D, a vertical transformation index (Vertical Transform Idx) and a horizontal transformation index (Horizontal Transform Idx) can be derived from the transformation index.

  As shown in FIG. 4B, if the vertical transformation index is “0”, the 1D transformation matrix A (1D_Transform_Matrix_A) or its transposed matrix is selected for vertical orthogonal transformation or vertical inverse orthogonal transformation. On the other hand, if the vertical transformation index is “1”, the aforementioned 1D transformation matrix B (1D_Transform_Matrix_B) or its transposed matrix is selected for vertical orthogonal transformation or vertical inverse orthogonal transformation.

  As shown in FIG. 4C, if the horizontal transformation index is “0”, the 1D transformation matrix A (1D_Transform_Matrix_A) or its transposed matrix is selected for horizontal orthogonal transformation or horizontal inverse orthogonal transformation. On the other hand, if the horizontal transformation index is “1”, the aforementioned 1D transformation matrix B (1D_Transform_Matrix_B) or its transposed matrix is selected for horizontal orthogonal transformation or horizontal inverse orthogonal transformation.

FIG. 4A illustrates an index (IntraNxNPredModeIndex) of each (intra) prediction mode, its name (Name of IntraNxNPredMode), and a corresponding vertical conversion index and horizontal conversion index. In FIG. 4A, “NxN” represents the size of the prediction target block (N = 4, 8, 16, etc.). The size of the prediction target block can be expanded to “MxN” (that is, a rectangle other than a square).
Here, FIG. 4E illustrates an index of each prediction mode, a name thereof, and a corresponding conversion index obtained by integrating FIGS. 4A and 4D.

  The 1D transformation matrix set unit 112 detects a prediction mode index from the prediction mode information included in the prediction information 126, and generates corresponding 1D transformation matrix set information 129. Note that the various tables shown in FIGS. 4A, 4B, 4C, 4D, and 4E are examples, and the 1D conversion matrix set unit 112 uses the 1D conversion matrix set information without using some or all of these tables. 129 may be generated.

  For example, when the TransformIdx indicates 0, it means that the Vertical Transform index indicates 0 and the Horizontal Transform index indicates 0. That is, it means that the 1D conversion matrix A is used for the vertical orthogonal transformation, and the 1D conversion matrix A is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix A is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix A is used for horizontal inverse orthogonal transformation.

  When the TransformIdx indicates 1, it means that the Vertical Transform index indicates 0 and the Horizontal Transform index indicates 1. That is, it means that the 1D conversion matrix A is used for the vertical orthogonal transformation, and the 1D transformation matrix B is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix A is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix B is used for horizontal inverse orthogonal transformation.

  When the TransformIdx indicates 2, it means that the Vertical Transform index indicates 1, and the Horizontal Transform index indicates 0. That is, it means that the 1D transformation matrix B is used for the vertical orthogonal transformation and the 1D transformation matrix A is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix B is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix A is used for horizontal inverse orthogonal transformation.

  When TransformIdx indicates 3, it means that Vertical Transform index indicates 1, and Horizontal Transform index indicates 1. That is, it means that the 1D transformation matrix B is used for the vertical orthogonal transformation, and the 1D transformation matrix B is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix B is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix B is used for horizontal inverse orthogonal transformation.

  In the table shown in FIG. 4A, 1D transformation matrix set information 129 is assigned in consideration of the tendency of each intra prediction mode described above. In other words, 0 is assigned to the Vertical Transform index in the prediction mode that shows the above-mentioned tendency in the vertical direction of the prediction error, and 0 is assigned to the Horizontal Transform index in the mode that shows the above-mentioned tendency in the horizontal direction. On the other hand, 1 is assigned to each direction in which the above tendency is not exhibited. By classifying the vertical direction and horizontal direction of each prediction mode into two classes according to the presence or absence of the above-mentioned trends, each prediction mode is applied by adaptively applying the 1D conversion matrix A or 1D conversion matrix B to each of the vertical direction and the horizontal direction. High conversion efficiency is achieved as compared with a case where fixed orthogonal transform such as DCT is uniformly applied to the mode.

Details of the coefficient order control unit 113 will be described below.
The coefficient order control unit 113 converts each element of the quantized transform coefficient 121 that is a two-dimensional representation into a quantized transform coefficient sequence 117 that is a one-dimensional representation by arranging the elements in a predetermined order. As an example, the coefficient order control unit 113 can perform common 2D-1D conversion regardless of the prediction mode. Specifically, the coefficient control unit 113 includes the H.264 standard. Similar to H.264, zigzag scanning can be used. In the zigzag scan, the respective elements of the quantized transform coefficients 121 are arranged in the order shown in FIG. 8A and converted into a quantized transform coefficient string 117 as shown in FIG. 8B. 8A and 8B, (i, j) indicates the coordinates (position information) in the quantized transform coefficient (matrix) 121 of each element. FIG. 8C shows 2D-1D conversion using a zigzag scan (in the case of a 4 × 4 pixel block). Specifically, FIG. 8C shows an index (idx) indicating the coefficient order (scan order) of the quantized transform coefficient sequence 117 subjected to 2D-1D conversion using zigzag scanning, and the corresponding quantized transform coefficient 121. Element (cij) is shown. In FIG. 8C, cij indicates an element of coordinates (i, j) in the quantized transform coefficient (matrix) 121.

  As another example, the coefficient order control unit 113 can perform individual 2D-1D conversion for each prediction mode. The coefficient order control unit 113 that performs such an operation is illustrated in FIG. 5A. The coefficient order control unit 113 includes a selection switch 501 and individual 2D-1D conversion units 502,..., 510 for each of nine types of prediction modes. The selection switch 501 converts the quantized transform coefficient 121 according to the prediction mode information included in the prediction information 126 (for example, the index of the prediction mode in FIG. 4A), and the 2D-1D conversion unit (502,. Any one of 510). For example, if the prediction mode index is 0, the selection switch 501 guides the quantized transform coefficient 121 to the 2D-1D transform unit 502. In FIG. 5A, each prediction mode and the 2D-1D conversion unit have a one-to-one correspondence, and the quantized transform coefficient 121 is guided to one 2D-1D conversion unit corresponding to the prediction mode. FIG. 9 illustrates 2D-1D conversion (in the case of a 4 × 4 pixel block) performed by each 2D-1D conversion unit 502,. A specific design method for 2D-1D conversion for each prediction mode as shown in FIG. 9 will be described later. An index (idx) indicating the coefficient order (scan order) of the quantized transform coefficient sequence 117 subjected to 2D-1D conversion by the 2D-1D transform unit corresponding to each prediction mode, and an element (cij) of the corresponding quantized transform coefficient 121 ). In FIG. 9, cij represents an element of coordinates (i, j) in the quantized transform coefficient (matrix) 121. Further, in FIG. 9, each prediction mode is represented by its name, but the correspondence with the prediction mode index is as shown in FIG. 4A. As described above, when the individual 2D-1D transform for each prediction mode is applied, for example, the coefficients are scanned in an order suitable for the generation tendency of the non-zero coefficient in the quantized transform coefficient 121 for each prediction mode. Will improve.

  For simplification, an example related to a 4 × 4 pixel block is shown, but individual 2D-1D conversion for each prediction mode can be defined similarly for an 8 × 8 pixel block, a 16 × 16 pixel block, and the like. Further, if the pixel block is a rectangular block expressed by M × N, M × N can be used as a block size for performing 2D-1D conversion. In this case, regarding the rectangular block, individual 2D-1D conversion as exemplified in FIG. 9 may be defined for each prediction mode.

  As yet another example, the coefficient order control unit 113 may dynamically update the scan order in the 2D-1D conversion. The coefficient order control unit 113 that performs such an operation is illustrated in FIG. 5B. The coefficient order control unit 113 includes a selection switch 501, individual 2D-1D conversion units 502,..., 510 for each of nine types of prediction modes, an occurrence frequency counting unit 511, and a coefficient order updating unit 512. Including. The selection switch 501 is as described with reference to FIG. 5A. The individual 2D-1D conversion units 502,..., 510 for each of the nine types of prediction modes differ from FIG. 5A in that the scan order is updated by the coefficient order update unit 512.

  The occurrence frequency counting unit 511 creates a histogram of the number of occurrences of non-zero coefficients in each element of the quantized transform coefficient sequence 117 for each prediction mode. The occurrence frequency counting unit 511 inputs the created histogram 513 to the coefficient order updating unit 512.

  The coefficient order update unit 512 updates the coefficient order based on the histogram 513 at a predetermined timing. The timing is, for example, the timing when the coding process of the coding tree unit is finished, the timing when the coding process for one line in the coding tree unit is finished, or the like.

  Specifically, the coefficient order update unit 512 refers to the histogram 513, and updates the coefficient order for the prediction mode having an element in which the number of occurrences of non-zero coefficients is counted more than a threshold. For example, the coefficient order update unit 512 updates the prediction mode having an element in which the occurrence of a non-zero coefficient is counted 16 times or more. By providing a threshold value for the number of occurrences, the coefficient order is updated globally, so that it is difficult to converge to a local optimum solution.

  The coefficient order update unit 512 sorts the elements in descending order of the occurrence frequency of the non-zero coefficient regarding the prediction mode to be updated. Sorting can be realized by existing algorithms such as bubble sort and quick sort. Then, the coefficient order update unit 512 inputs coefficient order update information 514 indicating the order of the sorted elements to the 2D-1D conversion unit corresponding to the prediction mode to be updated.

  When the coefficient order update information 514 is input, the 2D-1D conversion unit performs 2D-1D conversion in accordance with the updated scan order. When the scan order is dynamically updated, the initial scan order of each 2D-1D conversion unit needs to be determined in advance. For example, the zigzag scan or the scan order illustrated in FIG. 9 can be used as the initial scan order.

  In this way, by dynamically updating the scan order, the tendency of occurrence of non-zero coefficients in the quantized transform coefficients 121 changes according to the influence of the properties of the predicted image, quantization information (quantization parameters), and the like. Even in this case, high encoding efficiency can be expected stably. Specifically, the generated code amount of run-length encoding in the entropy encoding unit 114 can be suppressed.

  For simplification, H.C. In the example of H.264, there are 9 types of prediction modes. However, when the prediction modes are expanded to 17 types, 33 types, etc., a 2D-1D conversion unit corresponding to each expanded prediction mode is added. Then, individual 2D-1D conversion for each prediction mode can be performed.

  Hereinafter, processing performed by the image coding apparatus in FIG. 1 for a coding target block (coding tree unit) will be described with reference to FIGS. 10A and 10B. In the examples of FIGS. 10A and 10B, it is assumed that the orthogonal transformation and inverse orthogonal transformation (that is, adaptive orthogonal transformation and inverse orthogonal transformation based on the 1D transformation matrix set information 129) according to the present embodiment are effective. It is said. However, as described later, it may be specified that the orthogonal transform and the inverse orthogonal transform according to the present embodiment are invalidated by the syntax.

  When the input image 118 is input to the image encoding apparatus in FIG. 1 in units of encoding target blocks, the encoding target block encoding process starts (step S601). The intra prediction unit 108 and the inter prediction unit 109 generate an intra prediction image and an inter prediction image using the reference image 125 stored in the reference image memory 107 (step S602). The encoding control unit 116 determines the optimal prediction mode from the viewpoint of the above-described encoding cost and generates the prediction information 126 (step S603). The prediction information 126 is input from the prediction selection unit 110 to each element as described above. If the prediction information 126 generated in step S603 indicates intra prediction, the process proceeds to step S605. If the prediction information 126 indicates inter prediction, the process proceeds to step S605 '.

  In step S605, the subtraction unit 101 generates the prediction error 119 by subtracting the (intra) prediction image 127 from the encoding target block, and the process proceeds to step S606. On the other hand, in step S605 'as well, the subtracting unit 101 subtracts the (inter) predicted image 127 from the encoding target block to generate a prediction error 119, and the process proceeds to step S614'.

  In step S606, the 1D conversion matrix setting unit 112 extracts prediction mode information included in the prediction information 126 generated in step S603. The 1D transformation matrix set unit 112 generates 1D transformation matrix set information 129 based on the extracted prediction mode information (for example, referring to the table of FIG. 4A) (step S607). The 1D transform matrix set unit 112 inputs 1D transform matrix set information 129 to the orthogonal transform unit 102 and the inverse orthogonal transform unit 105.

  The selection switch 201 in the orthogonal transform unit 102 selects the 1D orthogonal transform unit A206 or the 1D orthogonal transform unit B207 based on the 1D transform matrix set information 129 (steps S608, S609, and S610). On the other hand, the selection switch 204 in the orthogonal transform unit 102 selects the 1D orthogonal transform unit A208 or the 1D orthogonal transform unit B209 based on the 1D transform matrix set information 129 (steps S611, S612, and S613). Thereafter, the process proceeds to step S614.

  For example, when the transformation index (TransformIdx) that is an example of the 1D transformation matrix set information 129 is 0, the selection switch 201 selects the 1D orthogonal transformation unit A206 in the vertical transformation unit 202 (step S609), and the selection switch 204 is horizontal. The 1D orthogonal transform unit A208 in the transform unit 205 is selected (step S612). When TransformIdx is 1, the selection switch 201 selects the 1D orthogonal transform unit A206 in the vertical transform unit 202 (step S609), and the selection switch 204 selects the 1D orthogonal transform unit B209 in the horizontal transform unit 205 (step S613). ). When TransformIdx is 2, the selection switch 201 selects the 1D orthogonal transform unit B207 in the vertical transform unit 202 (step S610), and the selection switch 204 selects the 1D orthogonal transform unit A208 in the horizontal transform unit 205 (step S612). ). When TransformIdx is 3, the selection switch 201 selects the 1D orthogonal transform unit B207 in the vertical transform unit 202 (step S610), and the selection switch 204 selects the 1D orthogonal transform unit B209 in the horizontal transform unit 205 (step S613). ).

  In step S614, the orthogonal transform unit 102 performs vertical transform and horizontal transform on the prediction error 119 according to the settings in step S608,..., Step S613, respectively, to generate the transform coefficient 120. Subsequently, the quantization unit 103 quantizes the transform coefficient 120 generated in step S614 to generate a quantized transform coefficient 121 (step S615), and the process proceeds to step S616.

  On the other hand, in step S <b> 614 ′, the orthogonal transform unit 102 performs a fixed orthogonal transform such as DCT on the prediction error 119 to generate a transform coefficient 120. Subsequently, the quantization unit 103 quantizes the transform coefficient 120 generated in step S614 'to generate a quantized transform coefficient 121 (step S615'), and the process proceeds to step S617 '. Note that the orthogonal transform performed in step S614 'may be realized by a DCT unit (not shown) or the like, or may be realized by the 1D orthogonal transform unit B207 and the 1D orthogonal transform unit B209.

  In step S616, the coefficient order control unit 113 scans based on the prediction mode information included in the prediction information 126 generated in step S603 (that is, the connection of the selection switch 501 in the example of FIGS. 5A and 5B). First) is set, and the process proceeds to step S617. However, if the coefficient control unit 113 performs common 2D-1D conversion regardless of the prediction mode, step S616 can be omitted.

  In step S617, the coefficient order control unit 113 performs 2D-1D conversion on the quantized transform coefficient 121 according to the setting in step S616 to generate a quantized transform coefficient sequence 117. Subsequently, the entropy encoding unit 114 performs entropy encoding on the encoding parameter including the quantized transform coefficient sequence 117 (step S618). The encoded data 130 is output at an appropriate timing managed by the encoding control unit 116. On the other hand, the inverse quantization unit 104 performs inverse quantization on the quantized transform coefficient 121 to generate a restored transform coefficient 122 (step S619), and the process proceeds to step S620.

  In step S617 ′, the coefficient order control unit 113 performs a fixed 2D-1D conversion such as a zigzag scan or a 2D-1D conversion corresponding to Intra_NxN_DC in FIG. A coefficient sequence 117 is generated. Subsequently, the entropy encoding unit 114 performs entropy encoding on the encoding parameter including the quantized transform coefficient sequence 117 (step S618 '). The encoded data 130 is output at an appropriate timing managed by the encoding control unit 116. On the other hand, the inverse quantization unit 104 performs inverse quantization on the quantized transform coefficient 121 to generate the restored transform coefficient 122 (step S619 '), and the process proceeds to step S626'.

  The selection switch 301 in the inverse orthogonal transform unit 105 selects the 1D inverse orthogonal transform unit A306 or the 1D inverse orthogonal transform unit B307 based on the 1D transform matrix set information 129 (step S620, step S621, and step S622). On the other hand, the selection switch 304 in the inverse orthogonal transform unit 105 selects the 1D inverse orthogonal transform unit A308 or the 1D inverse orthogonal transform unit B309 based on the 1D transform matrix set information 129 (steps S623, S624, and S625). Thereafter, the process proceeds to step S626.

  For example, when the transform index (TransformIdx), which is an example of the 1D transform matrix set information 129, is 0, the selection switch 301 selects the 1D inverse orthogonal transform unit A306 in the vertical inverse transform unit 302 (step S621), and the selection switch 304 Selects the 1D inverse orthogonal transform unit A308 in the horizontal inverse transform unit 305 (step S624). When TransformIdx is 1, the selection switch 301 selects the 1D inverse orthogonal transform unit A306 in the vertical inverse transform unit 302 (step S621), and the selection switch 304 selects the 1D inverse orthogonal transform unit B309 in the horizontal inverse transform unit 305. (Step S625). When TransformIdx is 2, the selection switch 301 selects the 1D inverse orthogonal transform unit B307 in the vertical inverse transform unit 302 (step S622), and the selection switch 304 selects the 1D inverse orthogonal transform unit A308 in the horizontal inverse transform unit 305. (Step S624). When TransformIdx is 3, the selection switch 301 selects the 1D inverse orthogonal transform unit B307 in the vertical inverse transform unit 302 (step S622), and the selection switch 304 selects the 1D inverse orthogonal transform unit B309 in the horizontal inverse transform unit 305. (Step S625).

  In step S626, the inverse orthogonal transform unit 105 performs the vertical inverse transform and the horizontal inverse transform according to the settings in step S620,..., Step S625 on the restored transform coefficient 122 to generate the restored prediction error 123. The process proceeds to step S627. In step S626 ', the inverse orthogonal transform unit 105 performs inverse orthogonal transform such as IDCT on the reconstructed transform coefficient 122 to generate a reconstructed prediction error 123, and the process proceeds to step S627. Note that the fixed inverse orthogonal transform performed in step S626 'may be realized by an IDCT unit (not shown) or the like, or may be realized by the 1D inverse orthogonal transform unit B307 and the 1D inverse orthogonal transform unit B309.

  In step S627, the adding unit 106 adds the reconstructed prediction error 123 generated in step S626 or step S626 ′ and the predicted image 127 to generate a local decoded image 124. The local decoded image 124 is used as a reference image as a reference image memory. In step S628, the encoding process of the block to be encoded is completed.

  Hereinafter, a design method of the above-described 1D conversion matrix A and 1D conversion matrix B will be described. H. In the H.264 4 × 4 pixel block and the 8 × 8 pixel block, nine types of prediction modes are defined, and in the 16 × 16 pixel block, four types of prediction modes are defined.

  First, a prediction error 119 for each prediction mode is generated. Of the prediction errors 119 in each prediction mode, those indicating the above-mentioned tendency that the absolute value of the prediction error increases as the distance from the reference pixel increases in the vertical direction or the horizontal direction are collected. Then, by setting the direction indicating this tendency vertically and performing singular value decomposition on the matrix in which the prediction errors 119 are arranged horizontally, a 1D orthogonal basis for removing the correlation in the vertical direction of the matrix is designed. A 1D conversion matrix A is generated by vertically arranging the 1D orthogonal bases as row vectors.

  On the other hand, a singular value decomposition is performed on a matrix in which prediction directions 119 are set vertically and a prediction error 119 is arranged horizontally, thereby generating a 1D orthogonal basis for removing the vertical correlation of the matrix. To do. A 1D conversion matrix B is generated by vertically arranging the 1D orthogonal bases as row vectors. The 1D conversion matrix B can be simply replaced with a matrix for DCT. Although a design for a 4 × 4 pixel block is illustrated for simplicity, 1D transformation matrices for 8 × 8 pixel blocks and 16 × 16 pixel blocks can be designed as well. The described design method is an example, and there is room for appropriate design in consideration of the properties of the prediction residual described above.

  Hereinafter, a specific design method of 2D-1D conversion (scan order) for each prediction mode as exemplified in FIG. 9 will be described. The scan order for each prediction mode is designed based on the quantized transform coefficient 121 generated by the quantization unit 103. For example, in the design related to a 4 × 4 pixel block, a plurality of training images are prepared and the prediction residuals 119 of each of the nine types of prediction modes are generated. Each of the prediction residuals 119 is subjected to orthogonal transformation shown in Equation (3) and Equation (4) to generate a transform coefficient 120, which is further quantized. The number of occurrences of non-zero coefficients is cumulatively added to each quantized transform coefficient 121 for each element in the 4 × 4 pixel block. This cumulative addition is performed on all training images, and a histogram indicating the frequency of occurrence of non-zero coefficients is created for every 16 elements of the 4 × 4 pixel block. Based on this histogram, indexes 0 to 15 are given in ascending order from the element with the highest occurrence frequency. Such index assignment is performed individually for all prediction modes. The order of the assigned indexes is used as the scan order corresponding to each prediction mode.

  For the sake of simplicity, the design related to the 4 × 4 pixel block is illustrated, but the scan order of the 8 × 8 pixel block and the 16 × 16 pixel block can be similarly designed. Further, even if the prediction modes are expanded to 17 types, 33 types, and an arbitrary number, the design can be performed by the same method. The method for dynamically updating the scan order is as described with reference to FIG. 5B.

Hereinafter, the syntax used by the image encoding device in FIG. 1 will be described.
The syntax indicates the structure of encoded data (for example, encoded data 130 in FIG. 1) when the image encoding apparatus encodes moving image data. When decoding this encoded data, the image decoding apparatus interprets the syntax with reference to the same syntax structure. FIG. 11 illustrates a syntax 700 used by the image coding apparatus in FIG.

  The syntax 700 includes three parts: a high level syntax 701, a slice level syntax 702, and a coding tree level syntax 703. The high level syntax 701 includes syntax information of a layer higher than the slice. A slice refers to a rectangular area or a continuous area included in a frame or a field. The slice level syntax 702 includes information necessary for decoding each slice. The coding tree level syntax 703 includes information necessary for decoding each coding tree (ie, each coding tree unit). Each of these parts includes more detailed syntax.

  High level syntax 701 includes sequence and picture level syntax, such as sequence parameter set syntax 704 and picture parameter set syntax 705. The slice level syntax 702 includes a slice header syntax 706, a slice data syntax 707, and the like. The coding tree level syntax 703 includes a coding tree unit syntax 708, a prediction unit syntax 709, and the like.

  The coding tree unit syntax 708 may have a quadtree structure. Specifically, the coding tree unit syntax 708 can be recursively called as a syntax element of the coding tree unit syntax 708. That is, one coding tree unit can be subdivided with a quadtree. The coding tree unit syntax 708 includes a transform unit syntax 710. The transform unit syntax 710 is invoked at each coding tree unit syntax 708 at the extreme end of the quadtree. The transform unit syntax 710 describes information related to inverse orthogonal transformation and quantization.

  FIG. 12 illustrates a slice header syntax 706 according to this embodiment. The slice_directive_unified_transform_flag shown in FIG. 12 is a syntax element indicating, for example, validity / invalidity of orthogonal transformation and inverse orthogonal transformation according to the present embodiment for the slice.

  When slice_directive_unified_transform_flag is 0, the orthogonal transform and inverse orthogonal transform according to the present embodiment in the slice are invalid. Therefore, the orthogonal transform unit 102 and the inverse orthogonal transform unit 105 perform fixed orthogonal transform and inverse orthogonal transform such as DCT and IDCT. This fixed orthogonal transform and inverse orthogonal transform are performed by the 1D orthogonal transform unit B207, the 1D orthogonal transform unit B209, the 1D inverse orthogonal transform unit 307, and the 1D inverse orthogonal transform unit 309 (that is, by the 1D transform matrix B). Alternatively, it may be performed by a DCT unit and an IDCT unit (not shown). The coefficient order control unit 113 also performs fixed 2D-1D conversion (for example, zigzag scanning). This fixed 2D-1D conversion may be performed by the 2D-1D conversion unit (mode 2) 504, or may be performed by a 2D-1D conversion unit (not shown).

  As an example, when slice_directive_unified_transform_flag is 1, the orthogonal transformation and inverse orthogonal transformation according to the present embodiment are effective in the entire area in the slice. That is, the encoding process is performed in the entire area in the slice according to the encoding flowchart described with reference to FIGS. 10A and 10B. That is, the selection switch 201 selects the 1D orthogonal transform unit A206 or the 1D orthogonal transform unit B207 based on the 1D transform matrix set information 129. The selection switch 204 selects the 1D orthogonal transform unit A208 or the 1D orthogonal transform unit B209 based on the 1D transform matrix set information 129. The selection switch 301 selects the 1D inverse orthogonal transform unit A306 or the 1D inverse orthogonal transform unit B307 based on the 1D transform matrix set information 129. The selection switch 304 selects the 1D inverse orthogonal transform unit A308 or the 1D inverse orthogonal transform unit B309 based on the 1D transform matrix set information 129. Further, the selection switch 501 selects one of the 2D-1D conversion units 502,..., 510 according to the prediction mode information included in the prediction information 126.

  As another example, when slice_directional_unified_transform_flag is 1, the orthogonal transform and inverse according to this embodiment are performed for each local region in the slice in the syntax of a lower layer (coding tree unit, transform unit, etc.). Validity / invalidity of orthogonal transformation may be defined.

  FIG. 13 illustrates a coding tree unit syntax 708 according to this embodiment. Ctb_directive_unified_transform_flag shown in FIG. 13 is a syntax element indicating validity / invalidity of orthogonal transform and inverse orthogonal transform according to the present embodiment with respect to the coding tree unit. Further, pred_mode shown in FIG. 13 is one of syntax elements included in the prediction unit syntax 709, and indicates the coding type in the coding tree unit or macroblock. MODE_INTRA indicates that the encoding type is intra prediction. ctb_directive_unified_transform_flag is encoded only when the above-mentioned slice_directional_unified_transform_flag is 1 and the coding type of the coding tree unit is intra prediction.

  When ctb_directive_unified_transform_flag is 0, the orthogonal transform and the inverse orthogonal transform according to the present embodiment in the coding tree unit are invalid. Therefore, the orthogonal transform unit 102 and the inverse orthogonal transform unit 105 perform fixed orthogonal transform and inverse orthogonal transform such as DCT and IDCT. This fixed orthogonal transform and inverse orthogonal transform are performed by the 1D orthogonal transform unit B207, the 1D orthogonal transform unit B209, the 1D inverse orthogonal transform unit 307, and the 1D inverse orthogonal transform unit 309 (that is, by the 1D transform matrix B). Alternatively, it may be performed by a DCT unit and an IDCT unit (not shown). The coefficient order control unit 113 also performs fixed 2D-1D conversion (for example, zigzag scanning). This fixed 2D-1D conversion may be performed by the 2D-1D conversion unit (mode 2) 504, or may be performed by a 2D-1D conversion unit (not shown).

  On the other hand, when ctb_directive_unified_transform_flag is 1, the orthogonal transform and the inverse orthogonal transform according to the present embodiment are valid in the coding tree unit, and the encoding process is performed according to the encoding flowchart described in FIGS. 10A and 10B. That is, the selection switch 201 selects the 1D orthogonal transform unit A206 or the 1D orthogonal transform unit B207 based on the 1D transform matrix set information 129. The selection switch 204 selects the 1D orthogonal transform unit A208 or the 1D orthogonal transform unit B209 based on the 1D transform matrix set information 129. The selection switch 301 selects the 1D inverse orthogonal transform unit A306 or the 1D inverse orthogonal transform unit B307 based on the 1D transform matrix set information 129. The selection switch 304 selects the 1D inverse orthogonal transform unit A308 or the 1D inverse orthogonal transform unit B309 based on the 1D transform matrix set information 129. Further, the selection switch 501 selects one of the 2D-1D conversion units 502,..., 510 according to the prediction mode information included in the prediction information 126.

  In the coding tree unit syntax 708, as shown in the example of FIG. 13, when a flag specifying validity / invalidity of orthogonal transformation and inverse orthogonal transformation according to the present embodiment is encoded, information is encoded compared to a case where this flag is not encoded. The amount (code amount) increases. However, by encoding this flag, it is possible to perform optimal orthogonal transform for each local region (ie, coding tree unit).

  FIG. 14 illustrates a transform unit syntax 710 according to this embodiment. A tu_directive_unified_transform_flag shown in FIG. 14 is a syntax element indicating validity / invalidity of the orthogonal transform and the inverse orthogonal transform according to the present embodiment with respect to the transform unit. Further, pred_mode shown in FIG. 14 is one of syntax elements included in the prediction unit syntax 709, and indicates the coding type in the coding tree unit or macroblock. MODE_INTRA indicates that the encoding type is intra prediction. Tu_directive_unified_transform_flag is encoded only when slice_directive_unified_transform_flag is 1 and the coding type of the coding tree unit is intra prediction.

  When tu_directive_unified_transform_flag is 0, the orthogonal transform and inverse orthogonal transform according to the present embodiment in the transform unit are invalid. Therefore, the orthogonal transform unit 102 and the inverse orthogonal transform unit 105 perform fixed orthogonal transform and inverse orthogonal transform such as DCT and IDCT. This fixed orthogonal transform and inverse orthogonal transform are performed by the 1D orthogonal transform unit B207, the 1D orthogonal transform unit B209, the 1D inverse orthogonal transform unit 307, and the 1D inverse orthogonal transform unit 309 (that is, by the 1D transform matrix B). Alternatively, it may be performed by a DCT unit and an IDCT unit (not shown). The coefficient order control unit 113 also performs fixed 2D-1D conversion (for example, zigzag scanning). This fixed 2D-1D conversion may be performed by the 2D-1D conversion unit (mode 2) 504, or may be performed by a 2D-1D conversion unit (not shown).

  On the other hand, when tu_directional_unified_transform_flag is 1, the orthogonal transform and inverse orthogonal transform according to the present embodiment in the transform unit are valid, and the encoding process is performed according to the encoding flowchart described in FIGS. 10A and 10B. That is, the selection switch 201 selects the 1D orthogonal transform unit A206 or the 1D orthogonal transform unit B207 based on the 1D transform matrix set information 129. The selection switch 204 selects the 1D orthogonal transform unit A208 or the 1D orthogonal transform unit B209 based on the 1D transform matrix set information 129. The selection switch 301 selects the 1D inverse orthogonal transform unit A306 or the 1D inverse orthogonal transform unit B307 based on the 1D transform matrix set information 129. The selection switch 304 selects the 1D inverse orthogonal transform unit A308 or the 1D inverse orthogonal transform unit B309 based on the 1D transform matrix set information 129. Further, the selection switch 501 selects one of the 2D-1D conversion units 502,..., 510 according to the prediction mode information included in the prediction information 126.

  As in the example of FIG. 14, in the transform unit syntax 710, when the flag that defines the validity / invalidity of the orthogonal transform and the inverse orthogonal transform according to the present embodiment is encoded, the information is compared with the case where the flag is not encoded. The amount (code amount) increases. However, by encoding this flag, it is possible to perform optimal orthogonal transform for each local region (that is, transform unit).

  It should be noted that syntax elements not defined in this embodiment may be inserted between the rows of the syntax tables illustrated in FIGS. 12, 13, and 14, or other conditional branch descriptions may be included. Good. Further, the syntax table may be divided into a plurality of tables, or a plurality of syntax tables may be integrated. Moreover, the term of each illustrated syntax element can be changed arbitrarily.

  As described above, the image encoding apparatus according to the present embodiment uses the tendency of intra prediction that the prediction accuracy decreases as the distance from the reference pixel increases. This image encoding apparatus classifies the vertical direction and horizontal direction of each prediction mode into two classes according to the presence or absence of the above-described tendency, and adaptively assigns the 1D conversion matrix A or 1D conversion matrix B to each of the vertical direction and horizontal direction. Apply. The 1D transform matrix A has high coefficient density when performing 1D orthogonal transform in the direction (vertical direction or horizontal direction) orthogonal to the line of the reference pixel group (that is, the ratio of non-zero coefficients in the quantized transform coefficient 121) Is generated in advance by designing a common transformation base so that On the other hand, the 1D transformation matrix B is generated by designing a general-purpose transformation matrix having no such property. For example, a generic conversion is DCT. Therefore, according to the image coding apparatus according to the present embodiment, high conversion efficiency is achieved as compared with a case where fixed orthogonal transform such as DCT is uniformly applied to each prediction mode.

Further, the orthogonal transform unit 102 and the inverse orthogonal transform unit 105 according to the present embodiment are suitable for both hardware implementation and software implementation.
Since Equations (3) to (6) represent multiplication of a fixed matrix, when the orthogonal transform unit and the inverse orthogonal transform unit are implemented by hardware, they are configured by hard-wired logic rather than a multiplier. Is assumed.

  If orthogonal transform and inverse orthogonal transform are performed using a dedicated transform basis for each of the nine types of intra prediction modes, nine 2D orthogonal transform units or 18 (= 9 × 2) as shown in FIG. ) 1D orthogonal transform units must be prepared. These 9 2D orthogonal transform units or 18 1D orthogonal transform units multiply each by a different transform matrix. In addition to dedicated hardware for DCT required by H.264, additional dedicated hardware for nine 2D orthogonal transform units or 18 1D orthogonal transform units is provided, which increases the circuit scale.

  On the other hand, as shown in FIGS. 2 and 3, two orthogonal transform units and inverse orthogonal transform units according to the present embodiment are used (when the vertical (inverse) transform unit and the horizontal (inverse) transform unit are shared in a time division manner). The four types of two-dimensional orthogonal transforms are executed by a combination of the 1D orthogonal transform unit and a circuit for transposing the matrix. Therefore, according to the orthogonal transformation part and the inverse orthogonal transformation part which concern on this embodiment, the increase in the circuit scale in hardware mounting can be suppressed significantly.

  Further, regarding software implementation, if orthogonal transform and inverse orthogonal transform are performed using dedicated transform bases for each of the nine types of intra prediction modes, nine 2D orthogonal transform matrices or 18 (= 9 × 2) It is assumed that 1D orthogonal transformation matrices are stored in a memory, and these transformation matrices are called for each prediction mode to implement orthogonal transformation using a general-purpose multiplier. Therefore, there is a possibility that the cost increases due to an increase in the memory size for storing the transformation matrix, or that the memory bandwidth is increased by loading the transformation matrix into the memory for each transformation.

  On the other hand, the orthogonal transform unit and the inverse orthogonal transform unit according to the present embodiment combine four types of two-dimensional by combining vertical transform and horizontal transform using two 1D orthogonal transform matrices as shown in FIGS. The orthogonal transformation of is performed. Therefore, according to the orthogonal transformation part and the inverse orthogonal transformation part which concern on this embodiment, the increase in the memory size in software mounting can be suppressed significantly.

  Also, as described in the present embodiment, preparing an individual scan order for each prediction mode contributes to an improvement in coding efficiency. The quantized transform coefficient 121 has a property that the generation tendency of non-zero coefficients is biased for each element. The occurrence tendency of such non-zero coefficients differs for each prediction direction of intra prediction. Furthermore, if the prediction directions are the same, the non-zero coefficient generation tendency is similar even when pixel blocks of different input images 118 are encoded. Therefore, the coefficient order control unit 113 converts the zero coefficient in the quantized transform coefficient sequence 122 by transforming the quantized transform coefficient sequence 122 into the one-dimensional quantized transform coefficient sequence 122 in order from the element having the highest non-zero coefficient occurrence probability. Are dense with high probability. That is, it is possible to reduce the amount of generated code by run length encoding in the entropy encoding unit 114. As described with reference to FIGS. 5A and 5B, the coefficient order control unit 113 may use the scan order learned in advance for each prediction mode, or dynamically update the scan order during the encoding process. You may use it. If the scan order optimized for each prediction mode is used, for example, H.264 is used. Compared with H.264, the amount of generated code based on the quantized transform coefficient sequence 122 can be reduced without causing a significant increase in the amount of computation.

(Second Embodiment)
The image encoding device according to the second embodiment differs from the image encoding device according to the first embodiment described above in the details of orthogonal transform and inverse orthogonal transform. In the following description, the same parts as those in the first embodiment are denoted by the same reference numerals in the present embodiment, and different parts will be mainly described. An image decoding apparatus corresponding to the image encoding apparatus according to the present embodiment will be described in a fifth embodiment.

  The image encoding apparatus according to the present embodiment includes an orthogonal transform unit 102 illustrated in FIG. 16 instead of the orthogonal transform unit 102 illustrated in FIG. 16 includes a selection switch 801, a vertical conversion unit 802, a transposition unit 203, a selection switch 804, and a horizontal conversion unit 805. The vertical transform unit 802 includes a 1D orthogonal transform unit C806, a 1D orthogonal transform unit D807, and a 1D orthogonal transform unit E808. The horizontal transform unit 805 includes a 1D orthogonal transform unit C809, a 1D orthogonal transform unit D810, and a 1D orthogonal transform unit E811. Note that the order of the vertical conversion unit 802 and the horizontal conversion unit 805 is an example, and these may be reversed.

  The 1D orthogonal transform unit C806 and the 1D orthogonal transform unit C809 have a common function in that the input matrix is multiplied by the 1D transform matrix C. The 1D orthogonal transform unit D807 and the 1D orthogonal transform unit D810 have a common function in that the input matrix is multiplied by the 1D transform matrix D. The 1D orthogonal transform unit E808 and the 1D orthogonal transform unit E811 have a common function in that the input matrix is multiplied by the 1D transform matrix E.

Hereinafter, the 1D conversion matrix C, the 1D conversion matrix D, and the 1D conversion matrix E according to the present embodiment will be described.
As described above, the prediction error 119 tends to increase in absolute value as the distance from the reference pixel increases. Although the tendency is the same regardless of the prediction direction, it cannot be said that the prediction pixel 119 in the DC prediction mode shows a tendency related to either the vertical direction or the horizontal direction. In the present embodiment, a 1D conversion matrix E described later with respect to the DC prediction mode is used. On the other hand, for prediction modes other than the DC prediction mode, the 1D conversion matrix C and the 1D conversion matrix D are adaptively used according to the presence or absence of the above-described tendency, as in the first embodiment.

  Specifically, the 1D conversion matrix C can be generated by the same design method as the 1D conversion matrix A described above. The 1D conversion matrix D can be generated by a design method similar to the 1D conversion matrix B described above. That is, the 1D transformation matrix D can be generated by performing the above-described design method for the 1D transformation matrix B after excluding the DC prediction mode.

  The 1D transformation matrix E may be a matrix for DCT. Alternatively, the 1D transform matrix E has higher coefficient density when performing 1D orthogonal transform in the vertical and horizontal directions with respect to the prediction error 119 in the DC prediction mode, compared to the 1D transform matrix D (ie, quantization). It may be generated by designing a common transformation base in advance so that the ratio of non-zero coefficients in the transformation coefficient 121 becomes smaller.

  The image encoding apparatus according to the present embodiment includes an inverse orthogonal transform unit 105 illustrated in FIG. 17 instead of the inverse orthogonal transform unit 105 illustrated in FIG. The inverse orthogonal transform unit 105 in FIG. 17 includes a selection switch 901, a vertical inverse transform unit 902, a transposition unit 303, a selection switch 904, and a horizontal inverse transform unit 905. The vertical inverse transform unit 902 includes a 1D inverse orthogonal transform unit C906, a 1D inverse orthogonal transform unit D907, and a 1D inverse orthogonal transform unit E908. The horizontal inverse transform unit 905 includes a 1D inverse orthogonal transform unit C909, a 1D inverse orthogonal transform unit D910, and a 1D inverse orthogonal transform unit E911. Note that the order of the vertical inverse transform unit 902 and the horizontal inverse transform unit 905 is an example, and these may be reversed.

  The 1D inverse orthogonal transform unit C906 and the 1D inverse orthogonal transform unit C909 have a common function in that an input matrix is multiplied by a transposed matrix of the 1D transform matrix C. The 1D inverse orthogonal transform unit D907 and the 1D inverse orthogonal transform unit D910 have a common function in that the input matrix is multiplied by the transposed matrix of the 1D transform matrix D. The 1D inverse orthogonal transform unit E908 and the 1D inverse orthogonal transform unit E911 have a common function in that the input matrix is multiplied by the transposed matrix of the 1D transform matrix E.

Hereinafter, details of the 1D conversion matrix set information 129 according to the present embodiment generated by the 1D conversion matrix set unit 112 will be described.
The 1D transformation matrix set information 129 includes a vertical transformation index for selecting a transformation matrix used for vertical orthogonal transformation and vertical inverse orthogonal transformation, and a transformation matrix used for horizontal orthogonal transformation and horizontal inverse orthogonal transformation. The horizontal transformation index for selecting is directly or indirectly indicated. For example, the 1D transformation matrix set information 129 can be expressed by a transformation index (TransformIdx) illustrated in FIG. 18D. With reference to the table in FIG. 18D, a vertical transformation index (Vertical Transform Idx) and a horizontal transformation index (Horizontal Transform Idx) can be derived from the transformation index.

  As shown in FIG. 18B, if the vertical transformation index is “0”, the 1D transformation matrix C (1D_Transform_Matrix_C) or its transposed matrix is selected for vertical orthogonal transformation or vertical inverse orthogonal transformation. On the other hand, if the vertical transformation index is “1”, the aforementioned 1D transformation matrix D (1D_Transform_Matrix_D) or its transposed matrix is selected for vertical orthogonal transformation or vertical inverse orthogonal transformation. Further, if the vertical transformation index is “2”, the 1D transformation matrix E (1D_transform_Matrix_E) or its transposed matrix is selected for vertical orthogonal transformation or vertical inverse orthogonal transformation.

  As shown in FIG. 18C, if the horizontal transformation index is “0”, the 1D transformation matrix C (1D_Transform_Matrix_C) or its transposed matrix is selected for horizontal orthogonal transformation or horizontal inverse orthogonal transformation. On the other hand, if the horizontal transformation index is “1”, the aforementioned 1D transformation matrix D (1D_Transform_Matrix_D) or its transposed matrix is selected for horizontal orthogonal transformation or horizontal inverse orthogonal transformation. Furthermore, if the horizontal transformation index is “2”, the 1D transformation matrix E (1D_Transform_Matrix_E) or its transposed matrix is selected for horizontal orthogonal transformation or horizontal inverse orthogonal transformation.

Further, FIG. 18A illustrates an index (IntraNxNPredModeIndex) of each (intra) prediction mode, its name (Name of IntraNxNPredMode), and a corresponding vertical conversion index and horizontal conversion index. In FIG. 18A, “NxN” represents the size of the prediction target block (N = 4, 8, 16, etc.). The size of the prediction target block can be expanded to “MxN” (that is, a rectangle other than a square).
Here, FIG. 18E illustrates an index of each prediction mode, its name, and a corresponding conversion index obtained by integrating FIGS. 18A and 18D.

  The 1D transformation matrix set unit 112 detects a prediction mode index from the prediction mode information included in the prediction information 126, and generates corresponding 1D transformation matrix set information 129. Note that the various tables shown in FIGS. 18A, 18B, 18C, 18D, and 18E are examples, and the 1D conversion matrix set unit 112 uses the 1D conversion matrix set information without using some or all of these tables. 129 may be generated.

  For example, when the TransformIdx indicates 0, it means that the Vertical Transform index indicates 0 and the Horizontal Transform index indicates 0. That is, it means that the 1D transformation matrix C is used for the vertical orthogonal transformation and the 1D transformation matrix C is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix C is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix C is used for horizontal inverse orthogonal transformation.

  When the TransformIdx indicates 1, it means that the Vertical Transform index indicates 0 and the Horizontal Transform index indicates 1. That is, it means that the 1D transformation matrix C is used for the vertical orthogonal transformation, and the 1D transformation matrix D is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix C is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix D is used for horizontal inverse orthogonal transformation.

  When the TransformIdx indicates 2, it means that the Vertical Transform index indicates 1, and the Horizontal Transform index indicates 0. That is, it means that the 1D transformation matrix D is used for the vertical orthogonal transformation, and the 1D transformation matrix C is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of the 1D transformation matrix D is used for the vertical inverse orthogonal transformation, and a 1D transformation matrix C is used for the horizontal inverse orthogonal transformation.

  When TransformIdx indicates 3, it means that Vertical Transform index indicates 2 and Horizontal Transform index indicates 2. That is, it means that the 1D transformation matrix E is used for the vertical orthogonal transformation and the 1D transformation matrix E is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix E is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix E is used for horizontal inverse orthogonal transformation.

  Here, when the prediction target block is a rectangular block represented by M × N, the block size for performing orthogonal transform may also be M × N.

  In the table shown in FIG. 18A, 1D transformation matrix set information 129 is assigned in consideration of the tendency of each intra prediction mode described above. That is, in the DC prediction mode, 2 is assigned to both the Vertical Transform index and the Horizontal Transform index. Therefore, in the DC prediction mode, orthogonal transformation or inverse orthogonal transformation in the vertical direction and the horizontal direction is performed using the above-described 1D transformation matrix E or its transposed matrix, and high transformation efficiency is achieved.

  Regarding prediction modes other than the DC prediction mode, 0 is assigned to the Vertical Transform index if the tendency is shown in the vertical direction of the prediction error, and 0 is assigned to the Horizontal Transform index if the tendency is shown in the horizontal direction. On the other hand, 1 is assigned to each direction in which the above tendency is not exhibited. Regarding prediction modes other than the DC prediction mode, the vertical direction and the horizontal direction of each prediction mode are classified into two classes according to the presence or absence of the above-described tendency, and the 1D conversion matrix C or 1D conversion matrix is adaptively applied to each of the vertical direction and the horizontal direction. By applying D, high conversion efficiency is achieved.

  As described above, the image coding apparatus according to this embodiment uses the tendency of intra prediction that the prediction accuracy decreases as the distance from the reference pixel increases as in the first embodiment, Differentiate predictions and apply orthogonal and inverse orthogonal transforms. This image encoding apparatus classifies the vertical direction and horizontal direction of each prediction mode into two classes according to the presence or absence of the above-described tendency, and adaptively sets the 1D conversion matrix C or 1D conversion matrix D for each of the vertical direction and the horizontal direction. Apply. This image encoding apparatus applies the 1D transformation matrix E to the DC prediction mode. The 1D transform matrix C has higher coefficient density when performing 1D orthogonal transform in the direction (vertical direction or horizontal direction) orthogonal to the line of the reference pixel group (that is, the ratio of non-zero coefficients in the quantized transform coefficient 121) Is generated in advance by designing a common transformation base so that The 1D transformation matrix D is generated by designing a general-purpose transformation matrix that does not have such a property after excluding the DC prediction mode. The 1D transformation matrix E may be a matrix for DCT. Alternatively, the 1D transform matrix E has a high coefficient density when performing 1D orthogonal transform in the vertical and horizontal directions with respect to the prediction error 119 in the DC prediction mode (that is, the non-zero coefficient of the quantized transform coefficient 121). It is generated by pre-designing a common transformation basis so that the ratio becomes smaller. Therefore, according to the image coding apparatus according to the present embodiment, high conversion efficiency is achieved as compared with a case where fixed orthogonal transform such as DCT is uniformly applied to each prediction mode.

(Third embodiment)
The image encoding device according to the third embodiment differs from the image encoding devices according to the first and second embodiments described above in details of orthogonal transform and inverse orthogonal transform. In the following description, the same reference numerals are given to the same parts as those in the first embodiment or the second embodiment in the present embodiment, and different parts will be mainly described. An image decoding apparatus corresponding to the image encoding apparatus according to the present embodiment will be described in a sixth embodiment.

  The image encoding apparatus according to the present embodiment includes an orthogonal transform unit 102 illustrated in FIG. 19 instead of the orthogonal transform unit 102 illustrated in FIG. 19 includes a selection switch 1201, a vertical conversion unit 1202, a transposition unit 203, a selection switch 1204, and a horizontal conversion unit 1205. The vertical transform unit 1202 includes a 1D orthogonal transform unit F1206, a 1D orthogonal transform unit G1207, and a 1D orthogonal transform unit H1208. The horizontal transformation unit 1205 includes a 1D orthogonal transformation unit F1209, a 1D orthogonal transformation unit G1210, and a 1D orthogonal transformation unit H1211. Note that the order of the vertical conversion unit 1202 and the horizontal conversion unit 1205 is an example, and these may be reversed.

  The 1D orthogonal transform unit F1206 and the 1D orthogonal transform unit F1209 have a common function in that the input matrix is multiplied by the 1D transform matrix F. The 1D orthogonal transform unit G1207 and the 1D orthogonal transform unit G1210 have a common function in that the input matrix is multiplied by the 1D transform matrix G. The 1D orthogonal transform unit H1208 and the 1D orthogonal transform unit H1211 have a common function in that the input matrix is multiplied by the 1D transform matrix H.

Hereinafter, the 1D conversion matrix F, the 1D conversion matrix G, and the 1D conversion matrix H according to the present embodiment will be described.
As described above, the prediction error 119 tends to increase in absolute value as the distance from the reference pixel increases. This tendency is the same regardless of the prediction direction, but in the intra prediction mode, only the reference pixel group on the left adjacent line of the prediction target block or only the reference pixel group on the upper adjacent line is referenced (a copy of the reference pixel value or There is a prediction mode in which interpolation is performed from a reference pixel value), and there is also a prediction mode in which reference pixel groups on the left adjacent line and the upper adjacent line of the prediction target block are referred to. It can be said that there is a difference in the appearance of the above-described tendency between the prediction mode that refers only to the reference pixel group on one line and the prediction mode that refers to the reference pixel group on two lines. Therefore, in this embodiment, orthogonal transformation and inverse orthogonal transformation are performed by distinguishing between a prediction mode that refers only to a reference pixel group on one line and a prediction mode that refers to a reference pixel group on two lines. Specifically, for a prediction mode that refers to a reference pixel group on two lines, a 1D conversion matrix H described later is used. On the other hand, for the prediction mode in which only the reference pixel group on one line is referred to, the 1D conversion matrix F and the 1D conversion matrix G are adaptively used according to the presence or absence of the above-described tendency, as in the first embodiment. To do.

  Specifically, the 1D conversion matrix F can be generated by a design technique similar to the 1D conversion matrix A described above. That is, the 1D transformation matrix F excludes a prediction mode (for example, mode 4, mode 5 and mode 6 in FIG. 7A) that refers to a reference pixel group on two lines, and then designs the 1D transformation matrix A described above. Can be generated. Further, the 1D conversion matrix G can be generated by the same design method as the 1D conversion matrix B described above. Alternatively, the 1D transformation matrix G may be a matrix for DCT.

  The 1D transform matrix H has high coefficient density when performing 1D orthogonal transform in the vertical direction and the horizontal direction on the prediction error 119 of the prediction mode that refers to the reference pixel group on two lines (that is, quantization transform). The common conversion base can be generated in advance so that the ratio of the non-zero coefficient in the coefficient 121 is reduced).

  The image encoding apparatus according to the present embodiment includes an inverse orthogonal transform unit 105 illustrated in FIG. 20 instead of the inverse orthogonal transform unit 105 illustrated in FIG. The inverse orthogonal transform unit 105 in FIG. 20 includes a selection switch 1301, a vertical inverse transform unit 1302, a transposition unit 303, a selection switch 1304, and a horizontal inverse transform unit 1305. The vertical inverse transform unit 1302 includes a 1D inverse orthogonal transform unit F1306, a 1D inverse orthogonal transform unit G1307, and a 1D inverse orthogonal transform unit H1308. The horizontal inverse transform unit 1305 includes a 1D inverse orthogonal transform unit F1309, a 1D inverse orthogonal transform unit G1310, and a 1D inverse orthogonal transform unit H1311. Note that the order of the vertical inverse transform unit 1302 and the horizontal inverse transform unit 1305 is an example, and these may be reversed.

  The 1D inverse orthogonal transform unit F1306 and the 1D inverse orthogonal transform unit F1309 have a common function in that an input matrix is multiplied by a transposed matrix of the 1D transform matrix F. The 1D inverse orthogonal transform unit G1307 and the 1D inverse orthogonal transform unit G1310 have a common function in that the input matrix is multiplied by the transposed matrix of the 1D transform matrix G. The 1D inverse orthogonal transform unit H1308 and the 1D inverse orthogonal transform unit H1311 have a common function in that an input matrix is multiplied by a transposed matrix of the 1D transform matrix H.

Hereinafter, details of the 1D conversion matrix set information 129 according to the present embodiment generated by the 1D conversion matrix set unit 112 will be described.
The 1D transformation matrix set information 129 includes a vertical transformation index for selecting a transformation matrix used for vertical orthogonal transformation and vertical inverse orthogonal transformation, and a transformation matrix used for horizontal orthogonal transformation and horizontal inverse orthogonal transformation. The horizontal transformation index for selecting is directly or indirectly indicated. For example, the 1D transformation matrix set information 129 can be expressed by a transformation index (TransformIdx) illustrated in FIG. 21D. With reference to the table of FIG. 21D, a vertical transformation index (Vertical Transform Idx) and a horizontal transformation index (Horizontal Transform Idx) can be derived from the transformation index.

  As shown in FIG. 21B, if the vertical transform index is “0”, the 1D transform matrix F (1D_Transform_Matrix_F) or its transposed matrix is selected for the vertical orthogonal transform or the vertical inverse orthogonal transform. On the other hand, if the vertical transformation index is “1”, the aforementioned 1D transformation matrix G (1D_Transform_Matrix_G) or its transposed matrix is selected for vertical orthogonal transformation or vertical inverse orthogonal transformation. Furthermore, if the vertical transformation index is “2”, the 1D transformation matrix H (1D_transform_Matrix_H) or its transposed matrix is selected for vertical orthogonal transformation or vertical inverse orthogonal transformation.

  As shown in FIG. 21C, if the horizontal transformation index is “0”, the 1D transformation matrix F (1D_Transform_Matrix_F) or its transposed matrix is selected for horizontal orthogonal transformation or horizontal inverse orthogonal transformation. On the other hand, if the horizontal transformation index is “1”, the aforementioned 1D transformation matrix G (1D_Transform_Matrix_G) or its transposed matrix is selected for horizontal orthogonal transformation or horizontal inverse orthogonal transformation. Furthermore, if the horizontal transformation index is “2”, the 1D transformation matrix H (1D_Transform_Matrix_H) or its transposed matrix is selected for horizontal orthogonal transformation or horizontal inverse orthogonal transformation.

Further, FIG. 21A illustrates an index (IntraNxNPredModeIndex) of each (intra) prediction mode, its name (Name of IntraNxNPredMode), and a corresponding vertical conversion index and horizontal conversion index. In FIG. 21A, “NxN” represents the size of the prediction target block (N = 4, 8, 16, etc.). The size of the prediction target block can be expanded to “MxN” (that is, a rectangle other than a square).
Here, FIG. 21E illustrates an index of each prediction mode, a name thereof, and a corresponding conversion index obtained by integrating FIGS. 21A and 21D.

  The 1D transformation matrix set unit 112 detects a prediction mode index from the prediction mode information included in the prediction information 126, and generates corresponding 1D transformation matrix set information 129. Note that the various tables shown in FIGS. 21A, 21B, 21C, 21D, and 21E are examples, and the 1D conversion matrix set unit 112 uses the 1D conversion matrix set information without using some or all of these tables. 129 may be generated.

  For example, when TransformIdx indicates 0, it means that Vertical Transform index indicates 2, and Horizonal Transform index indicates 2. That is, it means that the 1D transformation matrix H is used for the vertical orthogonal transformation and the 1D transformation matrix H is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix H is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix H is used for horizontal inverse orthogonal transformation.

  When the TransformIdx indicates 1, it means that the Vertical Transform index indicates 0 and the Horizontal Transform index indicates 1. That is, it means that the 1D transformation matrix F is used for the vertical orthogonal transformation and the 1D transformation matrix G is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of 1D transformation matrix F is used for vertical inverse orthogonal transformation, and a transposed matrix of 1D transformation matrix G is used for horizontal inverse orthogonal transformation.

  When the TransformIdx indicates 2, it means that the Vertical Transform index indicates 1, and the Horizontal Transform index indicates 0. That is, it means that the 1D transformation matrix G is used for the vertical orthogonal transformation, and the 1D transformation matrix F is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of the 1D transformation matrix G is used for the vertical inverse orthogonal transformation, and a 1D transformation matrix F is used for the horizontal inverse orthogonal transformation.

  When TransformIdx indicates 3, it means that Vertical Transform index indicates 1, and Horizontal Transform index indicates 1. That is, it means that the 1D conversion matrix G is used for the vertical orthogonal transformation, and the 1D conversion matrix G is used for the horizontal orthogonal transformation. Further, it means that a transposed matrix of the 1D transformation matrix G is used for the vertical inverse orthogonal transformation, and a transposed matrix of the 1D transformation matrix G is used for the horizontal inverse orthogonal transformation.

  Here, when the prediction target block is a rectangular block represented by M × N, the block size for performing orthogonal transform may also be M × N.

  In the table shown in FIG. 21A, 1D transformation matrix set information 129 is assigned in consideration of the tendency of each intra prediction mode described above. That is, 2 is assigned to both the vertical transform index and the horizontal transform index in the prediction mode for referencing the reference pixel group on two lines. Therefore, for the prediction mode that refers to the reference pixel group on two lines, the orthogonal transformation or inverse orthogonal transformation in the vertical direction and the horizontal direction is performed using the 1D transformation matrix H or its transpose matrix, and high transformation efficiency is achieved. The

  For prediction modes other than the prediction mode that refers to the reference pixel group on two lines, if the tendency is shown in the vertical direction of the prediction error, 0 is in the vertical transform index, and if the tendency is shown in the horizontal direction, the horizonal transform is shown. 0 is assigned to the index. On the other hand, 1 is assigned to each direction that does not show the above tendency. Regarding prediction modes other than the prediction mode that refers to the reference pixel group on two lines, the vertical direction and horizontal direction of each prediction mode are classified into two classes according to the presence or absence of the above-mentioned tendency, and adaptive in each of the vertical direction and the horizontal direction. By applying the 1D conversion matrix F or the 1D conversion matrix G to the above, high conversion efficiency is achieved.

  As described above, the image encoding apparatus according to the present embodiment uses the tendency of intra prediction that the prediction accuracy decreases as the distance from the reference pixel increases, as in the first embodiment. The prediction mode is distinguished by the number of lines of the reference pixel group, and orthogonal transformation and inverse orthogonal transformation are applied. This image encoding apparatus classifies the vertical direction and the horizontal direction into two classes according to the presence or absence of the above-described tendency with respect to the prediction mode except the prediction mode that refers to the reference pixel group on two lines, and each of the vertical direction and the horizontal direction. The 1D transformation matrix F or the 1D transformation matrix G is applied adaptively. On the other hand, this image encoding apparatus applies the 1D transformation matrix H to each prediction mode that refers to reference pixel groups on two lines. The 1D transformation matrix F has a coefficient density when performing 1D orthogonal transformation in a direction (vertical direction or horizontal direction) orthogonal to the line of the reference pixel group with respect to each prediction mode in which only the reference pixel group on one line is referred to. It is generated by designing a common transform base in advance so as to be high (that is, the ratio of non-zero coefficients in the quantized transform coefficient 121 is small). On the other hand, the 1D transformation matrix G is generated by designing a general-purpose transformation matrix having no such property. Further, the 1D transform matrix H has a high coefficient density when performing 1D orthogonal transform in the vertical direction and the horizontal direction with respect to the prediction error 119 of each prediction mode referring to the reference pixel group on two lines (that is, It is generated by designing a common transform base in advance so that the ratio of the non-zero coefficient in the quantized transform coefficient 121 becomes smaller. Therefore, according to the image coding apparatus according to the present embodiment, high conversion efficiency is achieved as compared with a case where fixed orthogonal transform such as DCT is uniformly applied to each prediction mode.

  In the first to third embodiments, two or three types of 1D conversion matrices are prepared, and 1D for vertical conversion (or vertical reverse conversion) and horizontal conversion (or horizontal reverse conversion) according to the prediction mode. Select a transformation matrix. However, the above-described two or three types of 1D transformation matrices are examples, and it is possible to prepare more transformation matrices to improve the encoding efficiency. For example, it is possible to prepare four types of 1D conversion matrices by combining the second embodiment and the third embodiment. However, since additional hardware is required as the types of transformation matrices to be prepared increase, it is desirable to consider the balance between the disadvantages associated with the increase in types of transformation matrices and the coding efficiency.

(Fourth embodiment)
The fourth embodiment relates to an image decoding apparatus. The image coding apparatus corresponding to the image decoding apparatus according to the present embodiment is as described in the first embodiment. That is, the image decoding apparatus according to the present embodiment decodes encoded data generated by the image encoding apparatus according to the first embodiment, for example.

  As shown in FIG. 22, the image decoding apparatus according to the present embodiment includes an input buffer 401, an entropy decoding unit 402, a coefficient order control unit 403, an inverse quantization unit 404, an inverse orthogonal transform unit 405, an addition unit 406, A reference image memory 407, an intra prediction unit 408, an inter prediction unit 409, a selection switch 410, a 1D transformation matrix setting unit 411, and an output buffer 412 are included.

  The image decoding apparatus in FIG. 22 decodes the encoded data 414 stored in the input buffer 401, stores the decoded image 419 in the output buffer 412, and outputs it as an output image 425. The encoded data 414 is output from, for example, the image encoding device in FIG. 1 and the like, and is temporarily stored in the input buffer 401 via a storage system or a transmission system (not shown).

  The entropy decoding unit 402 performs decoding based on the syntax for each frame or field for decoding the encoded data 414. The entropy decoding unit 402 sequentially entropy-decodes the code string of each syntax, and reproduces the encoding parameters of the encoding target block such as the prediction information 424 including the prediction mode information 421 and the quantized transform coefficient string 415. The encoding parameter is a parameter necessary for decoding such as prediction information 424, information on transform coefficients, information on quantization, and the like. The quantized transform coefficient sequence 415 is input to the coefficient order control unit 403. Similarly, the prediction mode information 421 included in the prediction information 424 is also input to the coefficient order control unit 403. The prediction information 424 is input to the 1D conversion matrix setting unit 411 and the selection switch 410.

  The coefficient order control unit 403 converts the quantized transform coefficient sequence 415 that is a one-dimensional representation into a quantized transform coefficient 416 that is a two-dimensional representation, and inputs the quantized transform coefficient sequence 415 to the inverse quantization unit 404. Details of the coefficient control unit 403 will be described later.

  The inverse quantization unit 404 performs inverse quantization on the quantized transform coefficient 416 from the coefficient order control unit 403 to obtain a restored transform coefficient 417. Specifically, the inverse quantization unit 404 performs inverse quantization according to the information related to quantization decoded by the entropy decoding unit 402. The inverse quantization unit 404 inputs the restored transform coefficient 417 to the inverse orthogonal transform unit 405.

  The inverse orthogonal transform unit 405 performs an inverse orthogonal transform corresponding to the orthogonal transform performed on the encoding side on the restored transform coefficient 417 from the inverse quantization unit 404 to obtain a restored prediction error 418. The inverse orthogonal transform unit 405 inputs the restoration prediction error 418 to the addition unit 406.

  Specifically, since the inverse orthogonal transform unit 405 according to the present embodiment is substantially the same as or similar to the inverse orthogonal transform unit 105 of FIG. 3, detailed description thereof is omitted. In particular, the inverse orthogonal transform unit 405 according to the present embodiment uses the 1D transform matrix A and the 1D transform matrix B common to the inverse orthogonal transform unit 105 of FIG. Note that the restored transform coefficient 122, 1D transform matrix set information 129, and the restored prediction error 123 in FIG. 3 respectively correspond to the restored transform coefficient 417, 1D transform matrix set information 422, and the restored prediction error signal 418 in the present embodiment. .

  The adding unit 406 adds the restored prediction error 418 and the corresponding predicted image 423 to generate a decoded image 419. The decoded image 419 is temporarily stored in the output buffer 412 for the output image 425 and also stored in the reference image memory 407 for the reference image 420. The decoded image 419 stored in the reference image memory 407 is referred to by the intra prediction unit 408 and the inter prediction unit 409 as the reference image 420 in units of frames or fields as necessary. The decoded image 419 temporarily stored in the output buffer 412 is output according to the output timing managed by the decoding control unit 413.

  The intra prediction unit 408, the inter prediction unit 409, and the selection switch 410 are substantially the same as or similar to the intra prediction unit 108, the inter prediction unit 109, and the selection switch 111 in FIG. The decoding control unit 413 controls each element of the image decoding device in FIG. Specifically, the decoding control unit 413 performs various controls for the decoding process including the above-described operation.

  The 1D transform matrix set unit 411 generates 1D transform matrix set information 422 based on the prediction mode information included in the prediction information 424 from the entropy decoding unit 402 and inputs the 1D transform matrix set information 422 to the inverse orthogonal transform unit 405.

Specifically, the 1D transformation matrix set unit 411 according to the present embodiment is substantially the same as or similar to the 1D transformation matrix set unit 112 according to the first embodiment, and thus detailed description thereof is omitted. That is, the 1D conversion matrix set unit 411 according to the present embodiment generates 1D conversion matrix set information 422 using, for example, the tables of FIGS. 4A, 4B, 4C, 4D, and 4E. Note that the prediction information 126 and the 1D transformation matrix set information 129 in the first embodiment correspond to the prediction information 424 and the 1D transformation matrix set information 422 in the present embodiment, respectively.
The image decoding apparatus in FIG. 22 uses the same or similar syntax as the syntax described with reference to FIGS. 11, 12, 13, and 14, and thus detailed description thereof is omitted.

Details of the coefficient order control unit 403 will be described below.
The coefficient order control unit 403 arranges the elements of the quantized transform coefficient sequence 415 that is a one-dimensional representation according to a predetermined order (that is, the order that corresponds to the encoding side), and thereby the quantized transform that is a two-dimensional representation. Convert to coefficient 416. As an example, if the common 2D-1D conversion is performed regardless of the prediction mode on the encoding side, the coefficient order control unit 403 can perform the common 1D-2D conversion regardless of the prediction mode. Specifically, the coefficient control unit 403 includes the H.264 standard. Similar to H.264, reverse zigzag scanning can be used. Inverse zigzag scanning is 1D-2D conversion corresponding to the above-described zigzag scanning. As another example, if individual 2D-1D conversion for each prediction mode is performed on the encoding side, the coefficient order control unit 403 Can also perform individual 1D-2D conversion for each prediction mode. The coefficient order control unit 403 that performs such an operation is illustrated in FIG. 23A. The coefficient order control unit 403 includes a selection switch 1001 and individual 1D-2D conversion units 1002, ..., 1010 for each of nine types of prediction modes. The selection switch 1001 converts the quantized transform coefficient sequence 415 according to the prediction mode information (for example, the prediction mode index in FIG. 4A) included in the prediction information 424, and the 1D-2D conversion unit (1002,. , 1010). For example, if the prediction mode index is 0, the selection switch 1001 guides the quantized transform coefficient sequence 415 to the 1D-2D transform unit 1002. In FIG. 23A, each prediction mode and the 1D-2D conversion unit have a one-to-one correspondence, and the quantized transform coefficient sequence 415 is guided to one 1D-2D transform unit corresponding to the prediction mode, and the quantized transform is performed. Converted to a coefficient 416.

  As yet another example, if the scan order in the 2D-1D conversion is dynamically updated on the encoding side, the coefficient order control unit 403 also causes the scan order in the 1D-2D conversion to correspond to the encoding side. It may be updated dynamically. The coefficient order control unit 403 that performs such an operation is illustrated in FIG. 23B. The coefficient order control unit 403 includes a selection switch 1001, individual 1D-2D conversion units 1002,..., 1010 for each of nine types of prediction modes, an occurrence frequency counting unit 1011, and a coefficient order update unit 1012. Including. The selection switch 1001 is as described with reference to FIG. 23A. The individual 1D-2D conversion units 1002,..., 1010 for each of the nine types of prediction modes differ from FIG. 23A in that the scan order is updated by the coefficient order update unit 1012.

  The occurrence frequency counting unit 1011 creates a histogram of the number of occurrences of non-zero coefficients in each element of the quantized transform coefficient 416 for each prediction mode. The occurrence frequency counting unit 1011 inputs the created histogram 1013 to the coefficient order updating unit 1012.

  The coefficient order update unit 1012 updates the coefficient order based on the histogram 1013 at a predetermined timing. The timing is, for example, the timing when the decoding process of the coding tree unit is completed, the timing when the decoding process for one line in the coding tree unit is completed, or the like.

  Specifically, the coefficient order update unit 1012 refers to the histogram 1013 and updates the coefficient order for a prediction mode having an element in which the number of occurrences of non-zero coefficients is counted above a threshold. For example, the coefficient order update unit 1012 updates the prediction mode having an element in which the occurrence of a non-zero coefficient is counted 16 times or more. By providing a threshold value for the number of occurrences, the coefficient order is updated globally, so that it is difficult to converge to a local optimum solution.

  The coefficient order update unit 1012 sorts the elements in descending order of the occurrence frequency of the non-zero coefficient with respect to the prediction mode to be updated. Sorting can be realized by existing algorithms such as bubble sort and quick sort. Then, the coefficient order update unit 1012 inputs coefficient order update information 1014 indicating the order of the sorted elements to the 1D-2D conversion unit corresponding to the prediction mode to be updated.

  When the coefficient order update information 1014 is input, the 1D-2D conversion unit performs 1D-2D conversion according to the updated scan order. Note that when the scan order is dynamically updated, it is necessary to determine in advance the initial scan order corresponding to the encoding side of each 1D-2D conversion unit.

  For simplification, H.C. H.264 has been described as an example of nine prediction modes. However, when the prediction mode is expanded to 17 types, 33 types, etc., a 1D-2D conversion unit corresponding to each expanded prediction mode is added. Then, individual 1D-2D conversion for each prediction mode can be performed.

  As described above, the image decoding apparatus according to the present embodiment has the same or similar inverse orthogonal transform unit as the image encoding apparatus according to the first embodiment described above. Therefore, according to the image decoding apparatus according to the present embodiment, the same or similar effects as those of the image encoding apparatus according to the first embodiment described above can be obtained.

(Fifth embodiment)
The image decoding apparatus according to the fifth embodiment differs from the image decoding apparatus according to the above-described fourth embodiment in the details of inverse orthogonal transform. In the following description, in this embodiment, the same parts as those in the fourth embodiment are denoted by the same reference numerals, and different parts will be mainly described. The image coding apparatus corresponding to the image decoding apparatus according to the present embodiment is as described in the second embodiment.

  Since the inverse orthogonal transform unit 405 according to the present embodiment is substantially the same as or similar to the inverse orthogonal transform unit 105 of FIG. 17, detailed description thereof is omitted. In particular, the inverse orthogonal transform unit 405 according to the present embodiment uses the 1D transform matrix C, the 1D transform matrix D, and the 1D transform matrix E common to the inverse orthogonal transform unit 105 in FIG. Note that the restored transform coefficient 122, 1D transform matrix set information 129, and the restored prediction error 123 in FIG. 17 correspond to the restored transform coefficient 417, 1D transform matrix set information 422, and the restored prediction error signal 418 in the present embodiment, respectively. .

  Since the 1D transformation matrix set unit 411 according to the present embodiment is substantially the same as or similar to the 1D transformation matrix set unit 112 according to the second embodiment, detailed description thereof is omitted. That is, the 1D conversion matrix set unit 411 according to the present embodiment generates 1D conversion matrix set information 422 using, for example, the tables of FIGS. 18A, 18B, 18C, 18D, and 18E. Note that the prediction information 126 and the 1D transformation matrix set information 129 in the second embodiment correspond to the prediction information 424 and the 1D transformation matrix set information 422 in the present embodiment, respectively.

  As described above, the image decoding apparatus according to the present embodiment has the same or similar inverse orthogonal transform unit as the image encoding apparatus according to the second embodiment described above. Therefore, according to the image decoding apparatus according to the present embodiment, the same or similar effects as those of the image encoding apparatus according to the second embodiment described above can be obtained.

(Sixth embodiment)
The image decoding device according to the sixth embodiment differs from the image decoding devices according to the fourth embodiment and the fifth embodiment described above in details of inverse orthogonal transform. In the following description, in this embodiment, the same parts as those in the fourth or fifth embodiment are denoted by the same reference numerals, and different parts will be mainly described. The image encoding device corresponding to the image decoding device according to the present embodiment is as described in the third embodiment.

  Since the inverse orthogonal transform unit 405 according to the present embodiment is substantially the same as or similar to the inverse orthogonal transform unit 105 of FIG. 20, detailed description thereof is omitted. In particular, the inverse orthogonal transform unit 405 according to the present embodiment uses the 1D transform matrix F, the 1D transform matrix G, and the 1D transform matrix H that are common to the inverse orthogonal transform unit 105 in FIG. Note that the restored transform coefficient 122, 1D transform matrix set information 129, and the restored prediction error 123 in FIG. 20 respectively correspond to the restored transform coefficient 417, 1D transform matrix set information 422, and the restored prediction error signal 418 in the present embodiment. .

  Since the 1D conversion matrix set unit 411 according to the present embodiment is substantially the same as or similar to the 1D conversion matrix set unit 112 according to the third embodiment, detailed description thereof is omitted. That is, the 1D conversion matrix set unit 411 according to the present embodiment generates 1D conversion matrix set information 422 using, for example, the tables of FIGS. 21A, 21B, 21C, 21D, and 21E. Note that the prediction information 126 and the 1D transformation matrix set information 129 in the third embodiment correspond to the prediction information 424 and the 1D transformation matrix set information 422 in the present embodiment, respectively.

  As described above, the image decoding apparatus according to this embodiment has the same or similar inverse orthogonal transform unit as that of the image encoding apparatus according to the third embodiment described above. Therefore, according to the image decoding apparatus according to the present embodiment, the same or similar effects as those of the image encoding apparatus according to the third embodiment described above can be obtained.

  In the fourth to sixth embodiments, two or three types of 1D conversion matrices are prepared, and a 1D conversion matrix for vertical inverse transformation and horizontal inverse transformation is selected according to the prediction mode. However, the above-described two or three types of 1D transformation matrices are examples, and it is possible to prepare more transformation matrices to improve the encoding efficiency. For example, it is possible to prepare four types of 1D conversion matrices by combining the fifth embodiment and the sixth embodiment. However, since additional hardware is required as the types of transformation matrices to be prepared increase, it is desirable to consider the balance between the disadvantages associated with the increase in types of transformation matrices and the coding efficiency.

Hereinafter, modifications of each embodiment will be listed and introduced.
In the first to sixth embodiments, an example is described in which a frame is divided into rectangular blocks of 16 × 16 pixel size and the like, and encoding / decoding is performed in order from the upper left block to the lower right side of the screen ( (See FIG. 6A). However, the encoding order and the decoding order are not limited to this example. For example, encoding and decoding may be performed sequentially from the lower right to the upper left, or encoding and decoding may be performed so as to draw a spiral from the center of the screen toward the screen end. Furthermore, encoding and decoding may be performed sequentially from the upper right to the lower left, or encoding and decoding may be performed so as to draw a spiral from the screen end toward the center of the screen.

  In the first to sixth embodiments, the description has been given by exemplifying the prediction target block size such as the 4 × 4 pixel block, the 8 × 8 pixel block, and the 16 × 16 pixel block, but the prediction target block is a uniform block. It does not have to be a shape. For example, the prediction target block size may be a 16 × 8 pixel block, an 8 × 16 pixel block, an 8 × 4 pixel block, a 4 × 8 pixel block, or the like. Also, it is not necessary to unify all the block sizes within one coding tree unit, and a plurality of different block sizes may be mixed. When a plurality of different block sizes are mixed in one coding tree unit, the amount of codes for encoding or decoding the division information increases as the number of divisions increases. Therefore, it is desirable to select the block size in consideration of the balance between the code amount of the division information and the quality of the locally decoded image or the decoded image.

  In the first to sixth embodiments, for the sake of simplification, a comprehensive description of the color signal component is described without distinguishing between the luminance signal and the color difference signal. However, when the prediction process is different between the luminance signal and the color difference signal, the same or different prediction methods may be used. If different prediction methods are used between the luminance signal and the chrominance signal, the prediction method selected for the chrominance signal can be encoded or decoded in the same manner as the luminance signal.

  In the first to sixth embodiments, for the sake of simplification, a comprehensive description of the color signal component is described without distinguishing between the luminance signal and the color difference signal. However, when the orthogonal transformation process is different between the luminance signal and the color difference signal, the same or different orthogonal transformation methods may be used. If different orthogonal transformation methods are used between the luminance signal and the color difference signal, the orthogonal transformation method selected for the color difference signal can be encoded or decoded in the same manner as the luminance signal.

  As described above, each embodiment realizes highly efficient orthogonal transformation and inverse orthogonal transformation while alleviating the difficulty in hardware implementation and software implementation. Therefore, according to each embodiment, the encoding efficiency is improved, and the subjective image quality is also improved.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  For example, it is possible to provide a program that realizes the processing of each of the above embodiments by storing it in a computer-readable storage medium. The storage medium may be a computer-readable storage medium such as a magnetic disk, optical disk (CD-ROM, CD-R, DVD, etc.), magneto-optical disk (MO, etc.), semiconductor memory, etc. For example, the storage format may be any form.

  Further, the program for realizing the processing of each of the above embodiments may be stored on a computer (server) connected to a network such as the Internet and downloaded to the computer (client) via the network.

DESCRIPTION OF SYMBOLS 101 ... Subtraction part 102 ... Orthogonal transformation part 103 ... Quantization part 104 ... Dequantization part 105 ... Inverse orthogonal transformation part 106 ... Addition part 107 ... Reference image memory 108 ... Intra prediction unit 109 ... Inter prediction unit 110 ... Prediction selection unit 111 ... Selection switch 112 ... 1D transform matrix set unit 113 ... Coefficient order control unit 114 ... Entropy coding Unit 115 ... output buffer 116 ... encoding control unit 117 ... quantized transform coefficient sequence 118 ... input image 119 ... prediction error 120 ... transform coefficient 121 ... quantized transform coefficient 122: Restoration conversion coefficient 123: Restoration prediction error 124 ... Local decoded image 125 ... Reference image 126 ... Prediction information 127 ... Prediction image 129 ... 1D variation Matrix set information 130 ... Encoded data 201, 204, 801, 804, 1101, 1104, 1201, 1204 ... Selection switch 202, 802, 1102, 1202 ... Vertical conversion unit 206, ..., 209 , 806,..., 811, 1206,..., 1211... 1D orthogonal transform unit 203, 1103, transpose unit 205, 805, 1105, 1205, horizontal transform unit 301, 304, 901,. 904, 1301, 1304 ... selection switch 302, 902, 1302 ... vertical reverse conversion unit 303 ... transposition unit 305, 905, 1305 ... horizontal reverse conversion unit 306, ..., 309, 906 ..., 911, 1306, ..., 1311 ... 1D inverse orthogonal transform unit 401 ... input buffer 402 ... entry P-decoding unit 403 ... Coefficient order control unit 404 ... Inverse quantization unit 405 ... Inverse orthogonal transformation unit 406 ... Addition unit 407 ... Reference image memory 408 ... Intra prediction unit 409 .. Inter prediction unit 410 ... selection switch 411 ... 1D transform matrix set unit 412 ... output buffer 413 ... decoding control unit 414 ... encoded data 415 ... quantized transform coefficient sequence 416: quantized transform coefficient 417: restored transform coefficient 418 ... restored prediction error 419 ... decoded image 420 ... reference image 421 ... prediction mode information 422 ... 1D transform matrix set information 423 ... Predicted image 424 ... Predicted information 425 ... Output image 501 ... Selection switch 502, ..., 510 ... 2D-1D converter 511 ... Occurrence frequency counting unit 512... Coefficient order update unit 513... Histogram 514... Coefficient order update information 700 ... Syntax 701 ... High level syntax 702 ... Slice level syntax 703 ... Coding tree Level syntax 704 ... Sequence parameter set syntax 705 ... Picture parameter set syntax 706 ... Slice header syntax 707 ... Slice data syntax 708 ... Coding tree unit syntax 709 ... Prediction unit syntax 710 ··· Transform unit syntax 1001 ··· Selection switch 1002, ···, 1010 ··· 1D-2D converter 1011 ··· Frequency counter 1012 ··· Coefficient order update section 1013 ... histogram 1014 ... coefficient order update information

Claims (15)

An accumulator that accumulates encoded data output from another device via a communication line;
A decoding unit for decoding transform coefficients to be decoded from the stored encoded data;
A set unit configured to set a combination of a vertical transformation matrix and a horizontal transformation matrix corresponding to the prediction mode to be decoded using only two types of matrices based on a predetermined relationship according to a predicted image generation method; ,
Using the set vertical transformation matrix and the horizontal transformation matrix, an inverse transformation unit that obtains a prediction error by performing vertical inverse transformation and horizontal inverse transformation on the transformation coefficient;
An adder that generates a decoded image based on the prediction error, and
The combination is a combination between a first transformation matrix, there is a combination between different second transformation matrix from the first transformation matrix,
The combination of the second transformation matrices is set to a plurality of intra prediction modes respectively corresponding to Diagonal down right, Vertical right and Horizontal down directions.
An image decoding apparatus characterized by that. Storing the encoded data output from the other device via the communication line in the storage unit;
Decoding transform coefficients to be decoded from the stored encoded data;
Setting a combination of a vertical transformation matrix and a horizontal transformation matrix corresponding to the prediction mode to be decoded using only two types of matrices based on a predetermined relationship according to a predicted image generation method;
Using the set vertical transformation matrix and the horizontal transformation matrix to obtain a prediction error by performing vertical inverse transformation and horizontal inverse transformation on the transformation coefficient;
Generating a decoded image based on the prediction error, and
The combination is a combination between a first transformation matrix, there is a combination between different second transformation matrix from the first transformation matrix,
The combination of the second transformation matrices is set to a plurality of intra prediction modes respectively corresponding to Diagonal down right, Vertical right and Horizontal down directions.
An image decoding method characterized by the above. Computer
Storage means for storing encoded data output from other devices via a communication line in a storage unit;
Means for decoding transform coefficients to be decoded from the stored encoded data;
Means for setting a combination of a vertical transformation matrix and a horizontal transformation matrix corresponding to the prediction mode to be decoded using only two types of matrices based on a predetermined relationship according to a predicted image generation method;
Means for obtaining a prediction error by performing vertical inverse transformation and horizontal inverse transformation on the transformation coefficient using the set vertical transformation matrix and horizontal transformation matrix;
Function as a means for generating a decoded image based on the prediction error,
The combination is a combination between a first transformation matrix, there is a combination between different second transformation matrix from the first transformation matrix,
The combination of the second transformation matrices is set to a plurality of intra prediction modes respectively corresponding to Diagonal down right, Vertical right and Horizontal down directions.
An image decoding program characterized by the above. An accumulator that accumulates encoded data output from another device via a communication line;
A decoding unit that decodes, from the accumulated encoded data, a transform coefficient of a decoding target block encoded by a prediction process according to a predetermined prediction direction or a prediction process without a prediction direction;
A prediction unit that generates a prediction image by prediction processing of the decoding target block;
A first conversion process that uses only two types of matrixes for the conversion coefficient and uses the first conversion matrix for vertical inverse conversion and horizontal inverse conversion, and a second conversion process different from the first conversion matrix. Inverse transformation that obtains a prediction error by applying any one of the plurality of transformation processes including a second transformation process that uses the transformation matrix for vertical inverse transformation and horizontal inverse transformation according to the prediction image generation method. And
An adder that generates a decoded image based on the prediction image and the prediction error;
The second transform process is applied to transform coefficients of a plurality of decoding target blocks encoded by prediction processes according to Diagonal down right, Vertical right, and Horizontal down directions, respectively.
Image decoding device. Storing the encoded data output from the other device via the communication line in the storage unit;
Decoding the transform coefficient of the decoding target block encoded by the prediction process according to a predetermined prediction direction or the prediction process having no prediction direction from the accumulated encoded data;
Generating a prediction image by prediction processing of the decoding target block;
A first conversion process that uses only two types of matrixes for the conversion coefficient and uses the first conversion matrix for vertical inverse conversion and horizontal inverse conversion, and a second conversion process different from the first conversion matrix. Obtaining a prediction error by applying any one of the plurality of conversion processes including the second conversion process using the conversion matrix for the vertical reverse conversion and the horizontal reverse conversion, corresponding to the prediction image generation method. ,
Generating a decoded image based on the prediction image and the prediction error, and
The second transform process is applied to transform coefficients of a plurality of decoding target blocks encoded by prediction processes according to Diagonal down right, Vertical right, and Horizontal down directions, respectively.
Image decoding method. Storage means for storing encoded data output from another device via a communication line in a storage unit,
Means for decoding transform coefficients of a decoding target block encoded by a prediction process according to a predetermined prediction direction or a prediction process having no prediction direction from the stored encoded data;
Means for generating a prediction image by prediction processing of the decoding target block;
A first conversion process that uses only two types of matrixes for the conversion coefficient and uses the first conversion matrix for vertical inverse conversion and horizontal inverse conversion, and a second conversion process different from the first conversion matrix. Means for obtaining a prediction error by applying any one of the conversion processes corresponding to the prediction image generation method among a plurality of conversion processes including a second conversion process using a conversion matrix for a vertical inverse transform and a horizontal inverse transform;
Function as means for generating a decoded image based on the prediction image and the prediction error;
The second transform process is applied to transform coefficients of a plurality of decoding target blocks encoded by prediction processes according to Diagonal down right, Vertical right, and Horizontal down directions, respectively.
Image decoding program.   Means for obtaining a two-dimensional transform coefficient from the one-dimensional transform coefficient to be decoded by inverse zigzag scanning and generating a decoded image based on a prediction error obtained by transforming the two-dimensional transform coefficient; The image decoding device according to claim 1 or 4.   The method further comprises obtaining a two-dimensional transform coefficient from the one-dimensional transform coefficient to be decoded by inverse zigzag scanning, and generating a decoded image based on a prediction error obtained by transforming the two-dimensional transform coefficient. The image decoding method according to claim 2 or 5. Computer
It further functions as means for obtaining a two-dimensional transform coefficient from the one-dimensional transform coefficient to be decoded by inverse zigzag scanning, and generating a decoded image based on a prediction error obtained by transforming the two-dimensional transform coefficient. ,
The image decoding program according to claim 3 or 6. The image decoding device according to claim 1, wherein the inverse transform unit is implemented by hardware. The image decoding device according to claim 1, wherein the inverse transform unit is software-implemented. A first program and a second program for causing a computer to execute a predetermined operation,
The first program is an image decoding program according to any one of claims 3, 6, and 9.
The second program is an image encoding program for causing a computer to perform an encoding process on an input video signal and store it in a storage medium.
A first program and a second program.   The encoding process of the image encoding program is a process of performing orthogonal transform processing on a prediction error signal obtained by prediction processing on a block of an input video signal to obtain transform coefficients, and rearranging the transform coefficients using a zigzag scan The first program and the second program according to claim 12, comprising: Network connection means for causing a computer to download the program stored in the storage unit via a network;
The program is an image decoding program according to any one of claims 3, 6, and 9, or a first program according to any one of claims 12 and 13. Including a second program,
Server system. A download control method in a server system having a connection means for connecting to a network,
The image decoding program according to any one of claims 3, 6, and 9, or the first program and the second program according to any one of claims 12 and 13. A download control method for causing a computer to download a program including

Images