JP7291471B2 - Image encoding device, image decoding device, and program - Google Patents

Description

The present invention relates to an image encoding device, an image decoding device, and a program.

A conventional image coding apparatus that encodes each encoding target block obtained by dividing an image into blocks performs motion compensation prediction using a plurality of reference images to generate a prediction image corresponding to the encoding target block. Then, orthogonal transform is performed on the prediction residual representing the pixel-by-pixel difference between the encoding target block and the predicted image to generate orthogonal transform coefficients, the orthogonal transform coefficients are quantized, and the quantized Entropy coding is performed on the orthogonal transform coefficients.

Specifically, entropy coding includes a process called serialization in which two-dimensionally arranged orthogonal transform coefficients are read out in a predetermined scan order and converted into a one-dimensional orthogonal transform coefficient sequence. This serialization refers to a process of sequentially encoding a one-dimensional orthogonal transform coefficient sequence from the leading orthogonal transform coefficient.

It is generally known that due to orthogonal transform in image coding, the energy of the orthogonal transform coefficients is concentrated in the low frequency range, and the energy after quantization (that is, the value of the quantized orthogonal transform coefficients) becomes zero in the high frequency range. It is For this reason, conventionally, the quantized orthogonal transform coefficients are read out in scanning order from low frequency to high frequency, and an end flag is given to the last significant coefficient (non-zero coefficient), so that only the significant coefficient is efficiently processed. It is encoded (see, for example, Non-Patent Document 1).

Recommendation ITU-T H. 265, (12/2016), "High efficiency video coding", International Telecommunications Union

In HEVC (see Non-Patent Document 1), in addition to a mode in which orthogonal transform processing is performed, a transform skip mode in which orthogonal transform processing is not performed can be applied. When the transform skip mode is applied, the prediction residual is not orthogonally transformed, so energy cannot be expected to be concentrated in the low frequency range.

For this reason, if entropy coding is performed in the transform skip mode by the same method as in the orthogonal transform mode, efficient entropy coding cannot be performed, resulting in a decrease in coding efficiency.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image encoding device, an image decoding device, and a program capable of improving encoding efficiency when motion compensation prediction is performed using a plurality of reference images.

An image encoding device according to a first feature is an image encoding device that encodes an encoding target block in block units obtained by dividing an input image, wherein the encoding target block is subjected to orthogonal transform processing. A motion compensation prediction unit that generates a prediction image corresponding to the encoding target block by performing motion compensation prediction using a plurality of reference images in a skip transform skip mode, and a similarity between the plurality of reference images. and a quantized prediction residual obtained by quantizing a prediction residual representing a pixel-by-pixel difference between the encoding target block and the predicted image or the quantized prediction residual calculating a degree of similarity representing a correlation between the restored prediction residual obtained by inversely quantizing the difference and the degree of similarity evaluated by the evaluation unit, and determining whether the degree of similarity is greater than or equal to a threshold; and an encoding unit that encodes the quantized prediction residuals in order according to the similarities evaluated by the evaluation unit when the determination unit determines that the similarity is equal to or greater than the threshold. The gist is to provide

An image decoding device according to a second feature is an image decoding device that decodes a decoding target block in units of blocks from encoded data, wherein in a transform skip mode for skipping inverse orthogonal transform processing of the decoding target block, a plurality of A motion compensation prediction unit that generates a prediction image corresponding to the block to be decoded by performing motion compensation prediction using a reference image, an evaluation unit that evaluates the similarity between the plurality of reference images on a pixel-by-pixel basis; a decoding unit that decodes the encoded data to obtain a quantized prediction residual representing a pixel-by-pixel difference between the decoding target block and the predicted image; and the quantization obtained by the decoding unit. calculating a degree of similarity representing a correlation between a prediction residual or a restored prediction residual obtained by inversely quantizing the quantized prediction residual and the degree of similarity evaluated by the evaluation unit, and calculating the degree of similarity; a determination unit that determines whether the similarity is equal to or greater than the threshold, and the quantization in order according to the similarity evaluated by the evaluation unit when the determination unit determines that the similarity is equal to or greater than the threshold and a rearrangement unit that rearranges and outputs the prediction residuals.

A program according to a third feature is summarized to cause a computer to function as the image encoding device according to the first feature.

A gist of a program according to a fourth feature is to cause a computer to function as the image decoding device according to the second feature.

Advantageous Effects of Invention According to the present invention, it is possible to provide an image encoding device, an image decoding device, and a program capable of improving encoding efficiency when motion compensation prediction is performed using a plurality of reference images.

1 is a diagram showing the configuration of an image encoding device according to an embodiment; FIG. It is a figure which shows the structure of the image decoding apparatus which concerns on embodiment. It is a figure which shows an example of the motion compensation prediction which concerns on embodiment. It is a figure which shows an example of the prediction image produced|generated by the motion compensation prediction which concerns on embodiment. It is a figure which shows an example of a structure of the evaluation part which concerns on embodiment. It is a figure which shows an example of a structure of the entropy coding part which concerns on embodiment. It is a figure which shows the example of generation|occurrence|production of the probability index which concerns on embodiment. It is a figure which shows an example of rearrangement of the prediction residual which concerns on embodiment. It is a figure which shows an example of a structure of the entropy code decoding part which concerns on embodiment. It is a figure which shows the processing flow in the image coding apparatus which concerns on embodiment. It is a figure which shows the processing flow in the image decoding apparatus which concerns on embodiment. It is a figure which shows an example of overlapped motion compensation based on the modification 1 of embodiment. It is a figure which shows the structure of the image coding apparatus which concerns on the modification 3 of embodiment. It is a figure which shows an example of operation|movement of the rearrangement part which concerns on the modification 3 of embodiment. It is a figure which shows the structure of the image decoding apparatus based on the modification 3 of embodiment. It is a figure which shows an example of operation|movement of the rearrangement part based on the modification 4 of embodiment.

An image encoding device and an image decoding device according to embodiments of the present invention will be described below with reference to the drawings. An image encoding device and an image decoding device according to the embodiments of the present invention encode and decode moving images represented by MPEG (Moving Picture Experts Group). In addition, in the description of the drawings, the same or similar reference numerals are given to the parts having the same or similar configurations.

<1. Configuration of image encoding device>
FIG. 1 is a diagram showing the configuration of an image encoding device 1A according to the embodiment. As shown in FIG. 1, the image coding device 1A includes a block division unit 100, a subtraction unit 101, a transform unit 102a, a quantization unit 102b, a switching unit 102c, an entropy coding unit 103A, an inverse quantization unit transforming unit 104a, inverse transforming unit 104b, switching unit 104c, synthesizing unit 105, intra prediction unit 106, loop filter 107, frame memory 108, motion compensation prediction unit 109, switching unit 110, evaluation and a section 111 .

The block division unit 100 divides an input image in units of frames (or pictures) into block-shaped small regions, and outputs the image blocks to the subtraction unit 101 (and the motion compensation prediction unit 109). The size of an image block is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels. However, the shape of the image block is not limited to a square, and may be rectangular, trapezoidal, triangular, or L-shaped. An image block is a unit for encoding by the image encoding device 1A and a unit for decoding by the image decoding device 2A (see FIG. 2), and such an image block is called a block to be encoded. Such an image block is also referred to as a coding unit (CU: Coding Unit) or a coding block (CB: Coding Block).

The subtraction unit 101 calculates a prediction residual representing a pixel-by-pixel difference between the encoding target block input from the block division unit 100 and a predicted image (prediction image block) corresponding to the encoding target block. Specifically, the subtracting unit 101 calculates a prediction residual by subtracting each pixel value of the predicted image from each pixel value of the encoding target block, and outputs the calculated prediction residual to the transforming unit 102a. Note that the predicted image is input to the subtraction unit 101 via the switching unit 110 from the intra prediction unit 106 or the motion compensation prediction unit 109 which will be described later.

The transformation unit 102a, the quantization unit 102b, and the switching unit 103c constitute a transformation/quantization unit 102 that performs orthogonal transformation processing and quantization processing on a block basis.

Transformation section 102a calculates an orthogonal transformation coefficient for each frequency component by orthogonally transforming the prediction residual input from subtraction section 101, and outputs the computed orthogonal transformation coefficient to quantization section 102b. For the orthogonal transform, for example, a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), or the like may be used. By such orthogonal transform, the residual signal in the pixel domain is transformed into the frequency domain. In general, the energy of the residual signal in the pixel region concentrates on the low frequency side due to the orthogonal transformation.

Note that if irrational numbers or floating point numbers are used for the orthogonal transform calculation, there may be a slight difference in calculation results between devices depending on the calculation methods and accuracy of the encoding devices and decoding devices. In order to avoid this difference, a transform simulating the orthogonal transform basis with integers or rational numbers may be used. Such transforms may not have orthogonality, but in this embodiment, the term orthogonal transform is used for transforms that do not have orthogonality, and inverse transforms that do not have orthogonality are referred to as inverse orthogonal transforms. use the term

The switching unit 102 c switches the input of the quantization unit 102 b between the output of the transform unit 102 a and the output of the subtraction unit 101 . Normally, the switching unit 102c selects the orthogonal transform coefficient output from the transform unit 102a, and selects the prediction residual output from the subtraction unit 101 in the case of a transform skip encoding target block, which will be described later. Therefore, in the case of the transform skip encoding target block, the prediction residual and the orthogonal transform coefficient can be regarded as the same signal. In the case of a transform skip encoding target block, the quantized prediction residual and the quantized orthogonal transform coefficients can be regarded as the same signal.

Quantization section 102b quantizes the orthogonal transform coefficients input from transform section 102a through switching section 102c using a quantization parameter (Qp) and a quantization matrix, and converts the quantized orthogonal transform coefficients (quantized orthogonal transform coefficients). A quantization parameter (Qp) is a parameter commonly applied to each orthogonal transform coefficient in a block, and is a parameter that determines coarseness of quantization. A quantization matrix is a matrix whose elements are quantization values when each orthogonal transform coefficient is quantized. The quantization section 102b outputs the generated quantized orthogonal transform coefficients to the entropy coding section 103A and the inverse quantization section 104a.

The entropy coding unit 103A performs entropy coding on the quantized orthogonal transform coefficients input from the quantization unit 102b, generates coded data (bitstream) by data compression of the entropy coding, and converts the coded data to Output to the outside of the image encoding device 1A. Huffman code, CABAC (Context-based Adaptive Binary Arithmetic Coding), or the like can be used for entropy coding.

Entropy coding includes a process called serialization in which the two-dimensionally arranged quantized orthogonal transform coefficients are read out in a predetermined scan order and converted into a one-dimensional quantized orthogonal transform coefficient sequence. The quantized orthogonal transform coefficients are encoded in order from the head of the column. Orthogonal transformation concentrates energy in the low frequency range, and quantization causes the energy in the high frequency range (quantized orthogonal transform coefficient values) to converge to zero. Only the significant coefficients are efficiently coded by reading out the transform coefficients and adding an end flag to the last significant coefficient in the scanning order.

The entropy coding unit 103A receives information about prediction from the intra prediction unit 106 and the motion compensation prediction unit 109 and receives information about filtering from the loop filter 107 . The entropy coding unit 103A also entropy codes these pieces of information. Further, when the transform skip mode is applied to the encoding target block, the entropy encoding unit 103A includes in the encoded data a transform skip flag indicating that the transform skip mode is applied to the encoding target block.

The inverse quantization unit 104a, the inverse transform unit 104b, and the switching unit 104c constitute an inverse quantization/inverse transform unit 104 that performs inverse quantization processing and inverse orthogonal transform processing for each block.

The inverse quantization unit 104a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a inversely quantizes the quantized orthogonal transform coefficients input from the quantization unit 102b using a quantization parameter (Qp) and a quantization matrix, thereby obtaining orthogonal transform coefficients is restored, and the restored orthogonal transform coefficients (restored orthogonal transform coefficients) are output to the inverse transform unit 104b.

The switching unit 104c switches the output of the inverse quantization/inverse transform unit 104 between the output of the inverse transform unit 104b and the output of the inverse quantization unit 104a. Normally, the restored orthogonal transform coefficient output from the inverse transform unit 104b is selected, and the restored prediction residual output from the inverse quantization unit 104a is selected in the case of a transform skip encoding target block, which will be described later. Therefore, in the case of a transform skip encoding target block, the reconstructed prediction residual and the reconstructed orthogonal transform coefficient can be regarded as the same signal. In the case of a transform skip encoding target block, the quantized orthogonal transform coefficient and the quantized prediction error can be regarded as the same signal.

The inverse transform unit 104b performs inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transform unit 102a. For example, when the transform unit 102a performs discrete cosine transform, the inverse transform unit 104b performs inverse discrete cosine transform. The inverse transform unit 104b performs inverse orthogonal transform on the restored orthogonal transform coefficients input from the inverse quantization unit 104a to restore the prediction residuals, and the restored prediction residuals, which are the restored prediction residuals, are synthesized by the synthesis unit 105. output to

The synthesizing unit 105 synthesizes the restored prediction residual input from the inverse transform unit 104b via the switching unit 104c with the predicted image input from the switching unit 110 on a pixel-by-pixel basis. Synthesizing section 105 adds each pixel value of the restored prediction residual and each pixel value of the predicted image to reconstruct the encoding target block, and outputs the reconstructed block, which is the reconstructed encoding target block, to intra prediction section 106 . and output to the loop filter 107 .

The intra prediction unit 106 performs intra prediction using the reconstructed block input from the synthesis unit 105 to generate an intra prediction image, and outputs the intra prediction image to the switching unit 110 . In addition, the intra prediction unit 106 outputs information about the selected intra prediction mode and the like to the entropy coding unit 103A.

Loop filter 107 performs filtering as post-processing on the reconstructed block input from synthesizing section 105 , and outputs the reconstructed block after filtering to frame memory 108 . Loop filter 107 also outputs information about filtering to entropy coding section 103A. Filtering includes deblocking filtering and sample adaptive offsetting.

The frame memory 108 stores the reconstructed block input from the loop filter 107 in frame units as a decoded image.

The motion compensation prediction unit 109 performs inter prediction using one or a plurality of reconstructed blocks (decoded images) stored in the frame memory 108 as reference images. Specifically, the motion-compensated prediction unit 109 calculates a motion vector by a method such as block matching, generates a motion-compensated predicted image based on the motion vector, and outputs the motion-compensated predicted image to the switching unit 110 . The motion compensation prediction unit 109 also outputs information about motion vectors to the entropy coding unit 103A.

The switching unit 110 switches between the intra prediction image input from the intra prediction unit 106 and the motion compensation prediction image input from the motion compensation prediction unit 109 and outputs the prediction image to the subtraction unit 101 and synthesis unit 105 .

On the other hand, in the transform skip mode in which the orthogonal transform processing of the encoding target block is skipped, the encoding target block output by the block division unit 100 is transformed into a prediction residual in the subtraction unit 101, and then is quantized without being orthogonally transformed. become. Specifically, the switching unit 102c selects the output of the subtraction unit 101, and the prediction residual output by the subtraction unit 101 skips the orthogonal transform in the transform unit 102a and is input to the quantization unit 102b. The quantization unit 102b quantizes the prediction residual for the encoding target block that skips the orthogonal transform (hereinafter referred to as the “transform skip encoding target block”), and converts the quantized prediction residual, which is the quantized prediction residual. is output to the entropy coding unit 103A and the inverse quantization unit 104a.

For the transform skip encoding target block, the entropy encoding unit 103A performs entropy encoding on the quantized prediction residual input from the quantization unit 102b, performs data compression, and outputs encoded data (bitstream). and outputs the encoded data to the outside of the image encoding device 1A.

The inverse quantization unit 104a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b on the transform skip encoding target block. For the transform skip block, the switching unit 104c selects the output of the inverse quantization unit 104a, so that the inverse transform unit 104b skips the inverse orthogonal transform processing. Therefore, the signal restored by inverse quantization section 104a is input to synthesis section 105 as a restored prediction residual without undergoing inverse orthogonal transform processing.

When the motion compensation prediction unit 109 performs motion compensation prediction using a plurality of reference images for a transform skip encoding target block, the evaluation unit 111 evaluates the similarity between the plurality of reference images on a pixel-by-pixel basis. Map information representing the result is output to entropy coding section 103A. Based on the map information input from the evaluation unit 111, the entropy encoding unit 103A rearranges the restored prediction residual input from the quantization unit 102b and then encodes the target block for transform skip encoding. Details of the evaluation unit 111 and the entropy coding unit 103A will be described later.

<2. Configuration of image decoding device>
FIG. 2 is a diagram showing the configuration of the image decoding device 2A according to the embodiment. As shown in FIG. 2, the image decoding device 2A includes an entropy code decoding unit 200A, an inverse quantization unit 201a, an inverse transform unit 201b, a switching unit 201c, a synthesis unit 202, an intra prediction unit 203, a loop It comprises a filter 204 , a frame memory 205 , a motion compensation prediction section 206 , a switching section 207 and an evaluation section 208 .

The entropy code decoding unit 200A decodes the encoded data generated by the image encoding device 1 and outputs the quantized orthogonal transform coefficients to the inverse quantization unit 201a. Further, the entropy code decoding unit 200A decodes the encoded data, acquires information on prediction (intra prediction and motion compensation prediction) and information on filtering, and transmits information on prediction to the intra prediction unit 203 and the motion compensation prediction unit. 206 and outputs information about filtering to the loop filter 204 .

The inverse quantization unit 201a, the inverse transform unit 201b, and the switching unit 201c constitute an inverse quantization/inverse transform unit 201 that performs inverse quantization processing and inverse orthogonal transform processing for each block.

The inverse quantization unit 201a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b of the image encoding device 1A. The inverse quantization unit 201a restores the orthogonal transform coefficients by inversely quantizing the quantized orthogonal transform coefficients input from the entropy code decoding unit 200A using the quantization parameter (Qp) and the quantization matrix, The restored orthogonal transform coefficients (restored orthogonal transform coefficients) are output to the inverse transform unit 201b.

The switching unit 201c switches the output of the inverse quantization/inverse transform unit 201 between the output of the inverse transform unit 201b and the output of the inverse quantization unit 201a. The output of the inverse transform unit 201b is normally selected, and the output of the inverse quantization unit 201a is selected in the case of a transform skip decoding target block, which will be described later. As a result, in the case of the target block for transform skip decoding, the restored prediction residual and the recovered orthogonal transform coefficients can be regarded as the same signal. can be regarded as the same signal.

The inverse transform unit 201b performs inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transform unit 102a of the image encoding device 1A. The inverse transform unit 201b performs inverse orthogonal transform on the restored orthogonal transform coefficients input from the inverse quantization unit 201a to restore prediction residuals, and the restored prediction residuals, which are the restored prediction residuals, are synthesized by the synthesis unit 202. output to

The synthesizing unit 202 synthesizes the restored prediction residual input from the inverse transform unit 201b via the switching unit 201c and the predicted image input from the switching unit 207 on a pixel-by-pixel basis, thereby reproducing the original target block. and outputs the reconstructed block to the intra prediction unit 203 and the loop filter 204 .

The intra prediction unit 203 refers to the reconstructed block input from the synthesizing unit 202 and performs intra prediction based on the intra prediction information input from the entropy code decoding unit 200A to generate an intra prediction image. The image is output to the switching unit 207 .

The loop filter 204 performs filtering similar to that performed by the loop filter 107 of the image encoding device 1A on the reconstructed block input from the synthesizing unit 202 based on the filtering information input from the entropy code decoding unit 200A. Filter processing is performed, and the reconstructed block after filtering is output to the frame memory 205 .

The frame memory 205 stores the reconstructed block input from the loop filter 204 in frame units, and outputs it as a decoded image to the outside of the image decoding device 2A.

The motion compensation prediction unit 206 uses the decoded image stored in the frame memory 205 as a reference image to perform motion compensation prediction (inter prediction) based on the motion vector information input from the entropy code decoding unit 200A. , generates a motion-compensated predicted image, and outputs the motion-compensated predicted image to the switching unit 207 .

The switching unit 207 switches between the intra prediction image input from the intra prediction unit 203 and the motion compensation prediction image input from the motion compensation prediction unit 206 and outputs the prediction image to the synthesis unit 202 .

On the other hand, for the decoding target block to which the transform skip mode is applied (transform skip decoding target block), the entropy code decoding unit 200A decodes the encoded data generated by the encoding device 1, and the quantized prediction residual is (Quantized prediction residual) is output to the inverse quantization unit 201a.

The inverse quantization unit 201a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b of the image coding device 1A for the transform skip blocks. For the transform skip blocks, the switching unit 201c selects the output of the inverse quantization unit 201a, so that the inverse transform unit 201b skips the inverse orthogonal transform processing. Therefore, the prediction residual (restored prediction residual) restored by the inverse quantization unit 201a is input to the synthesizing unit 202 without undergoing inverse orthogonal transform processing.

The evaluation unit 208 performs the same operation as the evaluation unit 111 of the image encoding device 1A. Specifically, when the motion-compensated prediction unit 206 performs motion-compensated prediction using a plurality of reference images for a transform-skip-decoding target block, the evaluation unit 208 calculates the degree of similarity between the plurality of reference images on a pixel-by-pixel basis. It evaluates and outputs map information representing the evaluation result to the entropy code decoding unit 200A. The entropy code decoding unit 200A decodes the encoded data for the transform skip encoding target block to acquire the quantized prediction residual, and based on the map information input from the evaluation unit 208, the quantized prediction residual Output after rearranging in the order of . Details of the evaluation unit 208 and the entropy code decoding unit 200A will be described later.

<3. Motion compensation prediction>
FIG. 3 is a diagram illustrating an example of motion compensated prediction. FIG. 4 is a diagram illustrating an example of a predicted image and prediction residuals. As a simple example of motion-compensated prediction, a case of using bi-prediction, especially forward and backward prediction (bidirectional prediction) used in HEVC will be described.

As shown in FIG. 3, motion compensated prediction refers to frames temporally before and after the frame being coded (the current frame). In the example of FIG. 3, motion compensated prediction of a block in an image of the tth frame, which is the current frame, is performed with reference to the t−1th frame and the t+1th frame. Motion compensation detects a portion (block) similar to the encoding target block from within a search range set by the system from within the t−1 and t+1 reference frames.

The detected location is the reference image. Information representing the relative position of the reference image with respect to the block to be coded is the arrow shown in the drawing, and is called a motion vector. The motion vector information is encoded by entropy encoding together with the frame information of the reference image in the image encoding device 1A. On the other hand, the image decoding device 2A detects a reference image based on the motion vector information generated by the image encoding device 1A.

As shown in FIGS. 3 and 4, the reference images 1 and 2 detected by motion compensation are similar partial images aligned within the reference frame to the block to be coded. The image resembles a block. In the example of FIG. 4, the encoding target block includes a star pattern and a partial circle pattern. Reference image 1 includes a star pattern and a general circle pattern. Reference image 2 contains a pattern of stars, but does not contain a pattern of circles.

A predicted image is generated from the reference images 1 and 2 . Since the prediction process is a process with a high processing load, it is common to generate a predicted image by averaging the reference images 1 and 2. FIG. However, a predicted image may be generated by using more advanced processing, for example, signal enhancement processing using a low-pass filter, high-pass filter, or the like. Here, since the reference image 1 includes a circle pattern and the reference image 2 does not include a circle pattern, when the predicted image is generated by averaging the reference images 1 and 2, the circle pattern in the predicted image is the same as that in the reference image The signal is halved compared to 1.

The difference between the prediction image obtained from the reference images 1 and 2 and the coding target block is the prediction residual. The difference between the predicted image obtained from the reference images 1 and 2 and the target image block (encoding target image) is the prediction residual. In the prediction residual shown in FIG. 4, a large difference occurs in the part where the circle pattern is deviated (hatched part), but the other parts are accurately predicted and no difference occurs.

The portion of the star pattern in which no difference has occurred is the portion where the degree of similarity between the reference image 1 and the reference image 2 is high, and where highly accurate prediction has been performed. On the other hand, a portion with a large difference is a portion unique to each reference image, that is, a portion with low similarity between reference image 1 and reference image 2 . Therefore, it can be said that the portion where the degree of similarity between the reference image 1 and the reference image 2 is low, that is, the “area A” in the prediction residual has low prediction accuracy and causes a large prediction residual.

Therefore, in the transform skip mode, the image encoding device 1A preferentially encodes a portion with a low degree of similarity between the reference image 1 and the reference image 2 as a portion with a large prediction residual. Specifically, the entropy encoding unit 103A encodes the quantized prediction residuals in order from the pixel position with the lowest similarity between the reference images, thereby preferentially encoding the significant coefficients, and sets the end flag earlier. make it possible to give

On the other hand, "area B" in the prediction residual is a portion with high similarity between the reference image 1 and the reference image 2, but produces a large prediction residual. For this reason, if only the region A, which is a portion with a low degree of similarity between the reference image 1 and the reference image 2, is regarded as a portion with a large quantized prediction residual and preferentially encoded, the coding of the “region B” will be postponed, and the addition of the end flag will be delayed.

In particular, in the conventional encoding of transform skip encoding target blocks used in HEVC, encoding is performed by diagonal scanning (a scanning method performed diagonally upward to the right, similar to zigzag scanning such as MPEG-2). conduct. Therefore, if the conventional diagonal scan is used, the region B is coded early and the end flag is added early. In other words, in such a case, rather than preferentially encoding a portion with low similarity between reference image 1 and reference image 2 as a portion with a large prediction residual, conventional diagonal scanning is used. If the encoding is performed in the order in which the data are encoded, the end flag will be added sooner.

Therefore, in the embodiment, in the transform skip mode, map information representing the distribution of the degree of similarity in units of pixels between the reference image 1 and the reference image 2 and quantized prediction that is "quantized prediction residual" The correlation with the residual distribution is obtained, and if the correlation is low, the encoding is performed according to the conventional scan order, and if the correlation is high, the encoding is performed in descending order of similarity based on the map information. This enables efficient entropy coding and improves coding efficiency.

An example of determining the correlation between map information representing the distribution of similarity in pixel units between reference image 1 and reference image 2 and the distribution of quantized prediction residuals will be described below. A reconstructed prediction residual may be used instead of the quantized prediction residual. That is, the correlation between the map information representing the similarity distribution in units of pixels between the reference image 1 and the reference image 2 and the distribution of the restored prediction residuals may be obtained.

<4. Evaluation Department>
FIG. 5 is a diagram showing an example of the configuration of the evaluation section 111 in the image encoding device 1A. As shown in FIG. 5, the evaluation unit 111 includes a difference calculation unit 111a and a normalization unit 111b.

The difference calculation unit 111a calculates the difference value (specifically, the absolute value of the difference value; the same shall apply hereinafter) between the reference images 1 and 2 input from the motion compensation prediction unit 109 on a pixel-by-pixel basis, and calculates the calculated difference value. Output to the normalization unit 111b.

Here, it can be said that the larger the difference value, the lower the degree of similarity and the lower the accuracy of prediction. It can be said that the smaller the difference value, the higher the degree of similarity and the higher the accuracy of prediction. The difference calculation unit 111a may calculate the difference value after performing filter processing on each reference image. The difference calculator 111a may calculate a statistic such as a squared error and output the statistic.

The normalization unit 111b normalizes the pixel-by-pixel difference value input from the difference calculation unit 111a by the maximum difference value in the encoding target block (that is, the maximum difference value in the encoding target block). and output. Based on at least one of a quantization parameter (Qp) that determines the coarseness of quantization and a quantization matrix to which a different quantization value is applied for each orthogonal transform coefficient, the normalization unit 111b outputs The input difference value may be adjusted and output.

The evaluation unit 111 calculates the accuracy Rij for each pixel in the encoding target block, for example, as shown in Equation (1) below.

Rij=1−(abs(Xij−Yij)/maxD×Scale(Qp)) (1)
However, Xij is the pixel value at pixel position ij in reference image 1, Yij is the pixel value at pixel position ij in reference image 2, and abs is a function for obtaining an absolute value.

Also, in equation (1), maxD is the maximum value of the difference values abs(Xij−Yij) in the encoding target block. In order to obtain maxD, it is necessary to obtain difference values for all pixel positions in the block to be coded, but in order to omit this process, the maximum value of adjacent blocks that have already been coded may be substituted. . Alternatively, maxD can be obtained from the quantization parameter (Qp) or the quantization value of the quantization matrix using a table that defines the correspondence between the quantization parameter (Qp) or the quantization value of the quantization matrix and maxD. good. Alternatively, a fixed value defined in advance by specifications may be used as maxD.

In equation (1), Scale(Qp) is a coefficient that is multiplied according to the quantization parameter (Qp) and the quantization value of the quantization matrix. Scale(Qp) is designed so that it approaches 1.0 when Qp or the quantization value of the quantization matrix is large, and approaches 0 when it is small, and its degree is adjusted by the system. Alternatively, a fixed value defined in advance by specifications may be used as Scale (Qp).

In addition, when a substitute value such as a fixed value is used as maxD or Scale (Qp), there may be cases where the accuracy Rij exceeds 1.0 or is less than 0. In that case, 1.0 or 0 clip processing to .

The evaluation unit 111 outputs map information including the accuracy Rij of each pixel position ij in the encoding target block to the entropy encoding unit 103A.

However, the evaluation unit 111 performs evaluation (calculation of the accuracy Rij) only when motion-compensated prediction using a plurality of reference images is applied to the transform-skip coding target block, and other modes such as unidirectional prediction and In intra-prediction processing, evaluation (calculation of accuracy Rij) may not be performed.

Note that the evaluation unit 208 in the image decoding device 2A is configured similarly to the evaluation unit 111 in the image encoding device 1A. Specifically, the evaluation unit 208 in the image decoding device 2A includes a difference calculation unit 208a and a normalization unit 208b. The evaluation unit 208 in the image decoding device 2A outputs map information including the accuracy Rij of each pixel position ij in the encoding target block to the entropy coding/decoding unit 200A.

<5. Entropy Encoding Unit>
FIG. 6 is a diagram showing an example of the configuration of entropy coding section 103A. FIG. 7 is a diagram illustrating an example of generating a probability index. FIG. 8 is a diagram illustrating an example of rearrangement of quantized prediction residuals.

As shown in FIG. 6, the entropy coding unit 103A includes a sorting unit 103a, a prediction residual rearrangement unit 103b, a serialization unit 103c, a probability rearrangement unit 103d, a determination unit 103e, a switching unit 103f, and an encoding unit 103g.

The sorting unit 103a sorts the accuracies Rij in the map information input from the evaluating unit 111 in ascending order. Specifically, as shown in FIG. 7A, the accuracy Rij is arranged two-dimensionally in the map information. is serialized by and set as a probability sequence. Then, as shown in FIG. 7B, the sorting unit 103a sorts the accuracy Rij in descending order, and uses the accuracy Rij as an index i to convert the index i and the pixel position (the X coordinate position and the Y coordinate position). The associated probability index information is output to the prediction residual rearrangement unit 103b and the probability rearrangement unit 103d.

In the example of FIG. 7A, the accuracy at pixel position (x, y)=(2, 2) is the lowest, and the accuracy at pixel position (x, y)=(2, 3) is the second lowest. , pixel position (x,y)=(3,2) has the third lowest probability, and pixel position (x,y)=(3,3) has the fourth lowest probability. It can be estimated that a region composed of such pixel positions has low prediction accuracy and generates a large prediction residual. On the other hand, it can be estimated that a pixel position with a certainty of 1 has high prediction accuracy and does not generate a prediction residual.

The prediction residual rearrangement unit 103b rearranges the quantized prediction residuals input from the quantization unit 102b for the transform skip blocks based on the probability index information input from the sorting unit 103a. Specifically, the prediction residual rearrangement unit 103b performs quantization so that the quantized prediction residuals are encoded in order from pixel positions with low accuracy (that is, pixel positions with low similarity between reference images). Sort the prediction residuals pixel by pixel.

The quantized prediction residuals shown in FIG. 8A are arranged two-dimensionally, and the prediction residual rearrangement unit 103b, as shown in FIG. is rearranged so that the quantized prediction residuals of pixel positions with low accuracy are concentrated in the upper left region. Here, zigzag scanning is assumed as the scanning order, and the prediction residual rearrangement unit 103b scans in order from the upper left region to the lower right region, and preferentially serializes the quantized prediction residuals at pixel positions with low accuracy. Then, a quantized prediction residual sequence in which quantized prediction residuals are arranged in descending order of accuracy (that is, the similarity between reference images is low) is output to the switching unit 103f and the determination unit 103e. However, the scanning order is not limited to zigzag scanning, and horizontal scanning or vertical scanning may be used. The prediction residual rearrangement unit 103b may rearrange after scanning instead of rearranging before scanning.

Alternatively, instead of a fixed scanning order such as zigzag scanning, horizontal scanning, or vertical scanning, the prediction residual rearrangement unit 103b performs variable scanning so that quantized prediction residuals are scanned in order from pixel positions with low accuracy. By determining the order and scanning in the determined scan order, a quantized prediction residual sequence in which quantized prediction residuals are arranged in descending order of accuracy may be output to the switching unit 103f and the determination unit 103e.

The serialization unit 103c performs the same serialization processing as that for conventional transform skip blocks on the quantized prediction residual input from the quantization unit 102b. That is, the serialization unit 103c converts the two-dimensional quantized prediction residual into a one-dimensional quantized prediction residual in a predetermined order (diagonal scan, zigzag scan, etc.), The column is output to the switching unit 103f.

The probability rearrangement unit 103d one-dimensionally rearranges the map information input from the evaluation unit 111 in the same manner as the prediction residual rearrangement unit 103b, based on the probability index information input from the sort unit 103a. Specifically, the accuracy sorting unit 103d outputs map information as a one-dimensional signal in order from pixel positions with low accuracy (that is, pixel positions with low similarity between reference images). This output signal is referred to herein as the "probability column". The probability rearrangement unit 103d outputs the probability sequence to the determination unit 103e.

The determination unit 103e determines the accuracy sequence and the quantization prediction from the accuracy sequence input from the accuracy rearrangement unit 103d and the absolute values of the quantized prediction residual sequence input from the prediction residual prediction residual rearrangement unit 103b. The degree of similarity representing the correlation between the residual sequence is obtained, and if the degree of similarity is equal to or greater than the threshold Th (or greater), a signal indicating application of sorting based on the map information is output to the switching unit 103f. If the degree is less than (or less than) the threshold Th, a signal indicating non-application of rearrangement based on the map information is output to the switching unit 103f.

Examples 1 to 3 of similarities obtained from the accuracy sequence and the quantized prediction residual sequence are shown below. In the following, N is the number of pixels in the transform skip block, p _i is the accuracy sequence, q _i is the sequence obtained by arranging the accuracy sequence in reverse order (reverse order accuracy sequence) (that is, q _i =p _N-1-i ), and quantization Let d _i be the prediction residual sequence and R be the degree of similarity to be obtained. i is an index and takes the range of .

(1) Example 1 of similarity used in determination unit 103e
An example using the cosine similarity between the reverse order accuracy sequence and the absolute value of the quantized prediction residual sequence as the similarity is shown.

(2) Example 2 of degree of similarity used in determination unit 103e
As another example, an example of similarity using the L2 distance is shown.

where Q _i is the probability sequence normalized from 0 to 1 (where 0 is the lowest probability and 1 is _the highest probability). The normalization of accuracy is the same as the normalization in the evaluation unit 111 . Note that Q _i =q _i if the probability sequence was originally normalized. D _i is a sequence obtained by normalizing the absolute values of the quantized prediction residual sequence to a range of 0-1. When the maximum possible pixel value of the target image format is minL and the minimum value is maxL, D _i is, for example,

can be obtained as

(3) Example 3 of similarity used in determination unit 103e
As another example, an example of similarity using the L1 distance will be shown.

In this way, as similarity, various degrees of similarity and distances that can be calculated between two sequences can be used. In the case of similarity (an index with a larger value as the two sequences are similar), a normalized value can be used as the similarity, and in the case of distance (an index with a smaller value as the two sequences are more similar) A value obtained by subtracting the normalized distance from 1 may be used as the degree of similarity.

Note that the threshold Th to be compared with the degree of similarity may be a predetermined constant or a variable value. When a variable value is used, the encoded data may be transmitted including the threshold value.

Based on the signal input from the determination unit 103e, the switching unit 103f sends the output of the prediction residual rearrangement unit 103b to the encoding unit 103g when rearrangement based on map information is applied, and performs rearrangement based on the map information. If not applicable, the output of the serialization unit 103c is sent to the encoding unit 103g. Note that when the rearrangement based on the map information is not applied, the encoding target block is processed by the same transform skip as in the conventional art.

The coding unit 103g codes the quantized prediction residual in the prediction residual sequence input from the switching unit 103f and outputs coded data. The encoding unit 103g determines the last significant coefficient included in the quantized prediction residual sequence input from the prediction residual rearrangement unit 103b, and determines the last significant coefficient from the beginning of the quantized prediction residual sequence to the last significant coefficient. encoding. The encoding unit 103g sequentially determines whether or not the quantized prediction residual sequence input from the prediction residual rearrangement unit 103b is a significant coefficient from the beginning, and assigns an end flag to the last significant coefficient. , efficiently encode the significant coefficients by not encoding the quantized prediction residuals (ie, zero coefficients) after the end flag.

For example, the encoding unit 103g, as shown in FIG. 8C, of the quantized prediction residual sequence input from the switching unit 103f, the last significant coefficient, that is, (X= 1, Y=2) is encoded as last_sig_coeff_x and y (end flag). After that, the encoding unit 103g determines whether or not significant coefficients exist in reverse scan order, that is, in order from (3, 3) to (0, 0), starting from the position (1, 2) of the last significant coefficient. or as sig_coeff_flag. In sig_coeff_flag, "1" indicates a coordinate position where a significant coefficient exists, and "0" indicates a coordinate position where no significant coefficient exists.

Furthermore, the encoding unit 103g encodes whether the significant coefficient is greater than 1 as coeff_abs_level_greater1_flag, and encodes whether the significant coefficient is greater than 2 as coeff_abs_level_greater2_flag. For significant coefficients greater than 2, the encoding unit 103g encodes a value obtained by subtracting 3 from the absolute value of the significant coefficient as coeff_abs_level_remaining, and encodes a flag indicating whether the significant coefficient is positive or negative as coeff_sign_flag. .

With such entropy encoding, the lower right region (later in the scan order) the position of the last significant coefficient is, the larger the values of last_sig_coeff_x and y and the larger the amount of Sig_coeff_flag. Such entropy coding increases the amount of generated information.

When sorting based on map information is applied, sorting is performed so that quantized prediction residuals are encoded in order from pixel positions with low accuracy (that is, pixel positions with low similarity between reference images). Therefore, the values of Last_sig_coeff_x and y are reduced and the amount of Sig_coeff_flag is reduced. As a result, the amount of information generated by entropy coding can be reduced.

Furthermore, as in the example shown in FIG. 4, the assumption that pixel positions with low similarity between a plurality of reference images have large prediction residuals does not hold, and the similarity is less than the threshold Th. , the sorting based on the map information is not applied. That is, the quantized prediction residuals are encoded in order according to conventional diagonal scanning. This makes it possible to avoid deterioration of coding efficiency due to applying rearrangement based on map information.

<6. Entropy Coding/Decoding Unit>
FIG. 9 is a diagram showing an example of the configuration of the entropy code decoding unit 200A. As shown in FIG. 9, the entropy code decoding unit 200A includes a decoding unit 200a, a sorting unit 200b, a prediction residual prediction residual rearranging unit 200c, a deserializing unit 200d, a probability rearranging unit 200e, and a determination It includes a section 200f and a switching section 200g.

The decoding unit 200a decodes the encoded data generated by the image encoding device 1A, and generates a quantized prediction residual sequence (quantized prediction residual), information on prediction (intra prediction and motion compensation prediction), and , outputs the quantized prediction residual sequence to the prediction residual rearrangement unit 200 c and the deserialization unit 200 d, and outputs information about prediction to the intra prediction unit 203 and the motion compensation prediction unit 206 . When the transform skip flag obtained from the encoded data indicates application of transform skip and the information on prediction indicates bi-prediction, the decoding unit 200a determines to perform rearrangement based on the map information. may As a result, there is no need to include in the encoded data a flag indicating that the rearrangement is to be performed based on the map information, and it is possible to suppress an increase in the number of flags.

The sorting unit 200b sorts the accuracies Rij in the map information input from the evaluating unit 208 in ascending order. Since the accuracies Rij are arranged two-dimensionally in the map information, the sorting unit 200b serializes the map information by, for example, zigzag scanning to form an accuracy sequence. Then, the sorting unit 200b sorts the accuracy Rij in descending order, sets the accuracy Rij as an index i, and converts the accuracy index information that associates the index i with the pixel position (X-coordinate position and Y-coordinate position) to the prediction residual. Output to the rearrangement section 200c and the probability rearrangement section 200e.

The prediction residual rearrangement unit 200c reverses the rearrangement processing performed by the prediction residual rearrangement unit 103b of the image encoding device 1A. The prediction residual rearrangement unit 200c deserializes the quantized prediction residual sequence input from the decoding unit 200a for the transform skip decoding target block by rearranging it based on the probability index information input from the sorting unit 200b. and outputs the two-dimensionally arranged quantized prediction residuals to the switching unit 200g.

The deserialization unit 200d performs the same deserialization processing as the conventional processing for the conversion skip blocks. That is, the deserialization unit 200d rearranges the quantized prediction residual sequence into two-dimensional quantized prediction residuals in a predetermined order (for example, zigzag scanning from upper left to lower right).

Based on the map information input from the evaluation unit 208, the probability rearrangement unit 200e performs the same processing as the probability rearrangement unit 103d of the image encoding device 1, and outputs the probability sequence to the determination unit 200f.

The determination unit 200 f performs the same processing as the determination unit 103 e of the image encoding device 1 .

The switching unit 200g selects the output of the rearrangement unit 200c when the determination unit 200f outputs a signal indicating the application of the rearrangement based on the map information, and the determination unit 200f selects the non-application of the rearrangement based on the map information. When the signal indicating is output, the output of the deserialization unit 200d is selected. The switching unit 200g outputs the selected output (two-dimensional quantized prediction residual) to the inverse quantization unit 201a.

In the embodiment, the prediction residual rearrangement unit 200c, the deserialization unit 200d, and the switching unit 200g constitute a rearrangement unit. When the determination unit 200f determines that the degree of similarity is equal to or greater than the threshold, the rearrangement unit 200c rearranges and outputs the quantized prediction residuals in order according to the degree of similarity evaluated by the evaluation unit 208. , when the determination unit 200f determines that the degree of similarity is less than the threshold, the quantized prediction residuals are rearranged in a predetermined order and output.

<7. Operation of image encoding>
FIG. 10 is a diagram showing a processing flow in the image encoding device 1A according to the embodiment. The image coding device 1A executes this processing flow when applying the transform skip mode and motion compensation prediction to the coding target block.

As shown in FIG. 10, in step S1101, the motion-compensated prediction unit 109 predicts an encoding target block by performing motion compensation prediction using a plurality of reference images, and generates a predicted image corresponding to the encoding target block. Generate.

In step S1102, the evaluation unit 111 evaluates the degree of similarity between a plurality of reference images for each pixel position, and generates map information representing the prediction accuracy (prediction accuracy) of each pixel position in the encoding target block.

In step S1103, the subtraction unit 101 calculates a prediction residual representing a pixel-by-pixel difference between the encoding target block and the prediction image.

In step S1104, the quantization unit 102b quantizes the prediction residual calculated by the subtraction unit 101 to generate a quantized prediction residual.

In step S1105, the entropy coding unit 103A calculates a degree of similarity representing the correlation between the quantized prediction residual and the map information, and determines whether or not the degree of similarity is equal to or greater than a threshold.

In step S1106, if the degree of similarity is equal to or greater than the threshold, the entropy encoding unit 103A encodes the quantized prediction residuals in ascending order of accuracy based on the map information, and outputs encoded data. When the degree of similarity is less than the threshold, entropy coding section 103A performs coding in a predetermined order and outputs coded data.

In step S1107, the inverse quantization unit 104a performs inverse quantization on the quantized prediction residual input from the quantization unit 102b to restore the prediction residual and generate a restored prediction residual.

In step S1108, the synthesizing unit 105 reconstructs the target block by synthesizing the restored prediction residual with the predicted image on a pixel-by-pixel basis to generate a reconstructed block.

In step S1109, the loop filter 107 filters the reconstructed block.

In step S1110, the frame memory 108 stores the reconstructed block after filtering in units of frames.

<8. Operation of image decoding>
FIG. 11 is a diagram showing a processing flow in the image decoding device 2A according to the embodiment. The image decoding device 2A executes this processing flow when applying the transform skip mode and motion compensation prediction to the target block.

As shown in FIG. 11, in step S1201, the decoding unit 200a of the entropy code decoding unit 200A decodes encoded data to acquire motion vector information, and outputs the acquired motion vector information to the motion compensation prediction unit 206. .

In step S1202, the motion compensation prediction unit 206 predicts the decoding target block by performing motion compensation prediction using a plurality of reference images based on motion vector information, and generates a prediction image corresponding to the decoding target block.

In step S1203, the evaluation unit 208 calculates the degree of similarity between a plurality of reference images for each pixel position, and generates map information representing the prediction accuracy (prediction accuracy) of each pixel position in the block.

In step S1204, the entropy code decoding unit 200A decodes the encoded data to acquire a quantized prediction residual sequence. The entropy code decoding unit 200A calculates a degree of similarity representing the correlation between the quantized prediction residual sequence and the map information, and determines whether or not the degree of similarity is equal to or greater than a threshold. If the degree of similarity is equal to or greater than the threshold, the entropy code decoding unit 200A rearranges the quantized prediction residuals based on the map information, and outputs the two-dimensionally arranged quantized prediction residuals to the inverse quantization unit 201a. Output. When the degree of similarity is less than the threshold, the entropy code decoding unit 200A rearranges the quantized prediction residuals in a predetermined order, and outputs the two-dimensionally arranged quantized prediction residuals to the inverse quantization unit 201a. do.

In step S1205, the inverse quantization unit 201a performs inverse quantization on the quantized prediction residual to restore the prediction residual and generate a restored prediction residual.

In step S1206, the synthesizing unit 202 reconstructs the target block by synthesizing the restored prediction residual with the predicted image on a pixel-by-pixel basis to generate a reconstructed block.

In step S1207, the loop filter 204 filters the reconstructed block.

In step S1208, the frame memory 205 stores and outputs the reconfigured block after filtering in units of frames.

<9. Summary of Embodiments>
In the image encoding device 1A, the evaluation unit 111 evaluates the degree of similarity between a plurality of reference images on a pixel-by-pixel basis, and outputs information (map information) of the evaluation result to the entropy encoding unit 103A. The entropy encoding unit 103A calculates a similarity that represents the correlation between the quantized prediction residual and the map information, and if the similarity is equal to or greater than a threshold, the entropy encoding unit 103A inputs from the quantization unit 102b based on the map information. The resulting quantized prediction residuals are coded in order from the pixel position with the lowest similarity between the reference images. By encoding the quantized prediction residuals in order from the pixel position with the lowest similarity between the reference images, it is possible to preferentially encode the significant coefficients and attach the end flag earlier. This makes it possible to perform efficient entropy coding on transform-skipped blocks and improve coding efficiency.

In the image decoding device 2A, the evaluation unit 208 evaluates the degree of similarity between a plurality of reference images on a pixel-by-pixel basis, and outputs information (map information) on the evaluation result to the entropy code decoding unit 200A. The entropy code decoding unit 200A decodes the encoded data to obtain a quantized prediction residual on a pixel-by-pixel basis, calculates a degree of similarity representing the correlation between the quantized prediction residual and the map information, and calculates the degree of similarity. If it is equal to or greater than the threshold, the quantized prediction residuals are rearranged and output based on the map information. In this way, by rearranging the quantized prediction residuals based on the evaluation result of the evaluation unit 208, the entropy coding/decoding unit 200A can be used even if the information specifying the details of the rearrangement is not transmitted from the image coding apparatus. can autonomously rearrange the quantized prediction residuals. This eliminates the need to transmit information designating the details of rearrangement from the image encoding device 1, thereby avoiding deterioration in encoding efficiency.

Also, if the degree of similarity representing the correlation between the prediction residuals and the map information is less than the threshold, the encoding efficiency can be further improved by rearranging in a predetermined order. Specifically, when the assumption that pixel positions with low similarity between multiple reference images have large prediction residuals does not hold and the similarity is less than the threshold, the map information , and encode the quantized prediction residuals in order according to conventional diagonal scanning. This makes it possible to avoid deterioration of coding efficiency due to applying rearrangement based on map information.

<10. Modification 1>
FIG. 12 is a diagram illustrating an example of overlapping motion compensation according to Modification 1 of the embodiment.

The evaluation unit 111 of the image encoding device 1A and the evaluation unit 208 of the image decoding device 2A may generate error maps as map information by the method described below and input them to the rearrangement unit 112 . When the error map is input to the sorting unit 112, the sorting unit 112 performs quantum permutation processing of the prediction residuals.

L0[i,j] and L1[i,j] (where [i,j] are the coordinates ), the error map map[i,j] and its maximum value max_map are calculated by the following equation (2).

map[i,j]=abs(L0[i,j]-L1[i,j])
max_map=max(map[i,j]) (2)

If max_map in formula (2) exceeds 6-bit precision (exceeds 64), the error map and maximum value are updated by shift set so that max_map falls within 6-bit precision according to formula (3) below. .

max_map = max_map >> shift
map[i, j] = map[i, j] >> shift (3)

<11. Modification 2>
The motion compensation prediction unit 109 of the image encoding device 1A and the motion compensation prediction unit 206 of the image decoding device 2A divide the target block (CU) into a plurality of small blocks, and apply a different motion vector to each small block, It may be possible to switch between unidirectional prediction and bi-prediction for each small block. In such a case, the evaluation unit 111 of the image encoding device 1A and the evaluation unit 208 of the image decoding device 2A do not calculate map information for a CU for which a predicted image is generated using both unidirectional prediction and bidirectional prediction. It can be a thing. On the other hand, when predicted images are generated by bi-prediction for all small blocks, the evaluation unit 111 of the image encoding device 1A and the evaluation unit 208 of the image decoding device 2A generate map information.

Also, the motion compensation prediction unit 109 of the image encoding device 1A and the motion compensation prediction unit 206 of the image decoding device 2A perform overlapping motion compensation (OBMC: Overlapped Block Motion Compensation) may be performed. The evaluation unit 111 of the image encoding device 1A and the evaluation unit 208 of the image decoding device 2A may consider correction of reference pixels by OBMC when generating map information.

For example, when the prediction mode of the surrounding blocks used for OBMC correction is bi-prediction, the evaluation unit 111 of the image encoding device 1A and the evaluation unit 208 of the image decoding device 2A determine the predicted image affected by the OBMC correction. , the map information is corrected using the motion vectors of the reference images (L0 and L1) used for generating predicted images for bi-prediction of surrounding blocks. Specifically, in the block boundary area, when the motion vectors of adjacent blocks are bi-predictive, weighted averaging is performed according to the position of the map information of the adjacent blocks. If the adjacent blocks are in the intra mode or in the case of unidirectional prediction, the map information is not corrected. In the case of FIG. 12, the map information is generated using L0 _a and L1 _a for the upper block boundary, and the weighted average of the map information of the CU and the map information of the CU is calculated for the lower area (the area overlapping with the CU). do. Since the prediction modes of the lower, right, and left CUs are unidirectional prediction, the map information is not corrected for regions overlapping these CUs.

<12. Modification 3>
In the above-described embodiment, an example of rearranging prediction residuals in units of pixels has been described, but prediction residuals may be rearranged in units of pixel groups (small blocks). Such a small block is a block made up of 4×4 pixels and is sometimes called CG.

FIG. 13 is a diagram showing the configuration of an image encoding device 1B according to Modification 3 of the embodiment. As shown in FIG. 13, the image coding device 1B includes a rearrangement unit 112 that rearranges prediction residuals in units of pixel groups (small blocks). If the target block (CU) generates a predicted image using bi-prediction and the CU applies the transform skip mode, the rearrangement unit 112 uses the error map described above. , the prediction residual of the CU is rearranged in units of small blocks (4×4).

FIG. 14 is a diagram illustrating an example of the operation of the sorting unit 112 according to Modification 3 of the embodiment.

As shown in FIG. 14A, the subtraction unit 101 of the image coding device 1B calculates prediction residuals corresponding to CUs. In the example of FIG. 14A, a prediction residual exists in the upper right region within the CU. The upper right region in the CU is a region with low similarity between reference images and can be regarded as a region with low prediction accuracy (certainty).

As shown in FIG. 14B, the evaluation unit 111 or the rearrangement unit 112 of the image encoding device 1B divides the error map into 4×4 CG units, and calculates the average value CGmap of the errors in CG units as follows. It is calculated by the formula (4).

Then, the rearrangement unit 112 rearranges the CGs in descending order of the error average value CGmap and assigns an index. In other words, the rearrangement unit 112 rearranges the CGs in descending order of similarity between the reference images and assigns an index. In the example of FIG. 14B, the numbers in each CG represent the index after rearrangement. Since the average value CGmap of the CG in the upper right area is large, the scan (encoding) priority is set high. Next, the rearrangement unit 112 rearranges the CGs so that the CGs are scanned (encoded) in ascending order of index, as shown in FIG. 14(C). As a result, the rearrangement section 112 outputs the prediction residuals rearranged in CG units to the quantization section 102b, as shown in FIG. 14(D).

Quantization section 102b quantizes the prediction residual input from rearrangement section 112, and outputs the quantized quantized prediction residual to entropy coding section 103B. The entropy coding unit 103B generates coded data by coding the CGs in descending order of the error average value CGmap.

Note that the rearrangement unit 112 rearranges the restored prediction residual output from the inverse quantization unit 104a in units of CG so as to restore the original arrangement order of the CGs, and rearranges the restored prediction residuals rearranged in units of CG. The difference is output to combining section 105 .

FIG. 15 is a diagram showing the configuration of an image decoding device 2B according to Modification 3 of the embodiment. As shown in FIG. 15, the image decoding device 2B includes a rearrangement unit 209 that rearranges the restored prediction residual output from the inverse quantization unit 201a in CG units.

If the target block (CU) generates a predicted image using bi-prediction and the CU applies the transform skip mode, the rearrangement unit 209 uses the error map described above. , the prediction residuals of the CU are rearranged in CG units.

Specifically, the evaluation unit 208 or the rearrangement unit 209 of the image decoding device 2B divides the error map into 4×4 CG units, and calculates the average value CGmap of the errors in CG units using the above equation (5). calculate. Then, the rearrangement unit 209 reverses the rearrangement processing performed by the rearrangement unit 112 of the image decoding device 2A based on the average value CGmap of errors in CG units, and synthesizes the prediction residuals after the rearrangement. Output to the unit 202 .

In addition, in this modification, when performing determination and switching based on similarity, the average value of the absolute values of the prediction residuals in the pixel group (CG) is set to one element of the prediction residual sequence in the above-described embodiment. should be treated as

<13. Modification 4>
In the above-described embodiment, an example in which prediction residuals are rearranged in pixel units or small block units has been described, but prediction residuals may be rearranged so as to be horizontally or vertically reversed.

As shown in FIG. 16, the subtraction unit 101 of the image coding device 1B calculates a prediction residual corresponding to the target block. In the example of FIG. 16, a prediction residual exists in the upper right area of the target block. Also, the upper right area of the target block can be regarded as an area where the degree of similarity between reference pixels is low and the prediction accuracy (certainty) is low.

The rearrangement unit 112 of Modification 4 of the embodiment calculates the center of gravity of the error map by the following equation (5).

When the calculated center of gravity ( _gi , _gj ) of the error map is located in the upper right region of the map, the rearrangement unit 112 sets the upper left coordinates to (0, 0) and the lower right coordinates to (m, n). Then,

and

, then horizontally flip the prediction residual.

When the center of gravity of the error map is located in the lower left area, the rearrangement unit 112 performs

and

, then vertically flip the prediction residual.

When the center of gravity of the error map is located in the lower right region, the rearrangement unit 112 performs

and

, flip the prediction residual horizontally and vertically.

Note that when the center of gravity of the error map is the lower right region, the rearrangement unit 112 may be configured to rotate the prediction residual by 180 degrees instead of horizontally and vertically inverting the prediction residual. Alternatively, the scanning order for coefficient encoding may be changed from the lower right to the upper left instead of from the upper left to the lower right.

In order to reduce the processing, the position of the maximum value of the error map is searched without calculating the center of gravity of the error map, and the position of the maximum value is regarded as the position of the center of gravity, and the above-mentioned inversion processing is performed. may

The prediction residual after the prediction residual is reversed by the rearrangement unit 112 of Modification 4 of the embodiment is output to the quantization unit 102b.

Quantization section 102b quantizes the prediction residual input from rearrangement section 112, and outputs the quantized quantized prediction residual to entropy coding section 103B. The entropy coding unit 103B generates coded data by coding the prediction residual in order from the upper left region to the lower right region.

Note that the rearrangement unit 112 performs an inversion process on the restored prediction residual output from the inverse quantization unit 104a based on the position of the center of gravity of the error map, and converts the rearranged restored prediction residual to Output to the synthesizing unit 105 .

<14. Other Embodiments>
In the above-described embodiment, an example has been described in which the entropy encoding unit 103A reads out all the prediction residuals arranged two-dimensionally in descending order of accuracy and serializes them. However, it is also possible to read out only the top several prediction residuals in descending order of accuracy among the two-dimensionally arranged prediction residuals, and read out the other prediction residuals in a fixed order determined by the system. Alternatively, the reading order of the two-dimensionally arranged prediction residuals may be moved up or down by a predetermined number according to the accuracy.

In the above-described embodiments, inter prediction was mainly described as motion compensated prediction. In inter prediction, a reference image in a frame different from the current frame is used for prediction of a target block in the current frame. However, it is also possible to apply a technique called intra block copy as motion compensated prediction. In intra-block copying, a reference image in the same frame as the current frame is used for prediction of the target block of the current frame.

A program that causes a computer to execute each process performed by the image encoding devices 1A and 1B and a program that causes a computer to execute each process performed by the image decoding devices 2A and 2B may be provided. Also, the program may be recorded on a computer-readable medium. A computer readable medium allows the installation of the program on the computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, but may be, for example, a recording medium such as CD-ROM or DVD-ROM. Alternatively, the image encoding devices 1A and 1B may be configured as semiconductor integrated circuits (chipsets, SoC) by integrating circuits for executing each process performed by the image encoding devices 1A and 1B. Similarly, a circuit for executing each process performed by the image decoding devices 2A and 2B may be integrated, and the image decoding devices 2A and 2B may be configured as semiconductor integrated circuits (chipsets, SoC).

Although the embodiments have been described in detail with reference to the drawings, the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the invention.

1, 1A, 1B: image encoding devices 2, 2A, 2B: image decoding device 100: block division unit 101: subtraction unit 102: transformation/quantization unit 102a: transformation unit 102b: quantization units 103A, 103B: entropy code Conversion unit 103a: Sorting unit 103b: Prediction residual rearrangement unit 103c: Serialization unit 103d: Accuracy rearrangement unit 103e: Judging unit 103f: Switching unit 103g: Encoding unit 104: Inverse quantization/inverse transform unit 104a: Inverse quantization conversion unit 104b: inverse transform unit 105: synthesis unit 106: intra prediction unit 107: loop filter 108: frame memory 109: motion compensation prediction unit 110: switching unit 111: evaluation unit 111a: difference calculation unit 111b: normalization unit 112: Sorting unit 200A: Entropy code decoding unit 200a: Decoding unit 200b: Sorting unit 200c: Prediction residual sorting unit 200d: Deserializing unit 200e: Accuracy sorting unit 200f: Judging unit 200g: Switching unit 201: Inverse quantization/ Inverse transform unit 201a: Inverse quantization unit 201b: Inverse transform unit 202: Synthesis unit 203: Intra prediction unit 204: Loop filter 205: Frame memory 206: Motion compensation prediction unit 207: Switching unit 209: Sorting unit

Claims (6)

An image decoding device that decodes encoded data in units of blocks,
A motion compensation prediction unit that generates a predicted image block corresponding to a block to be decoded by performing bi-prediction using a plurality of reference images;
an evaluation unit that calculates an evaluation value indicating a degree of similarity between the plurality of reference images for each small block that is a unit smaller than the block and is made up of a plurality of pixels only when the bi-prediction is performed;
an acquisition unit that decodes the encoded data and acquires a prediction residual corresponding to the block to be decoded;
a synthesizing unit that synthesizes the obtained prediction residual with the block of the predicted image to reconstruct the block to be decoded,
The image decoding device, wherein the evaluation value calculated by the evaluation unit is used to correct the composition target of the composition unit in units of the small blocks. In a transform skip mode that skips the orthogonal transform processing of the block to be decoded,
A degree of similarity representing the correlation between the quantized prediction residual or the restored prediction residual obtained by inverse quantizing the quantized prediction residual and the similarity evaluated by the evaluation unit and a determination unit that determines whether the degree of similarity is equal to or greater than a threshold;
Sorting for sorting and outputting the quantized prediction residuals in order according to the similarity evaluated by the evaluating unit when the determining unit determines that the similarity is equal to or greater than the threshold. The image decoding device according to claim 1, further comprising a unit. The rearrangement unit rearranges and outputs the quantized prediction residuals in a predetermined order when the determination unit determines that the degree of similarity is less than the threshold value. The image decoding device according to claim 2. An image encoding device that encodes an encoding target block in units of blocks obtained by dividing an input image,
A motion compensation prediction unit that generates a prediction image corresponding to the encoding target block by performing bi-prediction using a plurality of reference images;
an evaluation unit that calculates an evaluation value indicating a degree of similarity between the plurality of reference images for each small block that is a unit smaller than the block and is made up of a plurality of pixels only when the bi-prediction is performed;
a synthesizing unit that reconstructs the block to be encoded by synthesizing a prediction residual representing a pixel-by-pixel difference between the block to be encoded and the predicted image with the block of the predicted image;
The evaluation value calculated by the evaluation unit is used to correct the composition target of the composition unit in units of the small blocks,
In a transform skip mode that skips orthogonal transform processing of the encoding target block,
Correlation between a quantized prediction residual obtained by quantizing the prediction residual or a restored prediction residual obtained by inversely quantizing the quantized prediction residual and the similarity evaluated by the evaluation unit A determination unit that calculates a degree of similarity representing and determines whether the degree of similarity is greater than or equal to a threshold;
an encoding unit that encodes the quantized prediction residual in order according to the similarity evaluated by the evaluation unit when the determination unit determines that the similarity is equal to or greater than the threshold; An image encoding device, comprising: 5. The encoding unit according to claim 4, wherein, when the determination unit determines that the degree of similarity is less than the threshold, the encoding unit encodes the quantized prediction residual in a predetermined order. An image encoding device as described. A program that causes a computer to function as the image decoding device according to any one of claims 1 to 3.

Images