WO2020255688A1

WO2020255688A1 - Image encoding device, image decoding device and method, and program

Info

Publication number: WO2020255688A1
Application number: PCT/JP2020/021614
Authority: WO
Inventors: 真悟志摩
Original assignee: キヤノン株式会社
Priority date: 2019-06-20
Filing date: 2020-06-01
Publication date: 2020-12-24
Also published as: JP2021002722A

Abstract

The present invention reduces mounting complexity by limiting blocks to be able to be used in prediction processing to some blocks according to positional relationships of blocks. To this end, an image encoding device, which encodes a moving image according to the present invention, includes: a division unit which separates, from frame images in the moving image to be encoded, basic blocks of a preset size in a raster scan order and additionally divides the basic blocks into a plurality of sub-blocks; and an encoding unit which intra-prediction encodes and inter-prediction encodes the sub-blocks obtained from the division unit. Here, when a sub-block of interest in a basic block of interest is intra-prediction encoded and a sub-block in the upper right of the sub-block of interest is within a basic block in the upper right of the basic block of interest, the encoding unit performs intra-prediction encoding by excluding the sub-block in the upper right of the sub-block of interest from a reference target of the intra-prediction encoding.

Description

Image coding device and image decoding device, method and program

The present invention relates to an image coding device, an image decoding device, a method, and a program.

As a coding method for compressed recording of moving images, a HEVC (High Efficiency Video Coding) coding method (hereinafter referred to as HEVC) is known. In HEVC, in order to improve coding efficiency, a basic block having a size larger than that of a conventional macroblock (16 × 16 pixels) has been adopted. This large-sized basic block is called a CTU (Coding Tree Unit), and its size is a maximum of 64 × 64 pixels. The CTU is further divided into sub-blocks that serve as units for prediction and conversion.

Further, in HEVC, a process of predicting the motion vector of the coded block is used by using the motion vector of the coded block. By predicting the motion vector, it is possible to improve the compression efficiency of the motion vector by reducing the difference to be encoded. Patent Document 1 discloses a technique for predicting such a motion vector.

In recent years, as a successor to HEVC, activities to carry out international standardization of more efficient coding methods have started. JVET (Joint Video Experts Team) was established between ISO / IEC and ITU-T, and standardization is being promoted as a VVC (Versatile Video Coding) coding method (hereinafter, VVC).

Special Table 2014-506439

Similar to HEVC, VVC is also considering the introduction of prediction technology using encoded block pixels and motion vectors. In such a prediction technology, the pixels and motion vectors of blocks adjacent to the left and above, upper left and upper right of the block to be encoded are used, but especially when parallel processing is performed in block column units, some blocks There were cases where pixels and motion vectors could not be referenced. As a result, such prediction processing may hinder the implementation of parallel processing.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to reduce the complexity of implementation by limiting the blocks that can be used for prediction processing to some blocks.

In order to solve this problem, for example, the image coding apparatus of the present invention has the following configuration. That is,
An image coding device that encodes an image.
A division means for dividing a frame image to be encoded into basic blocks of a predetermined size and further dividing the basic block into a plurality of subblocks.
It has a prediction means for executing intra-prediction or inter-prediction for the subblocks obtained by the division means.
The prediction means is
In the case where the intra prediction is executed for the attention subblock in the attention basic block, and the subblock located at the upper right of the attention subblock is in the basic block located at the upper right of the attention basic block. The subblock located at the upper right of the attention subblock is always excluded from the reference target in the intra prediction for the attention subblock, and the intra prediction is performed.

According to the present invention, the complexity of implementation can be reduced by limiting the blocks that can be used for prediction processing to some blocks depending on the positional relationship between the blocks and adjacent blocks.

Other features and advantages of the present invention will be clarified by the following description with reference to the accompanying drawings. In the attached drawings, the same or similar configurations are designated by the same reference numbers.

The accompanying drawings are included in the specification, form a part thereof, show an embodiment of the present invention, and are used together with the description to explain the principle of the present invention.
The block block diagram of the image coding apparatus in 1st Embodiment. The block block diagram of the image decoding apparatus in 1st Embodiment. The flowchart which shows the image coding process in 1st Embodiment. The flowchart which shows the image decoding processing in 1st Embodiment. The block diagram which shows the hardware structure of the computer applicable to the image coding apparatus and decoding apparatus of 2nd Embodiment. The figure which shows an example of the data structure of a bit stream in an embodiment. The figure which shows an example of the sub-block division in an embodiment. The figure which shows an example of the sub-block division in an embodiment. The figure which shows an example of the sub-block division in an embodiment. The figure which shows an example of the sub-block division in an embodiment. The figure which shows an example of the sub-block division in an embodiment. The figure which shows an example of the sub-block division in an embodiment. The figure which shows the relationship of the block used for the prediction processing in the coding processing in embodiment. The figure which shows the relationship of the block used for the prediction processing in the coding processing in embodiment. The figure which shows the relationship of the block used for the prediction processing in the coding processing in embodiment. The figure which shows the relationship of the block used for the prediction processing in the coding processing in embodiment. The figure which shows the relationship of the block used for the prediction processing in the coding processing in embodiment. The figure which shows an example of the scanning method when performing parallel coding of a basic block.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted. In addition, the names such as basic block and sub-block are names used for convenience in each embodiment, and other names may be used as appropriate as long as their meanings do not change. For example, a basic block or subunit may be referred to as a basic unit or subunit, or may simply be referred to as a block or unit. In addition, a block of interest such as a block to be processed at present is referred to as a block of interest. For example, the basic block to be processed at present is called a attention block. Further, the subblock to be processed at present is referred to as a attention subblock.

Further, in each of the following embodiments, restrictions on subblocks that can be used in intra-prediction and restrictions on sub-blocks that can be selected as motion vector predictors in inter-prediction will be described. Only one of these restrictions may be applied, or both may be applied. Even when only one of them is applied, it is possible to contribute to the reduction of implementation complexity and the reduction of memory usage.

[First Embodiment]
FIG. 1 is a block composition of the image coding apparatus of this embodiment. The image coding device has a control unit 150 that controls the entire device. The control unit 150 includes a CPU, a ROM for storing a program executed by the CPU, and a RAM used as a work area of the CPU. Further, the image coding apparatus includes an input terminal 101, a block division unit 102, a prediction unit 103, a conversion / quantization unit 104, an inverse quantization / inverse conversion unit 105, an image reproduction unit 106, a frame memory 107, and an in-loop filter unit. It has 108, a coding unit 109, an integrated coding unit 110, an output terminal 111, and a motion vector memory 112.

The input terminal 101 inputs the image data to be encoded supplied from the image data supply unit (not shown) in frame units. The image data generation source may be of any type, such as an image pickup device that generates image data to be encoded, a file server or a storage medium that stores the image data to be encoded. Further, the output terminal 112 outputs the coded data to the output destination device, but the output destination device is not particularly limited to a storage medium, a file server, or the like. For example, the output terminal 112 may be a network interface, or may output encoded data to the outside via the network.

The block division unit 102 divides the input frame image into a plurality of basic blocks, and outputs the image data of each basic block unit to the prediction unit 103 in the subsequent stage. Specifically, the block division unit 102 outputs the image data of the basic block to the prediction unit 103 in the order of raster scan with the basic block as a unit.

Here, it should be noted that when coding is performed sequentially in the order of raster scan for each basic block, the upper left, upper, upper right, and upper left, upper, upper right, and among the eight basic blocks adjacent to the basic block of interest to be encoded, A total of four basic blocks adjacent to the left are encoded, and a total of four basic blocks adjacent to the right, lower left, lower, and lower right are uncoded.

The prediction unit 103 divides the image data in basic block units into sub-blocks, performs intra-frame prediction, which is intra-frame prediction, inter-prediction, which is inter-frame prediction, and the like, and generates prediction image data. .. Further, the prediction unit 103 calculates and outputs a prediction error from the attention subblock and the prediction image data in the input image data. For example, the prediction error is the difference between the attention subblock and the prediction image data of the attention subblock. Further, the prediction unit 103 also outputs information necessary for prediction, for example, information such as sub-block division, prediction mode, motion vector, and the like together with the prediction error. Hereinafter, the information necessary for this prediction is referred to as prediction information. This prediction information will be used for prediction in other blocks and for generating prediction image data on the decoding side.

The conversion / quantization unit 104 obtains a conversion coefficient by orthogonally converting the prediction error input from the prediction unit 103 in units of subblocks. Further, the conversion / quantization unit 104 performs quantization on the obtained conversion coefficient to obtain a quantization coefficient. It should be noted that the function of performing orthogonal conversion and the function of performing quantization may be configured separately.

The inverse quantization / inverse conversion unit 105 inversely quantizes the quantization coefficient output from the conversion / quantization unit 104 to reproduce the conversion coefficient, and further performs inverse orthogonal conversion to reproduce the prediction error. The function of performing inverse quantization and the function of performing inverse orthogonal transformation may be configured separately.

Based on the prediction information output from the prediction unit 103, the image reproduction unit 106 generates the prediction image data by appropriately referring to the frame memory 107, and generates the reproduction image data from the generated prediction image data and the input prediction error. Generate and store in frame memory 107. For example, the image reproduction unit 106 reproduces the reproduced image data by adding a prediction error to the predicted image data.

The in-loop filter unit 108 performs in-loop filter processing such as a deblocking filter and a sample adaptive offset on the reproduced image data stored in the frame memory 107, and re-stores the filtered image data in the frame memory 107. ..

The motion vector memory 112 holds the motion vector of the encoded subblock. Therefore, the motion vector memory 112 holds the motion vector in the prediction information output by the prediction unit 103 in association with the position of the subblock to be encoded.

The coding unit 109 encodes the quantization coefficient output from the conversion / quantization unit 104 and the prediction information output from the prediction unit 103, generates code data, and outputs the code data.

The integrated coding unit 110 generates header code data encoded in the header part of the sequence or picture, which is necessary for decoding the bit stream on the decoding side. Further, the integrated coding unit 110 forms a bit stream in which the code data output from the coding unit 109 is integrated with the header code data, and outputs the bit stream via the output terminal 111.

Here, the image coding operation in the image coding device will be described in more detail below. In the present embodiment, moving image data is input in frame units. Further, in the present embodiment, for the sake of explanation, the block division unit 102 will be described as being divided into basic blocks of 64 × 64 pixels, but the present invention is not limited to this. For example, the size of the basic block may be 128 × 128 pixels.

Note that a notation such as 64 x 64 pixels indicates that the height of the block in the vertical direction is 64 pixels and the width in the horizontal direction is 64 pixels. Similarly, the notation 32 × 64 pixels indicates that the height of the block in the vertical direction is 32 pixels and the width in the horizontal direction is 64 pixels. Hereinafter, the same notation shall indicate the vertical height and the horizontal width of the block as in the above example.

The image data for one frame input via the input terminal 101 is supplied to the block dividing unit 102. The block division unit 102 divides the input image data into a plurality of basic blocks, and outputs the image data in basic block units to the prediction unit 103. In this embodiment, it is assumed that an image of a basic block unit of 64 × 64 pixels is output.

The prediction unit 103 executes prediction processing on the image data of the basic block unit input from the block division unit 102. Specifically, the prediction unit 103 determines the sub-block division that divides the basic block into smaller sub-blocks, and further refers to the motion vector memory 112 and the frame memory 107 in sub-block units for intra-prediction and inter-prediction. Determine the prediction mode such as.

An example of the subblock division method is shown with reference to FIGS. 7A to 7F. The thick frames of blocks 700 to 705 in FIGS. 7A to 7F represent basic blocks (64 × 64 pixels in the embodiment), and each quadrangle divided by a thin line inside thereof represents a subblock. FIG. 7A is an example in which the basic block 700 is a subblock. That is, this is an example in which the basic block is not divided and the size of the subblock is 64 × 64 pixels. FIG. 7B shows a division example in which the basic block 701 is divided into four square sub-blocks, and the size of one sub-block is 32 × 32 pixels. 7C to 7F show an example of rectangular subblock division. FIG. 7C shows that the basic block 702 is divided into two sub-blocks (longitudinal in the vertical direction) having a size of 32 × 64 pixels. FIG. 7D shows that the basic block 703 is divided into two subblocks (horizontally longitudinal) having a size of 64 × 32 pixels. In the case of the

basic blocks

704 and 705 of FIGS. 7E and 7F, although the division directions are different, they are divided into three rectangular subblocks at a ratio of 1: 2: 1. In this way, not only squares but also rectangular sub-blocks are used for encoding processing.

In the embodiment, for simplification of the description, as shown in FIG. 7B, the basic block 701 of 64 × 64 pixels is divided into quadtrees and divided into four sub-blocks of 32 × 32 pixels. However, the subblock division method is not limited to this. A binary tree division as shown in FIGS. 7C and 7D, a ternary tree division as shown in FIGS. 7E and 7F, or an undivided tree as shown in FIG. 7A may be used.

The prediction process by the prediction unit 103 will be described in detail. In image coding technology such as HEVC, in order to reduce the amount of data in the encoded bitstream while maintaining the image quality of the reproduced image, the pixels of the encoded subblock are used to encode the subblock to be encoded. The process of predicting the pixels of In the prediction process, intra-prediction using pixels of the coded subblock in the frame in which the subblock to be encoded exists, or a sub of the coded frame different from the frame in which the subblock to be coded exists. There are inter-predictions that use block pixels. Also in this embodiment, these two types of prediction methods are used.

When performing intra prediction, the prediction unit 103 is spatially located in the periphery of the subblock to be encoded, and uses the encoded pixels to generate prediction pixels of the subblock to be encoded (hereinafter, hereinafter, The subblock to be encoded is also called a subblock of interest, and the basic block including the subblock to be encoded is also called a basic block of interest). At this time, the prediction unit 103 describes an intra prediction method such as horizontal prediction using the coded pixels of the subblock adjacent to the left of the subblock of interest and vertical prediction using the coded pixels of the subblock adjacent above. The method used for the prediction of the subblock of interest is determined from the inside, and the information indicating it is generated as the intra prediction mode. The peripheral sub-blocks used for intra-prediction are not limited to those located on the left and above, and those located on the lower left, upper left, and upper right are also used if they are encoded. The relationship between the subblock to be encoded and the surrounding subblocks used for intra-prediction will be described in more detail with reference to FIGS. 8A to 8E.

In FIG. 8A, the thick frame shows a basic block of 64 × 64 pixels, and the thin frame obtained by dividing the thick frame into quadtrees shows a subblock of 32 × 32 pixels. 8B to 8E show a subblock of interest (C: Current) to be encoded and five subblocks {BL: Bottom Left, L: Left, located in the periphery {lower left, left, upper left, upper, upper right}. It shows the relationship with UL: Upper Left, U: Upper, UR: Upper Right}.

FIG. 8B shows the positional relationship with the five surrounding subblocks when the attention subblock (C) is located at the upper left of the attention basic block. Whether or not the attention subblock (C) is located at the upper left of the attention basic block can be determined by whether or not the attention subblock (C) includes the upper left pixel of the attention basic block. it can. In FIG. 8B, since the coding processing of the five subblocks UL, U, UR, L, and BL is completed at the time of the prediction processing of the coding target subblock (C), these five subblocks are used for the intra. It is possible to perform prediction processing. Depending on the prediction method, the intra-prediction processing of the attention subblock (C) can be performed using all of these five subblocks, or the attention subblock using only a part of the five subblocks. The intra prediction process of (C) may be performed.

FIG. 8C shows the positional relationship with the five surrounding subblocks when the attention subblock (C) is located in the upper right of the attention basic block. Whether or not the attention subblock (C) is located at the upper right of the attention basic block can be determined by whether or not the attention subblock (C) includes the pixel at the upper right end of the attention basic block. In FIG. 8C, at the time of the prediction processing of the coding target subblock (C), the coding processing of the lower left (BL) subblock among the five subblocks is not completed. Therefore, the subblock (BL) cannot be used for the intra prediction process.

In addition, the upper right (UR) subblock is coded when the coding process is sequentially performed. Here, in order to increase the speed of the coding process, when the coding process is executed in parallel in units of basic blocks, in particular, each basic block line (block line) is executed in parallel with a delay of one basic block. Consider the case. That is, consider a case where a separate coding unit is used for each block line and the coding process is started with a delay of one basic block. FIG. 9 shows that the four parallel coding processes are performed by delaying the four basic block lines from left to right by one basic block. The four shaded areas in the figure indicate the basic blocks encoded in parallel. When one of the basic blocks in the 2nd to 4th lines, excluding the coding process of the top basic block line, is applied to FIG. 8C, the sub in the upper right of the subblock (C) of interest is predicted. It can be seen that the block (UR) coding process is not completed (during coding).

Here, in the case where the coding process is executed in parallel as described above and the attention subblock is a subblock located in the upper right of the attention basic block, the sub is located in the upper right of the attention subblock. It is conceivable to perform prediction processing without using a block (UR). However, only in such a case, if the use of the subblock (UR) located at the upper right of the subblock of interest in the prediction processing is prohibited (restricted), the difficulty level in implementing the coding method is correspondingly increased. There are problems such as high speed, complicated coding process, slowdown of coding process, and increase of memory usage.

Therefore, the prediction unit 103 of the present embodiment always considers the attention subblock when the attention subblock is a subblock located at the upper right of the attention basic block, regardless of whether or not the coding processing is executed in parallel. It is assumed that the prediction process of the subblock of interest is performed without using the subblock (UR) located at the upper right of. In other words, if the upper right subblock of the attention subblock is in the basic block located at the upper right of the attention basic block, the upper right subblock is excluded from the subblocks used in the prediction process. As a result, it is possible to reduce the mounting cost such as holding the pixel value caused by the reference of the upper right basic block regardless of whether or not the coding processing is executed in parallel. In short, in the embodiment, when the subblock (C) to be encoded matches the state of FIG. 8C, the prediction unit 103 excludes the upper right (UR) and the lower left (BL) during the prediction processing (these Intra-prediction processing is performed using three sub-blocks (restricted from using blocks), left (L), upper left (UL), and upper (U).

Similarly, FIG. 8D shows the positional relationship between the attention subblock (C) and the surrounding five subblocks when it is located at the lower left of the basic block. Whether or not the attention subblock (C) is located at the lower left in the attention basic block can be determined by whether or not the attention subblock (C) includes the pixel at the lower left end in the attention basic block. In FIG. 8D, at the time of the prediction processing of the attention subblock (C), the coding processing of the lower left (BL) subblock is not completed. Therefore, the subblock (BL) cannot be used for the intra prediction process. However, the remaining four subblocks, left (L), upper left (UL), upper (U), and upper right (UR), have been encoded. Therefore, the prediction unit 103 does not use the lower left (BL) subblock, but refers to these four subblocks of left (L), upper left (UL), upper (U), and upper right (UR) for intra-prediction. Perform processing.

Finally, FIG. 8E shows the positional relationship between the attention subblock (C) and the surrounding five subblocks when it is located at the lower right of the basic block. Whether or not the attention subblock (C) is located at the lower right of the attention basic block can be determined by whether or not the attention subblock (C) includes the lower right pixel in the attention basic block. .. In FIG. 8E, at the time of the prediction processing of the attention subblock (C), the coding processing of the lower left (BL) and upper right (UR) subblocks is not completed. Therefore, the subblock (BL) and the subblock (UR) cannot be used for the intra prediction process. Therefore, in the case of the figure, the prediction unit 103 does not use the lower left (BL) and upper right (UR) subblocks, and the remaining three subblocks, left (L), upper left (UL), and upper (U). Intra-prediction processing is performed using.

On the other hand, inter-prediction is a process of predicting the pixels of the block of interest (sub-block to be encoded) by referring to the pixels of the encoded frame (frame different from the frame to which the sub-block to be encoded belongs). For example, if there is no movement between the coded frame of the reference target and the coded frame of the reference target, the pixels of the coded target subblock are predicted using the pixels at the same position of the coded frame of the reference target. .. In such a case, a motion vector (0, 0) indicating that there is no motion is generated. On the other hand, when motion is generated between frames for the subblock to be encoded, a motion vector (MVx, MVy) indicating the motion is generated.

Image coding standards such as HEVC employ a technique called "motion vector prediction" in order to further reduce the amount of data related to this motion vector. This is a technique for predicting the motion vector of the subblock to be encoded by using the motion vector of another encoded subblock, and has the effect of reducing the amount of data of the motion vector. For example, when the sub-block C in FIG. 8B is a part of a moving object, the sub-blocks L, U, UR, L, BL, etc. around the sub-block C are moved by a motion vector predictor. It is selected as a candidate, and the motion vector closest to the motion vector of the subblock C is used as the motion vector predictor. Then, for example, the difference between the motion vector predictor and the motion vector of the subblock to be encoded is encoded.

As in the case of intra-prediction, the prediction unit 103 of the present embodiment limits the sub-blocks used for motion vector prediction according to the positional relationship between the sub-block C of interest to be encoded and the sub-blocks around it. In other words, the subblocks from which the motion vector can be selected (enabled) as the motion vector predictor are made different according to the positional relationship with the surrounding subblocks.

For example, in the case of FIG. 8B, that is, when the coded subblock (C) is located at the upper left of the basic block, the lower left (BL), left (L), upper left (UL), upper (U), upper right ( Since all five subblocks of UR) are encoded, their motion vectors (stored in the motion vector memory 112) can be selected as motion vector predictors. The prediction unit 103 performs motion vector prediction using the motion vectors of all five subblocks as candidates for the motion vector predictor. However, these sub-blocks may not have a motion vector, in which case the motion vector of that sub-block cannot be used. For example, if a subblock is encoded using intra-prediction, that block does not have a motion vector. This also applies to the following description of FIGS. 8C to 8E.

Further, in the present embodiment, a case where it is possible to perform motion vector prediction using the motion vectors of all five subblocks as candidates for the motion vector predictor will be described depending on the situation. The maximum number of subblocks that can be used for is not limited to these five subblocks, but may be a part of the five subblocks, or other subblocks may be added. However, in any case, as will be described later, the subblocks that can be used are limited depending on the position of the subblock (C) to be encoded.

On the other hand, in the case of FIG. 8C, that is, when the coded subblock (C) is located at the upper right of the basic block, the prediction unit 103 uses the unencoded lower left (BL) sub as in the case of intra prediction. The motion vector of the block and the motion vector of the upper right (UR) subblock belonging to the upper right basic block of the current basic block are not always used for the motion vector prediction of the coded subblock (C). In other words, in this case, it is prohibited (restricted) that the motion vector of the lower left (BL) block is a candidate for the motion vector predictor. It is also prohibited (restricted) to use the motion vector of the upper right (UR) block as a candidate for the motion vector predictor.

Therefore, the prediction unit 103 performs motion vector prediction using the motion vectors of the remaining three sub-blocks (L), upper left (UL), and upper (U) as candidates for the motion vector predictor. As a result, as in the case of intra prediction, the motion vector of the lower left (BL) subblock and the motion vector of the upper right (UR) subblock are different regardless of whether or not the coding processing is executed in parallel. Since it is prohibited to be a candidate for a motion vector predictor, it is possible to reduce the implementation cost such as holding the pixel value caused by the reference of the upper right basic block.

Similarly, in the case of FIG. 8D, that is, when the coded subblock (C) is located at the lower left of the basic block, the prediction unit 103 uses the unencoded lower left (BL) motion vector to predict the motion vector. Not used. That is, in this case, it is prohibited (restricted) that the motion vector at the lower left (BL) is a candidate for the motion vector predictor. Therefore, the prediction unit 103 performs motion vector prediction using the motion vectors of the remaining four sub-blocks (L), upper left (UL), upper (U), and upper right (UR) as candidates for the motion vector predictor.

Finally, in the case of FIG. 8E, that is, when the coded subblock (C) is located at the lower right of the basic block, the prediction unit 103 uses the unencoded lower left (BL) motion vector and upper right (UR). ) Motion vector is not used for motion vector prediction. That is, in this case, it is prohibited (restricted) that the motion vector of the lower left (BL) block is a candidate for the motion vector predictor. It is also prohibited (restricted) to use the motion vector of the upper right (UR) block as a candidate for the motion vector predictor. Therefore, the prediction unit 103 uses the motion vectors of the remaining three sub-blocks (L), upper left (UL), and upper (U) as candidates for the motion vector predictor to perform motion vector prediction.

The prediction unit 103 generates prediction image data from the intra prediction mode determined as described above, the motion vector, and the encoded pixels, and further generates a prediction error from the input image data and the prediction image data. Output to the conversion / quantization unit 104. Further, the prediction unit 103 outputs information such as subblock division, intra prediction mode, and motion vector as prediction information to the coding unit 109, the image reproduction unit 106, and the motion vector memory 112.

The conversion / quantization unit 104 performs orthogonal conversion / quantization on the input prediction error and generates a quantization coefficient. Specifically, the conversion / quantization unit 104 first performs an orthogonal conversion process corresponding to the size of the subblock to generate an orthogonal conversion coefficient. Next, the conversion / quantization unit 104 quantizes the orthogonal conversion coefficient and generates the quantization coefficient. Then, the conversion / quantization unit 104 outputs the generated quantization coefficient to the coding unit 109 and the inverse quantization / inverse conversion unit 105.

The inverse quantization / inverse conversion unit 105 inversely quantizes the input quantization coefficient to reproduce the conversion coefficient, and further performs inverse orthogonal conversion of the reproduced conversion coefficient to reproduce the prediction error. Then, the inverse quantization / inverse conversion unit 105 outputs the reproduced prediction error to the image reproduction unit 106.

The image reproduction unit 106 appropriately refers to the frame memory 107 and the motion vector memory 112 based on the prediction information input from the prediction unit 103, and reproduces the predicted image. When intra-prediction is used, the image reproduction unit 106 reproduces the predicted image using the pixels of the peripheral encoded subblocks stored in the frame memory 107. In this case, the subblocks used for intra-prediction are limited according to the positional relationship between the coded subblock and the peripheral coded subblocks, and the limiting method is the limiting method in the prediction processing of the prediction unit 103. Since the same is true, the description here will be omitted. Similarly, when inter-prediction is used, the image reproduction unit 106 uses the pixels of the encoded subblocks of other frames stored in the frame memory 107 based on the motion vector determined by the prediction unit 103. Is used to generate a predicted image. In this case, the subblocks used for motion vector prediction are limited according to the positional relationship between the coded subblock and the peripheral coded subblocks, and the limiting method is the limiting method in the prediction process of the prediction unit 103. Since it is the same as the above, the description here is omitted. The image reproduction unit 106 then reproduces the image data from the reproduced predicted image and the reproduced prediction error input from the inverse quantization / inverse conversion unit 105, and stores the reproduced image data in the frame memory 107.

The in-loop filter unit 108 reads the reproduced image from the frame memory 107 and performs in-loop filter processing such as a deblocking filter. Then, the in-loop filter unit 108 re-stores the filtered image data in the frame memory 107.

The coding unit 109 entropy-encodes the quantization coefficient generated by the conversion / quantization unit 104 and the prediction information input from the prediction unit 103 in block units to generate code data. The method of entropy coding is not particularly limited, but Golomb coding, arithmetic coding, Huffman coding and the like can be used. The coding unit 109 outputs the generated code data to the integrated coding unit 110.

The integrated coding unit 110 generates header code data encoded in the header part of the sequence or picture, which is necessary for decoding the bit stream on the decoding side. The integrated coding unit 110 further multiplexes (integrates) the code data and the like input from the coding unit 109 so as to follow the header code data to form a bit stream. Then, the coding unit 109 outputs the formed bit stream to the outside via the output terminal 111.

FIG. 6 shows an example of the data structure of the bitstream output in the first embodiment. Prior to the picture data, which is the coded data for each picture, there are a sequence header that includes header information for each sequence and a picture header that includes header information for each picture.

FIG. 3 is a flowchart showing the coding process for one frame of the control unit 150 in the image coding device of the embodiment.

First, prior to image coding, in S301, the control unit 150 controls the integrated coding unit 110 to determine the size of the image (the number of pixels in the horizontal and vertical directions of one frame), the size of the basic block, and the like. The header information necessary for encoding the image data and decoding the bit stream on the decoding side is encoded and output.

In S302, the control unit 150 controls the block division unit 102, divides the input frame image into basic block units, and outputs one of them to the prediction unit 103.

In S303, the control unit 150 controls the prediction unit 103 to execute prediction processing on the image data of the basic block unit generated in S302, and performs prediction information such as subblock division information and prediction mode and prediction image data. Generate. The prediction unit 103 further calculates a prediction error from the input image data and the predicted image data, and outputs the prediction error.

In S304, the control unit 150 controls the conversion / quantization unit 104 to perform orthogonal conversion and quantization. As a result, the conversion / quantization unit 104 executes orthogonal conversion with respect to the prediction error calculated in S303 and generates a conversion coefficient. Then, the conversion / quantization unit 104 performs quantization with respect to the conversion coefficient to generate a quantization coefficient.

In S305, the control unit 150 controls the inverse quantization / inverse conversion unit 105 to perform inverse quantization and inverse orthogonal conversion. As a result, the inverse quantization / inverse conversion unit 105 dequantizes the quantization coefficient generated in S304 and reproduces the conversion coefficient. Further, the inverse quantization / inverse conversion unit 105 performs inverse orthogonal conversion with respect to the conversion coefficient and reproduces the prediction error.

In S306, the control unit 150 controls the image reproduction unit 106 to reproduce the image. Specifically, the image reproduction unit 106 reproduces the predicted image based on the prediction information generated in S303. Further, the image reproduction unit 106 reproduces the image data from the reproduced predicted image and the prediction error generated in S305, and stores the image data in the frame memory 107.

In S307, the control unit 150 controls the coding unit 109 to encode the image. Specifically, the coding unit 109 encodes the prediction information generated in S303 and the quantization coefficient generated in S304 to generate code data. In addition, the coding unit 109 generates a bit stream including other code data.

In S308, the control unit 150 determines whether or not the coding of all the basic blocks in the frame of interest is completed, and if so, proceeds to S309 if the coding is completed, and there is an uncoded basic block. If it is determined, the process is returned to S302 in order to encode the basic block.

In S309, the control unit 150 controls the in-loop filter unit 108 to execute the filter processing. Specifically, the in-loop filter unit 108 performs in-loop filter processing on the image (image data reproduced in S306) stored in the frame memory 107, and transfers the filtered image data to the frame memory 107. Restore and finish this process.

In the above configuration and operation, especially in S303, the basic block to which the subblock to be encoded (the subblock of interest) belongs regardless of whether or not each block line composed of a plurality of basic blocks is encoded in parallel. By performing the prediction processing without referring to the subblocks belonging to the basic block adjacent to the upper right of the above, the complexity of implementation can be reduced and the feasibility of parallel processing of coding can be improved.

FIG. 2 is a block configuration diagram of an image decoding device that performs intra-predictive decoding and inter-predictive decoding in sub-block units from the coded image data generated by the image coding device in the embodiment. Hereinafter, the configuration and its operation related to the decoding process will be described with reference to the figure. In the decoding process for decoding the coded image data, the restrictions on the subblocks that can be used in the above-mentioned intra-prediction and the restrictions on the sub-blocks that can be selected as the motion vector predictor in the inter-prediction are the same.

The image decoding device has a control unit 250 that controls the entire device. The control unit 250 has a CPU, a ROM for storing a program executed by the CPU, and a RAM used as a work area of the CPU. Further, the image decoding device includes an input terminal 201, a separation decoding unit 202, a decoding unit 203, an inverse quantization / inverse conversion unit 204, an image reproduction unit 205, a frame memory 206, an in-loop filter unit 207, an output terminal 208, and the like. It has a motion vector memory 209.

The input terminal 201 inputs a coded bit stream, and the input source is, for example, a storage medium storing the coded stream, but it may be input from a network, and the type thereof does not matter.

The separation / decoding unit 202 separates the bitstream into code data related to information related to decoding processing and coefficients, and decodes the code data existing in the header part of the bitstream. The separation / decoding unit 202 performs the reverse operation of the integrated coding unit 110 of FIG.

The decoding unit 203 decodes the code data output from the separation decoding unit 202 and reproduces the quantization coefficient and the prediction information. Then, the decoding unit 203 outputs the obtained quantization coefficient and prediction information to the inverse quantization / inverse conversion unit 204 and the image reproduction unit 205. Further, the decoding unit 203 outputs and stores the motion vector information included in the prediction information in the motion vector memory 209.

The inverse quantization / inverse conversion unit 204 performs inverse quantization on the quantization coefficient input from the decoding unit 203 to obtain a conversion coefficient. Further, the inverse quantization / inverse conversion unit 204 performs inverse orthogonal transformation on the conversion coefficient and reproduces the prediction error.

The image reproduction unit 205 appropriately refers to the frame memory 206 based on the input prediction information to generate the prediction image data. Then, the image reproduction unit 205 generates the reproduction image data from the predicted image data and the prediction error reproduced by the inverse quantization / inverse conversion unit 204, and outputs (stores) the reproduced image data to the frame memory.

Similar to the in-loop filter unit 108 of FIG. 1, the in-loop filter unit 207 performs an in-loop filter process such as a deblocking filter on the reproduced image (stored in the frame memory 206), and the filtered image. Is output.

The output terminal 208 sequentially outputs the frame images stored in the frame memory 206 to the outside. The output destination is generally a display device, but other devices may be used.

The operation related to image decoding of the image decoding device of the above embodiment will be described in more detail. In the present embodiment, the encoded bit stream is input in frame units.

In FIG. 2, the separation / decoding unit 202 inputs or receives a bit stream for one frame via the input terminal 201. The separation / decoding unit 202 separates the input bit stream into information related to the decoding process and code data related to the coefficient, and decodes the code data existing in the header unit of the bit stream. Subsequently, the separation / decoding unit 202 reproduces the code data of the basic block unit of the picture data and outputs the code data to the decoding unit 203.

The decoding unit 203 decodes the code data and reproduces the quantization coefficient and the prediction information. Then, the decoding unit 203 outputs the quantization coefficient to the inverse quantization / inverse conversion unit 204 and the prediction information to the image reproduction unit 205, respectively. Further, the decoding unit 203 outputs and stores the motion vector in the prediction information to the motion vector memory 209 so as to correspond to the position of the subblock.

The inverse quantization / inverse conversion unit 204 performs inverse quantization on the input quantization coefficient to generate an orthogonal conversion coefficient. Further, the inverse quantization / inverse conversion unit 204 performs inverse orthogonal transformation on the generated orthogonal transformation coefficient to reproduce the prediction error. Then, the inverse quantization / inverse conversion unit 204 outputs the reproduced prediction error to the image reproduction unit 205.

The image reproduction unit 205 reproduces the predicted image by referring to the prediction information and the motion vector memory 209 input from the decoding unit 203 and appropriately referring to the frame memory 206. In this embodiment, two types of prediction methods, intra-prediction and inter-prediction, are used as in the prediction unit 103 of the image coding apparatus. The subblocks used for the prediction processing are limited according to the positional relationship between the decoding target subblock and the peripheral decoded subblocks. However, for specific prediction image reproduction processing, the image reproduction unit 106 in the image coding apparatus Since the same is true, the description thereof will be omitted. The image reproduction unit 205 reproduces the image data from the predicted image and the prediction error input from the inverse quantization / inverse conversion unit 204, and stores the reproduced image data in the frame memory 206. The stored image data will be used as a reference when making a prediction.

Note that the image reproduction unit 205 also limits the sub-blocks that can be used in the intra-prediction, as in the prediction unit 103 of the image coding device. Therefore, it is not necessary to store unnecessary data for reference in other blocks, and it is easy to determine whether to store data for other blocks, so that the memory usage amount It is possible to reduce the complexity of implementation and reduce the complexity of implementation. Further, similarly to the prediction unit 103 of the image coding device, the subblocks that can be selected as the motion vector predictor in the inter-prediction are also limited. This also contributes to the reduction of implementation complexity and the reduction of memory usage.

Like the in-loop filter unit 108 of the image coding device, the in-loop filter unit 207 reads the reproduced image data from the frame memory 206 and performs in-loop filter processing such as a deblocking filter. Then, the in-loop filter unit 207 stores the filtered image in the frame memory 206 again. The reproduced image stored in the frame memory 206 and filtered is finally output from the terminal 208 to the outside (display or the like).

FIG. 4 is a flowchart showing a one-frame image decoding process of the control unit 250 in the image decoding device according to the embodiment. The decoding process will be described below with reference to the same figure.

In S401, the control unit 250 controls the separation / decoding unit 202 to separate the bit stream into information related to the decoding process and code data of the image data, acquire information required for decoding, and code data of the image. Is output to the decoding unit 203.

In S402, the control unit 250 controls the decoding unit 203, decodes the code data separated in S401, and reproduces the quantization coefficient and the prediction information.

In S403, the control unit 250 controls the inverse quantization / inverse conversion unit 204 to perform inverse quantization and inverse orthogonal conversion. Specifically, the inverse quantization / inverse conversion unit 204 performs inverse quantization on the quantization coefficient of each subblock under the control of the control unit 250 to obtain a conversion coefficient, and further, the conversion coefficient is obtained. The prediction error is reproduced by performing the inverse orthogonal transformation on the.

In S404, the control unit 250 controls the image reproduction unit 205 to reproduce the image data. Specifically, the image reproduction unit 205 reproduces the predicted image based on the predicted information. Then, the image reproduction unit 205 reproduces the image data by adding the prediction error generated in S404 to the predicted image.

In S405, the control unit 250 determines whether or not the decoding process of the frame of interest has been completed, that is, whether or not the decoding of all the basic blocks in the frame of interest and all the subblocks inside the frame has been completed. Then, when the decoding of the frame of interest is completed, the control unit 250 proceeds to S406, and if there is an undecrypted basic block or subblock, the control unit 250 decodes the next basic block or subblock. , The process is returned to S402.

In S406, the control unit 250 controls the in-loop filter unit 207, performs in-loop filter processing on the image data (image data reproduced in S404) stored in the frame memory 206, and filters the image. Is generated and the process ends.

As described above, according to the image decoding apparatus of the present embodiment, regardless of whether or not each block line is encoded in parallel, the subblock to be decoded (the subblock of interest) belongs to the upper right of the basic block. It is possible to decode a bitstream that has undergone prediction processing without referring to subblocks that belong to adjacent basic blocks. As a result, the complexity of implementation is reduced, and the feasibility of parallel processing of decoding can be improved.

[Second Embodiment]
Each processing unit included in the image coding device and the image decoding device of the above embodiment has been described as being configured by hardware. However, the processing performed by each processing unit shown in these figures may be configured by a computer program.

FIG. 5 is a block diagram showing a configuration example of computer hardware applicable to the image coding device and the image decoding device according to the above embodiment.

The CPU 501 controls the entire computer by using the computer programs and data stored in the RAM 502 and the ROM 503, and executes each of the above-described processes as performed by the image processing device according to the above embodiment. That is, the CPU 501 functions as each processing unit shown in FIGS. 1 and 2.

The RAM 502 has an area for temporarily storing programs and data acquired from the outside via the external storage device 506 and the I / F (interface) 507. Further, the RAM 502 is also used as a work area used by the CPU 501 when executing various processes. The RAM 502 can be allocated as a frame memory, for example, or various other areas can be provided as appropriate.

The ROM 503 stores the setting data of this computer, the boot program, and the like. The operation unit 504 is composed of a keyboard, a mouse, and the like, and can be operated by a user of the computer to input various instructions to the CPU 501. The display unit 505 displays the processing result by the CPU 501. The display unit 505 is composed of, for example, a liquid crystal display.

The external storage device 506 is a large-capacity information storage device represented by a hard disk drive device. The external storage device 506 stores an OS (operating system) and a computer program (application program) for realizing the functions of the respective parts shown in FIGS. 1 and 2 in the CPU 501. Further, each image data as a processing target may be stored in the external storage device 506.

The computer programs and data stored in the external storage device 506 are appropriately loaded into the RAM 502 according to the control by the CPU 501, and are processed by the CPU 501. A network such as a LAN or the Internet, or other devices such as a projection device or a display device can be connected to the I / F 507, and the computer acquires and sends various information via the I / F 507. Can be done. Reference numeral 508 is a bus connecting the above-mentioned parts.

In the above configuration, when the power is turned on to the present device, the CPU 501 executes the boot program stored in the ROM 503, loads the OS stored in the external storage device 506 into the RAM 502, and executes the boot program. Then, under the control of the OS, the CPU 501 loads the application program related to encoding or decoding from the external storage device 506 into the RAM 502 and executes it. As a result, the CPU 501 functions as each processing unit of FIG. 1 or 2, and the present device functions as an image coding device or an image decoding device.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is used in a coding device / decoding device that encodes / decodes still images / moving images. In particular, it can be applied to coding methods and decoding methods that use intra-prediction and inter-prediction.

The present invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the present invention. Therefore, in order to make the scope of the present invention public, the following claims are attached.

This application claims priority based on Japanese Patent Application No. 2019-114933 submitted on June 20, 2019, and all the contents thereof are incorporated herein by reference.

Claims

An image coding device that encodes an image.
A division means for dividing a frame image to be encoded into basic blocks of a predetermined size and further dividing the basic block into a plurality of subblocks.
It has a prediction means for executing intra-prediction or inter-prediction for the subblocks obtained by the division means.
The prediction means is
In the case where the intra prediction is executed for the attention subblock in the attention basic block, and the subblock located at the upper right of the attention subblock is in the basic block located at the upper right of the attention basic block. An image coding apparatus characterized in that a subblock located at the upper right of the attention subblock is always excluded from the reference target in the intra prediction for the attention subblock to perform the intra prediction.
The prediction means is
When the attention subblock is in the first position in the attention basic block,
Pixels of a subblock adjacent to the upper left of the attention subblock, pixels of a subblock adjacent to the attention subblock, pixels of a subblock adjacent to the upper right of the attention subblock, and left of the attention subblock. The image coding apparatus according to claim 1, wherein a predicted image is generated by referring to pixels of a subblock adjacent to the image.
The prediction means is
When the attention subblock is in a second position different from the first position in the attention basic block,
It is characterized in that a prediction image is generated by referring to a subblock adjacent to the upper left of the attention subblock, a subblock adjacent to the attention subblock, and a subblock adjacent to the left of the attention subblock. The image coding apparatus according to claim 1 or 2.
An image decoding device that decodes image data from a bit stream generated by dividing a frame image into basic blocks of a predetermined size, further dividing the basic block into a plurality of subblocks, and encoding each subblock. And
It has a reproduction means for reproducing the image data of the sub-block by executing intra-prediction or inter-prediction using the coded data of the sub-block and the decoded image data.
The regeneration means
Whenever an intra prediction is executed in a attention subblock and the subblock located at the upper right of the attention subblock is within the basic block located at the upper right of the basic block to which the attention subblock belongs. An image decoding apparatus characterized in that a subblock located at the upper right of the attention subblock is excluded from the reference target in the intra prediction for the attention subblock to perform the intra prediction.
The regeneration means
When the attention subblock is in the first position in the attention basic block,
Pixels of a subblock adjacent to the upper left of the attention subblock, pixels of a subblock adjacent to the attention subblock, pixels of a subblock adjacent to the upper right of the attention subblock, and left of the attention subblock. The image decoding apparatus according to claim 4, wherein a predicted image is generated by referring to pixels of a subblock adjacent to the image.
The regeneration means
When the attention subblock is in a second position different from the first position in the attention basic block,
The predicted image is obtained by referring to the pixels of the subblock adjacent to the upper left of the attention subblock, the pixels of the subblock adjacent to the attention subblock, and the pixels of the subblock adjacent to the left of the attention subblock. The image decoding apparatus according to claim 4 or 5, characterized in that it is generated.
An image coding method that encodes an image.
A division step of dividing a frame image to be encoded into basic blocks of a predetermined size and further dividing the basic block into a plurality of sub-blocks.
It has a prediction step of executing intra-prediction or inter-prediction for the sub-blocks obtained in the division step.
In the prediction process,
In the case where the intra prediction is executed for the attention subblock in the attention basic block, and the subblock located at the upper right of the attention subblock is in the basic block located at the upper right of the attention basic block. An image coding method characterized in that a subblock located at the upper right of the attention subblock is always excluded from the reference target in the intra prediction for the attention subblock, and the intra prediction is performed.
An image decoding method for decoding image data from a bit stream generated by dividing a frame image into basic blocks of a predetermined size, further dividing the basic block into a plurality of subblocks, and encoding each subblock. And
It has a reproduction step of reproducing the image data of the sub-block by executing intra-prediction or inter-prediction using the coded data of the sub-block and the decoded image data.
In the regeneration step,
Whenever an intra prediction is executed in a attention subblock and the subblock located at the upper right of the attention subblock is within the basic block located at the upper right of the basic block to which the attention subblock belongs. An image decoding method characterized in that a subblock located at the upper right of the attention subblock is excluded from the reference target in the intra prediction for the attention subblock, and the intra prediction is performed.
A program for causing the computer to execute each step of the method according to claim 7 or 8 by being read and executed by the computer.