RU2808454C1 - Encoder, decoder, encoding method and decoding method - Google Patents

Abstract

FIELD: image encoding and decoding.

SUBSTANCE: result is ensured by determining whether the size of the current block, which is a unit for which a list of possible vector options is generated, is less than or equal to a threshold value; when the size of the current block is less than or equal to the threshold value, generating a candidate vector list by registering a candidate vector using a historic motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process; when the size of the current block exceeds a threshold value, a list of candidate vectors is generated by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table.

EFFECT: increasing the efficiency of image encoding/decoding.

22 cl, 124 dwg, 3 tbl

Description

Field of technology to which the invention relates

[0001] The present disclosure relates to video encoding, and in particular to systems, components and methods for encoding and decoding video.

State of the art

[0002] Due to advancements in video encoding technology, from H.261 and MPEG-1 to H.264/AVC (Advanced Video Coding Standard), MPEG-LA, H.265/HEVC (High Efficiency Video Coding Standard) and H. 266/VVC (Versatile Video Codec), there remains an ongoing need to provide improvements and optimizations to video encoding technology in order to handle the ever-increasing volume of digital video data in a variety of applications. The present disclosure relates to additional enhancements, enhancements, and optimizations in video encoding.

[0003] It should be noted that Non-Patent Document (NPL) 1 refers to one example of a conventional standard regarding the above-described video encoding technology.

List of bibliographic references

Non-patent documents

[0004] NPL 1. H.265 (ISO/IEC 23008-2 HEVC)/HEVC (High Efficiency Video Coding)

The essence of the invention

Technical challenge

[0005] With respect to the encoding scheme, as described above, proposals for new schemes are required in order to (i) improve encoding efficiency, improve image quality, reduce processing volumes, reduce circuit sizes, or (ii) appropriately select an element or operation. An element, for example, is a filter, block, dimension, motion vector, reference frame, or reference block.

[0006] The present disclosure provides, for example, a configuration or method that can facilitate at least one of increasing encoding efficiency, increasing image quality, reducing processing volume, reducing circuit size, appropriate element or operation selection, etc. It should be noted that the present disclosure may cover possible configurations or methods that may provide benefits other than the above benefits.

The solution of the problem

[0007] For example, an encoder according to an aspect of the present disclosure is an encoder including: circuitry; and a memory device connected to the circuit. The circuit, when operating: when forming a predictive image of the current block, uses the intra-block copying (IBC) mode, in which the processed area of the frame is accessed, which includes the current block; determines whether the size of the current block is less than or equal to a threshold value, the current block being a unit for which a list of possible vector options is generated, including possible vector options; when the current block size is less than or equal to the threshold, generates a candidate vector list by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein the HMVP table stores, in a first-in-first-out (FIFO) manner, each of the first vector candidates as a HMVP vector candidate, the first vector candidates having information regarding the first vector used for the processed block; when the size of the current block exceeds a threshold value, generates a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table; and encodes the current block using a list of possible vectors.

[0008] In video encoding technology, new methods must be proposed to improve encoding efficiency, improve image quality, and reduce circuit sizes.

[0009] Each of the embodiments or each of a portion of the constituent elements and methods in the present disclosure enables, for example, at least one of the following: increasing encoding efficiency, increasing image quality, reducing encoding/decoding processing, reducing circuit size , increasing the processing speed of encoding/decoding, etc. Alternatively, each of the embodiments or each of a subset of the constituent elements and methods in the present disclosure allows, during encoding and decoding, an appropriate selection of an element or operation. An element, for example, is a filter, block, dimension, motion vector, reference frame, or reference block. It should be noted that the present disclosure includes disclosure regarding configurations and methods that may provide advantages other than those described above. Examples of such configurations and methods include a configuration or method for increasing encoding efficiency while reducing processing increases.

[0010] Additional benefits and advantages according to an aspect of the present disclosure will become apparent from the detailed description and drawings. Benefits and/or advantages may be separately obtained through various embodiments and features of the detailed description and drawings, and not all of them must necessarily be provided in order to obtain one or more such benefits and/or advantages.

[0011] It should be noted that these general or specific aspects may be implemented using a system, integrated circuit, computer program or computer readable medium, such as a CD-ROM, or any combination of systems, methods, integrated circuits, computer programs and media .

Advantages of the invention

[0012] A configuration or method according to an aspect of the present disclosure enables, for example, at least one of the following: increasing encoding efficiency, increasing image quality, reducing processing volume, reducing circuit size, increasing processing speed, appropriate element or operation selection, and etc. It should be noted that the configuration or method according to an aspect of the present disclosure may provide advantages other than those described above.

Brief description of drawings

[0013] FIG. 1 is a circuit diagram illustrating one configuration example of a transmission system according to an embodiment.

Fig. 2 is a diagram illustrating one example of a hierarchical data structure in a stream.

Fig. 3 is a diagram illustrating one example of a slice configuration.

Fig. 4 is a diagram illustrating one example of a tile configuration.

Fig. 5 is a diagram illustrating one example of a coding structure in scalable coding.

Fig. 6 is a diagram illustrating one example of a coding structure in scalable coding.

Fig. 7 is a block diagram illustrating one example of a functional configuration of an encoder according to the embodiment.

Fig. 8 is a block diagram illustrating an installation example of the encoder.

Fig. 9 is a flowchart illustrating one example of a complete encoding process performed by the encoder.

Fig. 10 is a diagram illustrating one example of block partitioning.

Fig. 11 is a diagram illustrating one example of a functional configuration of a partitioning unit.

Fig. 12 is a diagram illustrating examples of partitioning patterns.

Fig. 13A is a diagram illustrating one example of a split pattern syntax tree.

Fig. 13B is a diagram illustrating another example of a split pattern syntax tree.

Fig. 14 is a diagram illustrating basic transformation functions for each type of transformation.

Fig. 15 is a diagram illustrating examples of SVT.

Fig. 16 is a flowchart illustrating one example of a process performed by the converter.

Fig. 17 is a flowchart illustrating another example of a process performed by the converter.

Fig. 18 is a block diagram illustrating one example of a functional configuration of a quantizer.

Fig. 19 is a flowchart illustrating one example of quantization performed by a quantizer.

Fig. 20 is a block diagram illustrating one example of a functional configuration of an entropy encoder.

Fig. 21 is a diagram illustrating the flow of CABAC operations in an entropy encoder.

Fig. 22 is a block diagram illustrating one example of a functional configuration of a loop filter.

Fig. 23A is a diagram illustrating one example of a filter shape used in an adaptive loop filter (ALF).

Fig. 23B is a diagram illustrating another example of a filter shape used in the ALF.

Fig. 23C is a diagram illustrating another example of a filter shape used in the ALF.

Fig. 23D is a diagram illustrating an example in which Y samples (first component) are used for cross-component ALF (CCALF) for Cb and CCALF for Cr (for components other than the first component).

Fig. 23E is a diagram illustrating a diamond filter.

Fig. 23F is a diagram illustrating an example for joint chrominance CCALF (JC-CCALF).

Fig. 23G is a diagram illustrating an example for possible JC-CCALF weight index options.

Fig. 24 is a block diagram illustrating one example of a particular configuration of a loop filter that functions as a DBF.

Fig. 25 is a diagram illustrating an example of a deblocking filter having a symmetrical filtering characteristic with respect to a block boundary.

Fig. 26 is a diagram for illustrating a block boundary on which a filtering process for deblocking is performed.

Fig. 27 is a diagram illustrating examples of Bs values.

Fig. 28 is a flowchart illustrating one example of a process performed by the encoder prediction module.

Fig. 29 is a flowchart illustrating another example of a process performed by the encoder prediction module.

Fig. 30 is a flowchart illustrating another example of a process performed by the encoder prediction module.

Fig. 31 is a diagram illustrating one example of sixty-seven intra prediction modes used in intra prediction.

Fig. 32 is a flowchart illustrating one example of a process performed by the intra prediction module.

Fig. 33 is a diagram illustrating examples of reference frames.

Fig. 34 is a diagram illustrating examples of reference frame lists.

Fig. 35 is a flowchart illustrating a basic flow of inter-prediction processing.

Fig. 36 is a flowchart illustrating one example of MV extraction.

Fig. 37 is a flowchart illustrating another example of MV extraction.

Fig. 38A is a diagram illustrating one example of mode categorization for MV extraction.

Fig. 38B is a diagram illustrating one example of mode categorization for MV extraction.

Fig. 39 is a flowchart illustrating an example of inter-prediction by normal inter-mode.

Fig. 40 is a flowchart illustrating an example of inter-prediction by normal combining mode.

Fig. 41 is a diagram for illustrating one example of an MV extraction process by the normal combining mode.

Fig. 42 is a diagram for illustrating one example of an MV extraction process by HMVP mode.

Fig. 43 is a flowchart illustrating one example of frame rate upconversion (FRUC).

Fig. 44 is a diagram for illustrating one example of pattern matching (bilateral matching) between two blocks located along a motion path.

Fig. 45 is a diagram for illustrating one example of pattern matching (pattern matching) between a pattern in the current frame and a block in the reference frame.

Fig. 46A is a diagram for illustrating one example of MV extraction in subblock units in affine mode in which two control points are used.

Fig. 46B is a diagram for illustrating one example of MV extraction in subblock units in affine mode in which three control points are used.

Fig. 47A is a conceptual diagram for illustrating one example of MV extraction at control points in affine mode.

Fig. 47B is a conceptual diagram for illustrating one example of MV extraction at control points in affine mode.

Fig. 47C is a conceptual diagram for illustrating one example of MV extraction at control points in affine mode.

Fig. 48A is a diagram for illustrating an affine mode in which two control points are used.

Fig. 48B is a diagram for illustrating an affine mode in which three control points are used.

Fig. 49A is a conceptual diagram for illustrating one example of a method for MV extraction at control points when the number of control points for a coded block and the number of control points for a current block are different from each other.

Fig. 49B is a conceptual diagram for illustrating another example of a method for MV extraction at control points when the number of control points for a coded block and the number of control points for a current block are different from each other.

Fig. 50 is a flowchart illustrating one example of a process in the affine combining mode.

Fig. 51 is a flowchart illustrating one example of an affine reciprocal mode process.

Fig. 52A is a diagram for illustrating the generation of two triangular projection images.

Fig. 52B is a conceptual diagram illustrating examples of the first part of the first segment and the first and second sets of samples.

Fig. 52C is a conceptual diagram illustrating the first part of the first segment.

Fig. 53 is a flowchart illustrating one example of a triangle mode.

Fig. 54 is a diagram illustrating one example of the ATMVP mode in which the MV is extracted in units of subblocks.

Fig. 55 is a diagram illustrating the relationship between the combining mode and dynamic motion vector updating (DMVR).

Fig. 56 is a conceptual diagram to illustrate one example of a DMVR.

Fig. 57 is a conceptual diagram for illustrating another example of a DMVR for determining MV.

Fig. 58A is a diagram illustrating one example of motion estimation in a DMVR.

Fig. 58B is a flowchart illustrating one example of motion estimation in a DMVR.

Fig. 59 is a flowchart illustrating one example of predictive image generation.

Fig. 60 is a flowchart illustrating another example of predictive image generation.

Fig. 61 is a flowchart illustrating one example of a prediction image correction process by overlapping block motion compensation (OBMC).

Fig. 62 is a conceptual diagram for illustrating one example of a process for correcting prediction images by OBMC.

Fig. 63 is a diagram to illustrate the model under the condition of uniform linear motion.

Fig. 64 is a flowchart illustrating one example of inter-prediction according to BIO.

Fig. 65 is a diagram illustrating one example of a functional configuration of an inter-prediction module that performs inter-prediction according to BIO.

Fig. 66A is a diagram for illustrating one example of a method for generating predictive images using a luminance correction process through local illuminance compensation (LIC).

Fig. 66B is a flowchart illustrating one example of a method for generating predictive images using a brightness correction process by LIC.

Fig. 67 is a block diagram illustrating a functional configuration of a decoder according to an embodiment.

Fig. 68 is a block diagram illustrating an installation example of a video decoder.

Fig. 69 is a flowchart illustrating one example of a complete decoding process performed by the decoder.

Fig. 70 is a diagram illustrating the relationship between the partition determination module and other constituent elements.

Fig. 71 is a block diagram illustrating one example of a functional configuration of an entropy decoder.

Fig. 72 is a diagram illustrating the flow of CABAC operations in an entropy decoder.

Fig. 73 is a block diagram illustrating one example of a functional configuration of an inverse quantizer.

Fig. 74 is a flowchart illustrating one example of inverse quantization performed by an inverse quantizer.

Fig. 75 is a flowchart illustrating one example of a process performed by an inverter.

Fig. 76 is a flowchart illustrating another example of a process performed by an inverter.

Fig. 77 is a block diagram illustrating one example of a functional configuration of a loop filter.

Fig. 78 is a flowchart illustrating one example of a process performed by the decoder prediction module.

Fig. 79 is a flowchart illustrating another example of a process performed by the decoder prediction module.

Fig. 80A is a flowchart illustrating part of another example of a process performed by a decoder prediction module.

Fig. 80B is a flowchart illustrating the remainder of another example of a process performed by the decoder prediction module.

Fig. 81 is a diagram illustrating one example of a process performed by the decoder intra prediction unit.

Fig. 82 is a flowchart illustrating one example of MV extraction in a decoder.

Fig. 83 is a flowchart illustrating another example of MV extraction in a decoder.

Fig. 84 is a flowchart illustrating an example of inter-prediction by normal inter-mode in a decoder.

Fig. 85 is a flowchart illustrating an example of inter-prediction by a normal combining mode in a decoder.

Fig. 86 is a flowchart illustrating an example of inter-prediction by FRUC mode in a decoder.

Fig. 87 is a flowchart illustrating an example of inter-prediction by affine combining mode in a decoder.

Fig. 88 is a flowchart illustrating an example of mutual prediction by affine mutual mode in a decoder.

Fig. 89 is a flowchart illustrating an example of inter-prediction by triangular mode in a decoder.

Fig. 90 is a flowchart illustrating an example of motion estimation by DMVR in a decoder.

Fig. 91 is a flowchart illustrating one specific example of motion estimation by DMVR in a decoder.

Fig. 92 is a flowchart illustrating one example of generating a prediction image in a decoder.

Fig. 93 is a flowchart illustrating another example of generating a prediction image in a decoder.

Fig. 94 is a flowchart illustrating another example of prediction image correction by OBMC in a decoder.

Fig. 95 is a flowchart illustrating another example of correcting a prediction image by BIO in a decoder.

Fig. 96 is a flowchart illustrating another example of prediction image correction by LIC in a decoder.

Fig. 97 is a diagram for describing the intra block copy (IBC) mode.

Fig. 98 is a flowchart illustrating one example of operations performed by the encoder and decoder according to the first aspect.

Fig. 99 is a flowchart illustrating one example of operations performed by the encoder and decoder according to the second aspect.

Fig. 100 is a flowchart illustrating operations performed by the encoder.

Fig. 101 is a flowchart illustrating operations performed by the decoder.

Fig. 102 is a diagram illustrating a general configuration of a content delivery system for implementing a content distribution service.

Fig. 103 is a diagram illustrating an example of a web page display screen.

Fig. 104 is a diagram illustrating an example of a web page display screen.

Fig. 105 is a diagram illustrating one example of a smartphone.

Fig. 106 is a block diagram illustrating an example configuration of a smartphone.

Detailed Description of Embodiments

[0014] Introduction

An encoder according to an aspect of the present disclosure is an encoder including: circuitry; and a memory device connected to the circuit. The circuit, when operating: when forming a predictive image of the current block, uses the intra-block copying (IBC) mode, in which the processed area of the frame is accessed, which includes the current block; determines whether the size of the current block is less than or equal to a threshold value, the current block being a unit for which a list of possible vector options is generated, including possible vector options; when the current block size is less than or equal to the threshold, generates a candidate vector list by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein the HMVP table stores, in a first-in-first-out (FIFO) manner, each of the first vector candidates as a HMVP vector candidate, the first vector candidates having information regarding the first vector used for the processed block; when the size of the current block exceeds a threshold value, generates a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table; and encodes the current block using a list of possible vectors.

[0015] Accordingly, the encoder generates a list of candidate vectors without performing a first pruning process (hereinafter simply referred to as a “pruning process”) when the current block size is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the coding efficiency of the encoder.

[0016] For example, the circuit may: generate a predictive image of the current block using the second vector; encode the second vector using one of the possible vectors included in the list of possible vectors; In the first pruning process performed when generating a list of possible vector variants, determine, for each of the possible HMVP vector variants, each of which is a possible HMVP vector variant stored in the HMVP table, whether or not the possible HMVP vector variant is a vector identical to any of one or more candidate vectors registered in the list of candidate vectors; and when the candidate HMVP vector is not identical to any of the one or more candidate vectors, generating a list of candidate vectors by registering the candidate HMVP vector in the list of candidate vectors.

[0017] Accordingly, the encoder generates a list of candidate vectors without performing a pruning process when the current block size is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the coding efficiency of the encoder.

[0018] For example, the circuit may further update the HMVP table using a second vector candidate having information regarding the second vector, and when updating the HMVP table, the circuit may: determine whether the size of the current block is less than or equal to a threshold value; update the HMVP table without performing a second pruning process when the current block size is less than or equal to the threshold; and update the HMVP table by executing a second pruning process when the size of the current block exceeds a threshold value.

[0019] Accordingly, the encoder updates the HMVP table without performing a second pruning process (hereinafter also referred to simply as a “pruning process”) when the size of the current block is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the coding efficiency of the encoder.

[0020] For example, in the second pruning process performed when updating the HMVP table, the circuit may: determine whether or not a candidate second vector is identical to any of the candidate HMVP vectors; and updating the HMVP table by storing the second vector candidate in the HMVP table when the second vector candidate is not identical to any of the HMVP vector candidates.

[0021] Due to this, the encoder can register, in the list of candidate vectors, a candidate vector that is more suitable for the current block.

[0022] For example, the size of the current block may be specified by the total number of pixels included in the current block. The threshold value may be, for example, 16 pixels.

[0023] Accordingly, the encoder generates a list of candidate vectors and updates the HMVP table without performing a pruning process when the total number of pixels included in the current block is less than or equal to a threshold value (for example, less than or equal to 16 pixels), and thereby processing volume is reduced. This improves the coding efficiency of the encoder.

[0024] For example, the size of the current block may be specified by at least one of the width or height of the current block. The threshold value may be, for example, 4 pixels x 4 pixels.

[0025] Accordingly, the encoder generates a list of candidate vectors and updates the HMVP table without performing a pruning process when at least one of the width or height of the current block is less than or equal to a threshold value (for example, when the width and height of the current block is less than or equal to are equal to 4 pixels), and due to this, the amount of processing is reduced. This improves the coding efficiency of the encoder.

[0026] For example, the list of candidate vectors need not be shared between the current block and a block adjacent to the current block.

[0027] Accordingly, the encoder uses lists of candidate vectors, each of which is generated for a different current block, and thereby the prediction accuracy is improved.

[0028] For example, the circuit may: when generating a prediction image of the current block, use an IBC mode when determining whether to use an IBC mode from a plurality of prediction modes, and use a prediction mode other than the IBC mode when determining whether to use a prediction mode other than IBC mode, from a variety of forecasting modes; when using a prediction mode other than the IBC mode, generate a list of possible vector variants by performing a first pruning process and registering a possible HMVP vector variant in the list of possible vector variants from the HMVP table; and encode the current block using a list of possible vectors.

[0029] This allows the encoder to appropriately switch the conditions under which the clipping process is performed according to the prediction mode used. This improves the coding efficiency of the encoder.

[0030] Moreover, the decoder according to an aspect of the present disclosure is a decoder including: circuitry; and a memory device connected to the circuit. The circuit, when operating: when forming a predictive image of the current block, uses the intra-block copying (IBC) mode, in which the processed area of the frame is accessed, which includes the current block; determines whether the size of the current block is less than or equal to a threshold value, the current block being a unit for which a list of possible vector options is generated, including possible vector options; when the current block size is less than or equal to the threshold, generates a candidate vector list by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein the HMVP table stores, in a first-in-first-out (FIFO) manner, each of the first vector candidates as a HMVP vector candidate, the first vector candidates having information regarding the first vector used for the processed block; when the size of the current block exceeds a threshold value, generates a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table; and decodes the current block using a list of candidate vectors.

[0031] Accordingly, the decoder generates a list of candidate vectors without performing a pruning process when the current block size is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the processing efficiency of the decoder.

[0032] For example, the circuit may: generate a predictive image of the current block using the second vector; decode the second vector using one of the candidate vectors included in the list of candidate vectors; In the first pruning process performed when generating a list of possible vector variants, determine, for each of the possible HMVP vector variants, each of which is a possible HMVP vector variant stored in the HMVP table, whether or not the possible HMVP vector variant is a vector identical to any of one or more candidate vectors registered in the list of candidate vectors; and when the candidate HMVP vector is not identical to any of the one or more candidate vectors, generating a list of candidate vectors by registering the candidate HMVP vector in the list of candidate vectors.

[0033] Accordingly, the decoder generates a list of candidate vectors without performing a pruning process when the current block size is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the processing efficiency of the decoder.

[0034] For example, the circuit may further: update the HMVP table using a second vector candidate having information regarding the second vector, and when updating the HMVP table, the circuit may: determine whether the current block size is less than or equal to a threshold value; update the HMVP table without performing a second pruning process when the current block size is less than or equal to the threshold; and update the HMVP table by executing a second pruning process when the size of the current block exceeds a threshold value.

[0035] Accordingly, the decoder updates the HMVP table without performing a pruning process when the current block size is less than or equal to the threshold value, and thereby the processing amount is reduced. This improves the processing efficiency of the decoder.

[0036] For example, in the second pruning process performed when updating the HMVP table, the circuit may: determine whether or not a candidate second vector is identical to any of the candidate HMVP vectors; and updating the HMVP table by storing the second vector candidate in the HMVP table when the second vector candidate is not identical to any of the HMVP vector candidates.

[0037] Due to this, the decoder can register, in the list of candidate vectors, a candidate vector that is more suitable for the current block.

[0038] For example, the size of the current block may be specified by the total number of pixels included in the current block. The threshold value may be, for example, 16 pixels.

[0039] Accordingly, the decoder generates a list of candidate vectors and updates the HMVP table without performing a pruning process when the total number of pixels included in the current block is less than or equal to a threshold value (for example, less than or equal to 16 pixels), and thereby processing volume is reduced. This improves the processing efficiency of the decoder.

[0040] For example, the size of the current block may be specified by at least one of the width or height of the current block. The threshold value may be, for example, 4 pixels x 4 pixels.

[0041] Accordingly, the decoder generates a list of candidate vectors and updates the HMVP table without performing a pruning process when at least one of the width or height of the current block is less than or equal to the threshold value (for example, when the width and height of the current block is less than or equal to are equal to 4 pixels), and due to this, the amount of processing is reduced. This improves the processing efficiency of the decoder.

[0042] For example, the list of candidate vectors need not be shared between the current block and a block adjacent to the current block.

[0043] Accordingly, the decoder uses lists of candidate vectors, each of which is generated for a different current block, and thereby the prediction accuracy is improved.

[0044] For example, the circuit may: when generating a prediction image of the current block, use an IBC mode when determining whether to use an IBC mode from a plurality of prediction modes, and use a prediction mode other than the IBC mode when determining whether to use a prediction mode other than IBC mode, from a variety of forecasting modes; when using a prediction mode other than the IBC mode, generate a list of possible vector variants by performing a first pruning process and registering a possible HMVP vector variant in the list of possible vector variants from the HMVP table; and decode the current block using a list of candidate vectors.

[0045] This allows the decoder to appropriately switch the conditions under which the clipping process is performed according to the prediction mode used. This improves the processing efficiency of the decoder.

[0046] In addition, the encoding method according to an aspect of the present disclosure is an encoding method including: when generating a prediction image of the current block, using an intra-block copy (IBC) mode in which a processed area of the frame that includes the current block is accessed. block; determining whether the size of the current block is less than or equal to or not a threshold value, the current block being a unit for which a list of candidate vectors is generated including the candidate vectors; when the size of the current block is less than or equal to the threshold value, generating a list of candidate vectors by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein the HMVP table stores, in a first-in-first-out (FIFO) manner, each of the first vector candidates as a HMVP vector candidate, the first vector candidates having information regarding the first vector used for the processed block; when the size of the current block exceeds a threshold value, generating a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table; and encoding the current block using the list of candidate vectors.

[0047] Accordingly, the apparatus that implements the encoding method generates a list of candidate vectors without performing a pruning process when the size of the current block is less than or equal to the threshold value, and thereby the processing amount is reduced. This improves the encoding efficiency of the device that implements the encoding method.

[0048] In addition, the decoding method according to an aspect of the present disclosure is a decoding method including: when generating a prediction image of the current block, using an intra-block copy (IBC) mode in which a processed area of the frame that includes the current block is accessed. block; determining whether the size of the current block is less than or equal to or not a threshold value, the current block being a unit for which a list of candidate vectors is generated including the candidate vectors; when the size of the current block is less than or equal to the threshold value, generating a list of candidate vectors by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein the HMVP table stores, in a first-in-first-out (FIFO) manner, each of the first vector candidates as a HMVP vector candidate, the first vector candidates having information regarding the first vector used for the processed block; when the size of the current block exceeds a threshold value, generating a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table; and decoding the current block using the list of candidate vectors.

[0049] Accordingly, the apparatus that implements the decoding method generates a list of candidate vectors without performing a pruning process when the size of the current block is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the processing efficiency of the device that implements the decoding method.

[0050] Definitions of terms

The corresponding terms may be specified as below as examples.

[0051] (1) image

An image is a unit of data configured with a set of pixels, represents a frame, or includes blocks smaller than a frame. Images include still image in addition to video.

[0052] (2) frame

A frame is an image processing unit configured with a set of pixels and is also called a "frame" or "field".

[0053] (3) block

A block is a processing unit that is a collection of a specific number of pixels. The block is also named as indicated in the examples below. Block shapes are not limited. Examples include a rectangular shape of MxN pixels and a square shape of MxM pixels primarily, and also include a triangular shape, a circular shape, and other shapes.

[0054] Example blocks

- cut/tile/brick

- CTU/superblock/basic partitioning unit

- VPDU/partitioning unit for hardware processing

- CU/processing block unit/prediction block unit (PU)/orthogonal transform block unit (TU)/unit

- subblock

[0055] (4) pixel/sample

A pixel or sample represents the smallest point in an image. The pixels or samples include not only the pixel at an integer position, but also the pixel at a subpixel position generated from the pixel at an integer position.

[0056] (5) pixel value/sample value

A pixel value or sample value is the intrinsic value of a pixel. Pixel or sample values naturally include a luma value, a chrominance value, an RGB gradation level, and also encompass a depth value or a binary value of 0 or 1.

[0057] (6) flag

A flag indicates one or more bits and, for example, may be a parameter or index represented by two or more bits. Alternatively, the flag may indicate not only a binary value represented by a binary number, but also a multiple value represented by a number other than the binary number.

[0058] (7) signal

A signal is a signal that has been converted into symbolic form or encoded with the ability to convey information. The signals include a discrete digital signal and an analog signal that takes a continuous value.

[0059] (8) stream/bitstream

A stream or bitstream is a string of digital data stream of digital data. The stream or bitstream may be a single stream or may be configured with multiple streams having multiple hierarchical layers. The stream or stream of bits may be transmitted in serial communication using a single transmission path, or may be transmitted in packet switched communication using multiple transmission paths.

[0060] (9) difference

In the case of a scalar quantity, all that is required is that the simple difference (xy) and the difference calculation must be included. Differences include the absolute value of the difference (|xy|), the squared difference ( ^x∧2 - ^y∧2 ), the square root of the difference (√(xy)), the weighted difference (ax-by: a and b are constants), the difference offsets (x-y+a: a is the offset).

[0061] (10) amount

In the case of a scalar quantity, all that is required is that the simple sum (x+y) and the calculation of the sums must be included. Sums include the absolute value of the sum (|x+y|), the squared sum (x ^∧ 2+y ^∧ 2), the square root of the sum (√ (x+y)), the weighted difference (ax+by: a and b are constants), the offset sum (x+y+a: a is the offset).

[0062] (11) based

The phrase "on the basis of something" means that a thing other than something else can be considered. In addition, "based on" can be used in a case in which a direct result is obtained, or in a case in which the result is obtained through an intermediate result.

[0063] (12) used, using

The phrase "something used" or "using something" means that a thing other than something else can be considered. In addition, "used" or "using" can be used in a case in which a direct result is obtained, or in a case in which a result is obtained through an intermediate result.

[0064] (13) prohibit, prohibit

The term "prohibit" or "prohibit" can be rephrased as "not allow" or "not allow." In addition, “non-prohibition, non-prohibition” or “permission/permission” do not always mean “obligation”.

[0065] (14) limit, set limit/set limit/with set limit

The term "to limit" or "to set a limit/to set a limit/with a set limit" can be rephrased as "not to allow/not to allow" or "not allowing/not allowing". In addition, “non-prohibition, non-prohibition” or “permission/permission” do not always mean “obligation”. Moreover, it is only necessary that part of something must be prohibited/prohibited quantitatively or qualitatively, and something can be completely prohibited/prohibited.

[0066] (15) chrominance signal

An adjective, represented by the symbols Cb and Cr, indicating that an array of samples or a single sample represents one of two color difference signals associated with the primary colors. The term "chrominance" may be used instead of the term "chrominance".

[0067] (16) brightness signal

An adjective, represented by the symbol or subscript Y or L, indicating that an array of samples or a single sample represents a monochrome signal associated with the primary colors. The term "luminance signal" may be used instead of the term "luminance".

[0068] Notes related to descriptions

In the drawings, identical reference numbers indicate identical or similar components. The sizes and relative locations of components are not necessarily drawn to identical scale.

[0069] Embodiments will be described hereinafter with reference to the drawings. It should be noted that the embodiments described below show a general or specific example. The numerical values, shapes, materials, components, arrangement and connection of components, steps, relationship and order of steps, etc., indicated in the following embodiments are merely examples and are therefore not intended to limit the scope of the claims.

[0070] Embodiments of the encoder and decoder are described below. The embodiments are examples of an encoder and decoder to which the processes and/or configurations presented in the description of aspects of the present disclosure are applicable. The processes and/or configurations may also be implemented in an encoder and decoder different from the encoder and decoder of the embodiments. For example, with respect to the processes and/or configurations applied to the embodiments, any of the following may be implemented:

[0071] (1) Any of the encoder or decoder components according to the embodiments presented in the description of aspects of this disclosure may be replaced or combined with another component presented anywhere in the description of aspects of this disclosure.

[0072] (2) In the encoder or decoder according to the embodiments, discretionary changes may be made to functions or processes performed by one or more components of the encoder or decoder, such as adding, replacing, deleting, etc. functions or processes. For example, any function or process may be substituted for or combined with another function or process presented anywhere in the description of aspects of this disclosure.

[0073] (3) In the methods implemented by the encoder or decoder according to the embodiments, discretionary changes such as adding, replacing and deleting one or more processes included in the method may be made. For example, any process in the method may be replaced by or combined with another process presented anywhere in the description of aspects of the present disclosure.

[0074] (4) One or more components included in an encoder or decoder according to embodiments may be combined with a component presented anywhere in the description of aspects of the present disclosure may be combined with a component including one or more functions presented anywhere in the description of aspects of the present disclosure, and may be combined with a component that implements one or more of the processes implemented by the component presented in the description of aspects of the present disclosure.

[0075] (5) A component that includes one or more encoder or decoder functions according to embodiments, or a component that implements one or more encoder or decoder processes according to embodiments, may be combined or replaced by a component presented anywhere in the description aspects of the present disclosure, a component that includes one or more of the functions presented anywhere in the description of aspects of the present disclosure, or a component that implements one or more processes presented anywhere in the description of aspects of the present disclosure.

[0076] (6) In methods implemented by an encoder or decoder according to embodiments, any of the processes included in the method may be replaced or combined with a process presented anywhere in the description of aspects of this disclosure, or with any corresponding or equivalent process.

[0077] (7) One or more processes included in the methods implemented by the encoder or decoder according to the embodiments may be combined with a process presented anywhere in the description of aspects of the present disclosure.

[0078] (8) The implementation of the processes and/or configurations presented in the description of aspects of the present disclosure is not limited to the encoder or decoder according to the embodiments. For example, the processes and/or configurations may be implemented in an apparatus used for a purpose other than the moving frame encoder or moving frame decoder disclosed in the embodiments.

[0079] System Configuration

Fig. 1 is a circuit diagram illustrating one configuration example of a transmission system according to an embodiment.

[0080] The Trs transmission system is a system that transmits a stream generated by image encoding and decodes the transmitted stream. The transmission system Trs as specified includes, for example, an encoder 100, a network Nw, and a decoder 200, as illustrated in FIG. 1.

[0081] The image is input to the encoder 100. The encoder 100 generates a stream by encoding the input image and outputs the stream to network Nw. The stream includes, for example, encoded image information and control information for decoding the encoded image. The image is compressed through encoding.

[0082] It should be noted that the previous image before encoding and input to the encoder 100 is also called the "original image", "original signal" or "original sample". The image may be a video or a still image. The image represents the general principle of sequence, frame and block and is therefore not limited to a spatial region having a specific size and a temporal region having a specific size unless otherwise specified. An image is an array of pixels or pixel values, and the signal representing the image or pixel values is also called "samples". The stream may be referred to as a "bitstream", "coded bitstream", "compressed bitstream" or "coded signal". Additionally, the encoder may be referred to as an "image encoder" or a "video encoder". The encoding method performed by the encoder 100 may be called “encoding method,” “image encoding method,” or “video encoding method.”

[0083] Network Nw transmits the stream generated by encoder 100 to decoder 200. Network Nw may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination of these networks. The Nw network is not always limited to a bidirectional communication network and may be a unidirectional communication network that transmits digital terrestrial broadcast, satellite broadcast, and the like broadcast waves. Alternatively, the Nw network may be replaced by a medium such as a digital versatile disc (DVD) and a Blu-Ray Disc (BD)(R), etc., on which the stream is recorded.

[0084] The decoder 200 generates, for example, a decoded image, which is an uncompressed image, by decoding a stream transmitted through the network Nw. For example, the decoder decodes the stream according to a decoding method corresponding to the encoding method by the encoder 100.

[0085] It should be noted that the decoder may also be referred to as an “image decoder” or a “video decoder,” and that the decoding method performed by the decoder 200 may also be referred to as a “decoding method,” “image decoding method,” or “video decoding method.”

[0086] Data structure

Fig. 2 is a diagram illustrating one example of a hierarchical data structure in a stream. The stream includes, for example, a video sequence. As illustrated in (a) of FIG. 2, a video sequence includes a video parameter set (VPS), a sequence parameter set (SPS), a frame parameter set (PPS), additional enhancement information (SEI), and a plurality of frames.

[0087] In a video having multiple layers, the VPS includes: an encoding parameter that is common among a portion of the plurality of layers; and an encoding parameter associated with some of the plurality of layers included in the video or with an individual layer.

[0088] SPS includes a parameter that is used for the sequence, i.e. an encoding parameter that is accessed by decoder 200 in order to decode the sequence. For example, the encoding parameter may indicate the width or height of the frame. It should be noted that multiple SPS may be present.

[0089] PPS includes a parameter that is used for the frame, i.e. an encoding parameter that is accessed by the decoder 200 in order to decode each of the frames in the sequence. For example, the encoding parameter may include a reference value for the quantization width that is used to decode the frame, and a flag indicating the use of weighted prediction. It should be noted that multiple PPS may be present. Each of the SPS and PPS may be referred to simply as a "parameter set".

[0090] As illustrated in (b) of FIG. 2, a frame may include a frame header and at least one slice. The frame header includes an encoding parameter that is accessed by the decoder 200 in order to decode the at least one slice.

[0091] As illustrated in (c) of FIG. 2, a slice includes a slice header and at least one brick. The slice header includes an encoding parameter that is accessed by the decoder 200 in order to decode the at least one brick.

[0092] As illustrated in (d) of FIG. 2, the brick includes at least one coding tree unit (CTU).

[0093] It should be noted that the frame may not include a slice and may include a group of tiles instead of a slice. In this case, the tile group includes at least one tile. In addition, the brick may include a cut.

[0094] A CTU is also called a "superblock" or "basic partitioning unit". As illustrated in (e) of FIG. 2, a CTU as specified includes a CTU header and at least one coding unit (CU). The CTU header includes an encoding parameter that is accessed by the decoder 200 in order to decode the at least one CU.

[0095] A CU may be broken down into multiple smaller CUs. As illustrated in (f) of FIG. 2, the CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting the CU, and the residual coefficient information is information indicating the prediction residual, which is described below. Although the CU is essentially identical to the prediction unit (PU) and transformation unit (TU), it should be noted that, for example, the SBT, which is described below, may include a plurality of TUs smaller than the CU. In addition, the CU may be processed for each virtual pipelined decoding unit (VPDU) included in the CU. A VPDU, for example, is a fixed unit that can be processed in a single stage when pipelining is done in hardware.

[0096] It should be noted that the stream may not include part of the hierarchical layers illustrated in FIG. 2. The order of the hierarchical layers can be changed, or any of the hierarchical layers can be replaced by another hierarchical layer. Here, a frame that represents a target for a process to be performed by a device such as an encoder 100 or a decoder 200 is called a “current frame.” The current frame means the current frame to be encoded when the process is an encoding process, and the current frame means the current frame to be decoded when the process is a decoding process. Likewise, for example, a CU or block CU that represents a target for a process to be performed by a device such as encoder 100 or decoder 200 is called a “current block.” The current block means the current block to be encoded when the process is an encoding process, and the current block means the current block to be decoded when the process is a decoding process.

[0097] Frame structure: slice/tile

A frame may be configured with one or more slice units or tile units in order to decode the frame in parallel.

[0098] Slices are basic coding units included in a frame. A frame may include, for example, one or more slices. In addition, a slice includes one or more consecutive coding tree units (CTUs).

[0099] FIG. 3 is a diagram illustrating one example of a slice configuration. For example, a frame includes an 11x8 CTU and is split into four slices (slices 1-4). Slice 1 includes sixteen CTUs, slice 2 includes twenty-one CTUs, slice 3 includes twenty-nine CTUs, and slice 4 includes twenty-two CTUs. Here, each CTU in the frame belongs to one of the slices. The shape of each slice is the shape obtained by splitting the frame horizontally. The boundary of each slice does not have to coincide with the end of the image and can coincide with any of the boundaries between CTUs in the image. The processing order of CTUs in a slice (encoding order or decoding order), for example, is the raster scan order. A slice includes a slice header and encoded data. Slice attributes can be written to the slice header. Features include the CTU address of the top CTU in the slice, the slice type, etc.

[0100] A tile is a unit of rectangular area included in a frame. Each of the tiles can be assigned a number called a "TileId" in raster scan order.

[0101] FIG. 4 is a diagram illustrating one example of a tile configuration. For example, a frame includes an 11x8 CTU and is divided into four tiles of rectangular areas (tiles 1-4). When tiles are used, the CTU processing order changes relative to the processing order if the tiles are not used. When tiles are not in use, the multiple CTUs in a frame are processed in raster scan order. When multiple tiles are used, at least one CTU in each of the plurality of tiles is processed in raster scan order. For example, as illustrated in FIG. 4B, the processing order of CTUs included in tile 1 is an order that starts from the left end of the first column of tile 1 to the right end of the first column of tile 1 and then starts from the left end of the second column of tile 1 to the right end of the second column of tile 1.

[0102] It should be noted that one tile may include one or more slices, and one slice may include one or more tiles.

[0103] It should be noted that a frame may be configured with one or more tile sets. A set of tiles may include one or more groups of tiles or one or more tiles. A frame can be configured with only one of a tile set, a tile group, and a tile. For example, the order for scanning multiple tiles for each set of tiles in raster scan order is presumably the basic order for encoding the tiles. A set of one or more tiles that are contiguous in the basic encoding order within each tile set is assumed to constitute a group of tiles. Such a frame may be configured by a splitter 102 (see FIG. 7), which is described below.

[0104] Scalable Coding

Fig. 5 and 6 are diagrams illustrating examples of scalable flow structures.

[0105] As illustrated in FIG. 5, encoder 100 may generate a temporally/spatially scalable stream by dividing each of a plurality of frames into any of a plurality of layers and encoding the frame within the layer. For example, encoder 100 encodes a frame for each layer, thereby achieving scalability in which an enhancement layer is present above the base layer. This encoding of each frame is also called "scalable encoding". Thus, the decoder 200 allows switching of image quality for an image that is displayed by decoding the stream. In other words, the decoder 200 determines down to which layer to decode based on internal factors such as the processing ability of the decoder 200 and external factors such as the bandwidth state of the communication links. As a result, decoder 200 allows content to be decoded while freely switching between low resolution and high resolution. For example, a stream user watches a video stream halfway using a smartphone on the way home and continues watching the video at home on a device such as an Internet-connected TV. It should be noted that each of the smartphone and device described above includes a decoder 200 having identical or different performance. In this case, when the device decodes layers up to the top layer in the stream, the user can watch the video in high quality at home. Thus, the encoder 100 does not have to generate multiple streams having different image qualities of identical content, and thereby the processing load can be reduced.

[0106] In addition, the enhancement layer may include meta-information based on statistical information regarding the image. The decoder 200 may generate video whose image quality is enhanced by performing super-resolution imaging on a frame in the base layer based on metadata. Super-resolution imaging can be any of an improvement in the SN ratio at identical resolution and an increase in resolution. The metadata may include information to identify a linear or nonlinear filtering coefficient used in a super-resolution process, or information identifying the value of a parameter in a filtering process, machine learning, or least squares method used in a super-resolution process.

[0107] Alternatively, a configuration may be provided in which the frame is divided, for example, into tiles, for example, according to the semantic meaning of an object in the frame. In this case, the decoder 200 can decode only a partial area in the frame by selecting a tile to be decoded. In addition, the attribute of the object (person, car, ball, etc.) and the position of the object in the frame (coordinates in identical images) can be stored as metadata. In this case, decoder 200 is capable of identifying the position of the desired object based on the metadata and determining a tile including the object. For example, as illustrated in FIG. 6, metadata may be stored using a data storage structure different from image data, such as SEI in HEVC. This metadata indicates, for example, the position, size or color of the underlying object.

[0108] Metadata may be stored in units of multiple frames, for example, as a stream, sequence, or random access unit. Thus, the decoder 200 is capable of obtaining, for example, the time at which a particular person appears in a video, and by fitting the time information to the frame unit information, is capable of identifying the frame in which the object is present and determining the position of the object in the frame.

[0109] Encoder

Next, the encoder 100 according to this embodiment will be described. Fig. 7 is a block diagram illustrating one example of a functional configuration of the encoder 100 according to this embodiment. Encoder 100 encodes the image in units of blocks.

[0110] As illustrated in FIG. 7, the encoder 100 is equipment that encodes an image in units of blocks, and includes a partitioner 102, a subtractor 104, a converter 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse converter 114, an adder 116, and a memory 118. blocks, a loop filter 120, a movie frame memory 122, an internal prediction module 124, an inter-prediction module 126, a prediction controller 128, and a prediction parameter generator 130. It should be noted that the intra prediction module 124 and the inter prediction module 126 are configured as part of the prediction execution module.

[0111] Encoder installation example

Fig. 8 is a block diagram illustrating an installation example of the encoder 100. The encoder 100 includes a processor a1 and a memory a2. For example, the plurality of constituent elements of the encoder 100 illustrated in FIG. 7 are mounted in the processor a1 and memory a2 illustrated in FIG. 8.

[0112] Processor a1 is a circuit that performs information processing and is available to storage device a2. For example, processor a1 is a specialized or general electronic circuit that encodes an image. Processor a1 may be a processor such as a CPU. In addition, processor a1 may be a collection of multiple electronic circuits. In addition, for example, the processor a1 may serve as two or more constituent elements other than the information storing constituent element of the plurality of constituent elements of the encoder 100 illustrated in FIG. 7, etc.

[0113] Memory a2 is a dedicated or general memory for storing information that is used by processor a1 to encode an image. The storage device a2 may be an electronic circuit and may be coupled to the processor a1. In addition, the storage device a2 may be included in the processor a1. In addition, the storage device a2 may be a collection of a plurality of electronic circuits. In addition, the storage device a2 may be a magnetic disk, an optical disk or the like. or may be represented as a storage device, media, etc. In addition, the storage device a2 may be a non-volatile storage device or a volatile storage device.

[0114] For example, the storage device a2 may store an image to be encoded or a stream corresponding to an encoded image. In addition, the storage device a2 may store a program for instructing the processor a1 to encode an image.

[0115] In addition, for example, the storage device a2 may serve as two or more constituent elements for storing information from a plurality of constituent elements of the encoder 100 illustrated in FIG. 7. More specifically, the memory a2 may serve as the block memory 118 and the movie frame memory 122 illustrated in FIG. 7. More specifically, the storage device a2 can store a reconstructed image (specifically, a reconstructed block, a reconstructed frame, etc.).

[0116] It should be noted that, in the encoder 100, not all of the plurality of constituent elements indicated in FIG. 7, etc. may be implemented, and not all of the processes described above may be performed. Some of the constituent elements shown in FIG. 7 may be included in another device, or part of the processes described above may be performed by another device.

[0117] Hereinafter, this document describes the complete flow of processes performed by the encoder 100, and thereafter describes each of the constituent elements included in the encoder 100.

[0118] General flow of the encoding process

Fig. 9 is a flowchart illustrating one example of a complete encoding process performed by encoder 100.

[0119] First, the partitioning unit 102 of the encoder 100 partitions each of the frames included in the original image into a plurality of blocks having a fixed size (eg, 128×128 pixels) (step Sa_1). The partitioning unit 102 then selects a partitioning pattern for the fixed size block (step Sa_2). In other words, the partitioning module 102 further partitions the fixed-size block into a plurality of blocks that form a selected partitioning pattern. Encoder 100 performs, for each of a plurality of blocks, steps Sa_3-Sa_9 for the block.

[0120] The prediction controller 128 and the prediction execution module, which is configured with the intra prediction module 124 and the inter prediction module 126, generate a prediction image of the current block (step Sa_3). It should be noted that the prediction image is also referred to as a "prediction signal", "prediction block" or "prediction samples".

[0121] Next, the subtractor 104 generates the difference between the current block and the prediction image as a prediction residual (step Sa_4). It should be noted that the prediction residual is also called “prediction error”.

[0122] Next, the converter 106 converts the prediction image, and the quantizer 108 quantizes the result to generate a plurality of quantized coefficients (step Sa_5).

[0123] Next, the entropy encoder 110 encodes (specifically, entropy encodes) a plurality of quantized coefficients and a prediction parameter associated with generating a prediction image to generate a stream (step Sa_6).

[0124] Next, the inverse quantizer 112 inversely quantizes the plurality of quantized coefficients, and the inverse transformer 114 inversely transforms the result to recover the prediction remainder (step Sa_7).

[0125] Next, the adder 116 adds the prediction image with the reconstructed prediction residual to reconstruct the current block (step Sa_8). Thus, a reconstructed image is formed. It should be noted that the reconstructed image is also called a “reconstructed block”, and in particular, that the reconstructed image generated by the encoder 100 is also called a “local decoded block” or a “local decoded image”.

[0126] When the reconstructed image is generated, the loop filter 120 filters the reconstructed image as necessary (step Sa_9).

[0127] The encoder 100 then determines whether or not encoding of the entire frame is complete (step Sa_10). When determining that encoding is not yet complete ("No" in step Sa_10), the processes from step Sa_2 are executed repeatedly.

[0128] Although the encoder 100 selects one partitioning pattern for a fixed size block and encodes each block according to the partitioning pattern in the above example, it should be noted that each block may be encoded according to a corresponding one of a plurality of partitioning patterns. In this case, encoder 100 may estimate costs for each of a plurality of split patterns and, for example, may select a stream obtained by encoding according to the split pattern that results in the lowest cost as the stream that is ultimately output.

[0129] Alternatively, the processes in steps Sa_1-Sa_10 may be executed sequentially by the encoder 100, or two or more processes may be executed in parallel or may be reordered.

[0130] The encoding process by encoder 100 is a hybrid encoding using predictive encoding and transform encoding. In addition, predictive encoding is performed by an encoding loop configured with a subtractor 104, a transformer 106, a quantizer 108, an inverse quantizer 112, an inverse transformer 114, an adder 116, a loop filter 120, a block memory 118, a movie frame memory 122, and an intra prediction unit 124. , a mutual prediction module 126 and a prediction controller 128 . In other words, the prediction execution unit configured with the intra prediction unit 124 and the inter prediction unit 126 is part of a coding loop.

[0131] Partition module

The partitioning module 102 splits each of the frames included in the original image into a plurality of blocks and outputs each block to the subtractor 104. For example, the partitioning module 102 first partitions the frame into blocks of a fixed size (eg, 128×128 pixels). A fixed size block is also called a "coding tree unit (CTU)". The partitioning module 102 then partitions each fixed-size block into variable-size blocks (eg, 64×64 pixels or less) based on recursive quadtree and/or binary block partitioning. In other words, the partitioning module 102 selects a partitioning pattern. The variable size block is also called a "coding unit (CU)", "prediction unit (PU)" or "transformation unit (TU)". It should be noted that, in various types of installation examples, it is not necessary to distinguish between CU, PU and TU; all or some of the blocks in a frame can be processed in CU, PU or TU units.

[0132] FIG. 10 is a diagram illustrating one example of block partitioning according to this embodiment. In fig. 10, solid lines represent block boundaries for blocks divided by quadtree blocking, and dotted lines represent block boundaries for blocks divided by binary tree blocking.

[0133] Here, block 10 is a square block having 128×128 pixels. This block 10 is first divided into four square 64x64 pixel blocks (quadtree blocking).

[0134] The top left 64x64 pixel block is further vertically split into two rectangular 32x64 pixel blocks, and the left 32x64 pixel block is further vertically split into two rectangular 16x64 pixel blocks (binary tree chunking ). As a result, the top left square 64x64 pixel block is split into two 16x64 pixel blocks 11 and 12 and one 32x64 pixel block 13.

[0135] The top right square 64x64 pixel block is horizontally split into two rectangular 64x32 pixel blocks 14 and 15 (binary tree blocking).

[0136] The lower left 64x64 pixel square block is first divided into four 32x32 pixel square blocks (quadtree blocking). The top left block and bottom right block of four 32x32 pixel square blocks are further broken down. The top left square 32x32 pixel block is vertically split into two rectangular 16x32 pixel blocks, and the right 16x32 pixel block is further split horizontally into two 16x16 pixel blocks (binary tree blocking). The bottom right 32x32 pixel block is horizontally split into two 32x16 pixel blocks (binary tree blocking). The top right 32x32 pixel square block is horizontally split into two 32x16 pixel rectangular blocks (binary tree blocking). As a result, the lower left square 64x64 pixel block is split into a rectangular 16x32 pixel block 16, two square 16x16 pixel blocks 17 and 18, two square 32x32 pixel blocks 19 and 20, and two rectangular blocks 32 ×16-pixel blocks 21 and 22.

[0137] The lower right 64x64 pixel block 23 is not split.

[0138] As described above, FIG. 10, block 10 is divided into thirteen blocks 11-23 of variable size based on recursive quadtree and binary tree blocking. This partitioning is also called "quadtree plus binary tree (QTBT) partitioning."

[0139] It should be noted that, in FIG. 10, one block is partitioned into four or two blocks (quadtree or binary tree partitioning), but partitioning is not limited to these examples. For example, one block can be split into three blocks (ternary block splitting). Partitioning that includes such ternary block partitioning is also called "multi-type tree (MBT) partitioning."

[0140] FIG. 11 is a diagram illustrating one example of a functional configuration of the partitioning unit 102. As illustrated in FIG. 11, the partitioning unit 102 may include a block partitioning determining unit 102a. The block division determination unit 102a may perform the following processes as examples.

[0141] For example, the block partition determining unit 102a collects block information from the block memory 118 or from the movie frame memory 122 and determines the above-described partitioning pattern based on the block information. The partitioning unit 102 partitions the original image according to the partitioning pattern and outputs at least one block obtained by partitioning to the subtractor 104.

[0142] In addition, for example, the block partition determining unit 102a outputs a parameter indicating the above-described partitioning pattern for the transformer 106, the inverse transformer 114, the intra prediction unit 124, the inter prediction unit 126, and the entropy encoder 110. The transformer 106 may transform the prediction residual based on parameter. The intra prediction module 124 and the inter prediction module 126 may generate a prediction image based on a parameter. In addition, entropy encoder 110 can entropy encode a parameter.

[0143] A parameter associated with a split pattern may be written to the stream, as set forth below as one example.

[0144] FIG. 12 is a diagram illustrating examples of partitioning patterns. Examples of partitioning patterns include: quadruple partitioning (QT), which splits a block into two areas, both horizontally and vertically; splitting into three areas (HT or VT), in which the block is split in an identical direction in a ratio of 1:2:1; splitting into two areas (HB or VB), in which the block is split in an identical direction in a 1:1 ratio; and no partitioning (NS).

[0145] It should be noted that the partitioning pattern does not have block partitioning direction in the case of four-area partitioning and no partitioning, and that the partitioning pattern has partitioning direction information in the case of two-area partitioning or three-area partitioning.

[0146] FIG. 13A and 13B are diagrams illustrating one example of a split pattern syntax tree. In the example of FIG. 13A, firstly, information indicating whether or not partitioning should be performed (S: partition flag) is present, and information indicating whether or not partitioning into four areas should be performed (QT: QT flag) is present after it. . Information indicating which of the three-area and two-area partitioning is to be carried out (TT: TT flag or BT: BT flag) is present after it, and finally, information indicating the direction of partitioning (Ver: vertical flag, or Hor: horizontal flag), present. It should be noted that each at least one block obtained by partitioning according to such a partitioning pattern may further be partitioned multiple times in a similar process. In other words, as one example, whether or not the partitioning is performed, whether or not the four-area partitioning is performed, which of the horizontal direction and the vertical direction is the direction in which the partitioning method should be performed, which of the three partitioning region and splitting into two regions must be performed, can be recursively determined, and the results of the determination can be encoded in the stream according to the encoding order disclosed by the syntax tree illustrated in FIG. 13A.

[0147] In addition, although information elements respectively indicating S, QT, TT and Ver are arranged in the listed order in the syntax tree illustrated in FIG. 13A, information elements respectively indicating S, QT, Ver and BT may be arranged in the listed order. In other words, in the example of FIG. 13B, firstly, information indicating whether or not to perform partitioning (S: partition flag) is present, and information indicating whether or not partitioning into four areas should be performed (QT: QT flag) is present after it. . Information indicating the direction of partitioning (Ver: vertical flag, or Hor: horizontal flag) is present after it, and finally, information indicating what of two-area partitioning and three-area partitioning should be carried out (BT: BT flag , or TT: TT flag), present.

[0148] It should be noted that the partitioning patterns described above are examples, and partitioning patterns other than the described partitioning patterns may be used, or a portion of the described partitioning patterns may be used.

[0149] Subtractor

The subtractor 104 subtracts the prediction image (the prediction image that is input from the prediction controller 128 ) from the original image in units of blocks input from the partitioning unit 102 and partitioned by the partitioning unit 102 . In other words, the subtractor 104 calculates the prediction residuals of the current block. Subtractor 104 then outputs the calculated prediction residuals to converter 106.

[0150] The source signal is an input signal that is input to the encoder 100 and represents an image of each frame included in the video (eg, a luma signal and two chrominance signals).

[0151] Converter

Transformer 106 converts spatial domain prediction residuals into frequency domain transform coefficients and outputs the transform coefficients to quantizer 108. More specifically, transformer 106 applies, for example, a predefined discrete cosine transform (DCT) or discrete sine transform (DST) to the prediction residuals in spatial area.

[0152] It should be noted that the transformer 106 can adaptively select a transform type from a plurality of transform types and convert the prediction residuals into transform coefficients by using a transform basis function corresponding to the selected transform type. This kind of transformation is also called "explicit multiple base transformation (EMT)" or "adaptive multiple transformation (AMT)". In addition, the basis transformation function is also called simply "basis".

[0153] Conversion types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I and DST-VII. It should be noted that these conversion types can be represented as DCT2, DCT5, DCT8, DST1 and DST7. Fig. 14 is a diagram illustrating basic transformation functions for each type of transformation. In fig. 14, N indicates the number of input pixels. For example, the selection of a transformation type from a plurality of transformation types may depend on the prediction type (one of intra-prediction and inter-prediction) and may depend on the intra-prediction mode.

[0154] Information indicating whether or not such EMT or AMT should be applied (called, for example, "EMT flag" or "AMT flag"), and information indicating the selected type of conversion are normally signaled at the CU level . It should be noted that signaling of this information does not necessarily have to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, brick level, or CTU level).

[0155] In addition, the converter 106 may re-convert the conversion coefficients (which represent the results of the conversion). This re-transformation is also called "adaptive secondary transform (AST)" or "non-separable secondary transform (NSST)". For example, transformer 106 performs re-conversion in units of a subblock (eg, a 4x4 pixel subblock) included in a block of transform coefficients corresponding to the intra prediction residual. Information indicating whether or not to apply NSST and information associated with a transformation matrix to be used in NSST are normally signaled at the CU level. It should be noted that signaling of this information does not necessarily have to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, brick level, or CTU level).

[0156] The converter 106 may use a separable transform and an inseparable transform. Here, separable transformation is a method in which the transformation is performed multiple times by separately performing the transformation for each of the directions according to the number of dimensions of the inputs. An inseparable transformation is a method of performing a collective transformation in which two or more dimensions in multidimensional inputs are collectively treated as a single dimension.

[0157] In one example of a non-separable transformation, when the input is a 4x4 pixel block, the 4x4 pixel block is treated as a single array including sixteen elements, and the transformation applies a 16x16 transformation matrix to the array.

[0158] In another example of a non-separable transform, an input block of 4x4 pixels is treated as a single array including sixteen elements, and a transform (hypercubic Givens transform) can then be performed in which a Givens rotation is performed on the array multiple times.

[0159] When converting in the converter 106, the conversion types for the transfer basis functions to be converted into the frequency domain according to the domains in the CU can be switched. Examples include spatially varying transform (SVT).

[0160] FIG. 15 is a diagram illustrating one example of an SVT.

[0161] In SVT, as illustrated in FIG. 5B, the CUs are divided into two equal regions horizontally or vertically, and only one of the regions is converted to the frequency domain. The transformation type can be specified for each area. For example, DST7 and DST8 are used. For example, from two areas obtained by dividing the CU vertically into two equal areas, DST7 and DCT8 can be used for the area at position 0. Alternatively, from two areas, DST7 is used for the area at position 1. Likewise, from two areas obtained by partitioning CU horizontally into two equal areas, DST7 and DCT8 are used for the area at position 0. Alternatively, of the two areas, DST7 is used for the area at position 1. Although only one of the two areas in the CU is converted and the other is not converted in the example illustrated in fig. 15, each of the two areas can be converted. In addition, the partitioning method may include not only partitioning into two areas, but also partitioning into four areas. In addition, the partitioning method can be more flexible. For example, information indicating the partitioning method may be encoded and signaled identical to the CU partitioning. It should be noted that SVT is also called "sub-block transform (SBT)".

[0162] AMT and EMT described above may be referred to as "MTS (Multiple Transform Selection)". When MTS is applied, a transformation type that is DST7, DCT8, etc. can be selected, and information indicating the selected transformation type can be encoded as index information for each CU. Another process called "IMTS (Implicit MTS)" is provided as a process for selecting, based on the shape of the CU, the type of transformation to be used for the orthogonal transformation performed without encoding the index information. When IMTS is applied, for example, when the CU is rectangular in shape, the orthogonal transformation of the rectangular shape is performed using DST7 for the short side and DST2 for the long side. In addition, for example, when the CU has a square shape, the orthogonal transformation of the rectangular shape is performed using DCT2 when the MTS is effective in the sequence, and using DST7 when the MTS is ineffective in the sequence. DCT2 and DST7 are just examples. Other transformation types may be used, and it is also possible to change the combination of transformation types to use to another combination of transformation types. IMTS can be used only for intra prediction blocks or can be used for intra prediction blocks and inter prediction block.

[0163] Three processes of MTS, SBT and IMTS are described above as selection processes for selectively switching transform types for use in orthogonal transform. However, all three selection processes may become effective, or only a subset of selection processes may selectively become effective. Whether or not each of the selection processes becomes effective can be identified based on flag information and the like. in a header such as SPS. For example, when all three selection processes are effective, one of the three selection processes is selected for each CU, and the orthogonal transformation of the CU is performed. It should be noted that the selection processes for selectively switching transformation types may be selection processes different from the above three selection processes, or each of the three selection processes may be replaced by a different process, provided that at least one of the following four functions [ 1]-[4] can be achieved. Function [1] is a function for performing an orthogonal transformation of the entire CU and encoding information indicating the type of transformation used in the transformation. Function [2] is a function for performing an orthogonal transformation of the entire CU and determining the transformation type based on a predefined rule without encoding information indicating the transformation type. Function [3] is a function for performing an orthogonal transformation of a partial CU region and encoding information indicating the type of transformation used in the transformation. Function [4] is a function for performing an orthogonal transformation of a partial CU region and determining the transformation type based on a predefined rule without encoding information indicating the transformation type used in the transformation.

[0164] It should be noted that whether or not each of the MTS, IMTS and SBT is applied may be determined for each processing unit. For example, whether or not each of MTS, IMTS, and SBT is applied may be determined on a per-sequence, frame, brick, slice, CTU, or CU basis.

[0165] It should be noted that the tool for selectively switching transformation types in the present disclosure may be rephrased as a method for selectively selecting a basis for use in a transformation process, in a selection process, or in a process for selecting a basis. In addition, the tool for selectively switching transformation types can be rephrased by a mode for adaptively selecting a transformation type.

[0166] FIG. 16 is a flowchart illustrating one example of a process performed by converter 106.

[0167] For example, the converter 106 determines whether or not to perform the orthogonal transform (step St_1). Here, upon determining whether to perform the orthogonal transform (Yes in step St_1), the transformer 106 selects a transform type to use for the orthogonal transform from a plurality of transform types (step St_2). Next, transformer 106 performs an orthogonal transform by applying the selected transform type to the prediction residual of the current block (step St_3). The converter 106 then outputs information indicating the selected transform type to the entropy encoder 110 so as to enable the entropy encoder 110 to encode the information (step St_4). On the other hand, when determining whether orthogonal transformation is not performed (“No” in step St_1), the converter 106 outputs information indicating that orthogonal transformation is not performed so as to allow the entropy encoder 110 to encode information (step St_5). It should be noted that whether or not the orthogonal transformation should be performed in step St_1 may be determined, for example, based on the size of the transformation block, the prediction mode applied to the CU, etc. Alternatively, the orthogonal transform may be performed using a predefined transform type without encoding information indicating the transform type to use in the orthogonal transform.

[0168] FIG. 17 is a flowchart illustrating another example of a process performed by the converter 106. It should be noted that the example illustrated in FIG. 17 is an example of an orthogonal transform in the case where the transform types to be used in the orthogonal transform are selectively switched, as in the case of the example illustrated in FIG. 16.

[0169] As one example, the first group of transformation types may include DCT2, DST7, and DCT8. As another example, the second group of transformation types may include DCT2. The transformation types included in the first transformation type group and the transformation types included in the second transformation type group may partially overlap each other or may be completely different from each other.

[0170] More specifically, the converter 106 determines whether the conversion size is less than or equal to a predetermined value (step Su_1). Here, upon determining that the transform size is less than or equal to a predetermined value ("Yes" in step Su_1), the transformer 106 performs an orthogonal transform of the prediction residual of the current block using a transform type included in the first transform type group (step Su_2). Next, the transformer 106 outputs information indicating a transform type to be used from at least one transform type included in the first transform type group to the entropy encoder 110, so as to enable the entropy encoder 110 to encode information (step Su_3) . On the other hand, when determining that the transformation size is not less than or equal to a predetermined value (“No” in step Su_1), the transformer 106 performs an orthogonal transformation of the prediction residual of the current block using the second group of transformation types (step Su_4).

[0171] In step Su_3, information indicating a transform type to be used in the orthogonal transform may be information indicating a combination of a transform type to be applied vertically in the current block and a transform type to be applied horizontally in the current block. The first group of types may include only one transformation type, and information indicating the transformation type to be used in the orthogonal transformation may not be encoded. The second transformation type group may include a plurality of transformation types, and information indicating a transformation type to be used for orthogonal transformation from one or more transformation types included in the second transformation type group may be encoded.

[0172] Alternatively, the transform type may be determined based only on the transform size. It should be noted that such determinations are not limited to determining whether the transform size is less than or equal to or not a predetermined value, and other processes are also possible, provided that the processes serve to determine the type of transform to use in the orthogonal transform based on the transform size.

[0173] Quantizer

The quantizer 108 quantizes the transform coefficients output from the transformer 106. More specifically, the quantizer 108 scans, in a specific scanning order, the transform coefficients of the current block and quantizes the scanned transform coefficients based on the quantization parameters (QPs) corresponding to the transform coefficients. Quantizer 108 then outputs the quantized transform coefficients (hereinafter also referred to as “quantized coefficients”) of the current block to entropy encoder 110 and inverse quantizer 112.

[0174] The determined scanning order is the order for quantizing/inverse quantizing the transform coefficients. For example, a specific scan order is specified as ascending frequency order (low to high frequency) or descending frequency order (high to low frequency).

[0175] The quantization parameter (QP) is a parameter that specifies the quantization step (quantization width). For example, when the value of the quantization parameter increases, the quantization step size also increases. In other words, when the value of the quantization parameter increases, the error in the quantized coefficients (quantization error) increases.

[0176] In addition, the quantization matrix can be used for quantization. For example, several kinds of quantization matrices can be used, respectively, to perform frequency conversion for sizes such as 4x4 and 8x8, prediction modes such as intra prediction and inter prediction, and pixel components such as luminance and pixel components. color signal. It should be noted that quantization means digitizing values sampled at predetermined intervals, respectively, into predetermined levels. In this art, quantization may be represented by other expressions, such as rounding and scaling.

[0177] Methods using quantization matrices include a method using a quantization matrix that is set directly on the side of the encoder 100, and a method using a quantization matrix that is set as a default value (default matrix). At the encoder 100 side, a quantization matrix suitable for image features may be specified by directly specifying the quantization matrix. However, this case has the disadvantage of increasing the encoding volume for encoding the quantization matrix. It should be noted that the quantization matrix to be used to quantize the current block may be generated based on the default quantization matrix or the encoded quantization matrix, instead of directly using the default quantization matrix or the encoded quantization matrix.

[0178] A method is provided for quantizing a high frequency coefficient and a low frequency coefficient in an identical manner without using a quantization matrix. It should be noted that this method is equivalent to the method using a quantization matrix (flat matrix) whose coefficients all have identical values.

[0179] The quantization matrix may be encoded, for example, at the sequence level, frame level, slice level, brick level, or CTU level.

[0180] When using the quantization matrix, quantizer 108 scales, for each transform factor, eg, the quantization width, which may be calculated based on the quantization parameter, etc., using the value of the quantization matrix. A quantization process performed without using quantization matrices at all may be a process of quantizing transform coefficients based on a quantization width calculated based on a quantization parameter, etc. It should be noted that, in a quantization process performed without using quantization matrices, the quantization width may be multiplied by a predetermined value that is common to all transform coefficients in the block.

[0181] FIG. 18 is a block diagram illustrating one example of a functional configuration of the quantizer 108.

[0182] For example, the quantizer 108 includes a difference quantization parameter generator 108a, a predicted quantization parameter generator 108b, a quantization parameter generator 108c, a quantization parameter storage device 108d, and a quantization execution unit 108e.

[0183] FIG. 19 is a flowchart illustrating one example of quantization performed by quantizer 108.

[0184] As one example, quantizer 108 may perform quantization on a per-CU basis based on the flowchart illustrated in FIG. 19. More specifically, the quantization parameter generator 108c determines whether or not to perform quantization (step Sv_1). Here, when determining whether to perform quantization (“Yes” in step Sv_1), the quantization parameter generator 108c generates a quantization parameter for the current block (step Sv_2) and stores the quantization parameter in the quantization parameter storage device 108d (step Sv_3).

[0185] Next, the quantization execution unit 108e quantizes the transform coefficients of the current block using the quantization parameter generated in step Sv_2 (step Sv_4). The predicted quantization parameter generator 108b then obtains the quantization parameter for a processing unit other than the current block from the quantization parameter storage device 108d (step Sv_5). The predicted quantization parameter generator 108b generates a predicted quantization parameter of the current block based on the received quantization parameter (step Sv_6). The difference quantization parameter generator 108a calculates the difference between the current block quantization parameter generated by the quantization parameter generator 108c and the predicted quantization parameter of the current block generated by the predicted quantization parameter generator 108b (step Sv_7). The difference quantization parameter is generated by calculating the difference. The difference quantization parameter generator 108a outputs the difference quantization parameter to the entropy encoder 110 so as to enable the entropy encoder 110 to encode the difference quantization parameter (step Sv_8).

[0186] It should be noted that the difference quantization parameter may be encoded, for example, at the sequence level, frame level, slice level, brick level, or CTU level. In addition, the initial value of the quantization parameter may be encoded at the sequence level, frame level, slice level, brick level, or CTU level. At this time, the quantization parameter can be generated using the initial value of the quantization parameter and the difference quantization parameter.

[0187] It should be noted that the quantizer 108 may include a plurality of quantizers and may employ dependent quantization, in which the transform coefficients are quantized using a quantization method selected from a plurality of quantization methods.

[0188] Entropy encoder

Fig. 20 is a block diagram illustrating one example of a functional configuration of the entropy encoder 110.

[0189] The entropy encoder 110 generates a stream by entropy encoding the quantized coefficients input from the quantizer 108 and the prediction parameter input from the prediction parameter generator 130 . For example, context adaptive binary arithmetic coding (CABAC) is used as entropy coding. More specifically, the entropy encoder 110 includes a binarization unit 110a, a context controller 110b, and a binary arithmetic encoder 110c. The binarization unit 110a performs binarization in which multi-level signals such as quantized coefficients and a prediction parameter are converted into binary signals. Examples of binarization methods include truncated Rice code binarization, exponential Golomb codes, and fixed-length code binarization. The context controller 110b retrieves the context value according to the attribute or surrounding state of the syntactic element, i.e. probability of occurrence of a binary signal. Examples of methods for retrieving context value include traversal, syntactic element access, top and left adjacent block access, hierarchical information access, etc. Binary arithmetic encoder 110c arithmetically encodes the binary signal using the extracted context value.

[0190] FIG. 21 is a diagram illustrating the CABAC flow of entropy encoder 110.

[0191] First, initialization is performed in CABAC in entropy encoder 110. When initializing, initialization is performed in binary arithmetic encoder 110c and setting an initial context value. For example, the binarization unit 110a and the binary arithmetic encoder 110c perform binarization and arithmetic encoding of a plurality of quantization coefficients in the CTU sequentially. At this time, the context controller 110b updates the context value every time arithmetic encoding is performed. The context controller 110b then stores the context value as post-processing. The stored context value is used, for example, to initialize the context value for the next CTU.

[0192] Inverse Quantizer

The inverse quantizer 112 inversely quantizes the quantized coefficients that are input from the quantizer 108. More specifically, the inverse quantizer 112 inversely quantizes, in a specific scan order, the quantized coefficients of the current block. Inverse quantizer 112 then outputs the inversely quantized transform coefficients of the current block to inverse quantizer 114.

[0193] Inverter

The inverter 114 recovers the prediction errors by inversely transforming the transform coefficients that are input from the inverse quantizer 112. More specifically, the inverter 114 recovers the prediction residuals of the current block by performing an inverse transform corresponding to the transform applied to the transform coefficients by the inverter 106. The inverter 114 then outputs the reconstructed prediction residuals to the adder 116.

[0194] It should be noted that since information is lost in quantization, the reconstructed prediction residuals do not match the prediction errors calculated by the subtractor 104. In other words, the reconstructed prediction residuals normally include quantization errors.

[0195] Totalizer

The adder 116 reconstructs the current block by adding the prediction residuals that are input from the inverter 114 and the prediction images that are input from the prediction controller 128 . Consequently, a reconstructed image is formed. Adder 116 then outputs the reconstructed image to block memory 118 and loop filter 120.

[0196] Block memory

The block memory 118 is a storage device for storing a block that is included in the current frame and is referred to in intra prediction. More specifically, the block memory 118 stores the reconstructed image output from the adder 116.

[0197] Motion Picture Memory

The movie frame memory 122, for example, is a storage device for storing reference frames for use in inter-prediction, and is also called a “motion picture buffer”. More specifically, the movie frame memory 122 stores the reconstructed image filtered by the edge filter 120.

[0198] Loop filter

The edge filter 120 applies the edge filter to the reconstructed image output by the adder 116 and outputs the filtered reconstructed image to the movie frame memory 122. A loop filter is a filter used in a coding loop (in-loop filter). Examples of loop filters include, for example, an adaptive loop filter (ALF), a deblocking filter (DF or DBF), a sampled adaptive offset (SAO), etc.

[0199] FIG. 22 is a block diagram illustrating one example of a functional configuration of loop filter 120.

[0200] For example, as illustrated in FIG. 22, the loop filter 120 includes a deblocking filtering execution unit 120a, an SAO execution unit 120b, and an ALF execution unit 120c. The deblocking filtering execution unit 120a performs a filtering process to remove blocking from the restored image. The SAO execution unit 120b performs the SAO process of the reconstructed image after subjecting it to a blocking removal filtering process. The ALF execution unit 120c performs the ALF process of the reconstructed image after being subjected to the SAO process. The ALF and filtering processes for deblocking are described in detail below. The SAO process is a process for improving image quality by reducing ringing (a phenomenon in which pixel values are distorted as waves around an edge) and correcting deviation in pixel value. Examples of SAO processes include the edge shift process and the band shift process. It should be noted that loop filter 120 need not always include all of the constituent elements disclosed in FIG. 22, and may include only part of the constituent elements. In addition, the loop filter 120 may be configured to perform the above processes in a processing order different from the processing order disclosed in FIG. 22.

[0201] Loop filter > adaptive loop filter

In ALF, a least squares error filter is applied to remove compression artifacts. For example, one filter, selected from a plurality of filters based on the direction and activity of local gradients, is applied to each of the 2x2-pixel subblocks in the current block.

[0202] More specifically, first, each sub-block (eg, each 2x2-pixel sub-block) is classified into one of a plurality of classes (eg, fifteen or twenty-five classes). Subblock categorization is based, for example, on the directionality and activity of gradients. In a specific example, the category index C (eg, C=5D+A) is calculated based on the directionality of the D gradients (eg, 0-2 or 0-4) and the activity of the A gradients (eg, 0-4). Then, based on the category index C, each subblock is classified into one of multiple classes.

[0203] For example, the directionality D of the gradients is calculated by comparing the gradients of multiple directions (eg, horizontal, vertical, and two diagonal directions). In addition, for example, the activity A of the gradients is calculated by summing the gradients of multiple directions and quantizing the result of the summation.

[0204] The filter to be used for each sub-block is determined from the plurality of filters based on the result of such classification.

[0205] The filter shape to be used in the ALF, for example, is a circular symmetrical filter shape. Fig. 23A-23C illustrate examples of filter shapes used in the ALF. Fig. 23A illustrates a 5x5 diamond shape filter, FIG. 23B illustrates a 7x7 diamond shaped filter, and FIG. 23C illustrates a 9x9 diamond shaped filter. Information indicating the shape of the filter is normally carried in frame-level signaling. It should be noted that the signaling of this information indicating the shape of the filter need not be performed at the frame level and may be performed at another level (eg, sequence level, slice level, brick level, CTU level, or CU level).

[0206] Enabling or disabling ALF is determined, for example, at the frame level or at the CU level. For example, a decision regarding whether or not to apply ALF to a luma signal may be made at the CU level, and a decision regarding whether or not to apply ALF to a chrominance signal may be made at the frame level. Information indicating whether ALF is enabled or disabled is normally carried in signaling at the frame level or at the CU level. It should be noted that signaling information indicating enabling or disabling ALF need not be performed at the frame level or the CU level and may be performed at another level (eg, sequence level, slice level, brick level, or CTU level).

[0207] In addition, as described above, one filter is selected from the plurality of filters, and the subblock ALF process is executed. A set of coefficients for the coefficients to be used for each of a plurality of filters (eg, up to the fifteenth or twenty-fifth filter) is normally signaled at the frame level. It should be noted that the set of coefficients does not necessarily need to be signaled at the frame level and may be signaled at another level (eg, sequence level, slice level, brick level, CTU level, CU level, or subblock level).

[0208] Loop Filter > Cross Component Adaptive Loop Filter

Fig. 23D is a diagram illustrating an example in which Y samples (first component) are used for cross-component ALF (CCALF) for Cb and CCALF for Cr (for components other than the first component). Fig. 23E is a diagram illustrating a diamond filter.

[0209] One example of CCALF operates by applying a linear, diamond-shaped filter (FIGS. 23D, 23E) to the luma signal channel for each chrominance signal component. Filter coefficients, for example, can be passed to APS, scaled by a factor of 2 ^∧ 10, and rounded for fixed-point representation. The application of filters is controlled for a variable block size and signaled via a context-coded flag received for each block of samples. The block size along with the CCALF enable flag is adopted at the cut level for each chroma component. The syntax and semantics for CCALF are provided in the appendix. In operation, the following block sizes are supported (in chroma samples): 16×16, 32×32, 64×64 and 128×128.

[0210] Loop Filter > Combined Cross Component Adaptive Chroma Loop Filter

Fig. 23F is a diagram illustrating an example for joint chrominance CCALF (JC-CCALF).

[0211] One example of JC-CCALF in which only one CCALF filter is used to generate one CCALF filtered output as the chrominance detail signal for only one color component, while a properly weighted version of the identical chrominance detail signal must be applied to another color component. This reduces the complexity of the existing CCALF by about half.

[0212] The weighting factor value is encoded as a sign flag and a weighting index. The weight index (denoted as weight_index) is encoded in 3 bits and indicates the absolute value of the JC-CCALF weighting coefficient JcCcWeight. It cannot be equal to 0. The absolute value of JcCcWeight is defined as follows.

[0213] - If weight_index is less than or equal to 4, JcCcWeight is weight_index>>2.

[0214] - Otherwise, JcCcWeight is 4/(weight_index-4).

[0215] The on/off control at the block level of ALF filtering for Cb and Cr is separate. This is identical to CCALF, and two separate sets of block-level enable/disable control flags must be encoded. Unlike CCALF, in this document, the sizes of the on/off control blocks Cb, Cr are identical, and therefore only one block size variable is encoded.

[0216] Loop Filter > Deblocking Filter

In the deblocking filtering process, the edge filter 120 performs a filtering process on a block boundary in the reconstructed image so as to reduce distortion that occurs at the block boundary.

[0217] FIG. 24 is a block diagram illustrating one example of a specific configuration of the deblocking filtering execution unit 120a.

[0218] For example, the deblocking filtering module 120a includes: a boundary determination module 1201; filter determination module 1203; filtering execution module 1205; processing determination module 1208; filter characteristics determination module 1207; and switches 1202, 1204 and 1206.

[0219] Boundary determination module 1201 determines whether or not a pixel to be filtered for deblocking (ie, the current pixel) is present near a block boundary. The boundary determination module 1201 then outputs the determination result to the switch 1202 and the processing determination module 1208 .

[0220] In the case where the edge detection module 1201 determines that the current pixel is present near the block boundary, the switch 1202 outputs an unfiltered image to the switch 1204. In the opposite case, in which the edge detection module 1201 determines that the current pixel is not present near the boundary block, switch 1202 outputs an unfiltered image to switch 1206. It should be noted that the unfiltered image is an image configured with the current pixel and at least one surrounding pixel located near the current pixel.

[0221] The filter determination unit 1203 determines whether or not to perform filtering to deblock the current pixel based on the pixel value of at least one surrounding pixel located near the current pixel. The filter determination module 1203 then outputs the determination result to the switch 1204 and the processing determination module 1208.

[0222] In case it is determined that the filter determination unit 1203 performs filtering to remove blockiness of the current pixel, the switch 1204 outputs the unfiltered image obtained through the switch 1202 to the filtering execution unit 1205. In the opposite case, in which it is determined that the filter determination module 1203 does not perform filtering to deblock the current pixel, the switch 1204 outputs the unfiltered image obtained through the switch 1202 to the switch 1206.

[0223] When receiving an unfiltered image through switches 1202 and 1204, filtering execution unit 1205 performs, for the current pixel, deblocking filtering having a filter characteristic determined by filter characteristic determining unit 1207. The filtering module 1205 then outputs the filtered pixel to the switch 1206.

[0224] Under the control of the processing determination unit 1208, the switch 1206 selectively outputs a pixel that is not filtered for deblocking and a pixel that is filtered for deblocking by the filtering execution unit 1205.

[0225] The processing determination module 1208 controls the switch 1206 based on the results of the determinations made by the boundary determination module 1201 and the filtering determination module 1203 . In other words, the processing determination module 1208 instructs the switch 1206 to output a pixel that is filtered to deblock when the edge determination module 1201 determines that the current pixel is present near a block boundary, and it is determined that the filter determination module 1203 performs filtering to deblock the current pixel. In addition, in a case other than the above-described case, the processing determination unit 1208 instructs the switch 1206 to output a pixel that is not filtered for deblocking. The filtered image is output from the switch 1206 by repeating the pixel output in this manner. It should be noted that the configuration illustrated in FIG. 24 is one example of a configuration in the blocking deblocking filtering module 120a. The deblocking filtering unit 120a may be configured differently.

[0226] FIG. 25 is a diagram illustrating an example of a deblocking filter having a symmetrical filtering characteristic with respect to a block boundary.

[0227] In a deblocking filtering process, for example, one of two deblocking filters having different characteristics, i.e. strong filter and weak filter, selected using pixel values and quantization parameters. In the case of a strong filter, pixels p0-p2 and pixels q0-q2 are present at the block boundary, as illustrated in FIG. 25, the pixel values of the corresponding pixels q0-q2 are changed to the pixel values q'0-q'2 by performing calculations according to the following expressions.

[0228] q'0=(p1+2xp0+2xq0+2xq1+q2+4)/8

q'1=(p0+q0+q1+q2+2)/4

q'2=(p0+q0+q1+3xq2+2xq3+4)/8

[0229] It should be noted that, in the above expressions, p0-p2 and q0-q2 represent the pixel values of the corresponding pixels p0-p2 and pixels q0-q2. In addition, q3 is the pixel value of the neighboring pixel q3 located on the opposite side of pixel q2 relative to the block boundary. In addition, on the right side of each of the expressions, coefficients that are multiplied by the corresponding pixel values of the pixels to be used for filtering to remove blocking are filtering coefficients.

[0230] Additionally, when filtering to remove blocking, clipping may be performed such that the calculated pixel values do not change above a threshold. In the clipping process, the pixel values calculated according to the above expressions are clipped to the value obtained according to the "pre-computed pixel value ± 2 x threshold value" using the threshold value determined based on the quantization parameter. In this way, excessive smoothing can be prevented.

[0231] FIG. 26 is a diagram for illustrating one example of a block boundary on which a filtering process to remove blocking is performed. Fig. 27 is a diagram illustrating examples of Bs values.

[0232] A block boundary on which the deblocking filtering process is performed is, for example, a boundary between CUs, PUs or TUs having 8x8 pixel blocks, as illustrated in FIG. 26. The deblocking filtering process is performed, for example, in units of four rows or four columns. First, the boundary intensity values (Bs) are determined as indicated in FIG. 27 for the P block and Q block illustrated in FIG. 26.

[0233] According to the Bs values in FIG. 27, it can be determined whether or not filtering processes for deblocking the boundaries of blocks belonging to the same image should be performed using different intensities. The deblocking filtering process for the chroma signal is performed when the Bs value is 2. The deblocking filtering process for the luma signal is performed when the Bs value is 1 or more and a certain condition is satisfied. It should be noted that the conditions for determining the Bs values are not limited to those indicated in FIG. 27, and the Bs value may be determined based on another parameter.

[0234] Prediction Module (Intra Prediction Module, Inter Prediction Module, Prediction Controller)

Fig. 28 is a flowchart illustrating one example of a process performed by the encoder prediction module 100. It should be noted that the prediction module, as one example, includes all or part of the following constituent elements: an intra prediction module 124; mutual prediction module 126; and a prediction controller 128. The prediction execution module includes, for example, an intra prediction module 124 and an inter prediction module 126.

[0235] The prediction module generates a prediction image of the current block (step Sb_1). It should be noted that the prediction image is, for example, an internal prediction image (internal prediction signal) or a mutual prediction image (mutual prediction signal). More specifically, the prediction module generates a prediction image of the current block using a reconstructed image that has already been obtained for another block, through prediction image generation, prediction residual generation, quantized coefficient generation, prediction residual reconstruction, and prediction image summation.

[0236] The reconstructed image, for example, may be an image in a reference frame or an image of a coded block (ie, another block described above) in the current frame, which is a frame including the current block. The coded block in the current frame, for example, is a block adjacent to the current block.

[0237] FIG. 29 is a flowchart illustrating another example of a process performed by the encoder prediction module 100.

[0238] The prediction module generates a prediction image using the first method (step Sc_1a), generates a prediction image using the second method (step Sc_1b), and generates a prediction image using the third method (step Sc_1c). The first method, the second method and the third method may be mutually different methods for generating a predictive image. Each of the first to third methods may be an inter-prediction method, an intra-prediction method, or another prediction method. The above-described reconstructed image can be used in these prediction methods.

[0239] Next, the prediction module evaluates the prediction images generated in steps Sc_1a, Sc_1b and Sc_1c (step Sc_2). For example, the prediction module calculates the costs C for the prediction images generated in steps Sc_1a, Sc_1b and Sc_1c, and evaluates the prediction images by comparing the costs C of the prediction images. It should be noted that the cost C is calculated according to the expression of the R-D optimization model, for example, C=D+λ x R. In this expression, D indicates the compression artifacts of the forecast image and is represented, for example, as the sum of the absolute differences between the pixel value of the current block and the pixel value the value of the forecast image. In addition, R indicates the bit rate of the stream. In addition, λ indicates, for example, a multiplier according to the Lagrange multiplier method.

[0240] The prediction module then selects one of the prediction images generated in steps Sc_1a, Sc_1b and Sc_1c (step Sc_3). In other words, the prediction module selects a method or mode to obtain the final prediction image. For example, the prediction module selects the prediction image having the lowest cost C based on the costs C calculated for the prediction images. Alternatively, the evaluation in step Sc_2 and the selection of the prediction image in step Sc_3 may be performed based on a parameter that is used in the encoding process. Encoder 100 may convert information to identify the selected prediction image, method, or mode into a stream. The information may, for example, be a flag or the like. Thus, the decoder 200 is capable of generating a prediction image according to a method or mode selected by the encoder 100 based on the information. It should be noted that, in the example illustrated in FIG. 29, the prediction module selects any of the prediction images after the prediction images are generated using appropriate methods. However, the prediction module can select a method or mode based on a parameter to be used in the above-described encoding process before generating prediction images, and can generate a prediction image according to the selected method or mode.

[0241] For example, the first method and the second method may be intra-prediction and inter-prediction, respectively, and the prediction module may select the final prediction image for the current block from the prediction images generated according to the prediction methods.

[0242] FIG. 30 is a flowchart illustrating another example of a process performed by the encoder prediction module 100.

[0243] First, the prediction module generates a prediction image using intra prediction (step Sd_1a) and generates a prediction image using inter prediction (step Sd_1b). It should be noted that the prediction image generated by intra prediction is also called "intra prediction image", and the prediction image generated by inter prediction is also called "mutual prediction image".

[0244] Next, the prediction module evaluates each of the internal prediction image and the mutual prediction image (step Sd_2). The C costs described above can be used in the estimation. The prediction module can then select the prediction image for which the least cost C is calculated from the internal prediction image and the mutual prediction image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode is selected to generate a prediction image for the current block.

[0245] Internal prediction module

The intra prediction module 124 generates a prediction image (ie, an intra prediction image) of the current block by performing intra prediction (also called “intra prediction”) of the current block by accessing the block or blocks in the current frame that are stored in the block memory 118. More specifically, the intra prediction module 124 generates an internal prediction image by performing intra prediction by referring to the pixel values (eg, luma and/or chroma values) of the block or blocks adjacent to the current block, and then outputs the internal prediction image to the controller 128 forecasting.

[0246] For example, the intra prediction module 124 performs intra prediction by using one mode of a plurality of intra prediction modes that are preset. Intra prediction modes typically include one or more non-directional prediction modes and a plurality of directed prediction modes.

[0247] The one or more non-directional prediction modes include, for example, a planar prediction mode and a DC prediction mode defined in the H.265/HEVC standard.

[0248] The plurality of directional prediction modes includes, for example, the thirty-three directional prediction modes defined in the H.265/HEVC standard. It should be noted that the plurality of directional prediction modes may further include thirty-two directional prediction modes in addition to the thirty-three directional prediction modes (for a total of sixty-five directional prediction modes). Fig. 31 is a diagram illustrating sixty-seven intra prediction modes in total used in intra prediction (two non-directional prediction modes and sixty-five directional prediction modes). The solid arrows represent the thirty-three directions defined in the H.265/HEVC standard, and the dotted arrows represent the additional thirty-two directions (the two non-directional prediction modes are not illustrated in FIG. 31).

[0249] In various kinds of installation examples, the luma signal block may be accessed when internally predicting the chrominance signal block. In other words, the chrominance signal component of the current block can be predicted based on the luminance signal component of the current block. This internal forecasting is also called "cross-component linear model (CCLM)". An intra-prediction mode for a chrominance block in which such a luma block is accessed (also called, for example, "CCLM mode") may be added as one of the intra-prediction modes for the chrominance blocks.

[0250] The internal prediction module 124 may adjust the internally predicted pixel values based on horizontal/vertical reference pixel gradients. The internal prediction that accompanies this type of correction is also called "position-dependent combining with internal prediction (PDPC)". Information indicating whether or not to use PDPC (called, for example, a "PDPC flag") is normally signaled at the CU level. It should be noted that signaling of this information does not necessarily have to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, brick level, or CTU level).

[0251] FIG. 32 is a flowchart illustrating one example of a process performed by the intra prediction module 124.

[0252] The intra prediction unit 124 selects one intra prediction mode from a plurality of intra prediction modes (step Sw_1). The intra prediction unit 124 then generates a prediction image according to the selected intra prediction mode (step Sw_2). Next, the internal prediction module 124 determines the most probable modes (MPM) (step Sw_3). MPMs include, for example, six intraprediction modes. Two modes of the six intra prediction modes may be a planar mode and a DC prediction mode, and the other four modes may be directional prediction modes. The intra prediction unit 124 determines whether or not the intra prediction mode selected in step Sw_1 is turned on in the MPM (step Sw_4).

[0253] Here, when determining that the intra prediction mode selected in step Sw_1 is included in the MPM (Yes in step Sw_4), the intra prediction unit 124 sets the MPM flag to 1 (step Sw_5) and generates information indicating selected internal prediction mode, from MPM (step Sw_6). It should be noted that the MPM flag set to 1 and information indicating the intra prediction mode are encoded as prediction parameters by the entropy encoder 110.

[0254] When determining that the selected intra prediction mode is not included in the MPM ("No" in step Sw_4), the intra prediction module 124 sets the MPM flag to 0 (step Sw_7). Alternatively, the intra prediction module 124 does not set the MPM flag at all. The intra prediction unit 124 then generates information indicating the selected intra prediction mode from at least one intra prediction mode that is not included in the MPM (step Sw_8). It should be noted that the MPM flag set to 0 and the information indicating the intra prediction mode are encoded as prediction parameters by the entropy encoder 110. The information indicating the intra prediction mode indicates, for example, any of 0-60.

[0255] Mutual Prediction Module

The inter prediction unit 126 generates a prediction image (cross prediction image) by performing inter prediction (also called “inter prediction”) of the current block by referring to a block or blocks in a reference frame that is different from the current frame and stored in the movie frame memory 122. Cross prediction is performed in units of the current block or the current subblock within the current block. A subblock is included in a block and represents a unit smaller than the block. The subblock size can be 4x4 pixels, 8x8 pixels, or other size. The sub-block size can be switched per unit, such as slice, brick, frame, etc.

[0256] For example, inter-prediction module 126 performs motion estimation in a reference frame for the current block or current sub-block and finds the reference block or reference sub-block that has the best match with the current block or current sub-block. The inter-prediction module 126 then receives motion information (eg, a motion vector) that compensates for the motion or change from the reference block or reference subblock to the current block or current subblock. The mutual prediction unit 126 generates a mutual prediction image of the current block or the current sub-block by performing motion compensation (or motion prediction) based on the motion information. The mutual prediction module 126 outputs the generated mutual prediction image to the prediction controller 128 .

[0257] Motion information used in motion compensation may be signaled as mutual prediction images in various forms. For example, the motion vector may be signaled. In other words, the difference between the motion vector and the motion vector predictor may be signaled.

[0258] List of reference frames

Fig. 33 is a diagram illustrating examples of reference frames. Fig. 34 is a conceptual diagram illustrating examples of reference frame lists. Each reference frame list is a list indicating at least one reference frame stored in the movie frame memory 122. It should be noted that in FIG. 33, each of the rectangles indicates a frame, each of the arrows indicates the reference relationship of the frames, the horizontal axis indicates time, I, P and B in the rectangles indicate the internal prediction frame, uni-prediction frame and bi-prediction frame, respectively, and the numbers in the rectangles indicate the decoding order. As illustrated in FIG. 33, the decoding order of frames is the order of I0, P1, B2, B3 and B4, and the display order of frames is the order of I0, B3, B2, B4 and P1. As illustrated in FIG. 34, the reference frame list is a list representing candidate reference frames. For example, one frame (or slice) may include at least one list of reference frames. For example, one reference frame list is used when the current frame is a uni-prediction frame, and two reference frame lists are used when the current frame is a bi-prediction frame. In the examples of FIGS. 33 and 34, frame B3, which is the current frame currPic, has two reference frame lists, which are an L0 list and an L1 list. When the current frame currPic is a B3 frame, the possible reference frames for the current frame currPic are I0, P1 and B2, and the reference frame lists (which are an L0 list and an L1 list) indicate these frames. Inter prediction module 126 or prediction controller 128 indicates which frame in each reference frame list should actually be accessed in the form of reference frame index refIdxLx. In fig. 34, reference frames P1 and B2 are indicated by reference frame indices refIdxL0 and refIdxL1.

[0259] Such a list of reference frames may be generated for each unit, such as a sequence, frame, slice, brick, CTU, or CU. In addition, of the reference frames indicated in the reference frame lists, a reference frame index indicating a reference frame accessed in inter prediction may be signaled at the sequence level, frame level, slice level, brick level, CTU level, or C.U. In addition, a common reference frame list can be used in a variety of inter-prediction modes.

[0260] Basic Interprediction Flow

Fig. 35 is a flowchart illustrating a basic flow of inter-prediction processing.

[0261] First, the inter-prediction module 126 generates a prediction signal (steps Se_1 to Se_3). Next, the subtractor 104 generates the difference between the current block and the prediction image as a prediction residual (step Se_4).

[0262] Here, when generating a prediction image, the inter-prediction unit 126 generates a prediction image, for example, by determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and motion compensation (step Se_3). In addition, when determining the MV, the inter-prediction unit 126 determines the MV, for example, by selecting a motion vector candidate (MV candidate) (step Se_1) and extracting the MV (step Se_2). MV candidate selection is performed, for example, by internal prediction module 126 generating a list of MV candidate candidates and selecting at least one MV candidate candidate from the list of MV candidate candidates. It should be noted that MVs extracted previously may be added to the list of possible MV variants. Alternatively, when extracting the MV, the inter-prediction module 126 may further select at least one candidate MV from the at least one candidate MV and determine the selected at least one candidate MV as the MV for the current block. Alternatively, inter-prediction module 126 may determine the MV for the current block by performing an estimate in a reference frame region indicated by each of the selected at least one MV candidate. It should be noted that the estimation in the reference frame region may be referred to as "motion estimation".

[0263] In addition, although steps Se_1 to Se_3 are performed by the inter-prediction unit 126 in the above example, the process which is, for example, step Se_1, step Se_2, etc. may be performed by another component included in the encoder 100.

[0264] It should be noted that a list of MV candidates may be generated for each process in an inter-prediction mode, or a common list of MV candidates may be used in multiple inter-prediction modes. The processes in steps Se_3 and Se_4 correspond to steps Sa_3 and Sa_4 illustrated in FIG. 9, respectively. The process in step Se_3 corresponds to the process in step Sd_1b in FIG. thirty.

[0265] Flow for MV Extraction

Fig. 36 is a flowchart illustrating one example of MV extraction.

[0266] The inter-prediction module 126 may extract the MV for the current block in a mode for encoding motion information (eg, MV). In this case, for example, motion information may be encoded as a prediction parameter and may be signaled. In other words, encoded motion information is included in the stream.

[0267] Alternatively, inter-prediction module 126 may extract MVs in a mode in which motion information is not encoded. In this case, motion information is not included in the stream.

[0268] Here, MV extraction modes may include normal mutual mode, normal pooling mode, FRUC mode, affine mode, etc., which are described below. Modes in which motion information is encoded include normal reciprocal mode, normal combining mode, affine mode (specifically, affine reciprocal mode and affine combining mode), etc. It should be noted that the motion information may include not only MVs, but also MV predictor selection information, which is described below. Modes in which motion information is not encoded include FRUC mode, etc. The inter-prediction unit 126 selects a mode for extracting the MV of the current block from a plurality of modes, and extracts the MV of the current block using the selected mode.

[0269] FIG. 37 is a flowchart illustrating another example of MV extraction.

[0270] The inter prediction unit 126 may extract the MV of the current block in a mode in which the MV difference is encoded. In this case, for example, the MV difference is encoded as a prediction parameter and signaled. In other words, the encoded MV difference is included in the stream. The MV difference is the difference between the MV of the current block and the MV predictor. It should be noted that the MV predictor is a motion vector predictor.

[0271] Alternatively, inter-prediction module 126 may extract MVs in a mode in which the MV difference is not encoded. In this case, the encoded MV difference is not included in the stream.

[0272] Here, as described above, MV extraction modes include normal mutual mode, normal pooling mode, FRUC mode, affine mode, etc., which are described below. Modes in which the MV difference is encoded include normal reciprocal mode, affine mode (in particular, affine reciprocal mode), etc. Modes in which the MV difference is not encoded include FRUC mode, normal concatenation mode, affine mode (specifically, affine concatenation mode), etc. The inter-prediction unit 126 selects a mode for extracting the MV of the current block from a plurality of modes, and extracts the MV for the current block using the selected mode.

[0273] MV Extraction Modes

Fig. 38A and 38B are diagrams illustrating one example of mode categorization for MV extraction. For example, as illustrated in FIG. 38A, the MV extraction modes are roughly categorized into three modes according to whether or not motion information should be encoded and whether or not MV differences should be encoded. The three modes are reciprocal mode, fusion mode, and frame rate upconversion (FRUC) mode. The reciprocal mode is a mode in which motion estimation is performed and in which motion information and MV difference are encoded. For example, as illustrated in FIG. 38B, the reciprocal mode includes an affine reciprocal mode and a normal reciprocal mode. The merging mode is a mode in which motion estimation is not performed, and in which the MV is selected from the encoded surrounding block, and the MV for the current block is extracted using the MV. The combining mode is a mode in which essentially motion information is encoded and the MV difference is not encoded. For example, as illustrated in FIG. 38B, the combining modes include normal combining mode (also called “normal combining mode” or “regular combining mode”), motion vector difference (MMVD) combining mode, combined inter combining and intra prediction (CIIP) mode, triangle mode, ATMVP mode and affine merging mode. Here, the MV difference is encoded exclusively in the MMVD mode from the modes included in the combining modes. It should be noted that affine union mode and affine reciprocal mode are modes included in affine modes. The affine mode is a mode for retrieving, as the MV of the current block, the MV of each of a plurality of sub-blocks included in the current block, subject to an affine transformation. The FRUC mode is a mode that serves to extract the MV of the current block by performing estimation between encoded regions, and in which neither motion information nor any MV difference is encoded. It should be noted that the corresponding modes are described in detail below.

[0274] It should be noted that the categorization of modes illustrated in FIG. 38A and 38B are examples, and the categorization is not limited to this. For example, when the MV difference is encoded in the CIIP mode, the CIIP mode is categorized into mutual modes.

[0275] MV extract > normal mutual mode

The normal mutual mode is a mutual prediction mode for extracting the MV of the current block by finding a block similar to the image of the current block from the reference frame area indicated by the MV candidate. In this normal reciprocal mode, the MV difference is encoded.

[0276] FIG. 39 is a flowchart illustrating an example of inter-prediction by normal inter-mode.

[0277] First, the inter-prediction module 126 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sg_1). In other words, the inter-prediction module 126 generates a list of possible MV options.

[0278] Next, the inter-prediction module 126 extracts N (an integer of 2 or more) candidate MVs from the plurality of candidate MVs obtained in step Sg_1 as candidate motion vector predictors according to a predetermined priority order (step Sg_2 ). It should be noted that the priority order is determined in advance for each of the N possible MV options.

[0279] Next, the inter-prediction module 126 selects one MV predictor candidate from the N MV predictor candidates as the MV predictor for the current block (step Sg_3). At this time, the inter-prediction module 126 encodes, in the stream, MV predictor selection information to identify the selected MV predictor. In other words, the inter-prediction module 126 outputs the MV predictor selection information as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130.

[0280] Next, the inter-prediction unit 126 extracts the MV of the current block by referring to the encoded reference frame (step Sg_4). At this time, the inter-prediction unit 126 further encodes, in the stream, the difference value between the extracted MV and the MV predictor as the MV difference. In other words, the inter-prediction module 126 outputs the MV difference as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 . It should be noted that an encoded reference frame is a frame including a plurality of blocks that are reconstructed after encoding.

[0281] Finally, the inter prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Sg_5). Processes in stages Sg_1-Sg_5 are executed for each block. For example, when the processes in steps Sg_1-Sg_5 are executed for each of all blocks in the slice, the slice mutual prediction using the normal mutual mode ends. For example, when the processes in steps Sg_1 to Sg_5 are executed for each of all blocks in a frame, frame inter-prediction using the normal inter-mode ends. It should be noted that not all blocks included in the slice may be subject to processes in steps Sg_1 to Sg_5, and slice mutual prediction using the normal mutual mode may end when a portion of the blocks are subject to processes. Likewise, frame inter-prediction using the normal inter-mode may end when the processes in steps Sg_1-Sg_5 are executed for a portion of the blocks in the frame.

[0282] It should be noted that the prediction image is a mutual prediction signal as described above. In addition, information indicating the inter-prediction mode (normal inter-mode in the above example) used to generate a prediction image, for example, is encoded as a prediction parameter in the encoded signal.

[0283] It should be noted that the list of possible MV options can also be used as a list for use in another mode. In addition, processes associated with a list of possible MV options can be applied to processes associated with a list for use in another mode. Processes associated with a list of MV candidates include, for example, retrieving or selecting a candidate MV from a list of candidate MVs, reordering candidate MVs, or removing a candidate MV.

[0284] MV extract > normal merge mode

The normal pooling mode is a cross-prediction mode for selecting a candidate MV from a list of candidate MVs as the MV for the current block, thereby extracting the MV. It should be noted that the normal combining mode is a combining mode in a narrow sense and is also called simply "combining mode". In this embodiment, the normal combining mode and the combining mode are different, and the combining mode is used in a broad sense.

[0285] FIG. 40 is a flowchart illustrating an example of inter-prediction by normal combining mode.

[0286] First, the inter-prediction module 126 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sh_1). In other words, the inter-prediction module 126 generates a list of possible MV options.

[0287] Next, the inter-prediction module 126 selects one candidate MV from the plurality of candidate MVs obtained in step Sh_1, thereby extracting the MV for the current block (step Sh_2). At this time, the inter-prediction module 126 encodes, in the stream, MV selection information to identify the selected MV candidate. In other words, the inter-prediction module 126 outputs the MV selection information as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 .

[0288] Finally, the inter prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Sh_3). The processes in stages Sh_1-Sh_3 are performed, for example, for each block. For example, when the processes in steps Sh_1-Sh_3 are executed for each of all blocks in a slice, cross-prediction of the slice using the normal pooling mode ends. In addition, when the processes in steps Sh_1 to Sh_3 are executed for each of all blocks in a frame, frame inter-prediction using the normal combining mode ends. It should be noted that not all blocks included in the slice may be subject to processes in steps Sh_1-Sh_3, and cross-prediction of the slice using the normal pooling mode may end when a portion of the blocks are subject to processes. Likewise, frame inter-prediction using the normal combining mode may end when the processes in steps Sh_1-Sh_3 are executed for a portion of the blocks in the frame.

[0289] In addition, information indicating the inter-prediction mode (normal combining mode in the above example) used to generate the prediction image, for example, is encoded as a prediction parameter in the stream.

[0290] FIG. 41 is a diagram for illustrating one example of an MV extraction process for a current frame by the normal combining mode.

[0291] First, the inter-prediction module 126 generates a list of possible MV options in which the possible MV options are registered. Examples of MV candidates include: spatially adjacent MV candidates, which are MVs of a plurality of coded blocks located spatially surrounding the current block; temporarily adjacent MV candidates, which are the MVs of the surrounding blocks onto which the position of the current block in the encoded reference frame is projected; combined candidate MVs, which are MVs formed by combining the MV value of a spatially adjacent MV predictor and the MV value of a temporally adjacent MV predictor; and a null candidate MV, which is an MV having a value of zero.

[0292] Next, the inter-prediction unit 126 selects one MV candidate from the plurality of MV candidates registered in the MV candidate list, and determines the MV candidate as the MV of the current block.

[0293] In addition, entropy encoder 110 records and encodes, on stream, merge_idx, which is a signal indicating which MV candidate is selected.

[0294] It should be noted that the MV candidate candidates registered in the MV candidate list described in FIG. 41 are examples. The number of possible MV options may be different from the number of possible MV options in the diagram, the list of possible MV options may be configured so that some kinds of possible MV options in the diagram may not be included, or so that one or more possible MVs -options other than the types of possible MV options in the diagram are included.

[0295] The final MV may be determined by performing dynamic motion vector updating (DMVR), which is described below, using the MV of the current block extracted by the normal merging mode. It should be noted that, in normal combining mode, the MV difference is not encoded, but the MV difference is encoded. In MMVD mode, one possible MV variant is selected from a list of possible MV variants, as in the case of normal combining mode, the MV difference is encoded. As illustrated in FIG. 38B, MMVD can be categorized into combining modes along with a normal combining mode. It should be noted that the MV difference in MMVD mode does not always have to be identical to the MV difference for use in reciprocal mode. For example, retrieving MV differences in MMVD mode may be a process that requires less processing than the amount of processing required for retrieving MV differences in reciprocal mode.

[0296] In addition, a combined intercombining and intra-prediction (CIIP) mode may be performed. The mode serves to overlap the prediction image generated by inter-prediction and the prediction image generated by intra-prediction to generate a prediction image for the current block.

[0297] It should be noted that the list of MV candidate options may be referred to as a “candidate list.” In addition, merge_idx represents MV selection information.

[0298] MV extraction > HMVP mode

Fig. 42 is a diagram for illustrating one example of an MV extraction process for a current frame by HMVP combining mode.

[0299] In normal combining mode, the MV, for example, for a CU that represents the current block is determined by selecting one candidate MV from a list of candidate MVs generated by referring to a coded block (eg, the CU). Here, another possible MV option may be registered in the list of possible MV options. The mode in which such other possible MV variant is registered is called "HMVP mode".

[0300] In HMVP mode, MV candidates are managed using a first-in, first-out (FIFO) buffer for HMVP, separate from the list of MV candidates for normal concatenation mode.

[0301] In the FIFO buffer, motion information such as MV of blocks processed previously is stored "newest first". In FIFO buffer management, each time one block is processed, the MV for the newest block (which is the CU processed immediately before) is stored in the FIFO buffer, and the MV of the oldest CU (that is, the CU processed earlier) total) is removed from the FIFO buffer. In the example illustrated in FIG. 42, HMVP1 is the MV for the newest block, and HMVP5 is the MV for the oldest MV.

[0302] The inter-prediction module 126 then, for example, checks whether each MV managed in the FIFO buffer is or is not an MV different from all possible MVs that are already registered in the list of possible MVs for the normal mode. association starting with HMVP1. When determining that the MV is different from all possible MV cases, the inter-prediction module 126 may add the MV managed in the FIFO buffer to the list of possible MV cases for the normal merge mode as a possible MV case. At this time, the candidate MV registered from the FIFO buffer may be one or more.

[0303] By using the HMVP mode in this manner, it is possible to add not only the MV of a block that is spatially or temporally adjacent to the current block, but also the MV for a block that was processed previously. As a result, the variety of possible MV options for the normal combining mode is expanded, which increases the possibility that coding efficiency can be improved.

[0304] It should be noted that the MV may be motion information. In other words, the information stored in the MV candidate list and the FIFO buffer may include not only the MV values, but also information about reference frames, reference directions, frame numbers, etc. In addition, a block, for example, represents a CU.

[0305] It should be noted that the MV candidate list and FIFO buffer illustrated in FIG. 42 are examples. The MV candidate list and FIFO buffer may be different in size from the MV candidate list and FIFO buffer in FIG. 42 or may be configured to register candidate MVs in an order different from that in FIG. 42. In addition, the process described here is common between encoder 100 and decoder 200.

[0306] It should be noted that the HMVP mode can be used for modes other than the normal combining mode. For example, it is also excellent if motion information, such as MVs of blocks processed in affine mode in the past, can be kept "newest first" and can be used as candidate MVs. The mode obtained by applying the HMVP mode to the affine mode may be called a "history-based affine mode".

[0307] MV extraction > FRUC mode

Motion information may be extracted at the decoder 200 side without being signaled from the encoder 100 side. For example, motion information may be extracted by performing motion estimation at the decoder 200 side. At this time, at the decoder 200 side, motion estimation is performed without using pixel values in current block. Modes in which motion estimation is performed on the decoder 200 side in this manner include Frame Rate Upconversion (FRUC) mode, Pattern Matching Based Motion Vector Extraction (PMMVD) mode, etc.

[0308] One example of a FRUC process is illustrated in FIG. 43. First, a list that indicates, as possible MV options, MVs for encoded blocks, each of which is adjacent to the current block spatially or temporally, is generated by accessing the MV (the list may be a list of possible MV options and also be used as a list of possible MV options for normal combining mode) (step Si_1). Then, the best possible MV option is selected from the set of possible MV options registered in the list of possible MV options (step Si_2). For example, the evaluation values of the corresponding MV candidates included in the MV candidate list are calculated, and one MV candidate is selected as the best possible MV candidate based on the evaluation values. Based on the selected best possible MV option, the motion vector for the current block is then extracted (step Si_4). More specifically, for example, the selected best possible MV is directly retrieved as the MV for the current block. In addition, for example, the MV for the current block may be retrieved using pattern matching in the surrounding region of a position that is included in the reference frame and corresponds to the selected best possible MV candidate. In other words, evaluation using template matching on the reference frame and evaluation values can be performed on the surrounding region of the best possible MV candidate, and when there is an MV that results in a better evaluation value, the best possible MV candidate can be updated to the MV that results in a better score value, and the updated MV can be determined as the final MV for the current block. The update to the MV that results in the best score value may not be performed.

[0309] Finally, the inter prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Si_5). Processes in stages Si_1-Si_5 are performed, for example, for each block. For example, when the processes in steps Si_1-Si_5 are executed for each of all blocks in a slice, cross-prediction of the slice using FRUC mode ends. For example, when the processes in steps Si_1-Si_5 are executed for each of all blocks in a frame, frame inter-prediction using FRUC mode ends. It should be noted that not all blocks included in the slice may be subject to processes in steps Si_1-Si_5, and cross-prediction of the slice using FRUC mode may end when a portion of the blocks are subject to processes. Likewise, frame inter-prediction using FRUC mode may end when processes in steps Si_1-Si_5 are executed for a portion of the blocks included in the frame.

[0310] Each sub-block may be processed similarly to the above-described case of processing each block.

[0311] The evaluation values may be calculated according to various kinds of methods. For example, a comparison is made between a reconstructed image in a region in a reference frame corresponding to the MV and a reconstructed image in a certain region (the region, for example, may be a region in another reference frame or a region in an adjacent block of the current frame, as specified below). The difference between the pixel values of the two reconstructed images can be used for the MV estimation value. It should be noted that the score value may be calculated using information other than the difference value.

[0312] Pattern matching is described in detail next. First, one MV candidate included in the MV candidate list (also called a “merge list”) is selected as the starting point for evaluation through template matching. As a pattern match, a first pattern match or a second pattern match may be used. The first template matching and the second template matching may be referred to as "bilateral matching" and "template matching", respectively.

[0313] MV extraction > FRUC > bilateral matching

During the first pattern matching, pattern matching is performed between two blocks that are located along the current block's motion path and are included in two different reference frames. Accordingly, in the first template matching, a region in another reference frame located along the motion path of the current block is used as a determined region for calculating the evaluation value of the above-described MV candidate.

[0314] FIG. 44 is a diagram for illustrating one example of first template matching (bilateral matching) between two blocks in two reference frames located along a motion path. As illustrated in FIG. 44, in the first template matching, two motion vectors (MV ₀ , MV ₁ ) are extracted by evaluating the pair that has the best match from among the pairs in two blocks that are included in two different reference frames (Ref ₀ , Ref ₁ ) and are located along the trajectory of the current block (Cur block). More specifically, the difference between the reconstructed image at a specified position in the first coded reference frame (Ref ₀ ) indicated by the MV option, and the reconstructed image at a specified position in the second coded reference frame (Ref ₁ ) indicated by the symmetric MV, obtained by scaling the MV candidate in the display time interval is retrieved for the current block, and a score value is calculated using the resulting difference value. It is excellent to select, as the best MV, a possible MV variant that results in the best evaluation value from the set of possible MV variants.

[0315] Assuming a continuous motion path, the motion vectors (MV ₀ , MV ₁ ) indicating the two reference blocks are proportional to the time distances (TD ₀ , TD ₁ ) between the current frame (Cur Pic) and the two reference frames (Ref ₀ ,Ref ₁ ). For example, when the current frame is temporarily between two reference frames, and the temporal distances from the current frame to the corresponding two reference frames are equal to each other, mirror-symmetric bidirectional MVs are extracted in the first template matching.

[0316] MV extraction > FRUC > reference matching

In the second pattern matching (reference matching), pattern matching is performed between a block in the reference frame and a reference in the current frame (the reference is the block adjacent to the current block in the current frame (the adjacent block, for example, represents the top and/or left adjacent block(s)). Therefore, in the second pattern matching, a block adjacent to the current block in the current frame is used as a predetermined region for the above-described calculation of the MV candidate evaluation value.

[0317] FIG. 45 is a diagram for illustrating one example of pattern matching (pattern matching) between a pattern in the current frame and a block in the reference frame. As illustrated in FIG. 45, in the second pattern matching, the MV of the current block (Cur block) is extracted by estimating, in the reference frame (Ref ₀ ), the block that has the best match to the block adjacent to the current block in the current frame (Cur Pic). More specifically, the difference between a reconstructed image in a coded region that borders both left and above or either left or above, and a reconstructed image that is in a corresponding region in a coded reference frame (Ref0) and is indicated by an MV candidate, is extracted and the score value is calculated using the resulting difference value. It is excellent to select, as the best possible MV option, the candidate MV option that results in the best score value from the set of possible MV options.

[0318] This information indicating whether or not to use FRUC mode (called, for example, a "FRUC flag") may be signaled at the CU level. In addition, when the FRUC mode is applied (eg, when the FRUC flag is true), information indicating the applicable pattern matching method (first pattern matching or second pattern matching) may be signaled at the CU level. It should be noted that signaling of this information does not necessarily need to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, brick level, CTU level, or subblock level).

[0319] MV extraction > affine mode

Affine mode is a mode for generating MVs using an affine transformation. For example, the MV may be extracted in units of sub-blocks based on the motion vectors of multiple neighboring blocks. This mode is also called "motion compensated affine prediction mode".

[0320] FIG. 46A is a diagram for illustrating one example of MV extraction in subblock units based on the MVs of multiple neighboring blocks. In fig. 46A, the current block includes sixteen 4x4 pixel subblocks. Here, the motion vector v ₀ at the upper left corner control point in the current block is extracted based on the MV of the adjacent block, and similarly, the motion vector v ₁ at the upper right corner control point in the current block is extracted based on the MV of the adjacent sub-block. Two motion vectors v ₀ and v ₁ are projected according to expression (1A) below, and motion vectors (v _x , v _y ) for the corresponding sub-blocks in the current block are extracted.

[0321] Mathematical expression 1

[0322] Here, x and y indicate the horizontal position and vertical position of the subblock, respectively, and w indicates a predetermined weight coefficient.

[0323] This information indicating the affine mode (eg, called an "affine flag") may be signaled at the CU level. It should be noted that signaling of this information does not necessarily need to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, brick level, CTU level, or subblock level).

[0324] In addition, the affine mode may include multiple modes for different methods for extracting MVs at the top left and top right corner control points. For example, affine modes include two modes, which are affine reciprocal mode (also called "affine normal reciprocal mode") and affine union mode.

[0325] FIG. 46B is a diagram for illustrating one example of MV extraction in subblock units in affine mode in which three control points are used. In fig. 46B, the current block includes, for example, sixteen 4x4 pixel subblocks. Here, the motion vector v ₀ at the top left corner control point in the current block is extracted based on the MV of the adjacent block. Here, the motion vector v ₁ at the upper right corner control point in the current block is extracted based on the MV of the adjacent block, and similarly, the motion vector v ₂ at the lower left corner control point for the current block is extracted based on the MV of the adjacent block. The three motion vectors v ₀ , v ₁ and v ₂ may be projected according to Expression (1B) below, and the motion vectors (v _x , v _y ) for the corresponding sub-blocks in the current block are extracted.

[0326] Mathematical expression 2

[0327] Here, x and y indicate a horizontal position and a vertical position of a sub-block, respectively, and w and h each indicate a predetermined weighting factor. Here, w can indicate the width of the current block, and h can indicate the height of the current block.

[0328] Affine modes that use different numbers of control points (eg, two and three control points) can be switched and signaled at the CU level. It should be noted that information indicating the number of control points in the affine mode used at the CU layer may be signaled at another layer (eg, sequence level, frame level, slice level, brick level, CTU level, or subblock level).

[0329] In addition, such an affine mode that uses three control points may include different methods for extracting MVs at the top left, top right, and bottom left corner control points. For example, affine modes that use three control points include two modes that are affine reciprocal mode and affine union mode, as is the case for affine modes that use two control points.

[0330] It should be noted that, in affine modes, the size of each sub-block included in the current block may not be limited to 4x4 pixels and may be a different size. For example, the size of each sub-block may be 8x8 pixels.

[0331] MV Extract > Affine Mode > Control Point

Fig. 47A, 47B and 47C are a conceptual diagram to illustrate one example of MV control point extraction in affine mode.

[0332] As illustrated in FIG. 47A, in the affine mode, for example, MV predictors at the corresponding control points of the current block are calculated based on a set of MVs corresponding to blocks encoded according to the affine mode from coded block A (left), block B (top), block C (top right ), block D (bottom left) and block E (top left) that are adjacent to the current block. More specifically, the encoded block A (left), block B (upper), block C (upper right), block D (lower left) and block E (upper left) are examined in the listed order, and the first effective block encoded according to the affine mode , is identified. The MV at each control point for the current block is calculated based on the set of MVs corresponding to the identified block.

[0333] For example, as illustrated in FIG. 47B, when block A, which is adjacent to the left of the current block, is encoded according to an affine mode that uses two control points, motion vectors v ₃ and v ₄ , projected at the upper left corner position and upper right corner position of the encoded block including block A itself is extracted. The motion vector v ₀ at the top left control point and the motion vector v ₁ at the top right control point for the current block are then calculated from the extracted motion vectors v ₃ and v ₄ .

[0334] For example, as illustrated in FIG. 47C, when block A, which is adjacent to the left of the current block, is encoded according to an affine mode that uses three control points, motion vectors v ₃ , v ₄ and v ₅ projected at the top left corner position, top right corner position and bottom the left corner position of the coded block including block A is retrieved. The motion vector v ₀ at the top left control point for the current block, the motion vector v ₁ at the top right control point for the current block, and the motion vector v ₂ at the bottom left control point for the current block are then calculated from the extracted vectors v ₃ , v ₄ , and v ₅ movements.

[0335] The MV extraction methods illustrated in FIG. 47A-47C may be used in MV extraction at each control point for the current block in step Sk_1 illustrated in FIG. 50, or may be used to extract an MV predictor at each control point for the current block in step Sj_1 illustrated in FIG. 51 described below.

[0336] FIG. 48A and 48B are a conceptual diagram for illustrating another example of MV extraction at control points in affine mode.

[0337] FIG. 48A is a diagram for illustrating an affine mode in which two control points are used.

[0338] In affine mode, as illustrated in FIG. 48A, the MV selected from the MVs in the coded block A, block B and block C that adjoin the current block are used as the motion vector v ₀ at the upper left corner control point for the current block. Likewise, an MV selected from the MVs of the coded block D and block E that adjoin the current block is used as the motion vector v ₁ at the upper right corner control point for the current block.

[0339] FIG. 48B is a diagram for illustrating an affine mode in which three control points are used.

[0340] In affine mode, as illustrated in FIG. 48B, the MV selected from the MVs in the coded block A, block B and block C that adjoin the current block are used as the motion vector v ₀ at the upper left corner control point for the current block. Likewise, an MV selected from the MVs of the coded block D and block E that adjoin the current block is used as the motion vector v ₁ at the upper right corner control point for the current block. In addition, an MV selected from the MVs of a coded block F and a block G that adjoin the current block is used as a motion vector v ₂ at the lower left corner control point for the current block.

[0341] It should be noted that the MV extraction methods illustrated in FIGS. 48A and 48B may be used in MV extraction at each control point for the current block in step Sk_1 illustrated in FIG. 50 described below, or may be used to extract the MV predictor at each control point for the current block in step Sj_1 illustrated in FIG. 51 described below.

[0342] Here, when affine modes that use different numbers of control points (for example, two and three control points) can be switched and signaled at the CU level, the number of control points for the coded block and the number of control points for the current block may differ from each other.

[0343] FIG. 49A and 49B are a conceptual diagram for illustrating one example of a method for MV extraction at control points when the number of control points for a coded block and the number of control points for a current block are different from each other.

[0344] For example, as illustrated in FIG. 49A, the current block has three control points at the upper left corner, the upper right corner and the lower left corner, and block A, which is adjacent to the left of the current block, is encoded according to an affine mode in which two control points are used. In this case, the motion vectors v ₃ and v ₄ projected at the upper left corner position and the upper right corner position in the coded block including block A are extracted. The motion vector v ₀ at the top left corner control point and the motion vector v ₁ at the top right corner control point for the current block are then calculated from the extracted motion vectors v ₃ and v ₄ . In addition, the motion vector v ₂ at the lower left corner control point is calculated from the extracted motion vectors v ₀ and v ₁ .

[0345] For example, as illustrated in FIG. 49B, the current block has two control points in the upper left corner and in the upper right corner, and block A, which is adjacent to the left of the current block, is encoded according to an affine mode in which three control points are used. In this case, motion vectors v ₃ , v ₄ and v ₅ projected at the upper left corner position in the coded block including block A, at the upper right corner position in the coded block and at the lower left corner position in the coded block are extracted . The motion vector v ₀ at the top left corner control point for the current block and the motion vector v ₁ at the top right corner control point for the current block are then calculated from the extracted motion vectors v ₃ , v ₄ and v ₅ .

[0346] It should be noted that the MV extraction methods illustrated in FIGS. 49A and 49B may be used in MV extraction at each control point for the current block in step Sk_1 illustrated in FIG. 50 described below, or may be used to extract the MV predictor at each control point for the current block in step Sj_1 illustrated in FIG. 51 described below.

[0347] MV extract > affine mode > affine join mode

Fig. 50 is a flowchart illustrating one example of the affine combining mode.

[0348] In the affine merge mode, first, the inter-prediction module 126 extracts the MVs at the corresponding control points for the current block (step Sk_1). The control points are the top left corner point of the current block and the top right corner point of the current block, as illustrated in FIG. 46A, or the top left corner point of the current block, the top right corner point of the current block, and the bottom left corner point of the current block, as illustrated in FIG. 46B. At this time, the inter-prediction module 126 may encode MV selection information to identify two or three extracted MVs in the stream.

[0349] For example, when the MV extraction methods illustrated in FIG. 47A-47C are used as illustrated in FIGS. 47A, the inter-prediction module 126 examines the coded block A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) in the listed order and identifies the first effective block coded according to the affine regime.

[0350] Inter prediction module 126 extracts the MV at the control point using the identified first effective block encoded according to the identified affine mode. For example, when block A is identified and block A has two control points, as illustrated in FIG. 47B, the mutual prediction unit 126 calculates a motion vector v ₀ at the upper left corner control point of the current block and a motion vector v ₁ at the upper right corner control point of the current block from the motion vectors v ₃ and v ₄ at the upper left corner of the coded block including block A, and in the upper right corner of the coded block. For example, the mutual prediction module 126 calculates the motion vector v ₀ at the top left corner control point of the current block and the motion vector v ₁ at the top right corner control point of the current block by projecting the motion vectors v ₃ and v ₄ into the top left corner and the top right corner coded block to the current block.

[0351] Alternatively, when block A is identified and block A has three control points, as illustrated in FIG. 47C, the mutual prediction unit 126 calculates the motion vector v ₀ at the top left corner control point of the current block, the motion vector v ₁ at the top right corner control point of the current block, and the motion vector v ₂ at the bottom left corner control point of the current block from the vectors v ₃ . v ₄ and v ₅ movements in the upper left corner of the coded block including block A, in the upper right corner of the coded block and in the lower left corner of the coded block. For example, the mutual prediction module 126 calculates the motion vector v ₀ at the top left corner control point of the current block, the motion vector v ₁ at the top right corner control point of the current block, and the motion vector v ₂ at the bottom left corner control point of the current block by projecting the vectors v ₃ , v ₄ and v ₅ moves in the top left corner, top right corner and bottom left corner of the coded block onto the current block.

[0352] It should be noted that, as illustrated in FIG. 49A described above, MVs at three control points can be calculated when block A is identified and block A has two control points, and that, as illustrated in FIG. 49B described above, MVs at two control points can be calculated when block A is identified and block A has three control points.

[0353] Next, the inter-prediction module 126 performs motion compensation of each of the plurality of sub-blocks included in the current block. In other words, the inter-prediction unit 126 calculates the MV for each of the plurality of sub-blocks as an affine MV using either the two motion vectors v ₀ and v ₁ and the above expression (1A) or the three motion vectors v ₀ , v ₁ and v ₂ and the above expression (1B) (step Sk_2). The inter-prediction module 126 then performs motion compensation of the sub-blocks using these affine MVs and coded reference frames (step Sk_3). When the processes in steps Sk_2 and Sk_3 are executed for each of all sub-blocks included in the current block, the process for generating a prediction image using the affine fusion mode for the current block ends. In other words, motion compensation of the current block is performed in order to generate a predictive image of the current block.

[0354] It should be noted that the above-described list of possible MV options can be generated at step Sk_1. The MV candidate list, for example, may be a list including MV candidate candidates extracted using a plurality of MV extraction methods for each control point. The plurality of MV extraction methods may be any combination of the MV extraction methods illustrated in FIG. 47A-47C, the MV extraction methods illustrated in FIGS. 48A and 48B, the MV extraction methods illustrated in FIGS. 49A and 49B, and other MV extraction methods.

[0355] It should be noted that the MV candidate lists may include MV candidate candidates in a mode in which prediction is performed in subblock units other than the affine mode.

[0356] It should be noted that, for example, a list of possible MV options including possible MV options in the affine pooling mode in which two control points are used and in the affine combining mode in which three control points are used can be generated as a list of possible MV options. Alternatively, a list of possible MV options including possible MV options in an affine join mode that uses two control points, and a list of possible MV options including possible MV options in an affine join mode that uses three control points can be formed separately. Alternatively, a list of possible MV options including possible MV options in one of an affine join mode that uses two control points and an affine join mode that uses three control points can be generated. The possible MV variant(s), for example, could be MV for coded block A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) or MV for an efficient block of blocks.

[0357] It should be noted that an index indicating one of the MVs in the list of possible MV options may be transmitted as MV selection information.

[0358] MV Extract > Affine Mode > Affine Reciprocal Mode

Fig. 51 is a flowchart illustrating one example of the affine mutual mode.

[0359] In the affine mutual mode, as illustrated, first, the mutual prediction module 126 extracts the MV predictors (v ₀ , v ₁ ) or (v ₀ , v ₁ , v ₂ ) of the corresponding two or three control points of the current block (step Sj_1 ). The control points are the top left corner point for the current block, the top right corner point for the current block, and the bottom left corner point for the current block, as illustrated in FIG. 46A or FIG. 46B.

[0360] For example, when the MV extraction methods illustrated in FIG. 48A and 48B are used, the inter-prediction module 126 retrieves the MV predictors (v ₀ , v ₁ ) or (v ₀ , v ₁ , v ₂ ) at the corresponding two or three control points for the current block by selecting the MVs of any of the encoded blocks blocks near the corresponding control points for the current block illustrated in FIG. 48A or in FIG. 48B. At this time, the inter-prediction module 126 encodes, in-stream, MV predictor selection information to identify the selected two or three MV predictors.

[0361] For example, the inter-prediction module 126 may determine, using a cost estimate or the like, a block from which an MV is selected as a control point predictor MV from among the coded blocks adjacent to the current block, and may record , in the bit stream, a flag indicating which MV predictor is selected. In other words, the inter-prediction module 126 outputs, as a prediction parameter, MV predictor selection information such as a flag to the entropy encoder 110 via the prediction parameter generator 130.

[0362] Next, the inter-prediction module 126 performs motion estimation (steps Sj_3 and Sj_4) while updating the MV vector predictor selected or extracted in step Sj_1 (step Sj_2). In other words, the inter-prediction unit 126 calculates, as an affine MV, the MV of each of the sub-blocks that corresponds to the updated MV predictor using either expression (1A) or expression (1B) described above (step Sj_3). The inter-prediction module 126 then performs motion compensation of the sub-blocks using these affine MVs and coded reference frames (step Sj_4). The processes in steps Sj_3 and Sj_4 are executed for all blocks in the current block whenever the MV predictor is updated in step Sj_2. As a result, for example, the mutual prediction unit 126 determines the MV predictor that results in the lowest cost as the MV at the control point in the motion estimation loop (step Sj_5). At this time, the inter-prediction unit 126 further encodes, in the stream, the difference value between the determined MV and the MV predictor as an MV difference. In other words, the inter-prediction module 126 outputs the MV difference as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130.

[0363] Finally, the inter prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the determined MV and the encoded reference frame (step Sj_6).

[0364] It should be noted that the above-described list of possible MV options may be generated in step Sj_1. The MV candidate list, for example, may be a list including MV candidate candidates extracted using a plurality of MV extraction methods for each control point. The plurality of MV extraction methods may be any combination of the MV extraction methods illustrated in FIG. 47A-47C, the MV extraction methods illustrated in FIGS. 48A and 48B, the MV extraction methods illustrated in FIGS. 49A and 49B, and other MV extraction methods.

[0365] It should be noted that the list of possible MV options may include possible MV options in a mode in which prediction is performed in subblock units other than the affine mode.

[0366] It should be noted that, for example, a list of possible MV options including possible MV options in the affine reciprocal mode in which two control points are used and in the affine reciprocal mode in which three control points are used may be generated as a list of possible MV options. Alternatively, a list of possible MV options including possible MV options in an affine reciprocal mode that uses two control points, and a list of possible MV options including possible MV options in an affine reciprocal mode that uses three control points can be formed separately. Alternatively, a list of possible MV options including possible MV options in one of an affine reciprocal mode in which two control points are used and an affine reciprocal mode in which three control points are used may be generated. The possible MV variant(s), for example, could be MV for coded block A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) or MV for an efficient block of blocks.

[0367] It should be noted that an index indicating one of the possible MV options in the list of possible MV options may be transmitted as MV predictor selection information.

[0368] MV extract > triangle mode

The mutual prediction unit 126 generates one rectangular prediction image for the rectangular current block in the above example. However, the mutual prediction unit 126 can generate a plurality of prediction images having a shape other than a rectangle for the rectangular current block, and can combine the plurality of prediction images to generate a final rectangular prediction image. A shape other than a rectangle may, for example, be a triangle.

[0369] FIG. 52A is a diagram for illustrating the generation of two triangular projection images.

[0370] The inter-prediction unit 126 generates a triangular prediction image by performing motion compensation of the first segment having a triangular shape in the current block by using the first MV of the first segment to generate the triangular prediction image. Likewise, the inter-prediction unit 126 generates a triangular prediction image by performing motion compensation of the second segment having a triangular shape in the current block by using the second MV of the second segment to generate the triangular prediction image. The mutual prediction unit 126 then generates a prediction image having a rectangular shape identical to the rectangular shape of the current block by combining these prediction images.

[0371] It should be noted that the first prediction image having a rectangular shape corresponding to the current block may be generated as a prediction image for the first segment using the first MV. In addition, a second prediction image having a rectangular shape corresponding to the current block may be generated as a prediction image for the second segment using the second MV. The prediction image for the current block may be generated by performing weighting summation of the first prediction image and the second prediction image. It should be noted that the portion that is subject to weighting summation may be a partial region at the boundary between the first segment and the second segment.

[0372] FIG. 52B is a conceptual diagram for illustrating examples of a first portion of a first segment that overlaps with a second segment, and first and second sets of samples that may be weighted as part of the correction process. The first part, for example, may be one-fourth the width or height of the first segment. In another example, the first portion may have a width corresponding to N samples adjacent to the edge of the first segment, where N is an integer greater than zero and N, for example, may be the integer 2. As illustrated, the left example of FIG. 52B shows a rectangular segment having a rectangular portion with a width that is one-fourth the width of the first segment, with a first set of samples including samples outside the first portion and samples within the first portion, and a second set of samples including samples within the first portion. parts. The central example of FIG. 52B shows a rectangular segment having a rectangular portion with a height that is one quarter of the height of the first segment, with a first set of samples including samples outside the first portion and samples within the first portion, and a second set of samples including samples within the first portion. parts. The right example in Fig. 52B shows a triangular segment having a polygonal portion with a height that corresponds to two samples, with a first set of samples including samples outside the first portion and samples inside the first portion, and a second set of samples including samples inside the first portion.

[0373] The first portion may constitute a portion of the first segment that overlaps with an adjacent segment. Fig. 52C is a conceptual diagram for illustrating a first part of a first segment, which constitutes a part of the first segment that overlaps with a part of an adjacent segment. For ease of illustration, a rectangular segment is shown having an overlapping portion with a spatially adjacent rectangular segment. Segments having other shapes, for example, triangular segments, can be used, and overlapping portions can overlap with a spatially adjacent or temporally adjacent segment.

[0374] In addition, although an example is given in which a prediction image is generated for each of two segments using inter prediction, a prediction image may be generated for at least one segment using intra prediction.

[0375] FIG. 53 is a flowchart illustrating one example of a triangle mode.

[0376] In the triangular mode, first, the inter-prediction unit 126 splits the current block into a first segment and a second segment (step Sx_1). At this time, the inter-prediction unit 126 may encode, in the stream, segment information, which is segmentation-related information, as a prediction parameter. In other words, the inter-prediction module 126 may output the segment information as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 .

[0377] First, inter-prediction module 126 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sx_2). In other words, the inter-prediction module 126 generates a list of possible MV options.

[0378] The inter-prediction module 126 then selects a candidate MV for the first segment and a candidate MV for the second segment as the first MV and the second MV, respectively, from the plurality of candidate MVs obtained in step Sx_2 (step Sx_3). At this time, the inter-prediction module 126 encodes, in the stream, MV selection information for identifying the selected MV candidate as a prediction parameter. In other words, the inter-prediction module 126 outputs the MV selection information as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 .

[0379] Next, the inter prediction unit 126 generates a first prediction image by performing motion compensation using the selected first MV and the encoded reference frame (step Sx_4). Likewise, the inter-prediction module 126 generates a second prediction image by performing motion compensation using the selected second MV and the encoded reference frame (step Sx_5).

[0380] Finally, the inter-prediction unit 126 generates a prediction image for the current block by performing weighting summation of the first prediction image and the second prediction image (step Sx_6).

[0381] It should be noted that although the first segment and the second segment are triangles in the example illustrated in FIG. 52A, the first segment and the second segment may be trapezoidal or other shapes different from each other. In addition, although the current block includes two segments in the example illustrated in FIG. 52A, the current block may include three or more segments.

[0382] In addition, the first segment and the second segment may overlap each other. In other words, the first segment and the second segment may include an identical pixel region. In this case, the prediction image for the current block may be generated using the prediction image in the first segment and the prediction image in the second segment.

[0383] In addition, although an example is illustrated in which a prediction image is generated for each of two segments using inter prediction, a prediction image may be generated for at least one segment using intra prediction.

[0384] It should be noted that the list of possible MV options for selecting the first MV and the list of possible MV options for selecting the second MV may be different from each other, or the list of possible MV options for selecting the first MV may also be used as the list of possible MVs -options for choosing the second MV.

[0385] It should be noted that the segment information may include an index indicating a partitioning direction in which at least the current block is partitioned into a plurality of segments. The MV selection information may include an index indicating the selected first MV and an index indicating the selected second MV. A single index can specify many pieces of information. For example, one index jointly indicating part or all of the segment information and part or all of the MV selection information may be encoded.

[0386] MV extraction > ATMVP mode

Fig. 54 is a diagram illustrating one example of the ATMVP mode in which the MV is extracted in units of subblocks.

[0387] The ATMVP mode is a mode categorized as an aggregation mode. For example, in ATMVP mode, a candidate MV for each subblock is registered in the list of candidate MVs for use in normal aggregation mode.

[0388] More specifically, in ATMVP mode, first, as illustrated in FIG. 54, a temporary reference MV block associated with the current block is identified in a coded reference frame indicated by the MV (MV0) of a neighboring block located at the lower left position relative to the current block. Then, in each sub-block in the current block, an MV used to encode the region corresponding to the sub-block in the temporary reference MV block is identified. The MV identified in this manner is included in the list of MV candidates as a candidate MV for the subblock in the current block. When a candidate MV for each sub-block is selected from the list of candidate MVs, the sub-block undergoes motion compensation in which the candidate MV is used as the MV for the sub-block. Thus, a forecast image for each subblock is formed.

[0389] Although the block located at the lower left position relative to the current block is used as the surrounding MV reference block in the example illustrated in FIG. 54, it should be noted that a different block may be used. In addition, the subblock size may be 4x4 pixels, 8x8 pixels, or other size. The sub-block size can be switched per unit, such as slice, brick, frame, etc.

[0390] Motion Estimation > DMVR

Fig. 55 is a diagram illustrating the relationship between the combining mode and the DMVR.

[0391] The inter-prediction unit 126 extracts the MV for the current block according to the combining mode (step Sl_1). Next, the mutual prediction unit 126 determines whether or not to perform MV estimation, i.e. motion assessment (stage Sl_2). Here, when determining whether to not perform motion estimation (“No” in step Sl_2), the inter-prediction unit 126 determines the MV extracted in step Sl_1 as the final MV for the current block (step Sl_4). In other words, in this case, the MV for the current block is determined according to the merge mode.

[0392] When determining whether to perform motion estimation in step Sl_1 (Yes in step Sl_2), the inter-prediction module 126 extracts the final MV for the current block by estimating the surrounding area of the reference frame indicated by the MV extracted in step Sl_1 (step Sl_3) . In other words, in this case, the MV for the current block is determined according to the DMVR.

[0393] FIG. 56 is a conceptual diagram for illustrating another example of a DMVR for determining MV.

[0394] First, in merge mode, for example, candidate MVs (L0 and L1) are selected for the current block. The reference pixel is identified from the first reference frame (L0), which is a coded frame in the L0 list according to the MV candidate (L0). Likewise, a reference pixel is identified from a second reference frame (L1), which is a coded frame in the L1 list according to the MV candidate (L1). The reference is formed by calculating the average of these reference pixels.

[0395] Next, each of the surrounding regions of candidate MVs of the first reference frame (L0) and the second reference frame (L1) is estimated using the reference, and the MV that results in the lowest cost is determined as the final MV. It should be noted that costs can be calculated, for example, using the difference value between each of the pixel values in the reference and the corresponding one of the pixel values in the evaluation area, the values of the MV candidate values, etc.

[0396] The completely identical processes described here do not always have to be performed. Any process to enable the extraction of the final MV by evaluating the surrounding areas of candidate MVs can be used.

[0397] FIG. 57 is a conceptual diagram for illustrating another example of a DMVR for determining MV. Unlike the example DMVR illustrated in FIG. 56, in the example illustrated in FIG. 57, costs are calculated without creating a standard at all.

[0398] First, the inter-prediction module 126 estimates the surrounding area of the reference block included in each of the reference frames in the L0 list and the L1 list based on the initial MV, which is a candidate MV obtained from each candidate list. MV options. For example, as illustrated in FIG. 57, the initial MV corresponding to the reference block in the L0 list is InitMV_L0, and the initial MV corresponding to the reference block in the L1 list is InitMV_L1. When estimating motion, the inter-prediction module 126 first sets a search position for the reference frame in the L0 list. Based on the position indicated by the vector difference indicating the search position to be specified, namely, the initial MV (ie, InitMV_L0), the vector difference for the search position is MVd_L0. The inter-prediction module 126 then determines the estimated position in the reference frame in the L1 list. This search position is indicated by a vector difference for the search position from the position indicated by the initial MV (ie, InitMV_L1). More specifically, the inter-prediction unit 126 determines the vector difference as MVd_L1 by mirroring MVd_L0. In other words, the inter-prediction unit 126 determines a position that is symmetrical with respect to the position indicated by the initial MV as a search position in each reference frame in the L0 list and L1 list. The inter-prediction module 126 calculates, for each search position, the total sum of absolute differences (SAD) between the pixel values at the search positions in the blocks as costs and learns the search position that results in the lowest cost.

[0399] FIG. 58A is a diagram illustrating one example of motion estimation in DMVR, and FIG. 58B is a flowchart illustrating one example of motion estimation.

[0400] First, in step 1, the inter-prediction module 126 calculates the costs between the search position (also called the “starting point”) indicated by the initial MV and the eight surrounding search positions. The mutual prediction module 126 then determines whether or not the costs at each of the search positions other than the starting point are the smallest. Here, upon determining that the cost at a search position other than the starting point is the smallest, the inter-prediction unit 126 changes the target to the search position at which the smallest cost is obtained and executes the process in step 2. When the cost at the starting point is the smallest , the mutual prediction module 126 skips the process in step 2 and executes the process in step 3.

[0401] In step 2, the mutual prediction module 126 performs a search similar to the process in step 1 regarding, as a new starting point, the search position after the target is changed according to the result of the process in step 1. The mutual prediction module 126 then determines whether or not the costs at each of the search positions other than the starting point are the smallest. Here, when determining that the cost at a search position other than the starting point is the smallest, the inter-prediction unit 126 executes the process in step 4. When the cost at the starting point is the smallest, the inter-prediction unit 126 carries out the process in step 3.

[0402] In step 4, the inter-prediction unit 126 considers the search position at the starting point as the final search position, and determines the difference between the position indicated by the starting MV and the final search position as a vector difference.

[0403] In step 3, the inter-prediction module 126 determines the pixel position with sub-pixel precision at which the lowest cost is obtained, based on the costs at four points located at the top, bottom, left and right positions relative to the starting point in step 1 or step 2, and considers the pixel position as the final search position. The pixel position is determined with sub-pixel precision by performing a summation weighting each of the four top, bottom, left and right vectors ((0, 1), (0,-1), (-1, 0) and (1, 0)), using, as a weighting factor, the costs corresponding to one of the four search positions. The inter-prediction unit 126 then determines the difference between the position indicated by the initial MV and the final search position as a vector difference.

[0404] Motion compensation > BIO/OBMC/LIC

Motion compensation includes a mode for generating a predictive image and correcting a predictive image. The mode, for example, is BIO, OBMC and LIC, which are described below.

[0405] FIG. 59 is a flowchart illustrating one example of predictive image generation.

[0406] The mutual prediction unit 126 generates a prediction image (step Sm_1) and corrects the prediction image according to any of the modes described above (step Sm_2).

[0407] FIG. 60 is a flowchart illustrating another example of predictive image generation.

[0408] The inter-prediction unit 126 extracts the MV of the current block (step Sn_1). Next, the mutual prediction unit 126 generates a prediction image using the MV (step Sn_2) and determines whether or not to perform the correction process (step Sn_3). Here, upon determining whether to carry out the correction process (“Yes” in step Sn_3), the inter-prediction unit 126 generates the final prediction image by correcting the prediction image (step Sn_4). It should be noted that in the LIC described below, the brightness and chrominance can be adjusted in step Sn_4. When determining whether to not perform the correction process (“No” in step Sn_3), the inter-prediction unit 126 outputs the prediction image as the final prediction image without correcting the prediction image (step Sn_5).

[0409] Motion compensation > OBMC

It should be noted that the mutual prediction image can be generated using motion information for an adjacent block in addition to motion information for the current block obtained by motion estimation. More specifically, a mutual prediction image can be generated for each sub-block in the current block by performing weighting summation of the prediction image based on the motion information obtained by motion estimation (in the reference frame) and the prediction image based on the motion information of the adjacent block (in the current frame) . This mutual prediction (motion compensation) is also called "overlapping block motion compensation (OBMC)" or "OBMC mode".

[0410] In OBMC mode, information indicating the subblock size for the OBMC (referred to, for example, as "OBMC block size") may be signaled at the sequence level. In addition, information indicating whether or not to use the OBMC mode (called, for example, an “OBMC flag”) is signaled at the CU level. It should be noted that the signaling of this information need not necessarily be performed at the sequence level and the CU level and may be performed at another level (eg, frame level, slice level, brick level, CTU level, or subblock level).

[0411] The OBMC mode is described in more detail below. Fig. 61 and 62 are a flowchart and a conceptual diagram for illustrating a brief view of the prediction image correction process performed by OBMC.

[0412] First, as illustrated in FIG. 62, a prediction image (Pred) by normal motion compensation is obtained using the MV assigned to the current block. In fig. 62, the arrow "MV" points to the reference frame and indicates what the current block of the current frame accesses to obtain the prediction image.

[0413] Next, the prediction image (Pred_L) is obtained by applying the motion vector (MV_L) that is already extracted for the coded block adjacent to the left of the current block to the current block (reusing the motion vector for the current block). The motion vector (MV_L) is indicated by an arrow "MV_L" indicating a reference frame from the current block. The first correction of the prediction image is performed by overlapping two prediction images Pred and Pred_L. This provides the advantage of blending the boundary between adjacent blocks.

[0414] Similarly, a prediction image (Pred_U) is obtained by applying the MV (MV_U) that is already extracted for the coded block adjacent above the current block to the current block (reusing the MV for the current block). MV (MV_U) is indicated by an arrow "MV_U" indicating a reference frame from the current block. The second correction to the prediction image is performed by overlapping the prediction image Pred_U with the prediction images (eg, Pred and Pred_L) for which the first correction is performed. This provides the advantage of blending the boundary between adjacent blocks. The prediction image obtained by the second correction is a prediction image in which the boundary between adjacent blocks is blended (smoothed) and therefore represents the final prediction image of the current block.

[0415] Although the above example is a two-path correction method using a left and an upper adjacent block, it should be noted that the correction method may be a three or more path correction method using also a right adjacent block and/or a lower adjacent block.

[0416] It should be noted that the area in which such overlap is performed may be only a portion of the area near the block boundary instead of the pixel area of the entire block.

[0417] Note that the above describes a process for correcting prediction images according to OBMC to obtain one prediction image Pred from one reference frame by overlapping additional prediction images Pred_L and Pred_U. However, when the prediction image is corrected based on a plurality of reference pictures, a similar process may be applied to each of the plurality of reference frames. In such a case, after corrected prediction images are obtained from the corresponding reference frames by performing OBMC image correction based on a plurality of reference frames, the obtained corrected prediction images are further overlapped to obtain a final prediction image.

[0418] It should be noted that, in OBMC, the current block unit may be a PU or a sub-block unit obtained by further partitioning the PU.

[0419] One example of a method for determining whether or not to apply OBMC is a method for using obmc_flag, which is a signal indicating whether or not to apply OBMC. As one specific example, encoder 100 may determine whether or not the current block belongs to a region having complex motion. The encoder sets obmc_flag to a value of "1" when the block belongs to a region having complex motion and applies OBMC when encoding, and sets obmc_flag to a value of "0" when the block does not belong to a region having complex motion and encodes the block without applying OBMC. Decoder 200 switches between applying and not applying OBMC by decoding the obmc_flag written to the stream.

[0420] Motion compensation > BIO

The following describes the MV extraction method. First, a mode for model-based MV extraction under the condition of uniform straight-line motion is described. This mode is also called "Bidirectional Optical Flow (BIO) mode". In addition, this bidirectional optical flow can be recorded as BDOF instead of BIO.

[0421] FIG. 63 is a diagram to illustrate the model under the condition of uniform linear motion. In fig. 63, (v _x , v _y ) indicates the velocity vector, and τ ₀ and τ ₁ indicate the time distances between the current frame (Cur Pic) and the two reference frames (Ref ₀ , Ref ₁ ). (MVx ₀ , MVy ₀ ) indicates MVs corresponding to the reference frame Ref ₀ , and (MVx ₁ , MVy ₁ ) indicates MVs corresponding to the reference frame Ref ₁ .

[0422] Here, according to the assumption of relatively uniform rectilinear motion demonstrated by the velocity vector, (v _x , v _y ), (MVx ₀ , MVy ₀ ) and (MVx ₁ , MVy ₁ ) are represented as (v _x τ ₀ , v _y τ ₀ ) and (-v _x τ ₁ , -v _y τ ₁ ), respectively, and the following optical flow equation (2) is specified:

[0423] Mathematical expression 3

[0424] Here, I ^(k) denotes the luminance signal value from reference image k (k=0, 1) after motion compensation. This optical flow equation indicates that the sum of (i) the time derivative of the luminance signal value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is zero. The motion vector of each block, obtained, for example, from a list of possible MV variants, can be adjusted in units of pixels, based on a combination of the optical flow equation and Hermitian interpolation.

[0425] It should be noted that the motion vector may be extracted at the decoder 200 side using a method other than model-based motion vector extraction under the condition of uniform straight-line motion. For example, a motion vector may be extracted in sub-block units based on the MVs of multiple neighboring blocks.

[0426] FIG. 64 is a flowchart illustrating one example of inter-prediction according to BIO. Fig. 65 is a diagram illustrating one example of a functional configuration of the inter-prediction unit 126 that performs inter-prediction according to the BIO.

[0427] As illustrated in FIG. 65, the mutual prediction unit 126 includes, for example, a storage unit 126a, an interpolated image extracting unit 126b, a gradient image extracting unit 126c, an optical flow extracting unit 126d, a correction value extracting unit 126e, and a prediction image corrector 126f. It should be noted that the memory device 126a may be a movie frame memory 122.

[0428] The inter-prediction unit 126 extracts two motion vectors (M ₀ , M ₁ ) using two reference frames (Ref ₀ , Ref ₁ ) different from the frame (Cur Pic) including the current block. The mutual prediction unit 126 then extracts the prediction image for the current block using two motion vectors (M0, M1) (step Sy_1). It should be noted that the motion vector M0 is a motion vector (MVx ₀ , MVy ₀ ) corresponding to the reference frame Ref ₀ , and the motion vector M ₁ is a motion vector (MVx ₁ , MVy ₁ ) corresponding to the reference frame Ref ₁ .

[0429] Next, the interpolated image extracting unit 126b extracts the interpolated image I ⁰ for the current block using the motion vector M0 and the reference frame L0 by accessing the storage device 126a. In addition, the interpolated image extracting unit 126b extracts the interpolated image I ¹ for the current block using the motion vector M1 and the reference frame L1 by accessing the storage device 126a (step Sy_2). Here, the interpolated image I ⁰ is an image included in the reference frame Ref ₀ and is to be extracted for the current block, and the interpolated image I ¹ is an image included in the reference frame Ref ₁ and is to be extracted for the current block. Each of the interpolated image I ⁰ and the interpolated image I ¹ may have a size identical to the size of the current block. Alternatively, each of the interpolated image I ⁰ and the interpolated image I ¹ may be an image larger than the current block. In addition, the interpolated image I ⁰ and the interpolated image I ¹ may include a prediction image obtained by using motion vectors (M ₀ , M ₁ ) and reference frames (L ₀ , L ₁ ) and applying a motion compensation filter.

[0430] In addition, the gradient image extracting unit 126c extracts the gradient images (Ix ⁰ , Ix ¹ , Iy ⁰ , Iy ¹ ) of the current block from the interpolated image I ⁰ and the interpolated image I ¹ . It should be noted that the gradient images in the horizontal direction are (Ix ⁰ , Ix ¹ ), and the gradient images in the vertical direction are (Iy ⁰ , Iy ¹ ). The gradient image extraction module 126c may extract each gradient image, for example, by applying a gradient filter to the interpolated images. It is only necessary that the gradient image indicates the amount of spatial variation of the pixel value along the horizontal direction or vertical direction.

[0431] Next, optical flow extractor 126d extracts, for each sub-block of the current block, optical flow (vx, vy), which is a velocity vector, using interpolated images (I ⁰ , I ¹ ) and gradient images (Ix ⁰ , Ix ¹ , Iy ⁰ , Iy ¹ ). The optical flow specifies coefficients for correcting the amount of spatial pixel movement and may be referred to as a “local motion estimation value,” “adjusted motion vector,” or “adjusted weight vector.” As one example, the subunit may be a 4x4 pixel sub-CU. It should be noted that optical flow extraction may be performed for each pixel unit and the like. instead of executing for each subblock.

[0432] Next, the inter prediction module 126 corrects the prediction image for the current block using the optical flow (vx, vy). For example, the correction value extracting unit 126e extracts a correction value for a pixel value included in the current block using optical flow (vx, vy) (step Sy_5). The prediction image corrector 126f can then correct the prediction image for the current block using the correction value (step Sy_6). It should be noted that the correction value may be retrieved in units of a pixel, or may be retrieved in units of a plurality of pixels or in units of sub-blocks.

[0433] It should be noted that the BIO process flow is not limited to the process disclosed in FIG. 64. Only part of the processes disclosed in FIG. 64 may be executed, or another process may be added or used as a replacement, or the processes may be executed in a different processing order.

[0434] Motion compensation > LIC

Next, one example of a mode for generating a forecast image using local illuminance compensation (LIC) is described.

[0435] FIG. 66A is a diagram for illustrating one example of a method for generating predictive images using a brightness correction process performed by LIC. Fig. 66B is a flowchart illustrating one example of a method for generating predictive images using LIC.

[0436] First, the inter-prediction unit 126 extracts the MV from the encoded reference frame and obtains a reference picture corresponding to the current block (step Sz_1).

[0437] Next, the inter-prediction unit 126 extracts, for the current block, information indicating how the luminance signal value is changed between the current block and the reference frame (step Sz_2). This extraction is performed based on the luminance pixel values of the encoded left adjacent reference region (surrounding reference region) and the encoded top adjacent reference region (surrounding reference region) in the current frame and the pixel values of the luminance signal at corresponding positions in the reference frame indicated by the extracted MVs. The inter-prediction unit 126 calculates the luminance correction parameter using information indicating how the luminance signal value is changed (step Sz_3).

[0438] The inter-prediction unit 126 generates a prediction image for the current block by performing a luminance correction process in which a luminance correction parameter is applied to the reference image in the reference frame indicated by the MV (step Sz_4). In other words, the prediction image, which is a reference image in the reference frame indicated by the MV, is subject to correction based on the luminance correction parameter. With this correction, the brightness can be adjusted, or the color can be adjusted. In other words, the chrominance correction parameter can be calculated using information indicating how the chrominance is changed, and the chrominance correction process can be performed.

[0439] It should be noted that the shape of the surrounding support region illustrated in FIG. 66A is one example; another form may be used.

[0440] In addition, although a process in which a prediction image is generated from a single reference frame is described here, cases in which a prediction image is generated from a plurality of reference frames may be similarly described. The prediction image can be generated after performing the brightness correction process of the reference images obtained from the reference frames, identical to that described above.

[0441] One example of a method for determining whether or not to apply LIC is a method for using lic_flag, which is a signal indicating whether or not to apply LIC. As one specific example, encoder 100 determines whether or not the current block belongs to an area having a brightness change. The encoder 100 sets lic_flag to a value of "1" when the block belongs to a region having a luminance change and applies LIC when encoding, and sets lic_flag to a value of "0" when the block does not belong to a region having a luminance change and performs encoding without applying LIC. The decoder 200 can decode the lic_flag recorded in the stream and decode the current block by switching between applying and not applying the LIC according to the value of the flag.

[0442] One example of another method for determining whether or not to apply an LIC process is a method for determining according to whether or not an LIC process is applied to a surrounding block. As one specific example, when the current block is processed in the merge mode, the inter-prediction unit 126 determines whether or not the encoded surrounding block selected in MV extraction in the merge mode is encoded using LIC. The inter-prediction unit 126 performs encoding by switching between applying and not applying LIC according to the result. It should be noted that also in this example, identical processes apply to the processes on the decoder 200 side.

[0443] The luma correction (LIC) process is described with reference to FIG. 66A and 66B and is further described below.

[0444] First, the inter-prediction module 126 extracts the MV to obtain a reference picture corresponding to the current block from a reference frame, which is a coded frame.

[0445] Next, inter-prediction module 126 extracts information indicating how the luminance signal value of the reference frame is changed to the luminance signal value of the current frame, using the pixel luminance signal values of encoded surrounding reference regions that are adjacent to the left of or above the current block, and pixel values of the luminance signal at corresponding positions in the reference frames indicated by the MV, and calculates a luminance correction parameter. For example, it is assumed that the pixel value of the luminance signal of a given pixel in the surrounding reference region in the current frame is p0, and that the pixel value of the luminance signal of the pixel corresponding to a given pixel in the surrounding reference region in the reference frame is p1. The inter-prediction unit 126 calculates coefficients A and B to optimize A*p1+B=p0 as a luminance correction parameter for a plurality of pixels in the surrounding reference region.

[0446] Next, the inter-prediction unit 126 performs a luminance correction process using the luminance correction parameter for the reference image in the reference frame indicated by the MV to generate a prediction image for the current block. For example, it is assumed that the pixel value of the luminance signal in the reference image is p2, and that the luminance-corrected pixel value of the luminance signal of the prediction image is p3. The mutual prediction unit 126 generates a prediction image after undergoing the brightness correction process by calculating A*p2+B=p3 for each of the pixels in the reference image.

[0447] For example, a region having a certain number of pixels extracted from each of the top neighboring pixel and the left neighboring pixel can be used as a surrounding reference region. In addition, the surrounding reference area is not limited to an area that is adjacent to the current block, and may be an area that is not adjacent to the current block. In the example illustrated in FIG. 66A, the surrounding reference area in the reference frame may be an area indicated by another MV in the current frame from the surrounding reference area in the current frame. For example, the other MV may be an MV in the surrounding reference region in the current frame.

[0448] Although the operations performed by the encoder 100 are described here, it should be noted that the decoder 200 performs similar operations.

[0449] It should be noted that LIC can be applied not only to the luma signal, but also to the chrominance signal. At this time, a correction parameter can be extracted separately for each of Y, Cb and Cr, or a common correction parameter can be used for any of Y, Cb and Cr.

[0450] In addition, the LIC process may be applied in subblock units. For example, the correction parameter may be extracted using a surrounding reference area in the current sub-block and a surrounding reference area in the reference sub-block in the reference frame indicated by the MV of the current sub-block.

[0451] Forecast controller

The prediction controller 128 selects one of the internal prediction image (an image or signal output from the intra prediction unit 124) and the inter prediction image (an image or signal output from the inter prediction unit 126) and outputs the selected prediction image to the subtractor 104 and the adder 116.

[0452] Prediction parameter generator

The prediction parameter generator 130 may output information related to intra prediction, inter prediction, prediction image selection in prediction controller 128, etc. as a prediction parameter to the entropy encoder 110. The entropy encoder 110 may generate a stream based on the prediction parameter that is input from the prediction parameter generator 130 and the quantized coefficients that are input from the quantizer 108. The prediction parameter may be used in the decoder 200. The decoder 200 may receive and decode the stream and perform processes identical to the prediction processes performed by the intra prediction module 124, the inter prediction module 126, and the prediction controller 128. The prediction parameter may include (i) a selectable prediction signal (e.g., MV, prediction type, or prediction mode used by intra prediction module 124 or inter prediction module 126), or (ii) an optional index, flag, or value that is based on a prediction process performed in each of the intra prediction module 124, the inter prediction module 126, and the prediction controller 128, or which indicates the prediction process.

[0453] Decoder

The following describes a decoder 200 capable of decoding a stream output from the encoder 100 described above. Fig. 67 is a block diagram illustrating a functional configuration of the decoder 200 according to this embodiment. The decoder 200 is equipment that decodes a stream that represents an encoded image in units of blocks.

[0454] As illustrated in FIG. 67, the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse converter 206, an adder 208, a block memory 210, a loop filter 212, a movie frame memory 214, an intra prediction module 216, an inter prediction module 218, a prediction controller 220, a prediction parameter generator 222 and a partition determination module 224. It should be noted that the intra prediction module 216 and the inter prediction module 218 are configured as part of the prediction execution module.

[0455] Decoder installation example

Fig. 68 is a block diagram illustrating an installation example of the decoder 200. The decoder 200 includes a processor b1 and a memory b2. For example, the plurality of constituent elements of the decoder 200 illustrated in FIG. 67 are mounted in the processor b1 and memory b2 illustrated in FIG. 68.

[0456] Processor b1 is a circuit that performs information processing and is available to storage device b2. For example, processor b1 is a specialized or general electronic circuit that decodes the stream. Processor b1 may be a processor such as a CPU. In addition, processor b1 may be a collection of multiple electronic circuits. In addition, for example, the processor b1 may serve as two or more constituent elements other than the information storing constituent element of the plurality of constituent elements of the decoder 200 illustrated in FIG. 67, etc.

[0457] Memory b2 is a dedicated or general memory for storing information that is used by processor b1 to decode the stream. Memory b2 may be an electronic circuit and may be coupled to processor b1. In addition, memory b2 may be included in processor b1. In addition, the storage device b2 may be a collection of a plurality of electronic circuits. In addition, the storage device b2 may be a magnetic disk, an optical disk or the like. or may be represented as a storage device, media, etc. In addition, the storage device b2 may be a non-volatile storage device or a volatile storage device.

[0458] For example, storage device b2 may store an image or stream. In addition, memory b2 may store a program for instructing processor b1 to decode the stream.

[0459] In addition, for example, the storage device b2 may serve as two or more constituent elements for storing information from a plurality of constituent elements of the decoder 200 illustrated in FIG. 67, etc. More specifically, the memory b2 may serve as the block memory 210 and the movie frame memory 214 illustrated in FIG. 67. More specifically, the storage device b2 can store a reconstructed image (specifically, a reconstructed block, a reconstructed frame, etc.).

[0460] It should be noted that, in decoder 200, not all of the plurality of constituent elements illustrated in FIG. 67, etc. may be implemented, and not all of the processes described above may be performed. Some of the constituent elements shown in FIG. 67, etc., may be included in another device, or part of the processes described above may be performed by another device.

[0461] Hereinafter, a complete flow of processes performed by the decoder 200 is described, and thereafter each of the constituent elements included in the decoder 200 is described. It should be noted that some of the constituent elements included in the decoder 200 perform processes identical to the processes performed by some of the constituent elements included in the encoder 100, and therefore the identical processes are not described in detail again. For example, the inverse quantizer 204, inverse converter 206, adder 208, block memory 210, movie frame memory 214, intra prediction module 216, inter prediction module 218, prediction controller 220, and loop filter 212 included in the decoder 200 perform processes similar to processes performed by the inverse quantizer 112, inverse converter 114, adder 116, block memory 118, movie frame memory 122, intra prediction module 124, inter prediction module 126, prediction controller 128, and loop filter 120 included in the encoder 100, respectively.

[0462] General flow of decoding process

Fig. 69 is a flowchart illustrating one example of a complete decoding process performed by decoder 200.

[0463] First, the partition determination unit 224 in the decoder 200 determines the partition pattern of each of the plurality of fixed size blocks (128×128 pixels) included in the frame based on a parameter that is input from the entropy decoder 202 (step Sp_1). This split pattern is a split pattern selected by the encoder 100. The decoder 200 then performs the processes of steps Sp_2 to Sp_6 for each of the plurality of split pattern blocks.

[0464] The entropy decoder 202 decodes (specifically, entropy decodes) the encoded quantized coefficients and the prediction parameter of the current block (step Sp_2).

[0465] Next, the inverse quantizer 204 inversely quantizes the plurality of quantized coefficients, and the inverse transformer 206 inversely transforms the result to recover prediction residuals of the current block (step Sp_3).

[0466] Next, a prediction execution unit including all or part of the intra prediction unit 216, the inter prediction unit 218, and the prediction controller 220 generates a prediction image of the current block (step Sp_4).

[0467] Next, the adder 208 adds the predicted image with the prediction residual to generate a reconstructed image (also called a “decoded image block”) of the current block (step Sp_5).

[0468] When the reconstructed image is generated, the contour filter 212 performs filtering on the reconstructed image (step Sp_6).

[0469] The decoder 200 then determines whether or not decoding of the entire frame is complete (step Sp_7). When determining that decoding is not yet completed (“No” in step Sp_7), the decoder 200 repeatedly executes processes starting from step Sp_1.

[0470] It should be noted that the processes of these steps Sp_1-Sp_7 may be executed sequentially by the decoder 200, or two or more processes may be executed in parallel. The processing order of two or more processes can be modified.

[0471] Partition detection module

Fig. 70 is a diagram illustrating the relationship between the partition determination unit 224 and other constituent elements. The partition determination unit 224 may perform the following processes as examples.

[0472] For example, partition determination module 224 collects block information from block memory 210 or movie frame memory 214 and further obtains a parameter from entropy decoder 202. Partition determination module 224 can then determine a fixed-size block partition pattern based on the information blocks and parameter. The partition determination module 224 may then output information indicating the determined partition pattern to the inverter 206, the intra prediction module 216, and the inter prediction module 218. The inverse converter 206 may perform inverse conversion of the transform coefficients based on the partitioning pattern indicated by the information from the partition determination unit 224 . The intra prediction unit 216 and the inter prediction unit 218 may generate a prediction image based on the split pattern indicated by the information from the split determination unit 224.

[0473] Entropy decoder

Fig. 71 is a block diagram illustrating one example of a functional configuration of the entropy decoder 202.

[0474] Entropy decoder 202 generates quantized coefficients, a prediction parameter, and a parameter associated with a partitioning pattern by entropy decoding the stream. For example, CABAC is used in entropy decoding. More specifically, the entropy decoder 202 includes, for example, a binary arithmetic decoder 202a, a context controller 202b, and a debinarization unit 202c. Binary arithmetic decoder 202a arithmetically decodes the stream using the context value retrieved by context controller 202b into a binary signal. The context controller 202b retrieves the context value according to the attribute or surrounding state of the syntactic element, i.e. probability of occurrence of a binary signal is identical to that performed by the context controller 110b of the encoder 100. The debinarization unit 202c performs debinarization to convert the binary signal output from the binary arithmetic decoder 202a into a multi-level signal indicating quantized coefficients, as described above. This binary conversion is performed according to the binary conversion method described above.

[0475] Consequently, the entropy decoder 202 outputs the quantized coefficients of each block to the inverse quantizer 204. The entropy decoder 202 may output the prediction parameter included in the stream (see FIG. 1) to the intra prediction module 216, the inter prediction module 218, and the controller. 220 forecasting. The intra prediction module 216, the inter prediction module 218, and the prediction controller 220 are capable of performing prediction processes identical to the prediction processes performed by the intra prediction module 124, the inter prediction module 126, and the prediction controller 128 on the encoder 100 side.

[0476] Entropy decoder

Fig. 72 is a diagram illustrating the CABAC flow of entropy decoder 202.

[0477] First, initialization is performed in CABAC in the entropy decoder 202. In initialization, initialization in the binary arithmetic decoder 202a and setting the initial value of the context is performed. The binary arithmetic decoder 202a and the debinarization unit 202c then perform arithmetic decoding and debinarization of, for example, the encoded CTU data. At this time, the context controller 202b updates the context value every time arithmetic decoding is performed. The context controller 202b then stores the context value as post-processing. The stored context value is used, for example, to initialize the context value for the next CTU.

[0478] Inverse Quantizer

The inverse quantizer 204 inversely quantizes the quantized coefficients of the current block, which are inputs from the entropy decoder 202. More specifically, the inverse quantizer 204 inversely quantizes the quantized coefficients of the current block, based on the quantization parameters corresponding to the quantized coefficients. The inverse quantizer 204 then outputs the inversely quantized transform coefficients (which are transform coefficients) of the current block to the inverse transformer 206.

[0479] FIG. 73 is a block diagram illustrating one example of the functional configuration of the inverse quantizer 204.

[0480] The inverse quantizer 204 includes, for example, a quantization parameter generator 204a, a predicted quantization parameter generator 204b, a quantization parameter storage device 204d, and an inverse quantization execution unit 204e.

[0481] FIG. 74 is a flowchart illustrating one example of inverse quantization performed by inverse quantizer 204.

[0482] The inverse quantizer 204 may perform the inverse quantization process as one example for each CU based on the process flow illustrated in FIG. 74. More specifically, the quantization parameter generator 204a determines whether or not to perform inverse quantization (step Sv_11). Here, upon determining whether to perform inverse quantization (“Yes” in step Sv_11), the quantization parameter generator 204a obtains the difference quantization parameter for the current block from the entropy decoder 202 (step Sv_12).

[0483] Next, the predicted quantization parameter generator 204b then obtains the quantization parameter for a processing unit other than the current block from the quantization parameter storage device 204d (step Sv_13). The predicted quantization parameter generator 204b generates a predicted quantization parameter of the current block based on the received quantization parameter (step Sv_14).

[0484] The quantization parameter generator 204a then sums the difference quantization parameter for the current block obtained from the entropy decoder 202 and the predicted quantization parameter for the current block generated by the predicted quantization parameter generator 204b (step Sv_15). This summation forms the quantization parameter for the current block. In addition, the quantization parameter generator 204a stores the quantization parameter for the current block in the quantization parameter storage device 204d (step Sv_16).

[0485] Next, the inverse quantization execution unit 204e inversely quantizes the quantized coefficients of the current block into transform coefficients using the quantization parameter generated in step Sv_15 (step Sv_17).

[0486] It should be noted that the difference quantization parameter may be decoded at the bit sequence level, frame level, slice level, brick level, or CTU level. In addition, the initial value of the quantization parameter may be decoded at the sequence level, frame level, slice level, brick level, or CTU level. At this time, the quantization parameter can be generated using the initial value of the quantization parameter and the difference quantization parameter.

[0487] It should be noted that the inverse quantizer 204 may include a plurality of inverse quantizers and may inversely quantize the quantized coefficients using an inverse quantization method selected from a plurality of inverse quantization methods.

[0488] Inverter

The inverse transformer 206 recovers the prediction residuals by inversely transforming the transform coefficients that are the inputs from the inverse quantizer 204.

[0489] For example, when information parsed from the stream indicates that EMT or AMT should be applied (eg, when the AMT flag is true), the inverter 206 inverts the transform coefficients of the current block based on the information indicating the parsed type transformations.

[0490] In addition, for example, when information parsed from the stream indicates that NSST should be applied, inverter 206 applies a secondary inverse transform to the transform coefficients.

[0491] FIG. 75 is a flowchart illustrating one example of a process performed by the inverter 206.

[0492] For example, the inverter 206 determines whether or not information indicating that the orthogonal transform is not performed is present in the stream (step St_11). Here, upon determining that this information is not present (“No” in step St_11), the inverter 206 obtains information indicating the type of transformation decoded by the entropy decoder 202 (step St_12). Then, based on the information, the inverse transformer 206 determines the transform type used for the orthogonal transform in the encoder 100 (step St_13). The inverter 206 then performs the inverse orthogonal transform using a certain transform type (step St_14).

[0493] FIG. 76 is a flowchart illustrating another example of a process performed by the inverter 206.

[0494] For example, the inverter 206 determines whether the transform size is less than or equal to a predetermined value (step Su_11). Here, upon determining that the transform size is less than or equal to a predetermined value (“Yes” in step Su_11), the inverter 206 obtains, from the entropy decoder 202, information indicating what type of transform is used by the encoder 100 at least , from one transformation type included in the first group of transformation types (step Su_12). It should be noted that this information is decoded by the entropy decoder 202 and output to the inverter 206.

[0495] Based on the information, the inverse transformer 206 determines the transform type used for the orthogonal transform in the encoder 100 (step Su_13). The inverter 206 then performs an inverse orthogonal transform on the transform coefficients of the current block using a certain transform type (step Su_14). Upon determining that the transform size is not less than or equal to a predetermined value ("No" in step Su_11), the inverse converter 206 inversely converts the transform coefficients of the current block using the second group of transform types (step Su_15).

[0496] It should be noted that the inverse orthogonal transform by the inverter 206 can be performed according to the process flow illustrated in FIG. 75 or fig. 76 for each TU, as one example. In addition, the inverse orthogonal transform can be performed by using a predetermined transform type without decoding information indicating the transform type used for the orthogonal transform. In addition, the conversion type is specifically DST7, DCT8, and the like. For an inverse orthogonal transform, the inverse transform basis function corresponding to the transform type is used.

[0497] Totalizer

The adder 208 reconstructs the current block by summing the prediction residual, which is the input from the inverter 206, and the prediction image, which is the input from the prediction controller 220. In other words, a reconstructed image of the current block is formed. Adder 208 then outputs the reconstructed image of the current block to block memory 210 and loop filter 212.

[0498] Block memory

The block memory 210 is a storage device for storing a block that is included in the current frame and is referred to in intra prediction. More specifically, the block memory 210 stores the reconstructed image output from the adder 208.

[0499] Loop filter

The contour filter 212 applies the contour filter to the reconstructed image generated by the adder 208, and outputs the filtered reconstructed image to the movie frame memory 214 and display device, etc.

[0500] When information indicating ALF on or off parsed from the stream indicates that ALF is on, one filter from the plurality of filters is selected based on the direction and activity of the local gradients, and the selected filter is applied to the reconstructed image.

[0501] FIG. 77 is a block diagram illustrating one example of a functional configuration of loop filter 212. It should be noted that loop filter 212 has a configuration similar to that of loop filter 120 of encoder 100.

[0502] For example, as illustrated in FIG. 77, the loop filter 212 includes a deblocking filtering execution unit 212a, an SAO execution unit 212b, and an ALF execution unit 212c. The deblocking filtering execution unit 212a performs a filtering process to remove blocking from the restored image. The SAO execution unit 212b performs the SAO process of the reconstructed image after subjecting it to the deblocking filtering process. The ALF execution unit 212c performs the ALF process of the reconstructed image after being subjected to the SAO process. It should be noted that loop filter 212 need not always include all of the constituent elements disclosed in FIG. 77, and may include only part of the constituent elements. In addition, the loop filter 212 may be configured to perform the above processes in a processing order different from the processing order disclosed in FIG. 77.

[0503] Motion Picture Memory

The movie frame memory 214, for example, is a storage device for storing reference frames for use in inter-prediction, and is also called a “motion picture buffer”. More specifically, the movie frame memory 214 stores the reconstructed image filtered by the edge filter 212.

[0504] Prediction Module (Intra Prediction Module, Inter Prediction Module, Prediction Controller)

Fig. 78 is a flowchart illustrating one example of a process performed by the prediction module of the decoder 200. It should be noted that the prediction execution module includes all or part of the following constituent elements: an intra prediction module 216; mutual prediction module 218; and a prediction controller 220. The prediction execution module includes, for example, an intra prediction module 216 and an inter prediction module 218.

[0505] The prediction module generates a prediction image of the current block (step Sq_1). This prediction image is also called a "prediction signal" or a "prediction block". It should be noted that the prediction signal is, for example, an internal prediction signal or a mutual prediction signal. More specifically, the prediction module generates a prediction image of the current block using a reconstructed image that has already been obtained for another block through generating the prediction image, reconstructing the prediction residual, and summing the prediction image. The prediction module of the decoder 200 generates a prediction image identical to the prediction image generated by the prediction module of the encoder 100. In other words, the prediction images are generated according to a method common among prediction modules or mutually corresponding methods.

[0506] The reconstructed image, for example, may be an image in a reference frame or an image of a decoded block (ie, another block described above) in the current frame, which is a frame including the current block. The decoded block in the current frame, for example, is a block adjacent to the current block.

[0507] FIG. 79 is a flowchart illustrating another example of a process performed by the decoder prediction module 200.

[0508] The prediction module determines a method or mode for generating a prediction image (step Sr_1). For example, the method or mode may be determined, for example, based on a prediction parameter, etc.

[0509] When determining the first method as the mode for generating a prediction image, the prediction module generates a prediction image according to the first method (step Sr_2a). When determining the second method as the mode for generating the prediction image, the prediction module generates the prediction image according to the second method (step Sr_2b). When determining the third method as the mode for generating the prediction image, the prediction module generates the prediction image according to the third method (step Sr_2c).

[0510] The first method, the second method and the third method may be mutually different methods for generating a predictive image. Each of the first to third methods may be an inter-prediction method, an intra-prediction method, or another prediction method. The above-described reconstructed image can be used in these prediction methods.

[0511] FIG. 80A and FIG. 80B illustrates a flowchart illustrating another example of a process performed by the decoder prediction module 200.

[0512] The prediction module may perform the prediction process according to the flow illustrated in FIG. 80A and FIG. 80B as one example. It should be noted that the intra-block copying illustrated in FIG. 80A and 80B is one mode that belongs to inter-prediction, and in which a block included in the current frame is called a “reference picture” or a “reference block”. In other words, a frame different from the current frame is not accessed during intra-block copying. In addition, the PCM mode illustrated in FIG. 80A is one mode that belongs to intra prediction, and in which conversion and quantization are not performed.

[0513] Internal prediction module

The intra prediction unit 216 performs intra prediction by accessing a block in the current frame stored in the block memory 210 based on the intra prediction mode parsed from the stream to generate a prediction image of the current block (ie, an intra prediction image). More specifically, the intra prediction module 216 performs intra prediction by accessing pixel values (eg, luma and/or chroma values) of a block or blocks adjacent to the current block to generate an internal prediction image, and then outputs the internal prediction image to the controller. 220 forecasting.

[0514] It should be noted that when an intra prediction mode is selected in which the luminance signal block is accessed in intra prediction of a chroma block, the intra prediction unit 216 can predict the chrominance signal component of the current block based on the luminance signal component of the current block.

[0515] In addition, when information parsed from the stream indicates that PDPC should be applied, intra prediction module 216 adjusts internally predicted pixel values based on horizontal/vertical reference pixel gradients.

[0516] FIG. 81 is a diagram illustrating one example of a process performed by the intra prediction unit 216 of the decoder 200.

[0517] The intra prediction unit 216 first determines whether or not an MPM flag indicating 1 is present in the stream (step Sw_11). Here, upon determining that the MPM flag indicating 1 is present (“Yes” in step Sw_11), the intra prediction unit 216 obtains, from the entropy decoder 202, information indicating the intra prediction mode selected in the encoder 100 from the MPM ( stage Sw_12). It should be noted that this information is decoded by the entropy decoder 202 and output to the intra prediction module 216. Next, the intra prediction unit 216 determines the MPM (step Sw_13). MPMs include, for example, six intraprediction modes. The intra prediction unit 216 then determines an intra prediction mode that is included in a plurality of intra prediction modes included in the MPM and is indicated by the information obtained in step Sw_12 (step Sw_14).

[0518] Upon determining that the MPM flag indicating 1 is not present (“No” in step Sw_11), the intra prediction unit 216 obtains information indicating the intra prediction mode selected in the encoder 100 (step Sw_15). In other words, the intra prediction module 216 receives, from the entropy decoder 202, information indicating the intra prediction mode selected in the encoder 100 from among at least one intra prediction mode that is not included in the MPM. It should be noted that this information is decoded by the entropy decoder 202 and output to the intra prediction module 216. The intra prediction unit 216 then determines an intra prediction mode that is not included in the plurality of intra prediction modes included in the MPM and is indicated by the information obtained in step Sw_15 (step Sw_17).

[0519] The intra prediction unit 216 generates a prediction image according to the intra prediction mode determined in step Sw_14 or in step Sw_17 (step Sw_18).

[0520] Mutual Prediction Module

The inter-prediction unit 218 predicts the current block by referring to a reference frame stored in the movie frame memory 214 . Prediction is performed in units of the current block or the current subblock within the current block. It should be noted that a subblock is included in a block and represents a unit smaller than the block. The subblock size can be 4x4 pixels, 8x8 pixels, or other size. The sub-block size can be switched per unit, such as slice, brick, frame, etc.

[0521] For example, the mutual prediction module 218 generates a mutual prediction signal of the current block or sub-block by performing motion compensation by using motion information (e.g., a motion vector) parsed from the encoded bitstream (e.g., a prediction parameter output from the entropy decoder 202) , and outputs the mutual prediction signal to the prediction controller 220.

[0522] When information parsed from the stream indicates that the OBMC mode should be applied, the inter-prediction unit 218 generates a inter-prediction image using the motion information of the adjacent block in addition to the motion information of the current block obtained through motion estimation.

[0523] In addition, when information parsed from the stream indicates that the FRUC mode should be applied, the inter-prediction module 218 extracts the motion information by performing motion estimation in accordance with a template matching method (bilateral matching or template matching) , parsed from the stream. Mutual prediction module 218 then performs motion compensation (prediction) using the extracted motion information.

[0524] In addition, when the BIO mode is to be applied, the inter-prediction module 218 extracts the MV based on the model under the condition of uniform straight-line motion. In addition, when information parsed from the stream indicates that the affine mode should be applied, the inter-prediction module 218 extracts the MV for each sub-block based on the MVs of multiple neighboring blocks.

[0525] Sequence of operations for MV extraction

Fig. 82 is a flowchart illustrating one example of MV extraction in decoder 200.

[0526] The inter-prediction unit 218 determines, for example, whether or not motion information (eg, MV) should be decoded. For example, inter-prediction module 218 may make a determination according to a prediction mode included in the stream, or may make a determination based on other information included in the stream. Here, when determining whether to decode the motion information, the inter-prediction unit 218 extracts the MV for the current block in a mode in which the motion information is decoded. When determining whether to not decode the motion information, the inter-prediction module 218 extracts the MV in a mode in which the motion information is not decoded.

[0527] Here, MV extraction modes may include normal mutual mode, normal pooling mode, FRUC mode, affine mode, etc., which are described below. Modes in which motion information is decoded include normal reciprocal mode, normal combining mode, affine mode (specifically, affine reciprocal mode and affine combining mode), etc. It should be noted that the motion information may include not only MVs, but also MV predictor selection information, which is described below. Modes in which motion information is not decoded include FRUC mode, etc. The inter-prediction unit 218 selects a mode for extracting the MV for the current block from a plurality of modes, and extracts the MV for the current block using the selected mode.

[0528] FIG. 83 is a flowchart illustrating another example of MV extraction in the decoder 200.

[0529] For example, the inter-prediction module 218 may determine whether or not to decode the MV difference, i.e., for example, may make a determination according to a prediction mode included in the stream, or may make a determination based on other information included in flow. Here, when determining whether to decode the MV difference, the inter-prediction unit 218 may extract the MV for the current block in a mode in which the MV difference is decoded. In this case, for example, the MV difference included in the stream is decoded as a prediction parameter.

[0530] When determining whether to decode the MV difference at all, the inter-prediction module 218 extracts the MV in a mode in which the MV difference is not decoded. In this case, the encoded MV difference is not included in the stream.

[0531] Here, as described above, MV extraction modes include normal mutual mode, normal pooling mode, FRUC mode, affine mode, etc., which are described below. Modes in which the MV difference is encoded include normal reciprocal mode and affine mode (specifically, affine reciprocal mode), etc. Modes in which the MV difference is not encoded include FRUC mode, normal concatenation mode, affine mode (specifically, affine concatenation mode), etc. The inter-prediction unit 218 selects a mode for extracting the MV for the current block from a plurality of modes, and extracts the MV for the current block using the selected mode.

[0532] MV extract > normal mutual mode

For example, when the information parsed from the stream indicates that the normal reciprocal mode should be applied, the reciprocal prediction module 218 extracts the MV based on the information parsed from the stream and performs motion compensation (prediction) using the MV.

[0533] FIG. 84 is a flowchart illustrating an example of inter-prediction by normal inter-mode in the decoder 200.

[0534] The inter-prediction module 218 of the decoder 200 performs motion compensation for each block. At this time, first, the inter-prediction unit 218 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Sg_11). In other words, the inter-prediction module 218 generates a list of possible MV options.

[0535] Next, the inter-prediction module 218 extracts N (an integer of 2 or more) MV candidates from the set of MV candidates obtained in step Sg_11 as motion vector predictor candidates (also called “MV candidates”). predictors") according to predefined ranks in order of priority (step Sg_12). It should be noted that the ranks in priority order are determined in advance for the corresponding N candidate MV predictors.

[0536] Next, the inter-prediction unit 218 decodes the MV predictor selection information from the input stream and selects one MV predictor candidate from the N candidate MV predictor candidates as the MV predictor for the current block using the decoded MV predictor selection information ( stage Sg_13).

[0537] Next, the inter-prediction unit 218 decodes the MV difference from the input stream and extracts the MV for the current block by summing the difference value, which is the decoded MV difference, and the selected MV predictor (step Sg_14).

[0538] Finally, the inter prediction unit 218 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the decoded reference frame (step Sg_15). Processes in stages Sg_11-Sg_15 are executed for each block. For example, when the processes in steps Sg_11 to Sg_15 are executed for each of all blocks in the slice, the slice reciprocal prediction using the normal reciprocal mode ends. For example, when the processes in steps Sg_11 to Sg_15 are executed for each of all blocks in a frame, frame mutual prediction using the normal mutual mode ends. It should be noted that not all blocks included in the slice may be subject to processes in steps Sg_11 to Sg_15, and slice mutual prediction using the normal mutual mode may end when a portion of the blocks are subject to processes. Likewise, frame inter-prediction using the normal inter-mode may end when the processes in steps Sg_11-Sg_15 are executed for a portion of the blocks in the frame.

[0539] MV extract > normal merge mode

For example, when information parsed from the stream indicates that the normal combining mode should be applied, the inter-prediction module 218 extracts the MV and performs motion compensation (prediction) using the MV.

[0540] FIG. 85 is a flowchart illustrating an example of inter-prediction by the normal combining mode in the decoder 200.

[0541] At this time, first, the inter-prediction unit 218 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Sh_11). In other words, the inter-prediction module 218 generates a list of possible MV options.

[0542] Next, the inter-prediction module 218 selects one candidate MV from the plurality of candidate MVs obtained in step Sh_11, thereby extracting the MV of the current block (step Sh_12). More specifically, the inter-prediction unit 218 receives the MV selection information included as a prediction parameter in the stream, and selects a candidate MV identified by the MV selection information as the MV for the current block.

[0543] Finally, the inter prediction unit 218 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the decoded reference frame (step Sh_13). The processes in stages Sh_11-Sh_13 are performed, for example, for each block. For example, when the processes in steps Sh_11 to Sh_13 are executed for each of all blocks in a slice, cross-prediction of the slice using the normal pooling mode ends. In addition, when the processes in steps Sh_11 to Sh_13 are executed for each of all blocks in a frame, frame inter-prediction using the normal combining mode ends. It should be noted that not all blocks included in the slice are processed in steps Sh_11 to Sh_13, and cross-prediction of the slice using the normal pooling mode may end when a portion of the blocks are processed. Likewise, frame inter-prediction using the normal combining mode may end when the processes in steps Sh_11-Sh_13 are executed for a portion of the blocks in the frame.

[0544] MV extraction > FRUC mode

For example, when information parsed from the stream indicates that the FRUC mode should be applied, the inter-prediction module 218 extracts the MV in the FRUC mode and performs motion compensation (prediction) using the MV. In this case, motion information is extracted at the decoder 200 side without being signaled from the encoder 100 side. For example, the decoder 200 may extract motion information by performing motion estimation. In this case, decoder 200 performs motion estimation without using the pixel values in the current block at all.

[0545] FIG. 86 is a flowchart illustrating an example of inter-prediction by FRUC mode in the decoder 200.

[0546] First, the inter-prediction module 218 generates a list indicating the MVs of decoded blocks spatially or temporally adjacent to the current block by referring to the MVs as MV candidates (the list is a list of MV candidates and can also be used as a list of possible MV options for the normal combining mode (step Si_11).Next, the best possible MV option is selected from the plurality of possible MV options registered in the list of possible MV options (step Si_12). For example, the inter-prediction unit 218 calculates the evaluation value of each candidate MV option included in the list of candidate MV options, and selects one of the candidate MV options as the best possible MV option based on the evaluation values.Based on the selected best possible MV option, the inter-prediction unit 218 then extracts the MV for the current block (step Si_14) More specifically, for example, the selected best possible MV is directly extracted as the MV for the current block. In addition, for example, the MV for the current block may be retrieved using pattern matching in the surrounding region of a position that is included in the reference frame and corresponds to the selected best possible MV candidate. In other words, evaluation using template matching on the reference frame and evaluation values can be performed on the surrounding region of the best possible MV candidate, and when there is an MV that results in a better evaluation value, the best possible MV candidate can be updated to the MV that results in a better score value, and the updated MV can be determined as the final MV for the current block. The update to the MV that results in the best score value may not be performed.

[0547] Finally, the inter prediction unit 218 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the decoded reference frame (step Si_15). The processes in stages Si_11-Si_15 are performed, for example, for each block. For example, when the processes in steps Si_11-Si_15 are executed for each of all blocks in a slice, the cross-prediction of the slice using the FRUC mode ends. For example, when the processes in steps Si_11 to Si_15 are executed for each of all blocks in a frame, frame inter-prediction using the FRUC mode ends. Each sub-block can be processed similarly to the above-described case of processing each block.

[0548] MV extract > affine join mode

For example, when information parsed from the stream indicates that the affine combining mode should be applied, the inter-prediction module 218 extracts the MV in the affine combining mode and performs motion compensation (prediction) using the MV.

[0549] FIG. 87 is a flowchart illustrating an example of inter-prediction via the affine combining mode in the decoder 200.

[0550] In the affine merge mode, first, the inter-prediction module 218 extracts the MVs at the corresponding control points for the current block (step Sk_11). The control points are the top left corner point of the current block and the top right corner point of the current block, as illustrated in FIG. 46A, or the top left corner point of the current block, the top right corner point of the current block, and the bottom left corner point of the current block, as illustrated in FIG. 46B.

[0551] For example, when the MV extraction methods illustrated in FIG. 47A-47C are used as illustrated in FIGS. 47A, inter-prediction module 218 examines the decoded block A (left), block B (upper), block C (upper right), block D (lower left), and block E (upper left) in that order and identifies the first effective block decoded according to the affine regime.

[0552] Inter prediction module 218 extracts the MV at the control point using the identified first effective block decoded according to the affine mode. For example, when block A is identified and block A has two control points, as illustrated in FIG. 47B, the mutual prediction unit 218 calculates the motion vector v ₀ at the top left corner control point of the current block and the motion vector v ₁ at the top right corner control point of the current block by projecting the motion vectors v ₃ and v ₄ at the top left corner and the top right corner of the decoded block, including block A, to the current block. Thus, the MV at each control point is extracted.

[0553] It should be noted that, as illustrated in FIG. 49A, MVs at three control points can be calculated when block A is identified and block A has two control points, and that, as illustrated in FIG. 49B, MVs at two control points can be calculated when block A is identified and when block A has three control points.

[0554] In addition, when MV selection information is included as a prediction parameter in a stream, inter-prediction module 218 can extract MVs at each control point for the current block using the MV selection information.

[0555] Next, the inter-prediction module 218 performs motion compensation of each of the plurality of sub-blocks included in the current block. In other words, the inter-prediction module 218 calculates the MV for each of the plurality of sub-blocks as an affine MV using either the two motion vectors v ₀ and v ₁ and the above expression (1A) or the three motion vectors v ₀ , v ₁ and v ₂ and the above expression (1B) (step Sk_12). The inter-prediction module 218 then performs motion compensation of the sub-blocks using these affine MVs and the decoded reference frames (step Sk_13). When the processes in steps Sk_12 and Sk_13 are executed for each of all sub-blocks included in the current block, inter-prediction using the affine join mode for the current block is completed. In other words, motion compensation of the current block is performed in order to generate a predictive image of the current block.

[0556] It should be noted that the above-described list of possible MV options can be generated at step Sk_11. The MV candidate list, for example, may be a list including MV candidate candidates extracted using a plurality of MV extraction methods for each control point. The plurality of MV extraction methods may be any combination of the MV extraction methods illustrated in FIG. 47A-47C, the MV extraction methods illustrated in FIGS. 48A and 48B, the MV extraction methods illustrated in FIGS. 49A and 49B, and other MV extraction methods.

[0557] It should be noted that the list of possible MV options may include possible MV options in a mode in which prediction is performed in subblock units other than the affine mode.

[0558] It should be noted that, for example, a list of possible MV options including possible MV options in the affine pooling mode in which two control points are used and in the affine combining mode in which three control points are used can be generated as a list of possible MV options. Alternatively, a list of possible MV options including possible MV options in an affine join mode that uses two control points, and a list of possible MV options including possible MV options in an affine join mode that uses three control points can be formed separately. Alternatively, a list of possible MV options including possible MV options in one of an affine join mode that uses two control points and an affine join mode that uses three control points can be generated.

[0559] MV extraction > affine reciprocal mode

For example, when information parsed from the stream indicates that the affine reciprocal mode should be applied, the reciprocal prediction module 218 extracts the MV in the affine reciprocal mode and performs motion compensation (prediction) using the MV.

[0560] FIG. 88 is a flowchart illustrating an example of mutual prediction by affine mutual mode in the decoder 200.

[0561] In the affine mutual mode, first, the mutual prediction unit 218 extracts the MV predictors (v ₀ , v ₁ ) or (v ₀ , v ₁ , v ₂ ) of the corresponding two or three control points of the current block (step Sj_11). The control points are the top left corner point of the current block, the top right corner point of the current block, and the bottom left corner point of the current block, as illustrated in FIG. 46A or FIG. 46B.

[0562] The inter-prediction unit 218 receives the MV predictor selection information included as a prediction parameter in the stream, and extracts an MV predictor at each control point for the current block using the MV identified by the MV predictor selection information. For example, when the MV extraction methods illustrated in FIG. 48A and 48B are used, the inter-prediction module 218 retrieves the ( _v0 , _v1 ) or ( _v0 , _v1 , _v2 ) motion vector predictors at the control points for the current block by selecting the MV block identified by the MV predictor selection information. , from the decoded blocks around the corresponding control points for the current block illustrated in FIG. 48A or in FIG. 48B.

[0563] Next, the inter-prediction module 218 receives each MV difference included as a prediction parameter in the stream, and sums the MV predictor at each control point for the current block and the MV difference corresponding to the MV predictor (step Sj_12). Thus, the MV at each control point for the current block is retrieved.

[0564] Next, the inter-prediction module 218 performs motion compensation of each of the plurality of sub-blocks included in the current block. In other words, the inter-prediction module 218 calculates the MV for each of the plurality of sub-blocks as an affine MV using either the two motion vectors v ₀ and v ₁ and the above expression (1A) or the three motion vectors v ₀ , v ₁ and v ₂ and the above expressions (1B) (step Sj_13). The inter-prediction module 218 then performs motion compensation of the sub-blocks using these affine MVs and the decoded reference frames (step Sj_14). When the processes in steps Sj_13 and Sj_14 are executed for each of all sub-blocks included in the current block, inter-prediction using the affine join mode for the current block is completed. In other words, motion compensation of the current block is performed in order to generate a predictive image of the current block.

[0565] It should be noted that the above-described list of possible MV options can be generated in step Sj_11, similar to step Sk_11.

[0566] MV extract > triangle mode

For example, when information parsed from the stream indicates that the triangle mode should be applied, the inter-prediction module 218 extracts the MV in the triangle mode and performs motion compensation (prediction) using the MV.

[0567] FIG. 89 is a flowchart illustrating an example of inter-prediction by triangular mode in the decoder 200.

[0568] In the triangular mode, first, the inter-prediction unit 218 splits the current block into a first segment and a second segment (step Sx_11). At this time, the inter-prediction unit 218 can obtain, from the stream, segment information, which is partition-related information, as a prediction parameter. The inter-prediction module 218 can then split the current block into a first segment and a second segment according to the segment information.

[0569] Then, first, the inter-prediction module 218 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Sx_12). In other words, the inter-prediction module 218 generates a list of possible MV options.

[0570] The inter-prediction module 218 then selects a candidate MV for the first segment and a candidate MV for the second segment as the first MV and the second MV, respectively, from the plurality of candidate MVs obtained in step Sx_11 (step Sx_13). At this time, the inter-prediction module 218 may obtain, from the stream, MV selection information to identify a selected MV candidate as a prediction parameter. The inter-prediction unit 218 can then select the first MV and the second MV according to the MV selection information.

[0571] Next, the inter prediction unit 218 generates a first prediction image by performing motion compensation using the selected first MV and the decoded reference frame (step Sx_14). Likewise, the inter-prediction module 218 generates a second prediction image by performing motion compensation using the selected second MV and the decoded reference frame (step Sx_15).

[0572] Finally, the inter-prediction unit 218 generates a prediction image for the current block by performing weighting summation of the first prediction image and the second prediction image (step Sx_16).

[0573] Motion Estimation > DMVR

For example, the information parsed from the stream indicates that the DMVR should be applied, the inter-prediction module 218 performs motion estimation using the DMVR.

[0574] FIG. 90 is a flowchart illustrating an example of motion estimation by the DMVR in the decoder 200.

[0575] The inter-prediction unit 218 extracts the MV for the current block according to the combining mode (step Sl_11). Next, the inter-prediction unit 218 extracts the final MV for the current block by searching in the area surrounding the reference frame indicated by the MV extracted at Sl_11 (step Sl_12). In other words, the MV of the current block is determined according to the DMVR.

[0576] FIG. 91 is a flowchart illustrating a specific example of motion estimation by DMVR in decoder 200.

[0577] First, in step 1, illustrated in FIG. 58A, the inter-prediction module 218 calculates the costs between the search position (also called the “starting point”) indicated by the initial MV and the eight surrounding search positions. The mutual prediction module 218 then determines whether or not the costs at each of the search positions other than the starting point are the smallest. Here, upon determining that the cost at one of the search positions other than the starting point is the smallest, the inter-prediction unit 218 changes the target to the search position at which the smallest cost is obtained and carries out the process in step 2 illustrated in FIG. 58A. When the cost at the starting point is the smallest, the inter-prediction module 218 skips the process in step 2 illustrated in FIG. 58A, and carries out the process in step 3.

[0578] In step 2, illustrated in FIG. 58A, the inter-prediction unit 218 performs a search similar to the process in step 1 regarding the search position after the target is changed as a new starting point according to the result of the process in step 1. The inter-prediction unit 218 then determines whether or not the costs are in each of the search positions other than the starting point, the smallest. Here, when determining that the cost at one of the search positions other than the starting point is the smallest, the inter-prediction unit 218 executes the process in step 4. When the cost at the starting point is the smallest, the inter-prediction unit 218 executes the process in step 3.

[0579] In step 4, the inter-prediction unit 218 considers the search position at the starting point as the final search position, and determines the difference between the position indicated by the starting MV and the final search position as a vector difference.

[0580] In step 3, illustrated in FIG. 58A, the inter-prediction module 218 determines the sub-pixel precision pixel position at which the lowest cost is obtained based on the costs at four points located at the top, bottom, left and right positions relative to the starting point in step 1 or step 2, and considers the pixel position position as the final search position. The pixel position is determined with sub-pixel precision by performing a summation weighting each of the four top, bottom, left and right vectors ((0, 1), (0,-1), (-1, 0) and (1, 0)), using, as a weighting factor, the costs corresponding to one of the four search positions. The inter-prediction unit 218 then determines the difference between the position indicated by the initial MV and the final search position as a vector difference.

[0581] Motion compensation > BIO/OBMC/LIC

For example, when information parsed from the stream indicates that correction of the prediction image should be performed when generating the prediction image, the inter-prediction unit 218 corrects the prediction image based on the mode for correction. The mode, for example, is one of BIO, OBMC and LIC described above.

[0582] FIG. 92 is a flowchart illustrating one example of generating a prediction image in the decoder 200.

[0583] The mutual prediction unit 218 generates a prediction image (step Sm_11) and corrects the prediction image according to any of the modes described above (step Sm_12).

[0584] FIG. 93 is a flowchart illustrating another example of generating a prediction image in the decoder 200.

[0585] The inter-prediction unit 218 extracts the MV for the current block (step Sn_11). Next, the mutual prediction unit 218 generates a prediction image using the MV (step Sn_12) and determines whether or not to perform the correction process (step Sn_13). For example, the inter-prediction unit 218 receives the prediction parameter included in the flow and determines whether or not to perform the correction process based on the prediction parameter. This prediction parameter, for example, is a flag indicating whether each of the above modes should be applied or not. Here, upon determining whether to perform the correction process (Yes in step Sn_13), the inter-prediction unit 218 generates the final prediction image by correcting the prediction image (step Sn_14). It should be noted that, in LIC, the brightness and color of the prediction image can be adjusted in step Sn_14. When determining whether to not perform the correction process ("No" in step Sn_13), the inter-prediction unit 218 outputs the final prediction image without correcting the prediction image (step Sn_15).

[0586] Motion compensation > OBMC

For example, when information parsed from the stream indicates that OBMC should be performed when generating a prediction image, inter-prediction module 218 adjusts the prediction image according to the OBMC.

[0587] FIG. 94 is a flowchart illustrating an example of prediction image correction by OBMC in the decoder 200. It should be noted that the flowchart in FIG. 94 indicates a prediction image correction flow using the current frame and the reference frame illustrated in FIG. 62.

[0588] First, as illustrated in FIG. 62, the mutual prediction unit 218 obtains a prediction image (Pred) by normal motion compensation using the MV assigned to the current block.

[0589] Next, the prediction image (Pred_L) is obtained by applying the motion vector (MV_L) that is already extracted for the coded block adjacent to the left of the current block to the current block (reusing the motion vector for the current block). The mutual prediction module 218 then performs a first correction of the prediction image by overlapping the two prediction images Pred and Pred_L. This provides the advantage of blending the boundary between adjacent blocks.

[0590] Similarly, the inter prediction unit 218 obtains the prediction image (Pred_U) by applying the MV (MV_U) that is already extracted for the decoded block adjacent above the current block to the current block (reusing the motion vector for the current block). The inter-prediction module 218 then performs a second correction to the prediction image by overlapping the prediction image Pred_U with the prediction images (eg, Pred and Pred_L) for which the first correction has been performed. This provides the advantage of blending the boundary between adjacent blocks. The prediction image obtained by the second correction is a prediction image in which the boundary between adjacent blocks is blended (smoothed) and therefore represents the final prediction image of the current block.

[0591] Motion compensation > BIO

For example, when information parsed from the stream indicates that a BIO should be executed when generating a prediction image, the inter-prediction module 218 adjusts the prediction image according to the BIO.

[0592] FIG. 95 is a flowchart illustrating an example of prediction image correction by BIO in the decoder 200.

[0593] As illustrated in FIG. 63, the inter-prediction unit 218 extracts two motion vectors (M ₀ , M ₁ ) using two reference frames (Ref ₀ , Ref ₁ ) different from the frame (Cur Pic) including the current block. The mutual prediction unit 218 then extracts the prediction image for the current block using two motion vectors (M ₀ , M ₁ ) (step Sy_11). It should be noted that the motion vector M0 is a motion vector (MVx ₀ , MVy ₀ ) corresponding to the reference frame Ref ₀ , and the motion vector M1 is a motion vector (MVx ₁ , MVy ₁ ) corresponding to the reference frame Ref ₁ .

[0594] Next, the inter-prediction module 218 extracts the interpolated image I ⁰ for the current block using the motion vector M0 and the reference frame L0. In addition, the inter-prediction unit 218 extracts the interpolated image I ¹ for the current block using the motion vector M1 and the reference frame L1 (step Sy_12). Here, the interpolated image I ⁰ is an image included in the reference frame Ref ₀ and is to be extracted for the current block, and the interpolated image I ¹ is an image included in the reference frame Ref ₁ and is to be extracted for the current block. Each of the interpolated image I ⁰ and the interpolated image I ¹ may have a size identical to the size of the current block. Alternatively, each of the interpolated image I ⁰ and the interpolated image I ¹ may be an image larger than the current block. In addition, the interpolated image I ⁰ and the interpolated image I ¹ may include a prediction image obtained by using motion vectors (M ₀ , M ₁ ) and reference frames (L ₀ , L ₁ ) and applying a motion compensation filter.

[0595] In addition, the inter-prediction module 218 extracts the gradient images (Ix ⁰ , Ix ¹ , Iy ⁰ , Iy ¹ ) of the current block from the interpolated image I ⁰ and the interpolated image I ¹ (step Sy_13). It should be noted that the gradient images in the horizontal direction are (Ix ⁰ , Ix ¹ ), and the gradient images in the vertical direction are (Iy ⁰ , Iy ¹ ). Mutual prediction module 218 may extract gradient images, for example, by applying a gradient filter to the interpolated images. The gradient images may be gradient images each indicating an amount of spatial variation in a pixel value along a horizontal direction or an amount of spatial variation in a pixel value along a vertical direction.

[0596] Next, inter-prediction module 218 extracts, for each sub-block of the current block, optical flow (vx, vy), which is a velocity vector, using interpolated images (I ⁰ , I ¹ ) and gradient images (Ix ⁰ , Ix ¹ , Iy ⁰ , Iy ¹ ). As one example, the subunit may be a 4x4 pixel sub-CU.

[0597] Next, the inter prediction module 218 corrects the prediction image for the current block using the optical flow (vx, vy). For example, the inter-prediction module 218 extracts a correction value for a pixel value included in the current block using optical flow (vx, vy) (step Sy_15). The mutual prediction unit 218 can then correct the prediction image for the current block using the correction value (step Sy_16). It should be noted that the correction value may be retrieved in units of a pixel, or may be retrieved in units of a plurality of pixels or in units of sub-blocks.

[0598] It should be noted that the BIO process flow is not limited to the process disclosed in FIG. 95. Only part of the processes disclosed in FIG. 95 may be executed, or another process may be added or used as a replacement, or the processes may be executed in a different processing order.

[0599] Motion compensation > LIC

For example, when information parsed from the stream indicates that LIC should be performed when generating a prediction image, the inter-prediction module 218 adjusts the prediction image according to the LIC.

[0600] FIG. 96 is a flowchart illustrating an example of prediction image correction by LIC in the decoder 200.

[0601] First, the inter prediction unit 218 obtains a reference picture corresponding to the current block from the decoded reference frame using the MV (step Sz_11).

[0602] Next, the inter-prediction unit 218 extracts, for the current block, information indicating how the luminance signal value is changed between the current frame and the reference frame (step Sz_12). This extraction is performed based on the luminance signal pixel values for the decoded left adjacent reference region (surrounding reference region) and the decoded upper adjacent reference region (surrounding reference region) and the luminance signal pixel values at corresponding positions in the reference frame indicated by the extracted MVs. The inter-prediction unit 218 calculates the luminance correction parameter using information indicating how the luminance signal value is changed (step Sz_13).

[0603] The inter-prediction unit 218 generates a prediction image for the current block by performing a luminance correction process in which a luminance correction parameter is applied to the reference image in the reference frame indicated by the MV (step Sz_14). In other words, the prediction image, which is a reference image in the reference frame indicated by the MV, is subject to correction based on the luminance correction parameter. With this correction, the brightness can be adjusted, or the color can be adjusted.

[0604] Forecast controller

The prediction controller 220 selects either an internal prediction image or a cross prediction image and outputs the selected image to the adder 208. In general, the configurations, functions, and processes of the prediction controller 220, the intra prediction module 216, and the cross prediction module 218 on the decoder 200 side may correspond to the configurations the functions and processes of the prediction controller 128, the internal prediction module 124, and the inter-prediction module 126 on the side of the encoder 100.

[0605] Description of intra block copy (IBC) mode

The inter-prediction module 126 (see FIG. 7) of the encoder 100 according to the first aspect may access a decoded pixel (i.e., a pixel that is encoded and decoded) in a frame that includes the current block (i.e., in the current frame), in addition to the previous and future frames. For example, inter-prediction module 126 may only access a pixel included in the frame. This prediction mode is called "intra-block copying (IBC)". The IBC mode can be used as one normal inter-prediction mode, and can be used, for example, with an aggregation mode, a normal mutual mode, a skip mode, and the like. Later in this document, the IBC mode is described in more detail with reference to the drawings.

[0606] FIG. 97 is a diagram for describing the IBC mode. Part (a) of FIG. 97 is a diagram illustrating one example of an area that can be accessed in the IBC mode. Part (b) of FIG. 97 is a diagram illustrating one example of a block vector (BV).

[0607] In the inter-prediction process for the current block, the inter-prediction module 126 refers to a block that is encoded and decoded and is in a frame different from the frame including the current block (ie, the current frame). However, in the prediction process using the IBC mode, the inter prediction unit 126 only accesses the pixels that are encoded and decoded and located in the frame including the current block, similar to the intra prediction process. For example, as illustrated in part (a) of FIG. 97, the pixels that can be accessed in a frame are pixels located at the top or left side of the current block. The region including such pixels is called the "linkable region".

[0608] In addition, to identify a reference block in a movie frame or frame that includes the current block, encoder 100 may determine a vector that indicates an offset of the reference block relative to the current block. This vector is called the "block vector (BV)". The following describes how BV is expressed.

[0609] As illustrated in part (b) of FIG. 97, the block vector includes, for example, an X-axis component and a Y-axis component. The X-axis component (BVx) indicates the horizontal offset of the reference block relative to the current block in the frame. The Y-axis component (BVy) indicates the vertical offset of the reference block relative to the current block in the frame. For example, when the origin point (ie, starting point) of the BV is a pixel position in the upper left corner of the current block, the end point of the BV is a pixel position in the upper left corner of the reference block. In this case, BV is a vector that connects the top left corner of the current block and the top left corner of the reference block. The encoder 100 signals BV in the encoded bitstream to enable the decoder 200, when decoding the encoded bitstream, to identify the reference block selected by the encoder 100.

[0610] HMVP mode

The HMVP mode is described in more detail next, again with reference to FIG. 42. The MV candidate list (hereinafter also referred to as a "motion vector candidate list") illustrated in the drawing may be a list of MV candidate candidates for the merge mode, or may be a list of MV candidate candidates for normal mutual regime.

[0611] In merge mode and normal reciprocal mode, the motion vector (MV) of the current block is determined by selecting one MV candidate (hereinafter also referred to as a “prediction candidate” or “motion vector candidate”) from a list of candidate MV variants formed by accessing processed blocks. For example, when generating a prediction image of the current block in normal reciprocal mode, encoder 100 selects one candidate MV from a list of candidate MVs, receives the difference between the MV of the current block and the candidate MV, and encodes the difference and the index of the candidate MV. In addition, for example, when generating a prediction image of a current block in a fusion mode, the encoder 100 selects one fusion index from a list of possible MV fusion candidates to select a candidate MV candidate and a reference frame index that are associated with the fusion index. Encoder 100 encodes the selected join index. The list of MV candidates records: for example, spatially adjacent motion vector predictors (also called “spatially adjacent MV candidates”), which are motion vectors of encoded blocks that spatially surround the current block; and temporally adjacent motion vector predictors (also referred to as “temporally adjacent MV candidates”), which represent the motion vectors of neighboring blocks that are in the encoded reference frame and onto which the position of the current block is projected. MV candidates in HMVP mode (hereinafter also referred to as “HMVP motion vector candidates”) are one example of MV candidates registered in the MV candidate list.

[0612] In HMVP mode, MV candidate lists are managed using the HMVP mode FIFO buffer (also referred to as the "HMVP table" in this document), separate from the MV candidate lists for merged mode and normal mutual mode. In the FIFO buffer, a predetermined number (for example, 5 in FIG. 42) of prediction candidates (also called "MV candidates") having MV-related information (hereinafter called "MV information") of the blocks processed in the past (that is, blocks processed before the current block) are stored in block processing order, starting with the newest one. For example, as illustrated in FIG. 42, each time encoder 100 processes one block, encoder 100 stores, in a FIFO buffer, a prediction candidate having MV information of the newest block (in other words, the block processed immediately before one block). It should be noted that each time the encoder 100 finishes processing one block, the encoder 100 may store, in a FIFO buffer, a prediction candidate having the MV information of one block.

[0613] It should be noted that the list of block vector candidates for the IBC mode is generated using BV instead of MV, and the HMVP table stores block vector candidates (BV candidates) instead of MV candidates. Hereinafter in this document, MV and BV are referred to as “vectors,” MV candidates and BV candidates are referred to as “vector candidates,” and the motion vector candidate list and the block vector candidate list are referred to as “vector candidate lists.” Optionally, MV and possible MV variants can be read as BV and possible BV variants, and BV and possible BV variants can be read as MV and possible MV variants. It should be noted that a combination of BV candidates and MV candidates may be registered in the motion vector candidate list and the block vector candidate list.

[0614] When storing a prediction candidate having new MV information in the FIFO buffer, encoder 100 removes from the FIFO buffer the prediction candidate having the MV information of the oldest block (i.e., the block processed earliest) in the FIFO -buffer. Due to this, the encoder 100 can manage the prediction options in the FIFO buffer in an up-to-date state. It should be noted that in the example illustrated in FIG. 42, HMVP1 in the FIFO buffer is a prediction candidate having the MV information of the newest block, and HMVP5 in the FIFO buffer is a prediction candidate having the MV information of the oldest block.

[0615] Next, with reference to FIG. 42, a pruning process is described in the process of registering HMVP mode MV candidates in the MV candidate lists for the merged mode and the normal mutual mode and in the FIFO buffer update process.

[0616] Cutting process

The pruning process is a process of comparing the MV to be registered, for example, in a list, and the MV registered, for example, in the list, and determining whether the MV values (or MV information) of these MVs are identical or not. In HMVP mode, there are two types of pruning processes: (1) a pruning process to register the list of possible MVs from the HMVP table (called the "first pruning process"), and (2) a pruning process to register the MVs of the immediately previous processed block (i.e. the newest MV) in the HMVP table (i.e. updating the HMVP table) (this is called the “second pruning process”).

[0617] Referring to FIG. 42, for example, in the first pruning process described in (1) above, the encoder 100 determines, for each of the candidate MVs (eg, HMVP1 to HMVP5) in the FIFO buffer, whether or not the candidate MV has MV- information different from the MV information of all possible MV variants registered in the list of possible MV variants. More specifically, encoder 100 determines, for each of the candidate MVs (HMVP1-HMVP5 in this example) in the FIFO buffer, whether or not the MV information of the candidate MV is different from the MV information of all candidate predictions (i.e., e. MV candidates) registered in the list of MV candidates, in order starting with the MV candidate having the MV information of the newest block (here, HMVP1). Upon determining that the MV information of HMVP1 is different from the MV information of all possible MV options in the list of possible MV options, the encoder 100 registers HMVP1 in the list of possible MV options. Next, encoder 100 performs an identical process for each of the remaining candidate MVs (here, HMVP2-HMVP5) in the FIFO buffer. It should be noted that the total number of HMVP motion vector candidates registered in the MV candidate list from the FIFO buffer may be 1 or may be 2 or more.

[0618] Using the HMVP mode as described above allows not only MV candidates having MV information of a block spatially or temporally adjacent to the current block to be registered in the MV candidate list, but also an MV candidate , having the MV information of a block processed in the past. This leads to more variations in the possible MV options for the merge mode and the normal reciprocal mode. Accordingly, the encoder 100 can select a more appropriate MV candidate for the current block, and thereby the encoding efficiency is improved.

[0619] Additionally, for example, in the second pruning process described in (2) above, encoder 100 compares the MV of the block immediately before being processed (HMVP0) with all possible MVs registered in the FIFO buffer (HMVP1-HMVP5 in the drawing) to determine whether or not the MV information HMVP0 is different from the MV information of all possible MV variants. Upon determining that the MV information of HMVP0 is different from the MV information of all possible MVs in the FIFO buffer, encoder 100 removes the MV of the oldest block in the FIFO buffer (HMVP5 in this example) and registers HMVP0 in the FIFO buffer as MV of the newest block. On the other hand, when determining that the MV information HMVP0 is identical to the MV information of any of the possible MV variants, the encoder 100 does not register HMVP0 in the FIFO buffer, but maintains the current FIFO buffer (i.e., the FIFO buffer in which HMVP1-HMVP5 are registered) as is.

[0620] It should be noted that the MV information may include not only the MV value, but also information regarding the reference frame, information such as the reference direction and the number of reference frames, and the like.

[0621] Although lists of possible MV options for the merge mode and the normal mutual mode are described here, the present disclosure is not limited to this example. For example, during the IBC mode prediction process, encoder 100 generates, for each current block, a list of possible BV cases in which the possible BV cases are recorded.

[0622] It should be noted that the MV candidate list and FIFO buffer illustrated in FIG. 42 are merely an example; the MV candidate list and FIFO buffer may be of a different size than the illustrated size, and may be configured such that the MV candidates are registered in an order different from the illustrated order.

[0623] Although encoder 100 is described here as an example, the above processes are common to both encoder 100 and decoder 200.

[0624] First aspect

The encoder 100 and decoder 200 according to the first aspect will be described below.

[0625] Encoder 100: when generating a prediction image of the current block, uses an IBC mode in which the encoded area of the frame that includes the current block is accessed; determines whether the size of the current block is less than or equal to a threshold value, the current block being a unit for which a list of possible vector options is generated, including possible vector options; when the size of the current block is less than or equal to the threshold value, generates a list of candidate vectors by registering the candidate HMVP vector in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein the HMVP table is stored in a first-in, first-out manner. first out (FIFO), each of the possible first vectors as a possible HMVP vector, wherein the possible first vectors have information regarding the first vector used for the processed block; when the size of the current block exceeds a threshold value, generates a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table; and encodes the current block using a list of possible vectors.

[0626] In addition, the decoder 200: when generating a prediction image of the current block, uses an IBC mode in which the coded region of the frame that includes the current block is accessed; determines whether the size of the current block is less than or equal to a threshold value, the current block being a unit for which a list of possible vector options is generated, including possible vector options; when the size of the current block is less than or equal to the threshold value, generates a list of candidate vectors by registering the candidate HMVP vector in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein the HMVP table is stored in a first-in, first-out manner. first out (FIFO), each of the possible first vectors as a possible HMVP vector, wherein the possible first vectors have information regarding the first vector used for the processed block; when the size of the current block exceeds a threshold value, generates a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table; and decodes the current block using a list of candidate vectors.

[0627] FIG. 98 is a flowchart 1000 illustrating one example of operations performed by the encoder 100 and decoder 200 according to the first aspect.

[0628] During the prediction process for the current frame, encoder 100 begins a round-robin process that is performed on a block-by-block basis (not illustrated). First, the encoder 100 determines the size of the current block in the IBC mode (S1001). The IBC mode is a prediction mode in which the processed region of the frame including the current block is accessed when generating a prediction image of the current block.

[0629] Next, the encoder 100 determines whether the size of the current block is less than or equal to the threshold value (S1002). For example, the size of the current block may be specified by the total number of pixels included in the current block. In such a case, the threshold value is, for example, 16 pixels. In addition, for example, the size of the current block may be specified by at least one of the width or height of the current block. In such a case, the threshold value is, for example, 4 pixels x 4 pixels.

[0630] Then, upon determining that the size of the current block is not less than or equal to the threshold value; in other words, when determining that the size of the current block exceeds the threshold value ("No" in S1003), the encoder 100 generates a list of candidate block vectors by performing a first pruning process and registering the candidate HMVP block vector in the list of candidate block vectors ( S1004). For example, when the encoder 100 generates a list of candidate block vectors for the IBC mode, the encoder 100 performs, in step S1004, a first pruning process for each of the candidate HMVP block vectors in order starting from the newest candidate HMVP block vector, i.e. e. starting from the HMVP block vector candidate that is last entered into the HMVP table, to the HMVP block vector candidate that is first entered into the HMVP table. More specifically, encoder 100 compares each candidate HMVP block vector with all candidate block vectors registered in the candidate vector list table of the current block in order starting with the candidate HMVP block vector that was last entered into the HMVP table, and determines whether or not a possible variant of a block HMVP vector differs from all possible variants of block vectors. For example, the vector (MV or BV) used for an adjacent block may be registered in the list of candidate block vectors. For example, when the list of block vector candidates includes a block vector candidate that is identical to the HMVP block vector candidate that was last entered into the HMVP table, encoder 100 does not add the HMVP block vector candidate to the list of block vector candidates. . On the other hand, when the list of block vector candidates does not include a block vector candidate identical to the HMVP mode block vector candidate that was last entered into the HMVP table, encoder 100 adds the HMVP mode block vector candidate to the candidate list. block vectors. An identical process is performed for other possible HMVP block vectors stored in the HMVP table. The total number of candidate HMVP block vectors added to the list of candidate block vectors can be 1, or it can be 2 or more depending on the total number of vacancies in the list.

[0631] On the other hand, when determining that the current block size is less than or equal to the threshold value ("Yes" in S1003), the encoder 100 generates a block vector candidate list by registering the HMVP block vector candidate in the block vector candidate list without performing the first clipping process (S1005). In other words, the encoder 100 in step S1005 adds the HMVP block vector candidates stored in the HMVP table to the block vector candidate list in order starting from the newest one. The total number of candidate HMVP block vectors added to the list of candidate block vectors can be 1, or it can be 2 or more depending on the number of vacancies in the list. Thus, when the current block size is less than or equal to the threshold, encoder 100 skips the pruning process, thereby reducing the total number of processing cycles required to generate the list of candidate block vectors. This improves the coding efficiency of encoder 100. Accordingly, in order to reduce the total number of processing cycles required to generate a list of block vector candidates, encoder 100 need not necessarily have one list of block vector candidates shared between the current block and the block adjacent to the current block. For example, in merge mode, encoder 100 need not necessarily have a merge list shared between the current block and a block adjacent to the current block.

[0632] Referring to FIG. 98 again, the encoder 100 encodes the current block using the block vector candidate list generated in step S1004 or step S1005 (S1006).

[0633] After repeating the processes of steps S1001-S1006 for all blocks included in the current frame, the encoder 100 ends the round-robin process on a block-by-block basis (not illustrated).

[0634] Although encoder 100 is described here as an example, the above processes are common to both encoder 100 and decoder 200.

[0635] Here, the encoder 100 determines whether or not to skip the pruning process based on whether the current block size is less than or equal to or not a threshold value when generating a list of candidate vectors in IBC mode; however, encoder 100 may also perform an identical process when generating a list of candidate vectors in the merge mode and the normal reciprocal mode. For example, when generating a list of candidate vectors in a merge mode (a list of candidate MV merges), the encoder 100 determines whether or not to skip the pruning process based on whether the size of the current block is less than or equal to or not a threshold value. In the list of possible MV combination options, the possible MV options and reference frame information (reference frame indices) are registered in association with each other. Thus, even when the candidate vectors have an identical MV value, the pruning process determines that the candidate vectors are not identical if the candidate vectors have different reference frame information.

[0636] Although an example is described here in which the encoder 100 performs a prediction process using the IBC mode, the encoder 100 may perform the prediction process using the IBC mode when determining whether to use an IBC mode from a plurality of prediction modes and perform the prediction process using the mode prediction other than the IBC mode when determining whether to use a prediction mode other than the IBC mode from among the plurality of prediction modes. For example, when using a prediction mode other than the IBC mode, encoder 100 may generate a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table and encode the current block using the candidate list vector options.

[0637] Technical advantages of the first aspect

The first aspect of the present disclosure introduces a limitation, according to the size of the current block, into the pruning process performed on candidate HMVP vectors when generating a list of candidate vectors. This allows encoder 100 and decoder 200, when generating a list of candidate vectors, to skip the pruning process to compare candidate HMVP vectors stored in the HMVP table with candidate vectors registered in the list of candidate vectors when the current block size is smaller. or equal to the threshold value. As a result, the total number of cycles required to perform the pruning process to generate a list of candidate vectors decreases when the current block size is less than or equal to the threshold value. Accordingly, the encoder 100 and decoder 200 according to the first aspect reduce the processing amount, and thereby the encoding efficiency of the encoder 100 and the processing efficiency of the decoder 200 are improved.

[0638] Although the first aspect describes an example in which the above-described size constraint is introduced into the pruning process performed when registering a candidate HMVP block vector in the list of candidate block vectors from the HMVP table when generating a list of candidate block vectors for the IBC mode ; however, the above-described size limitation may also be introduced into the generation of the list of possible MV options for the merge mode and the normal mutual mode.

[0639] Second aspect

Hereinafter, this document will describe the encoder 100, the decoder 200, the encoding method, and the decoding method according to the second aspect. While the first aspect describes the process of generating a list of possible vectors, the second aspect describes the process of updating the HMVP table.

[0640] The encoder 100 according to the second aspect further performs an update of the HMVP table using a second vector candidate having information regarding the second vector. When updating the HMVP table, encoder 100 determines whether the size of the current block is less than or equal to the threshold value, updates the HMVP table without performing a second pruning process when the size of the current block is less than or equal to the threshold value, and updates the HMVP table by executing the second pruning process when the size of the current block exceeds a threshold value.

[0641] The decoder 200 according to the second aspect further performs an update of the HMVP table using the second vector candidate having information regarding the second vector. When updating the HMVP table, decoder 200 determines whether the size of the current block is less than or equal to the threshold value, updates the HMVP table without performing a second pruning process when the size of the current block is less than or equal to the threshold value, and updates the HMVP table by executing the second pruning process when the size of the current block exceeds a threshold value.

[0642] FIG. 99 is a flowchart of a method 2000 illustrating one example of operations performed by the encoder 100 and decoder 200 according to the second aspect.

[0643] During the prediction process for the current frame, encoder 100 begins a round-robin process that is performed on a block-by-block basis (not illustrated). First, the encoder 100 determines the size of the current block in the IBC mode (S2001).

[0644] Next, encoder 100 determines a block vector (the above second vector) of the current block (not illustrated). In IBC mode, the second vector is a vector that indicates the position, in the frame, of the reference block corresponding to the current block.

[0645] Next, the encoder 100 determines whether the size of the current block is less than or equal to the threshold value (S2002). For example, the size of the current block may be specified by the total number of pixels included in the current block. In such a case, the threshold value is, for example, 16 pixels. In addition, for example, the size of the current block may be specified by at least one of the width or height of the current block. In such a case, the threshold value is, for example, 4 pixels x 4 pixels.

[0646] When determining that the size of the current block is not less than or equal to the threshold value; in other words, when determining that the size of the current block exceeds the threshold value ("No" in S2003), the encoder 100 updates the HMVP table by performing a second pruning process (S2004). At this time, the encoder 100 updates the HMVP table using a second block vector candidate having information regarding the second block vector. The second block vector is a block vector used to form a prediction image of the current block. For example, in the second pruning process, encoder 100 determines whether or not a second block vector candidate is identical to any of the first block vector candidates (i.e., HMVP block vector candidates) in the HMVP table, and when the candidate the second block vector variant is not identical to any of the possible HMVP block vector variants, updates the HMVP table by storing the second block vector candidate in the HMVP table. On the other hand, when the second block vector candidate is identical to any of the HMVP block vector candidates in the HMVP table, the encoder 100 discards the second block vector candidate. The process of updating the HMVP table by performing a second pruning process includes a process of discarding a candidate of the second block vector. It should be noted that the threshold value is identical to the threshold value described in the first aspect, and therefore description thereof will be omitted.

[0647] It should be noted that, as described in the first aspect, in the combining mode and the normal mutual mode, the motion vector candidate includes MV information having information of the MV candidates and reference frames. Thus, when the prediction process is performed for the current block in the merging mode or the normal reciprocal mode, the fact that the second motion vector candidate and the HMVP motion vector candidate are identical to each other in the second clipping process means that the MV information these possible variants of motion vectors are identical. In contrast, when the prediction process is performed for the current block in IBC mode, the fact that the second block vector candidate and the HMVP block vector candidate are identical to each other means that the vector values of these block vector candidates are identical. More specifically, in step S2004, the encoder 100 compares the block vector candidate having block vector information that indicates the position of the reference block corresponding to the current block in the frame (i.e., the second block vector candidate) with all the block vector candidates , stored in the HMVP table (i.e., with the candidate first block vectors or with the candidate HMVP block vectors), and determines whether or not the candidate second block vector is identical to any of the candidate HMVP block vectors . Encoder 100 discards a second block vector candidate when the HMVP table includes an HMVP block vector candidate having a vector value identical to the vector value of a second block vector candidate of the current block. In other words, the second block vector candidate is not added to the HMVP table. On the other hand, encoder 100 adds a second block vector candidate to the HMVP table when the HMVP table does not include a candidate HMVP block vector having a vector value identical to the vector value of the second block vector candidate. At this time, the oldest possible HMVP block vector in the HMVP table is removed.

[0648] Referring to FIG. 99 again, upon determining that the size of the current block is less than or equal to the threshold value ("Yes" in S2003), the encoder 100 updates the HMVP table without performing the second pruning process (S2005). For example, encoder 100 updates the HMVP table by storing a second block vector candidate having information regarding the second block vector in the HMVP table and deleting the oldest candidate HMVP block vector in the HMVP table. More specifically, when performing the prediction process for the current block in the IBC mode, the encoder 100 in step S2005 adds a block vector candidate having information regarding the block vector of the current block (ie, a second block vector candidate) to the HMVP table without performing the cutting process.

[0649] Next, the encoder 100 encodes the current block after performing the HMVP table update process illustrated in step S2004 and step S2005 (step S2006). It should be noted that the HMVP table update process can be performed after the current block has been encoded.

[0650] After repeating the processes of steps S2001-S2006 for all blocks included in the current frame, the encoder 100 ends the round-robin process on a block-by-block basis (not illustrated).

[0651] Although encoder 100 is described here as an example, the above processes are common to both encoder 100 and decoder 200.

[0652] Although the IBC mode is described here as an example, even in the union mode and the normal mutual mode (for example, in the AMVP mode), the HMVP table is used to generate a list of possible MV options according to whether the size is less than, equal to, or not the current block to the threshold size, as described in the first aspect. Therefore, even in federated mode and AMVP mode, the above HMVP table update process can be performed identically to IBC mode.

[0653] Technical advantages of the second aspect

The second aspect of the present disclosure introduces a limit, according to the size of the current block, into the pruning process performed when updating the HMVP table. This allows encoder 100 and decoder 200, when updating the HMVP table, to skip the process of pruning the comparison of the second vector used to predict the current block with all possible HMVP vectors stored in the HMVP table when the size of the current block is less than or equal to a threshold value . As a result, the total number of cycles required to perform the pruning process when updating the HMVP table decreases when the current block size is less than or equal to the threshold value. Accordingly, the encoder 100 and decoder 200 according to the second aspect reduce the processing amount, and thereby the encoding efficiency of the encoder 100 and the processing efficiency of the decoder 200 are improved.

[0654] Although the second aspect describes an example of introducing the above-described size constraint into the pruning process performed when updating the HMVP table using the block vector of the current block predicted in IBC mode, the above-described size constraint can also be introduced in the union mode and normal mutual mode .

[0655] Combination with other aspects

One or more aspects disclosed herein may be combined with at least a portion of other aspects of the present disclosure. In addition, portions of the processes illustrated in the flowcharts, portions of device configurations, portions of syntax and/or other functions according to one or more aspects disclosed herein may be combined with other aspects.

[0656] All processes/elements described above are not necessarily required. The device/method may include part of the processes/elements. The processes described above can be performed by a decoder, similar to an encoder.

[0657] Additional notes

In conventional inter-prediction, encoder 100 refers to a block included in a frame different from the current frame; whereas in IBC mode, the encoder 100 refers to the processed block included in the processed area in the frame including the current block, i.e. in the current frame. At this time, the area being accessed (also called “referenceable area”) is an area that is located near the current block and is being processed relatively recently, and thereby storing the referenceable area in the storage device is a quickly accessible area in the hardware engine , and not in a storage device outside the hardware mechanism, is a sign of IBC mode. As a result, during the IBC mode prediction process, the encoder 100 does not necessarily have to access the storage device outside the hardware engine; in other words, storage bandwidth becomes optional. Therefore, the encoder 100 accesses the memory only in the hardware engine and can therefore perform the prediction process at a relatively high speed. Accordingly, in IBC mode, the size of the current block can take a smaller value than in other mutual prediction modes. However, when the current block size is small, the number of times the vector candidate list generation process is executed may increase, and the amount of processing for generating the vector candidate list may become larger than in other inter-prediction modes. In addition, the number of times the HMVP table update process is performed may also increase, and the processing volume may become larger than in other inter-prediction modes. In view of the above, in the case of performing the prediction process in IBC mode, it is possible to reduce the amount of processing by skipping the pruning process when generating a list of candidate vectors and updating the HMVP table similar to the present embodiment. This may resolve the bottleneck of the inter-prediction module 126.

[0658] It should be noted that the technology according to the present embodiment is not limited to the IBC mode and can also be used in other inter-prediction modes in an identical manner. Processes identical to those described in the above aspects may be performed in all inter-prediction modes used, or may be performed in one or more prediction modes of all inter-prediction modes used. For example, identical processes may be executed, from all inter-prediction modes, in a mode in which the current block size may be equal to a threshold value (eg, 16 pixels, ie = 4 pixels x 4 pixels).

[0659] Typical example of processes

The following describes a representative example of processes performed by the above-described encoder 100 and decoder 200.

[0660] FIG. 100 is a flowchart illustrating operations performed by the encoder 100. For example, the encoder 100 includes circuitry and a memory device coupled to the circuitry. The circuit and memory included in the encoder 100 may correspond to the processor a1 and memory a2 illustrated in FIG. 8. The encoder circuit 100, when in operation, does the following. When generating a predictive image of the current block, the encoder circuit 100 uses an IBC mode in which the processed region of the frame including the current block is accessed.

[0661] For example, the encoder circuit 100 determines whether the current block size is less than or equal to a threshold value (S3001). When the size of the current block is less than or equal to the threshold value ("Yes" in S3001), the circuit generates a candidate vector list by registering a candidate HMVP vector in the list of candidate vectors from the HMVP table without performing the first pruning process (S3002), and when the size of the current block exceeds the threshold value ("No" in S3001), the circuit generates a list of candidate vectors by performing a first pruning process and registers the candidate HMVP vector in the list of candidate vectors from the HMVP table (S3003) and encodes the current block using list of possible vector options (S3004).

[0662] In the HMVP table, candidate first vectors having information regarding the first vector used for a processed block are stored as candidate HMVP vectors by a FIFO method, and a list of candidate vectors is generated for each current block.

[0663] Accordingly, the encoder 100 generates a list of candidate vectors without performing a first pruning process (hereinafter referred to as a “pruning process”) when the size of the current block is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the coding efficiency of the encoder 100. In addition, since the encoder 100 uses the IBC mode when generating a prediction image of the current block, it is possible to identify a reference block that is encoded and decoded and located in a frame that includes the current block based on the motion vector. indicating a reference block, and reads the identified reference block. Because of this, the encoder 100 can identify, from the encoded blocks in a frame that includes the current block, a block to be accessed for the current block based on a vector indicating a position corresponding to the current block, and read the identified block.

[0664] For example, encoder circuit 100 may generate a predictive image of the current block using a second vector, and encode the second vector using one of the candidate vectors included in the list of candidate vectors. In the first pruning process performed in generating the list of candidate vectors, the encoder circuit 100 may determine, for each of the candidate HMVP vectors stored in the HMVP table, whether or not the candidate HMVP vector is identical to any of one or more possible vectors registered in the list of possible vectors. When a candidate HMVP vector is not identical to any of one or more candidate vectors, the encoder circuit 100 may generate a list of candidate vectors by registering the candidate HMVP vector in the list of candidate vectors.

[0665] Accordingly, the encoder 100 generates a list of candidate vectors without performing a pruning process when the current block size is less than or equal to the threshold value, and therefore the amount of processing is reduced. This improves the coding efficiency of the encoder 100.

[0666] For example, the encoder circuit 100 may further perform an update of the HMVP table using a second vector candidate having information regarding the second vector. When updating the HMVP table, the encoder circuitry 100 may determine whether the current block size is less than, equal to, or not a threshold value. When the current block size is less than or equal to the threshold, the encoder circuit 100 may update the HMVP table without performing a second pruning process. On the other hand, when the size of the current block exceeds a threshold value, the encoder circuit 100 may update the HMVP table by performing a second pruning process.

[0667] Accordingly, the encoder 100 updates the HMVP table without performing a second pruning process (hereinafter also referred to as a “pruning process”) when the size of the current block is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the coding efficiency of the encoder 100.

[0668] For example, the circuitry of encoder 100 may determine, in a second pruning process performed when updating the HMVP table, whether or not a candidate candidate of the second vector is identical to any of the candidate candidate HMVP vectors. When the second vector candidate is not identical to any of the HMVP vector candidates, the encoder circuit 100 may update the HMVP table by storing the second vector candidate in the HMVP table.

[0669] Due to this, the encoder 100 can register, in the list of candidate vectors, a candidate vector that is more suitable for the current block.

[0670] For example, the size of the current block may be specified by the total number of pixels included in the current block. For example, the threshold could be 16 pixels.

[0671] Accordingly, the encoder 100 generates a list of candidate vectors and updates the HMVP table without performing a pruning process when the total number of pixels included in the current block is less than or equal to a threshold value (for example, less than or equal to 16 pixels), and due to This reduces the processing volume. This improves the coding efficiency of the encoder 100.

[0672] For example, the size of the current block may be specified by at least one of the width or height of the current block. For example, the threshold could be 4 pixels x 4 pixels.

[0673] Accordingly, encoder 100 generates a list of candidate vectors and updates the HMVP table without performing a pruning process when at least one of the width or height of the current block is less than or equal to a threshold value (for example, when the width and height of the current block is less than or equal to 4 pixels), and due to this, the amount of processing is reduced. This improves the coding efficiency of the encoder 100.

[0674] For example, the list of candidate vectors need not be shared between the current block and a block adjacent to the current block.

[0675] Accordingly, encoder 100 uses lists of candidate vectors, each generated for a different current block, and thereby improves prediction accuracy.

[0676] For example, when generating a prediction image of a current block, the encoder circuit 100 may use an IBC mode when determining whether to use an IBC mode from a plurality of prediction modes, and use a prediction mode different from the IBC mode when determining whether to use a prediction mode different from from IBC mode, from many forecasting modes. When using a prediction mode other than the IBC mode, the encoder circuit 100 may generate a list of candidate vectors by performing a first pruning process and registering a candidate HMVP vector in the list of candidate vectors from the HMVP table, and encode the current block using the candidate list. vectors.

[0677] This allows the encoder 100 to appropriately switch the conditions under which the clipping process is performed according to the prediction mode being used. This improves the coding efficiency of the encoder 100.

[0678] FIG. 101 is a flowchart illustrating operations performed by the decoder 200. For example, the decoder 200 includes circuitry and a memory device coupled to the circuitry. The circuit and memory included in the decoder 200 may correspond to the processor b1 and memory b2 illustrated in FIG. 68. The decoder circuit 200, in operation, does the following. When generating a predictive image of the current block, the decoder circuit 200 uses an IBC mode in which the processed region of the frame including the current block is accessed.

[0679] For example, the decoder circuit 200 determines whether the current block size is less than or equal to a threshold value (S4001). When the current block size is less than or equal to the threshold value ("Yes" in S4001), the decoder circuit 200 generates a vector candidate list by registering the HMVP vector candidate in the vector candidate list from the HMVP table without performing the first pruning process (S4002). On the other hand, when the size of the current block exceeds the threshold value ("No" in S4001), the decoder circuit 200 generates a list of candidate vectors by performing a first pruning process and registering the candidate HMVP vector in the list of candidate vectors from the HMVP table (S4003 ). Next, the decoder circuit 200 decodes the current block using the candidate vector list generated in step S4002 or S4003 (S4004).

[0680] Accordingly, the decoder 200 generates a list of candidate vectors without performing a pruning process when the current block size is less than or equal to the threshold value, and therefore the amount of processing is reduced. This improves the processing efficiency of the decoder 200. In addition, since the decoder 200 uses the IBC mode when generating a prediction image of the current block, it is possible to identify, from the decoded blocks in the frame that includes the current block, the block to be accessed for the current block on based on a vector indicating the position corresponding to the current block, and read the identified block. Because of this, the decoder 200 can identify, from the decoded blocks in a frame that includes the current block, a block to be accessed for the current block based on a vector indicating a position corresponding to the current block, and read the identified block.

[0681] For example, decoder circuit 200 may generate a predictive image of the current block using a second vector and decode the second vector using one of the candidate vectors included in the list of candidate vectors. In the first pruning process performed in generating the list of candidate vectors, the decoder circuit 200 may determine, for each of the candidate HMVP vectors stored in the HMVP table, whether or not the candidate HMVP vector is identical to any of one or more possible vectors registered in the list of possible vectors. When a candidate HMVP vector is not identical to any of one or more candidate vectors, the decoder circuit 200 may generate a list of candidate vectors by registering the candidate HMVP vector in the list of candidate vectors.

[0682] Accordingly, the decoder 200 generates a list of candidate vectors without performing a pruning process when the current block size is less than or equal to the threshold value, and therefore the amount of processing is reduced. This improves the processing efficiency of the decoder 200.

[0683] For example, the decoder circuit 200 may further perform an update of the HMVP table using a second vector candidate having information regarding the second vector. When updating the HMVP table, the decoder circuitry 200 may determine whether the current block size is less than, equal to, or not a threshold value. When the current block size is less than or equal to the threshold, the decoder circuit 200 may update the HMVP table without performing a second pruning process. On the other hand, when the size of the current block exceeds a threshold value, the decoder circuit 200 may update the HMVP table by performing a second pruning process.

[0684] Accordingly, the decoder 200 updates the HMVP table without performing a pruning process when the current block size is less than or equal to the threshold value, and thereby the amount of processing is reduced. This improves the processing efficiency of the decoder 200.

[0685] For example, decoder circuitry 200 may determine, in a second pruning process performed when updating the HMVP table, whether or not a candidate second vector is identical to any of the candidate HMVP vectors. When the second vector candidate is not identical to any of the HMVP vector candidates, the decoder circuit 200 may update the HMVP table by storing the second vector candidate in the HMVP table.

[0686] Due to this, the decoder 200 can register, in the list of candidate vectors, a candidate vector that is more suitable for the current block.

[0687] For example, the size of the current block may be specified by the total number of pixels included in the current block. For example, the threshold could be 16 pixels.

[0688] Accordingly, the decoder 200 generates a list of candidate vectors and updates the HMVP table without performing a pruning process when the total number of pixels included in the current block is less than or equal to a threshold value (for example, less than or equal to 16 pixels), and due to This reduces the processing volume. This improves the processing efficiency of the decoder 200.

[0689] For example, the size of the current block may be specified by at least one of the width or height of the current block. For example, the threshold could be 4 pixels x 4 pixels.

[0690] Accordingly, decoder 200 generates a list of candidate vectors and updates the HMVP table without performing a pruning process when at least one of the width or height of the current block is less than or equal to a threshold value (for example, when the width and height of the current block is less than or equal to 4 pixels), and due to this, the amount of processing is reduced. This improves the processing efficiency of the decoder 200.

[0691] For example, the list of candidate vectors need not be shared between the current block and a block adjacent to the current block.

[0692] Accordingly, decoder 200 uses lists of candidate vectors, each generated for a different current block, and thereby improves prediction accuracy.

[0693] For example, when generating a prediction image of a current block, the decoder circuit 200 may use an IBC mode when determining whether to use an IBC mode from a plurality of prediction modes, and use a prediction mode different from the IBC mode when determining whether to use a prediction mode different from from IBC mode, from many forecasting modes. When using a prediction mode other than the IBC mode, the decoder circuit 200 may generate a candidate vector list by performing a first pruning process and registering a candidate HMVP vector in the list of candidate vectors from the HMVP table, and decode the current block using the candidate list. vectors.

[0694] This allows the decoder 200 to appropriately switch the conditions under which the clipping process is performed according to the prediction mode being used. This improves the processing efficiency of the decoder 200.

[0695] Each element may be a circuit, as described above. These circuits may work together to form a single circuit, or they may be separate circuits. Each element may be implemented using a general purpose processor or may be implemented using a specialized processor.

[0696] A process performed by a particular element may be performed by another element. In addition, the order in which processes are executed may vary, or multiple processes may be executed in parallel. The encoder decoding device may include an encoder 100 and a decoder 200.

[0697] Although aspects of the encoder 100 and decoder 200 are described above based on more than one example, aspects of the encoder 100 and decoder 200 are not limited to such examples. Aspects of the encoder 100 and decoder 200 may also include various modifications of each example that are possible for those skilled in the art, and embodiments implemented by combining elements in the various examples, so long as such modifications and embodiments do not depart from the spirit true revelation of the essence.

[0698] The present aspect may be accomplished by combining one or more aspects disclosed herein with at least a portion of other aspects according to the present disclosure. In addition, the present aspect may be performed by combining, with other aspects, portions of the processes indicated in any of the flowcharts of the aspects, portions of the configuration of any of the devices, portions of syntaxes, etc.

[0699] Implementations and Applications

As described in each of the above-described embodiments, each functional or operating unit can typically be implemented, for example, as an MPU (microprocessor) and a memory device. In addition, processes performed by each of the functional blocks may be implemented as a program execution unit, such as a processor, that reads and executes software (program) recorded on a medium such as ROM. Software can be distributed. The software may be recorded on a variety of media, such as a semiconductor storage device. It should be noted that each functional block can also be implemented as hardware (specialized circuitry).

[0700] In addition, the processing described in each of the embodiments may be implemented through integrated processing using a single piece of equipment (system) and, alternatively, may be implemented through decentralized processing using multiple pieces of equipment. Moreover, the processor that executes the above-described program may be a single processor or a plurality of processors. In other words, integrated processing may be performed, and alternatively, decentralized processing may be performed.

[0701] Embodiments of the present disclosure are not limited to the above exemplary embodiments; Various modifications may be made to the exemplary embodiments, the effects of which are also included within the scope of the embodiments of the present disclosure.

[0702] The following describes application examples of the moving frame encoding method (image encoding method) and the moving frame decoding method (image decoding method) described in each of the above embodiments, as well as various systems that implement the example applications. Such a system may be characterized as including an image encoder that uses an image encoding method, an image decoder that uses an image decoding method, and an image codec that includes both an image encoder and an image decoder. Other configurations of such a system can be modified for each individual case.

[0703] Examples of use

Fig. 102 illustrates a general configuration of a content delivery system ex100 suitable for implementing a content distribution service. The area in which the communication service is provided is divided into cells of required sizes, and base stations ex106, ex107, ex108, ex109 and ex110, which are fixed wireless stations in the illustrated example, are located in the corresponding cells.

[0704] In the content delivery system ex100, devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 through an Internet service provider ex102 or a communications network ex104 and base stations ex106- ex110. The content delivery system ex100 can combine and connect any of the above devices. In various implementations, devices may directly or indirectly communicate with each other through the telephone network or short-range communications rather than through the ex106-ex110 base stations. Additionally, the streaming server ex103 can connect with devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115, for example, via the Internet ex101. The ex103 streaming server can also connect, for example, to a terminal on a public hotspot on an ex117 aircraft via an ex116 satellite.

[0705] It should be noted that instead of base stations ex106-ex110, wireless access points or public access points may be used. The streaming server ex103 can connect to the communications network ex104 directly rather than via the Internet ex101 or Internet service provider ex102, and can connect to the aircraft ex117 directly rather than via a satellite ex116.

[0706] The camera ex113 is a device capable of capturing still images and video, such as a digital camera. The ex115 smartphone is a smartphone, cellular phone or personal mobile phone system (PHS) phone that can operate according to mobile communication system standards for 2G, 3G, 3.9G and 4G systems, as well as the following 5G system generations.

[0707] Home appliance ex114, for example, is a refrigerator or device included in a home fuel cell cogeneration system.

[0708] In the content delivery system ex100, a terminal including an image and/or video capture function allows, for example, live streaming through a connection to a streaming server ex103, for example, through a base station ex106. When live streaming, a terminal (eg, a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or an airplane terminal ex117) may perform the encoding processing described in the above embodiments for still image content or video content captured by the user through the terminal, can multiplex video data obtained through encoding and audio data obtained through audio encoding corresponding to the video, and can transmit the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to one aspect of the present disclosure.

[0709] Streaming server ex103 streams transmitted content data to clients that request the stream. Examples of the client include a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and airplane terminals ex117 that are capable of decoding the above-described encoded data. Devices that receive streaming data decode and play back the received data. In other words, the devices may function as an image decoder in accordance with one aspect of the present disclosure.

[0710] Decentralized processing

The ex103 streaming server can be implemented as a plurality of servers or computers that share tasks such as processing, recording, and streaming data. For example, the ex103 streaming server may be implemented as a content delivery network (CDN) that streams content over a network connecting multiple edge servers located around the world. In a CDN, an edge server physically near the client is dynamically assigned to the client. Content is cached and streamed to the edge server to reduce loading times. In the event of, for example, a certain type of error or connection change, such as due to a spike in traffic, data can be streamed reliably at high speeds because affected parts of the network can be avoided, for example by dividing processing among multiple edge servers or switching streaming loads transfer to another edge server and continue streaming.

[0711] Decentralization is not limited to simply sharing processing for streaming; Encoding of the captured data can be shared between and performed via terminals, on the server side, or both. In one example, in typical encoding, processing is performed in two loops. The first loop serves to detect how complex the image is on a frame-by-frame or scene-by-scene basis, or to detect encoding load. The second circuit provides processing that preserves image quality and improves encoding efficiency. For example, it is possible to reduce the processing load of the terminals and improve the quality and efficiency of content encoding by causing the terminals to perform a first encoding loop and causing the server side that receives the content to perform a second encoding loop. In such a case, when a decoding request is received, the encoded data resulting from the first loop performed by one terminal can be received and reproduced at the other terminal in approximately real time. This allows for smooth, real-time streaming.

[0712] In another example, camera ex113, etc. extracts the feature count from the image, compresses the data associated with the feature count as metadata, and transmits the compressed metadata to the server. For example, the server determines the saliency of an object based on the number of features and changes the quantization precision accordingly to perform compression appropriate for the semantic meaning (or content salience) of the image. These feature counts are particularly effective in improving the accuracy and efficiency of predicting motion vectors during a second pass of compression performed by the server. In addition, coding that has a relatively low processing load, such as variable length coding (VLC), can be processed by the terminal, and coding that has a relatively high processing load, such as context-adaptive binary arithmetic coding (CABAC), can be processed via the server.

[0713] In yet another example, cases arise in which multiple videos of an approximately identical scene are captured through multiple terminals, for example, in a stadium, a shopping mall, or a factory. In such a case, for example, encoding can be decentralized by dividing processing tasks among multiple terminals that capture video and, if necessary, other terminals that do not capture video, and a server on a per-unit basis. The units, for example, can be groups of frames (GOPs), frames, or tiles resulting from dividing a frame. This allows you to reduce loading times and achieve streaming that is closer to real time.

[0714] Since the videos have approximately the same scene, control and/or instructions can be performed by the server such that videos captured through the terminals can be cross-referenced. In addition, the server may receive encoded data from the terminals, change the reference relationship between data elements, or adjust or replace the frames themselves, and then perform encoding. This allows you to generate a stream with increased quality and efficiency for individual data elements.

[0715] In addition, the server may stream video data after performing transcoding to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (eg, VP9) and can convert H.264 to H.265.

[0716] Thus, encoding may be performed by a terminal or one or more servers. Accordingly, although the device that performs the encoding is referred to as a “server” or “terminal” in the following description, some or all of the processes performed by the server may be performed by the terminal, and similarly, some or all of the processes performed by the terminal may be performed by the server . This also applies to decoding processes.

[0717] 3D Multi-View Mode

There is an increase in the use of images or videos that are combined from images or videos of different scenes captured in parallel, or an identical scene captured from different angles through multiple terminals, such as the ex113 camera and/or the ex115 smartphone. Video captured by the terminals is combined, for example, based on a separately obtained relative positional relationship between terminals or regions in the video having matching feature points.

[0718] In addition to encoding two-dimensional moving frames, the server may encode the still image based on scene analysis of the moving frame, either automatically or at a time specified by the user, and transmit the encoded still image to the receiving terminal. In addition, when the server can obtain the relative positional relationship between video capture terminals, in addition to two-dimensional moving frames, the server can generate three-dimensional scene geometry based on video of an identical scene captured from different angles. The server may separately encode 3D data generated from, for example, a point cloud, and, based on the result of recognizing or tracking a person or object using the 3D data, may select or reconstruct and generate videos to be transmitted to the receiving terminal from the videos captured through multiple terminals.

[0719] This allows the user to enjoy the scene by freely selecting videos corresponding to the video capture terminals, and allows the user to enjoy the content obtained by extracting video at a selected viewpoint from 3D data recovered from a plurality of images or video, video. Also, similar to video, audio can be recorded from relatively different angles, and the server can multiplex the audio from a particular angle or space with the corresponding video and transmit the multiplexed video and audio.

[0720] In recent years, content that is a composite of the real world and the virtual world, such as virtual reality (VR) and augmented reality (AR) style content, has also become popular. In the case of VR images, the server may create images from the left and right eye viewpoints and perform encoding that allows a link between the two viewpoint images, such as multi-view encoding (MVC), and alternatively, may encode the images as separate streams without a link . When images are decoded as separate streams, the streams can be synchronized during playback in such a way as to recreate the virtual 3D space according to the user's point of view.

[0721] In the case of AR images, the server overlays virtual object information existing in virtual space with camera information representing real world space based on three-dimensional position or movement from the user's perspective. The decoder can receive or store virtual object information and 3D data, generate 2D images based on movement from the user's point of view, and then generate overlay data by transparently connecting the images. Alternatively, the decoder may transmit, to the server, motion from the user's point of view in addition to a request for virtual object information. The server can generate overlay data based on three-dimensional data stored in the server according to the received motion, and encode and stream the generated overlay data to the decoder. It should be noted that the overlay data includes, in addition to RGB values, an α value indicating transparency, and the server sets the α value for sections other than the object formed from the 3D data, for example, to 0, and may encode to the time when these sections are transparent. Alternatively, the server may set the background to a specific RGB value, such as a chroma key, and generate data in which areas other than the object are set as the background.

[0722] Decoding of similarly streamed data may be performed by the client (ie, terminals), on the server side, or shared between them. In one example, one terminal may transmit a receive request to a server, the requested content may be received and decoded by the other terminal, and the decoded signal may be transmitted to a device having a display. It is possible to reproduce data with high image quality through decentralization of processing and proper selection of content regardless of the processing ability of the communication terminal itself. In yet another example, while a television set, for example, receives image data that is large in size, an area of a frame such as a tile obtained by dividing the frame can be decoded and displayed on the personal terminal or terminals of the viewer or viewers of the television set. This allows viewers to share a view with large frames, as well as each viewer to check their assigned area or explore an area in more detail with a close-up view.

[0723] In situations in which multiple wireless connections are possible over short, medium and long distances, indoors or outdoors, it may be possible to transparently receive content using a streaming system standard such as MPEG DASH. The user can switch between real-time data by freely selecting a decoder or display equipment, including a user terminal, indoor or outdoor displays, etc. In addition, for example, using information regarding the user's position, decoding can be performed when switching which terminal processes decoding and which terminal processes content display. This allows information to be converted and displayed while the user is on the move on a route to a destination, on the wall of a nearby building into which a device capable of displaying the content is embedded, or on a piece of ground. In addition, it is also possible to switch the bit rate of the received data based on the reachability of the encoded data over the network, for example, when the encoded data is cached on a server readily accessible from the receiving terminal, or when the encoded data is copied to an edge server in a content delivery service.

[0724] Web page optimization

Fig. 103, for example, illustrates an example of a web page display screen on computer ex111. Fig. 104, for example, illustrates an example of a web page display screen on a smartphone ex115. As illustrated in FIG. 103 and fig. 104, a web page may include a plurality of image links that are links to image content, and the appearance of the web page differs depending on the device used to view the web page. When multiple image links are viewable on the screen until the user explicitly selects the image link, either until the image link is approximated in the center of the screen, or the entire image link fits into the screen, display hardware (decoder) may display, as image links, still images included in the content, or I-frames, may display video, such as an animated GIF using multiple still images or I-frames, for example, or may receive only base layer and decode and display the video.

[0725] When an image reference is selected by the user, the display equipment performs decoding when providing the highest priority to the base layer. It should be noted that if there is information in the HTML code of the web page indicating that the content is scalable, the display equipment can decode down to the enhancement layer. Additionally, to ensure real-time playback, before a selection is made, or when bandwidth is severely limited, the display equipment may reduce the delay between the time at which the leading frame is decoded and the time at which the decoded frame is displayed (i.e., e. the delay from the start of decoding the content until the content is displayed) by decoding and displaying only direct reference frames (I-frame, P-frame, forward reference B-frame). Still further, the display equipment may deliberately ignore the reference relationship between frames and approximately decode all B and P frames as direct reference frames and then perform normal decoding as the number of frames received in time increases.

[0726] Autonomous driving

When transmitting and receiving still image or video data such as two- or three-dimensional map information for autonomous or assisted driving, the receiving terminal may receive, in addition to image data belonging to one or more layers, information such as regarding weather or roadworks as metadata, and associate the metadata with image data when decoding. It should be noted that the metadata may be assigned on a per-layer basis and, alternatively, may simply be multiplexed with the image data.

[0727] In such a case, since the car, unmanned vehicle, aircraft, etc. containing the receiving terminal is mobile, the receiving terminal can transparently receive and perform decoding when switching between base stations among the base stations ex106 to ex110 by transmitting information , indicating the position of the receiving terminal. In addition, according to the selection made by the user, the user's situation and/or the connection bandwidth, the receiving terminal can dynamically select the extent to which metadata is received or the extent to which, for example, map information is updated.

[0728] In the content delivery system ex100, the client can receive, decode and reproduce, in real time, encoded information transmitted by the user.

[0729] Streaming content from people

In the content delivery system ex100, in addition to long-lasting content with high image quality distributed by a video distribution entity, unicast or multicast streaming of short content with low image quality from a person is also possible. The popularity of such content from people is likely to increase even more. The server may first perform editing processing on the content before encoding processing to refine content from people. This can be achieved, for example, using the following configuration.

[0730] In real time, when video or image content is captured or after the content is captured and accumulated, the server performs recognition processing based on the raw data or encoded data, such as capture error processing, scene retrieval processing, semantic meaning analysis, and /or object detection processing. Thereafter, based on the result of the recognition processing, the server (either directed or automatically) edits the content, examples of which include: correction such as correction of focus and/or motion blur; Removing scenes with low priority, such as scenes that have low brightness compared to other frames or are out of focus; adjusting the edges of objects; and regulation of color tones. The server encodes the edited data based on the result of the edit. It is known that overly long videos tend to have fewer views. Accordingly, in order to keep content within a specific duration that scales with the duration of the source video, the server can, in addition to the low priority scenes described above, automatically cut scenes with minor movement based on the result of image processing. Alternatively, the server can generate and encode a video digest based on the result of the analysis of the semantic meaning of the scene.

[0731] There may be cases in which content from people may include content that violates copyright, moral law, book rights, etc. Such a case may lead to an unfavorable situation for the creator, for example, when the content is shared beyond the scope intended by the creator. Accordingly, before encoding, the server, for example, can edit the images so as to, for example, blur the faces of people on the periphery of the screen or blur the inside of a house. Additionally, the server may be configured to recognize faces of people other than the registered person in images to be encoded, and when such faces appear in the image, may apply a mosaic filter to, for example, the person's face. Alternatively, as pre-processing or post-processing for encoding, the user may specify, for copyright reasons, an image area including a person or a background area to be processed. The server may process the specified area, for example, by replacing the area with another image or blurring the area. If the area includes a person, the person may be tracked in a moving frame, and the user's head region may be replaced with another image as the person moves.

[0732] Since there is a demand for real-time viewing of human-generated content, which often has a small data size, the decoder first takes the base layer as the highest priority and performs decoding and playback, although this may differ depending on the bandwidth. When the content is played back two or more times, for example, when the decoder receives the enhancement layer during decoding and playback of the base layer and loops playback, the decoder can play back the high image quality video including the enhancement layer. If a stream is encoded using such scalable encoding, the video may be low quality in the unselected state or at the beginning of the video, but it may offer an experience in which the quality of the images of the stream is gradually increased in an intelligent manner. This is not limited to just scalable coding; An identical experience can be offered by configuring one stream of the low quality stream played back the first time and a second stream encoded using the first stream as a reference.

[0733] Other implementation examples and applications

Encoding and decoding may be performed by an LSI ex500 (Large Integrated Circuit) (see FIG. 102), which is typically included in each terminal. The LSI ex500 can be configured with a single chip or multiple chips. Software for encoding and decoding moving frames may be integrated into some type of media (such as a CD-ROM, floppy disk, or hard disk) that is readable, for example, by a computer ex111, and encoding and decoding may be performed using the software. In addition, when the ex114 smartphone is equipped with a camera, video data captured by the camera can be transmitted. In this case, video data is encoded using the LSI ex500 included in the ex115 smartphone.

[0734] It should be noted that the LSI ex500 may be configured to download and activate an application. In such a case, the terminal first determines whether or not it is compatible with the scheme used to encode the content, or whether or not it is capable of performing a particular service. When the terminal is not compatible with the content encoding scheme, or when the terminal does not allow execution of a particular service, the terminal first downloads the codec or application software, and then receives and plays the content.

[0735] In addition to the example of a content provision system ex100 that uses the Internet ex101, at least a moving frame encoder (picture encoder) or a moving frame decoder (picture decoder) described in the above embodiments may be implemented in a digital broadcast system. Identical encoding processing and decoding processing can be used to transmit and receive broadcast radio waves overlaid with multiplexed audio and video data using, for example, a satellite, even though it accommodates multicast transmission, while unicast transmission is simpler with the ex100 system providing content.

[0736] Hardware configuration

Fig. 105 illustrates additional information regarding the smartphone ex115 shown in FIG. 102. FIG. 106 illustrates an ex115 smartphone configuration example. The ex115 smartphone includes an ex450 antenna for transmitting and receiving radio waves to/from the ex110 base station, an ex465 camera that allows video and still image capture, and an ex458 display that displays decoded data such as video captured through the ex465 camera and videos received via antenna ex450. The ex115 smartphone further includes a user interface ex466 such as a touch panel, an audio output module ex457 such as a speaker for outputting speech or other audio, an audio input module ex456 such as a microphone for audio input, a storage device ex467 capable of storing decoded data such as captured video or still images, recorded audio, received video or still images and mail messages, as well as decoded data, and the ex464 socket, which is an interface for the ex468 SIM card to authorize access to the network and various data. It should be noted that an external storage device can be used instead of the ex467 storage device.

[0737] Main controller ex460, which comprehensively controls the display ex458 and user interface ex466, power supply circuit ex461, user interface input controller ex462, video signal processor ex455, camera interface ex463, display controller ex459, modulator/demodulator ex452, multiplexer/demultiplexer ex453 , the ex454 audio processor, ex464 socket and ex467 memory are connected via the ex470 bus.

[0738] When the user turns on the power button of the power supply circuit ex461, the smartphone ex115 is turned on to an operational state and each component is supplied with power from the battery pack.

[0739] The ex115 smartphone performs processing, for example, for calls and data transfers, based on control performed by the ex460 main controller, which includes a CPU, ROM and RAM. When calls are made, the audio signal recorded by the audio input module ex456 is converted to a digital audio signal by the audio signal processor ex454, which is applied with spread spectrum processing by the ex452 modulator/demodulator and applied with digital-to-analog conversion and frequency conversion processing by the ex451 transceiver. and the resulting signal is transmitted through the ex450 antenna. The received data is amplified, frequency-converted and A/D converted, despread spectrum processed by the ex452 modulator/demodulator, converted to an analog audio signal by the ex454 audio processor, and then output from the ex457 audio output module. In the data transmission mode, text, still image data or video data are transmitted by the main body controller ex460 through the input controller ex462 from the user interface based on the operation of the main body user interface ex466. Similar transmission and reception processing is performed. In the data transfer mode, when sending video, still image, or video and audio, the video signal processor ex455 compressively encodes, through the moving frame encoding method described in the above embodiments, the video signal stored in the storage device ex467 or the video signal input from the camera ex465 , and transmits the encoded video data to the ex453 multiplexer/demultiplexer. The ex454 audio processor encodes the audio signal recorded by the ex456 audio input module, while the ex465 camera captures video or still image, and transmits the encoded audio data to the ex453 multiplexer/demultiplexer. The ex453 multiplexer/demultiplexer multiplexes encoded video data and encoded audio data using a specific circuit, modulates and converts the data using the ex452 modulator/demodulator (modulator/demodulator circuit) and ex451 transceiver and transmits the result through the ex450 antenna.

[0740] When a video attachment in an email message or chat, or a video containing a link from a web page, is received, for example, to decode multiplexed data received through antenna ex450, the ex453 multiplexer/demultiplexer demultiplexes the multiplexed data to separate multiplexes data onto a video bitstream and an audio bitstream, provides encoded video data to the ex455 video processor via the ex470 synchronous bus, and provides encoded audio data to the ex454 audio processor via the ex470 synchronous bus. The video signal processor ex455 decodes the video signal using a moving frame decoding method corresponding to the moving frame encoding method described in the above embodiments, and the video or still image included in the associated moving frame file is displayed on the display ex458 through the display controller ex459. The audio signal processor ex454 decodes the audio signal and outputs audio from the audio output module ex457. As live streaming becomes more popular, there may be cases in which audio playback may not be socially feasible, depending on the user's environment. Accordingly, as an initial value, for example, a configuration in which only video data is reproduced, i.e., may be preferred. no audio signal is played; and the audio can only be synchronized and played when input is received from the user clicking on the video data.

[0741] Although the smartphone ex115 is used in the above example, three other implementations are possible: a transceiver terminal including both an encoder and a decoder; a transmission terminal including only an encoder; and a receiving terminal including only a decoder. In the description of a digital broadcast system, an example is given in which multiplexed data obtained by multiplexing video data with audio data is received or transmitted. However, the multiplexed data may be video data multiplexed with data other than audio data, such as text data associated with the video. Additionally, direct video data rather than multiplexed data may be received or transmitted.

[0742] Although the ex460 main controller, which includes a CPU, is described as controlling encoding or decoding processes, various terminals often include GPUs. Accordingly, it is acceptable to have a configuration in which a large area is processed at once by using performance characteristics of the GPU through a memory shared by the CPU and the GPU, or through a memory including an address that is controlled so as to enable a wide use via CPU and GPU. This reduces encoding time, maintains the real-time nature of streaming, and reduces latency. In particular, processing related to motion estimation, deblocking filtering, sampled adaptive offset (SAO), and transform/quantization can be efficiently performed by the GPU instead of the CPU, for example, in frame units, simultaneously.

Industrial applicability

[0743] The present disclosure is applicable, for example, to television receivers, digital video recorders, automobile navigation systems, mobile phones, digital cameras, digital video cameras, teleconferencing systems, electronic mirrors, etc.

List of reference numbers

[0744] 100 - encoder

102 - partitioning module

102a - block partition determination module

104 - subtractor

106 - converter

108 - quantizer

108a - difference quantization parameter generator

108b, 204b - predicted quantization parameters generator

108c, 204a - quantization parameters generator

108d, 204d - quantization parameters storage device

108e - quantization execution module

110 - entropy encoder

110a - binary conversion module

110b, 202b - context controller

110c - binary arithmetic encoder

112, 204 - inverse quantizer

114, 206 - inverse converter

116, 208 - adder

118, 210 - block storage device

120, 212 - contour filter

120a, 212a - filtering module to remove blocking

120b, 212b - SAO execution module

120c, 212c - ALF execution module

122, 214 - film frame storage device

124, 216 - internal forecasting module

126, 218 - mutual prediction module

126a, a2, b2 - storage device

126b - module for extracting interpolated images

126c - gradient image extraction module

126d - optical flow extraction module

126e - correction value extraction module

126f - predictive image corrector

128, 220 - forecasting controller

130, 222 - forecasting parameters generator

200 - decoder

202 - entropy decoder

202a - binary arithmetic decoder

202c - debinarization module

204e - inverse quantization module

224 - partition determination module

1201 - boundary detection module

1202, 1204, 1206 - switch

1203 - filter detection module

1205 - filtering module

1207 - module for determining filter characteristics

1208 - processing definition module

a1, b1 – processor.

Claims (96)

1. Encoder containing: - diagram; And - a storage device connected to the circuit, In this case, the scheme during operation: when generating a predictive image of the current block, uses the intra-block copying (IBC) mode, in which the processed area of the frame, which includes the current block, is accessed; - determines whether the size of the current block is less than or equal to the threshold value, and the current block is a unit for which a list of possible vector options is generated, including possible vector options; - when the size of the current block is less than or equal to the threshold value, generates a list of candidate vectors by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein HMVP- the table stores, in a first-in-first-out (FIFO) manner, each of the possible first vectors as a possible HMVP vector, each of the possible first vectors having information regarding the first vector used for the processed block; - when the size of the current block exceeds a threshold value, generates a list of possible vector options by performing the first pruning process and registering a possible HMVP vector option in the list of possible vector options from the HMVP table; And - encodes the current block using a list of possible vector options. 2. Encoder according to paragraph 1, in which the scheme: - generates a forecast image of the current block using the second vector; - encodes the second vector using one of the possible vector options included in the list of possible vector options; in the first pruning process performed when generating a list of possible vector options, - determines, for each of the possible HMVP vector variants, each of which represents a possible HMVP vector variant stored in the HMVP table, whether or not a possible HMVP vector variant is identical to any of one or more possible vector variants , registered in the list of possible vector options; And - when a possible HMVP vector variant is not identical to any of one or more possible vector variants, generates a list of possible vector variants by registering the possible HMVP vector variant in the list of possible vector variants. 3. Encoder according to clause 2, in which the circuit additionally: - updates the HMVP table using a candidate version of the second vector having information regarding the second vector, and when updating the HMVP table, the scheme: - determines whether the size of the current block is less than or equal to the threshold value; - updates the HMVP table without performing a second pruning process when the current block size is less than or equal to the threshold value; And - updates the HMVP table by performing a second pruning process when the current block size exceeds a threshold. 4. Encoder according to clause 3, in which, in the second pruning process performed when updating the HMVP table, the scheme is: - determines whether or not a possible variant of the second vector is identical to any of the possible variants of HMVP vectors; And - updates the HMVP table by storing a candidate second vector in the HMVP table when the candidate second vector is not identical to any of the candidate HMVP vectors. 5. Encoder according to any one of paragraphs. 1-4, wherein the size of the current block is specified by the total number of pixels included in the current block. 6. Encoder according to clause 5, in which the threshold value is 16 pixels. 7. Encoder according to any one of paragraphs. 1-4, wherein the size of the current block is specified by at least one of the width or height of the current block. 8. Encoder according to clause 7, in which the threshold is 4 pixels x 4 pixels. 9. Encoder according to any one of paragraphs. 1-4, in which the list of possible vectors is not shared between the current block and a block adjacent to the current block. 10. Encoder according to any one of paragraphs. 1-4, in which the scheme: - when generating a forecast image of the current block, uses the IBC mode when determining the need to use an IBC mode from a variety of forecasting modes and uses a forecasting mode different from the IBC mode when determining the need to use a forecast mode different from the IBC mode from a set of modes forecasting; - when using a prediction mode different from the IBC mode, generates a list of possible vector options by performing the first pruning process and registering a possible HMVP vector option in the list of possible vector options from the HMVP table; And - encodes the current block using a list of possible vector options. 11. Decoder containing: - diagram; And - a storage device connected to the circuit, In this case, the scheme during operation: when generating a predictive image of the current block, uses the intra-block copying (IBC) mode, in which the processed area of the frame, which includes the current block, is accessed; - determines whether the size of the current block is less than or equal to the threshold value, and the current block is a unit for which a list of possible vector options is generated, including possible vector options; - when the size of the current block is less than or equal to the threshold value, generates a list of candidate vectors by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein HMVP- the table stores, in a first-in-first-out (FIFO) manner, each of the possible first vectors as a possible HMVP vector, each of the possible first vectors having information regarding the first vector used for the processed block; - when the size of the current block exceeds a threshold value, generates a list of possible vector options by performing the first pruning process and registering a possible HMVP vector option in the list of possible vector options from the HMVP table; And - decodes the current block using a list of possible vector options. 12. Decoder according to clause 11, in which the scheme: - generates a forecast image of the current block using the second vector; - decodes the second vector using one of the possible vector options included in the list of possible vector options; - in the first pruning process performed when generating a list of possible vector options, - determines, for each of the possible HMVP vector variants, each of which represents a possible HMVP vector variant stored in the HMVP table, whether or not a possible HMVP vector variant is identical to any of one or more possible vector variants , registered in the list of possible vector options; And - when a possible HMVP vector variant is not identical to any of one or more possible vector variants, generates a list of possible vector variants by registering the possible HMVP vector variant in the list of possible vector variants. 13. Decoder according to clause 12, in which the circuit additionally: - updates the HMVP table using a candidate version of the second vector having information regarding the second vector, and when updating the HMVP table, the scheme: - determines whether the size of the current block is less than or equal to the threshold value; - updates the HMVP table without performing a second pruning process when the current block size is less than or equal to the threshold value; And - updates the HMVP table by performing a second pruning process when the current block size exceeds a threshold. 14. Decoder according to clause 13, - in which, in the second pruning process performed when updating the HMVP table, the scheme is: - determines whether or not a possible variant of the second vector is identical to any of the possible variants of HMVP vectors; And - updates the HMVP table by storing a candidate second vector in the HMVP table when the candidate second vector is not identical to any of the candidate HMVP vectors. 15. Decoder according to any one of paragraphs. 11-14, wherein the size of the current block is specified by the total number of pixels included in the current block. 16. Decoder according to clause 15, in which the threshold value is 16 pixels. 17. Decoder according to any one of paragraphs. 11-14, wherein the size of the current block is specified by at least one of the width or height of the current block. 18. Decoder according to clause 17, in which the threshold is 4 pixels x 4 pixels. 19. Decoder according to any one of paragraphs. 11-14, in which the list of possible vectors is not shared between the current block and a block adjacent to the current block. 20. Decoder according to any one of paragraphs. 11-14, in which the scheme: - when generating a forecast image of the current block, uses the IBC mode when determining the need to use an IBC mode from a variety of forecasting modes and uses a forecasting mode different from the IBC mode when determining the need to use a forecast mode different from the IBC mode from a set of modes forecasting; - when using a prediction mode different from the IBC mode, generates a list of possible vector options by performing the first pruning process and registering a possible HMVP vector option in the list of possible vector options from the HMVP table; And - decodes the current block using a list of possible vector options. 21. An encoding method containing the steps of: when generating a predictive image of the current block, the intra-block copying (IBC) mode is used, in which the processed area of the frame, which includes the current block, is accessed; - determine whether the size of the current block is less than or equal to the threshold value, and the current block is a unit for which a list of possible vector options is generated, including possible vector options; - when the size of the current block is less than or equal to the threshold value, generating a list of candidate vectors by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein HMVP- the table stores, in a first-in-first-out (FIFO) manner, each of the possible first vectors as a possible HMVP vector, each of the possible first vectors having information regarding the first vector used for the processed block; - when the size of the current block exceeds a threshold value, a list of possible vector variants is generated by performing the first pruning process and registering a possible HMVP vector variant in the list of possible vector variants from the HMVP table; And - encode the current block using a list of possible vector options. 22. A decoding method comprising the steps of: - when forming a predictive image of the current block, use the intra-block copying (IBC) mode, in which the processed area of the frame is accessed, which includes the current block; - determine whether the size of the current block is less than or equal to the threshold value, and the current block is a unit for which a list of possible vector options is generated, including possible vector options; - when the size of the current block is less than or equal to the threshold value, generating a list of candidate vectors by registering a candidate vector using a history-based motion vector predictor (HMVP) in the list of candidate vectors from the HMVP table without performing a first pruning process, wherein HMVP- the table stores, in a first-in-first-out (FIFO) manner, each of the possible first vectors as a possible HMVP vector, each of the possible first vectors having information regarding the first vector used for the processed block; - when the size of the current block exceeds a threshold value, a list of possible vector variants is generated by performing the first pruning process and registering a possible HMVP vector variant in the list of possible vector variants from the HMVP table; And - decode the current block using a list of possible vector options.

Images