BR122023027391A2 - ENCODER AND DECODER - Google Patents

Abstract

The present invention relates to methods, and decoding device (decoder) and encoding device (encoder) (100) which includes: a set of circuits (a1); and a memory (a2) coupled to the set of circuits (a1). In operation, the set of circuits (a1): derives a base motion vector to be used in predicting a current block to be encoded; derives a first motion vector different from the base motion vector; derives a motion vector difference based on a difference between the base motion vector and the first motion vector; determines whether the motion vector difference is greater than a threshold; modifies the first motion vector when the motion vector difference is determined to be greater than the threshold, and does not modify the first motion vector when the motion vector difference is determined to be not greater than the threshold; and encodes the current block using the first modified motion vector or the first unmodified motion vector.

Description

Technical Field

[001] The present description refers to encoder and decoder for encoding and decoding video, and associated methods. Fundamental Technique

[002] As a video coding standard, there is H.265, which

it is also referred to as High Efficiency Video Coding (HEVC) (Non-Patent Literature (NPL) 1). Citation List Non-Patent Literature

[003] NPL 1: H.265 (ISO/IEC 23008-2 HEVC)HEVC (High Efficiency Video Coding)

Summary of the Invention Technical problems

[004] For such a coding method and decoding method, it is desired to propose a new scheme in order to increase processing efficiency and improve image quality, reduce the scale of the circuit set, etc.

[005] Each of the configurations and methods described in the embodiments or part of the embodiments, according to the present description, can contribute to, for example, at least one of: an improvement in coding efficiency; a reduction in the amount of encoding/decoding processing; a reduction in the scale of the circuit set; an improvement in encoding/decoding speed; the appropriate selection of a constituent element/operation for a filter; a block, a size, a motion vector, a reference image, a reference block, etc. in encoding and decoding.

[006] It should be noted that the present description also includes descriptions of a configuration or a method that may provide benefits in addition to the benefits described above. An example of such descriptions includes a configuration or method that can improve coding efficiency while reducing an increase in the amount of processing. Solutions to Problems

[007] An encoder, according to one aspect of the present description, is an encoder that encodes a video. The encoder includes a set of circuits; and a memory coupled to the set of circuits. In operation, the circuitry derives a base motion vector to be used in predicting a current (current) block to be encoded; derives a first motion vector, different from the base motion vector, derives a difference from the motion vector, based on a difference between the basis motion vector and the first motion vector; determines whether the motion vector difference is greater than a threshold; modifies the first motion vector when the motion vector difference is determined to be greater than the threshold, and does not modify the first motion vector when the motion vector difference is determined to be not greater than the threshold; and encodes the current block using the first modified motion vector or the first unmodified motion vector.

[008] A decoder, according to one aspect of the present description, is a decoder that decodes a video. The decoder includes a set of circuitry, and a memory coupled to the circuitry. In operation, the circuitry derives a base motion vector to be used in predicting a current block to be decoded; derives a different first motion vector from the base motion vector; derives a motion vector difference based on a difference between the base motion vector and the first motion vector; determines whether the motion vector difference is greater than a threshold; modifies the first motion vector when the motion vector difference is determined to be greater than the threshold, and does not modify the first motion vector when the motion vector difference is determined to be not greater than the threshold; and decodes the current block using the first modified motion vector or the first unmodified motion vector.

[009] Note that these general and specific aspects can be implemented using a system, a device, a method, an integrated circuit, a computer program, a non-transitory computer-readable recording medium, such as CD- ROM, or any combination of systems, devices, methods, integrated circuits, computer programs or recording media.

[0010] Additional benefits and advantages of the described embodiments will become apparent from the specification and drawings. Benefits and/or advantages may be obtained individually by various embodiments and features of the specification and drawings, not all of which are necessary in order to obtain one or more of said benefits and/or advantages.

Advantageous Effects of the Invention

[0011] The present description is capable of providing an encoder, a decoder, a coding method, or a decoding method that allows an increase in coding efficiency.

Brief Description of the Drawings

[0012] Figure 1 is a block diagram illustrating a functional configuration of an encoder, according to one embodiment;

[0013] Figure 2 is a flowchart indicating an example of a general coding process performed by the coder;

[0014] Figure 3 is a diagram illustrating an example of block division;

[0015] Figure 4A is a diagram illustrating an example of a slice configuration;

[0016] Figure 4B is a diagram illustrating an example of a tile configuration;

[0017] Figure 5A is a graph illustrating the basic transformation functions for each type of transformation;

[0018] Figure 5B is a diagram illustrating spatially variable transformations (SVT);

[0019] Figure 6A is a diagram illustrating an example of a filter format used in an adaptive circuit filter (ALF);

[0020] Figure 6B is a diagram illustrating another example of a filter format used in an ALF;

[0021] Figure 6C is a diagram illustrating another example of a filter format used in an ALF;

[0022] Figure 7 is a block diagram indicating an example of a specific configuration of a circuit filter that functions as a DBF;

[0023] Figure 8 is a diagram indicating an example of an unblocking filter having a symmetric filtering characteristic with respect to a block boundary;

[0024] Figure 9 is a diagram for illustrating a block boundary in which an unblocking filter process is performed;

[0025] Figure 10 is a diagram indicating examples of Bs values;

[0026] Figure 11 is a diagram illustrating an example of a process performed by an encoder prediction processor;

[0027] Figure 12 is a diagram illustrating another example of a process performed by the encoder prediction processor;

[0028] Figure 13 is a diagram illustrating another example of a process performed by the encoder prediction processor;

[0029] Figure 14 is a diagram illustrating sixty-seven intraforecast modes used in intraforecast;

[0030] Figure 15 is a flowchart illustrating a basic interforecast processing flow;

[0031] Figure 16 is a flowchart illustrating an example of deriving motion vectors;

[0032] Figure 17 is a flowchart illustrating another example of deriving motion vectors;

[0033] Figure 18 is a flowchart illustrating another example of deriving motion vectors;

[0034] Figure 19 is a flowchart illustrating an example of interprediction in normal intermode;

[0035] Figure 20 is a flowchart illustrating an example of interprediction in mixing mode;

[0036] Figure 21 is a diagram to illustrate an example of a motion vector derivation process in mixing mode;

[0037] Figure 22 is a flowchart illustrating an example of frame rate upconversion (FRUC);

[0038] Figure 23 is a diagram to illustrate an example of pattern matching (bilateral matching) between two blocks along a motion path;

[0039] Figure 24 is a diagram to illustrate an example of pattern combination (template combination) between a template in a current image and a block in a reference image;

[0040] Figure 25A is a diagram to illustrate an example of deriving a motion vector of each sub-block based on the motion vectors of a plurality of neighboring blocks;

[0041] Figure 25B is a diagram to illustrate an example of deriving a motion vector of each sub-block in affine mode in which three control points are used;

[0042] Figure 26A is a conceptual diagram to illustrate an affine mixing mode;

[0043] Figure 26B is a conceptual diagram to illustrate an affine mixing mode in which two control points are used;

[0044] Figure 26C is a conceptual diagram to illustrate an affine mixing mode in which three control points are used;

[0045] Figure 27 is a flowchart illustrating an example of a process in affine mixing mode;

[0046] Figure 28A is a diagram to illustrate a related intermode in which two control points are used;

[0047] Figure 28B is a diagram to illustrate a related intermode in which three control points are used;

[0048] Figure 29 is a flowchart illustrating an example of a process in the related intermode;

[0049] Figure 30A is a diagram to illustrate an affine intermode in which a current block has three control points and a neighboring block has two control points;

[0050] Figure 30B is a diagram to illustrate a related intermode in which a current block has two control points and a neighboring block has three control points;

[0051] Figure 31A is a diagram illustrating a relationship between a mixing mode and a dynamic motion vector update (DMVR);

[0052] Figure 31B is a conceptual diagram to illustrate an example of a DMVR process;

[0053] Figure 32 is a flowchart illustrating an example of generating a prediction image;

[0054] Figure 33 is a flowchart illustrating another example of generating a prediction image;

[0055] Figure 34 is a flowchart illustrating another example of generating a prediction image;

[0056] Figure 35 is a flowchart illustrating an example of a predictive image correction process performed by an overlapping block motion compensation (OBMC) process;

[0057] Figure 36 is a conceptual diagram to illustrate an example of a predictive image correction process performed by an OBMC process;

[0058] Figure 37 is a diagram to illustrate the generation of two triangular prediction images;

[0059] Figure 38 is a diagram to illustrate a model assuming uniform linear motion;

[0060] Figure 39 is a diagram to illustrate an example of a predictive image generation method using a luminescence correction process performed by a local illumination compensation (LIC) process;

[0061] Figure 40 is a block diagram illustrating an example of encoder assembly;

[0062] Figure 41 is a block diagram illustrating a functional configuration of a decoder, according to one embodiment;

[0063] Figure 42 is a flowchart illustrating an example of a general decoding process performed by the decoder;

[0064] Figure 43 is a diagram illustrating an example of a process performed by a decoder prediction processor;

[0065] Figure 44 is a diagram illustrating another example of a process performed by the decoder prediction processor;

[0066] Figure 45 is a flowchart illustrating an example of normal inter-mode interprediction in the decoder;

[0067] Figure 46 is a block diagram illustrating an example of decoder assembly;

[0068] Figure 47 is a flowchart illustrating an example of interforecast, according to the primary aspect;

[0069] Figure 48 is a diagram illustrating an example of a current block;

[0070] Figure 49 is a diagram illustrating the calculated limit examples;

[0071] Figure 50 is a diagram illustrating a general configuration of a content delivery system for implementing a content distribution service;

[0072] Figure 51 is a diagram illustrating an example of a coding structure in scalable coding;

[0073] Figure 52 is a diagram illustrating an example of a coding structure in scalable coding;

[0074] Figure 53 is a diagram illustrating an example of a web page display screen;

[0075] Figure 54 is a diagram illustrating an example of a web page display screen;

[0076] Figure 55 is a diagram illustrating an example of a smartphone;

[0077] Figure 56 is a block diagram illustrating an example of a smartphone configuration.

Description of Illustrative Embodiments Underlying Knowledge Formation Base of This Description

[0078] For example, in video coding, etc., an encoder, which encodes a video, derives a prediction error by subtracting a prediction image from an image included in the video, when the video is encoded by performing a additional subdivided prediction process, while preventing an increase in the amount of processing. Subsequently, the encoder converts the frequency and quantizes the prediction error, and encodes the resulting image data. In doing so, with respect to a motion in a current unit to be encoded, such as a block included in a video, when motion prediction is performed for each block or each sub-block included in the block, the coding accuracy can be improved by controlling the variation of movement vectors.

[0079] However, when the variation of motion vectors is not controlled in coding the block included in the video, etc., an increase in the amount of processing occurs, thereby reducing coding efficiency.

[0080] In view of this, an encoder, according to an aspect of the present description, is an encoder that encodes a video. The encoder includes a set of circuits, and a memory coupled to the set of circuits. In operation, the circuitry derives a base motion vector to be used in predicting a current block to be encoded; derives a first motion vector, different from the base motion vector; derives a motion vector difference based on a difference between the base motion vector and the first motion vector; determines whether the motion vector difference is greater than a threshold; modifies the first motion vector when the motion vector difference is determined to be greater than the threshold, and does not modify the first motion vector when the motion vector difference is determined to be not greater than the threshold; and encodes the current block using the first modified motion vector or the first unmodified motion vector.

[0081] With this, the encoder can adjust the movement vector so that the variation of the movement vectors in the current block, divided into sub-blocks, is within a predetermined range. The number of pixels to be referred to becomes lower than the case in which the variation of motion vectors is greater than the predetermined range and, in this way, it is possible to restrict the amount of data transferred from memory to the predetermined range. Accordingly, the encoder can reduce the memory access size in interprediction, and thus the coding efficiency is improved.

[0082] For example, the base motion vector may correspond to a first set of pixels in the current block, and the first motion vector may correspond to a second set of pixels, different from the first set of pixels, in the current block.

[0083] With this, the encoder can adequately control the variation of the motion vectors in the current block using the base motion vector determined based on the first set of pixels and the first motion vector determined based on the second set of pixels.

[0084] For example, the circuitry may modify the first motion vector using a value obtained by fixing the motion vector difference, when the motion vector difference is determined to be greater than the threshold.

[0085] With this, the encoder can modify the first motion vector so that the variation of the first motion vectors is within the predetermined range. Accordingly, the encoder can reduce the memory access size in interprediction, and thus the coding efficiency is improved.

[0086] For example, a prediction mode for the current block may be an affine mode.

[0087] With this, the encoder can perform the prediction in the current block for each sub-block and, in this way, the prediction accuracy is improved.

[0088] For example, the threshold may be determined such that a memory access size, in a worst case when the current block, which is predicted in prediction mode is less than or equal to a memory access size , in a worst case, when the current block is predicted in another prediction mode.

[0089] Thereby, the encoder smoothly performs a transfer process by restricting the memory access size in which data is transferred from memory, to the predetermined range, and thus the processing efficiency is improved.

[0090] For example, the worst-case memory access size when the current block is predicted in another prediction mode may be a memory access size when the current block is predicted bidirectionally based on 8 x 8 pixels in the other prediction mode.

[0091] In doing so, for example, when the prediction is performed without using the affine mode, the encoder generates the prediction image for each block from a coded image. When bidirectional prediction is performed for each 8 x 8 pixel block using a prediction mode in addition to affine mode, the memory access size is maximum. Accordingly, also in affine mode, the affine mode limit is determined to be within the memory access size. In this way, the encoder smoothly performs a transfer process by restricting the memory access size, in which data is transferred from memory to the predetermined range, and thus processing efficiency is improved.

[0092] For example, the limit may differ between a case in which the current block is predicted unidirectionally and a case in which the current block is predicted bidirectionally.

[0093] Therefore, in the encoder, the access size becomes larger in the case in which the current block is predicted bidirectionally than in the case where the current block is predicted unidirectionally. Therefore, coding efficiency is improved by setting the threshold to a lower value for a larger access size.

[0094] For example, the circuitry may determine whether the motion vector difference is greater than the threshold as follows: the motion vector difference is determined to be greater than the threshold when (i) a value absolute value of a horizontal component of the motion vector difference is greater than a first threshold value or (ii) an absolute value of a vertical component of the motion vector difference is greater than a second threshold value; and the motion vector difference is determined so that it is not greater than the threshold when (i) the absolute value of the horizontal component of the motion vector difference is not greater than the first value of the threshold and (ii) the absolute value of the vertical component of the motion vector difference is not greater than the second threshold value.

[0095] With this, the limit is represented by the first value which is a horizontal component (hereinafter, also referred to as the first limit value) and the second value which is a vertical component (hereinafter, also referred to as the second limit value), and in this way, the encoder can determine the variation of the first motion vectors in a bidirectional manner. Additionally, the encoder determines whether the variation of the first motion vectors in the current block is within the predetermined range and, in this way, it is possible to adjust the first vector in the current block to an appropriate value. Accordingly, coding efficiency is improved.

[0096] For example, the circuitry modifies the first motion vector using the base motion vector and a value obtained by fixing the motion vector difference, when the motion vector difference is determined to be greater than the threshold , and a new motion vector difference between the first modified motion vector and the base motion vector need not be greater than the threshold.

[0097] In doing so, when the variation of the first motion vectors exceeds the predetermined range, for example, the encoder sets, at the limit value, the absolute value of a component that exceeds the limit between the horizontal and vertical components of the difference. motion vector icon. Subsequently, for example, the encoder may determine, as the first modified motion vector, the value obtained by adding the fixed motion vector difference to the base motion vector. As a result, the variation of the first movement vectors in the current block is within the predetermined range. Accordingly, coding efficiency is improved.

[0098] For example, the base motion vector may be an average of a plurality of first motion vectors of the current block, the plurality of first motion vectors each being the first motion vector.

[0099] Thus, for each of the first motion vectors in the current block, the encoder can derive a difference from the reference that is the average of all the first motion vectors in the current block.

[00100] For example, the base motion vector may be one of the first motion vectors of the current block.

[00101] Thus, for each of the first motion vectors in the current block, the encoder can derive a difference from the reference that is one of the first motion vectors in the current block.

[00102] For example, the limit can be adaptive to a current block size.

[00103] Therefore, in the encoder, the memory access size varies depending on the size of the current block and, therefore, the limit can be reduced with an increase in the memory access size. For example, the limit is reduced with an increase in the current block size. Accordingly, the encoder can make the memory access size fall within the predetermined range even when the current block size increases, and thus the coding efficiency is improved.

[00104] For example, the limit can be predetermined, and does not need to be encoded in a sequence.

[00105] With this, the encoder can use the predetermined limit and, therefore, it is not necessary to encode the limit each time the prediction is made. Accordingly, coding efficiency is improved.

[00106] A decoder, according to one aspect of the present description, is a decoder that decodes a video. The decoder includes a set of circuits; and a memory coupled to the set of circuits. In operation, the circuitry derives a base motion vector to be used in predicting a current block to be decoded; derives a different first motion vector from the base motion vector; derives a difference motion vector based on a difference between the base motion vector and the first motion vector; determines whether the motion vector difference is greater than a threshold; modifies the first motion vector when the motion vector difference is determined to be greater than the threshold, and does not modify the first motion vector when the motion vector difference is determined to be not greater than the threshold; and decodes the current block using the first modified motion vector or the first unmodified motion vector.

[00107] With this, the decoder can adjust the movement vector so that the variation of the movement vectors in the current block, divided into sub-blocks, is within a predetermined range. The number of pixels to be referred to becomes lower than in the case in which the variation of the motion vectors is greater than the predetermined range, and, therefore, it is possible to restrict the amount of data transferred from memory to the predetermined range. Accordingly, the decoder can reduce the memory access size in interprediction and thus the coding efficiency is improved.

[00108] For example, the base motion vector may correspond to a first set of pixels in the current block, and the first motion vector may correspond to a second set of pixels, different from the first set of pixels, in the current block.

[00109] With this, the decoder can adequately control the variation of the motion vectors in the current block using the base motion vector determined based on the first set of pixels, and the first motion vector determined based on the second set of pixels.

[00110] For example, the circuitry may modify the first motion vector using a value obtained by fixing the motion vector difference, when the motion vector difference is determined to be greater than the threshold.

[00111] With this, the decoder can modify the first motion vector, so that the variation of the first motion vector is within the predetermined range. Accordingly, the decoder can reduce the memory access size in interprediction, and thus the coding efficiency is improved.

[00112] For example, a prediction mode for the current block may be an affine mode.

[00113] With this, the decoder can perform the prediction in the current block for each sub-block, and thus the prediction accuracy is improved.

[00114] For example, the threshold may be determined such that a memory access size, in a worst case, when the current block is predicted in prediction mode, is less than or equal to a memory access size, in a worst case, when the current block is predicted in another prediction mode.

[00115] Thereby, the decoder smoothly performs a transfer process by restricting the memory access size in which data is transferred from memory, to the predetermined range, and thus processing efficiency is improved.

[00116] For example, the worst-case memory access size when the current block is predicted in the other prediction mode may be a memory access size when the current block is predicted bidirectionally based on 8 x 8 pixels, in the other prediction mode.

[00117] In doing so, for example, when the prediction is performed without using the affine mode, the decoder generates the prediction image for each block from a decoded image. When bidirectional prediction is performed for each 8 x 8 pixel block, using a prediction mode in addition to the affine mode, the memory access size is maximum. Accordingly, also in affine mode, the affine mode limit is determined to lie within the memory access size. Accordingly, the decoder smoothly performs a transfer process by restricting the memory access size, in which the data is transferred from the memory to the predetermined range, and thus the processing efficiency is improved.

[00118] For example, the limit may differ between a case in which the current block is predicted unidirectionally and a case in which the current block is predicted bidirectionally.

[00119] Therefore, in the decoder, the access size becomes larger in the case in which the current block is predicted bidirectionally than in the case in which the current block is predicted unidirectionally. Therefore, coding efficiency is improved by setting the threshold to a lower value for a larger access size.

[00120] For example, the circuitry may determine whether the motion vector difference is greater than the threshold as follows: the motion vector difference is determined to be greater than the threshold when (i) an absolute value of a horizontal component of the motion vector difference is greater than a first threshold value or (ii) an absolute value of a vertical component of the motion vector difference is greater than a second threshold value; and the motion vector difference is determined to be no greater than the threshold when (i) the absolute value of the horizontal component of the motion vector difference is not greater than the first threshold value and (ii) the absolute value of the vertical component of the motion vector difference is not greater than the second threshold value.

[00121] With this, the limit is represented by the first value which is a horizontal component (hereinafter, also referred to as the first limit value) and the second value which is a vertical component (hereinafter, also referred to as the second limit value), and in this way, the encoder can determine the variation of the first motion vectors in a two-dimensional way. Additionally, the decoder determines whether the variation of the first motion vectors in the current block is within the predetermined range, and, in this way, it is possible to adjust the first vector in the current block to an appropriate value. Accordingly, the processing efficiency is improved.

[00122] For example, the circuitry may modify the first motion vector using the base motion vector and a value obtained by fixing the motion vector difference, when the motion vector difference is determined to be greater than the threshold, and a new motion vector difference, between the first modified motion vector and the base motion vector, need not be greater than the threshold.

[00123] In doing so, when the variation of the first motion vectors exceeds the predetermined range, for example, the decoder sets, at the limit value, the absolute value of a component exceeding the limit between the horizontal and vertical components of the difference in motion vector. Subsequently, for example, the decoder may determine, as the first modified motion vector, the value obtained by adding the fixed motion vector difference to the base motion vector. As a result, the variation of the first movement vectors in the current block is within the predetermined range. Accordingly, the processing efficiency is improved.

[00124] For example, the base motion vector may be an average of a plurality of first motion vectors of the current block, the plurality of first motion vectors each being the first motion vector.

[00125] Thus, for each of the first motion vectors in the current block, the decoder can derive a difference from the reference that is the average of all the first motion vectors in the current block.

[00126] For example, the base motion vector may be one of the first motion vectors of the current block.

[00127] Thus, for each of the first motion vectors in the current block, the decoder can derive a difference from the reference that is one of the first motion vectors in the current block.

[00128] For example, the limit can be adaptive to a current block size.

[00129] Therefore, in the decoder, the memory access size varies depending on the size of the current block, and, therefore, the limit can be reduced with an increase in the memory access size. For example, the limit is reduced with an increase in the current block size. Accordingly, the decoder can make the memory access size fall within the predetermined range even when the current block size increases, and thus the processing efficiency is improved.

[00130] For example, the limit can be predetermined, and does not need to be decoded from a sequence.

[00131] With this, the decoder can use the predetermined limit, and, therefore, it is not necessary to encode the limit each time the prediction is made. Accordingly, the processing efficiency is improved.

[00132] An encoding method, according to an aspect of the present description, is a encoding method for encoding a video. The encoding method includes deriving a base motion vector to be used in predicting a current block to be encoded; deriving a first motion vector different from the base motion vector; deriving a motion vector difference based on a difference between the base motion vector and the first motion vector; determine whether the motion vector difference is greater than a threshold; modifying the first motion vector when the motion vector difference is determined to be greater than the threshold and not modifying the first motion vector when the motion vector difference is determined to be not greater than the threshold; and encoding the current block using the first modified motion vector or the first unmodified motion vector.

[00133] With this, the coding method can adjust the movement vector so that the variation of the movement vectors in the current block, divided into sub-blocks, is within the predetermined range. The number of pixels to be referred to becomes lower than the case in which the variation of the motion vectors is greater than the predetermined range, and, in this way, it is possible to restrict the amount of data transferred from memory to the predetermined range. Accordingly, the coding method can reduce the memory access size in interprediction, and thus the coding efficiency is improved.

[00134] An encoding method, according to an aspect of the present description, is a decoding method, for decoding a video. The decoding method includes deriving a base motion vector to be used in predicting a current block to be decoded; deriving a first motion vector, different from the base motion vector; deriving a motion vector difference based on a difference between the base motion vector and the first motion vector; determine whether the motion vector difference is greater than a threshold; modifying the first motion vector when the motion vector difference is determined to be greater than the threshold, and not modifying the first motion vector when the motion vector difference is determined to be not greater than the threshold; and decode the current block using the first modified motion vector or the first unmodified motion vector.

[00135] With this, the decoding method can adjust the motion vector so that the variation of the motion vectors in the current block, divided into sub-blocks, is within a predetermined range. The number of pixels to be referred to becomes lower than the case in which the variation of the motion vectors is greater than the predetermined range, and, in this way, it is possible to restrict the amount of data transferred from memory to the predetermined range. Accordingly, the decoding method can reduce the memory access size in interprediction, and thus the processing efficiency is improved.

[00136] Additionally, these general and specific aspects can be implemented using a system, a device, a method, an integrated circuit, a computer program, a non-transitory computer readable recording medium, such as a CD-ROM , or any combination of systems, devices, methods, integrated circuits, computer programs or recording media.

[00137] Hereinafter, embodiments will be described with reference to the drawings. Note that the embodiments described below each illustrate a general or specific example. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, the relationship and order of the steps, etc., indicated in the following embodiments are mere examples, and should not limit the scope of the embodiments.

[00138] Embodiments of an encoder and a decoder will be described below. Embodiments are examples of an encoder and a decoder to which the processes and/or configurations set forth in the description of aspects of the present disclosure are applicable. The processes and/or configurations can also be implemented in an encoder and a decoder different from those referring to the modalities. For example, with reference to the processes and/or configurations, as applied to the embodiments, any of the following may be implemented:

[00139] (1) Any of the components of the encoder or decoder, according to the embodiments presented in the description of aspects of the present description, may be replaced or combined with another component presented elsewhere in the description of aspects of the present description.

[00140] (2) In the encoder or decoder, according to embodiments, discretionary changes may be made to the functions or processes performed by one or more components of the encoder or decoder, such as addition, substitution, removal, etc. functions or processes. For example, any function or process may be substituted for or combined with another function or process set forth elsewhere in the description of aspects of the present disclosure.

[00141] (3) In the methods implemented by the encoder or decoder, according to the embodiments, discretionary changes can be made such as addition, replacement and removal of one or more of the processes included in the method. For example, any process in the method may be substituted for or combined with another process set forth elsewhere in the description of aspects of the present disclosure.

[00142] (4) One or more components included in the encoder or decoder, according to embodiments, may be combined with a component set forth elsewhere in the description of aspects of the present description, may be combined with a component including one or more functions presented elsewhere in the description of aspects of the present description, and may be combined with a component that implements one or more processes implemented by a component presented in the description of aspects of the present description.

[00143] (5) A component, including one or more encoder or decoder functions, according to embodiments, or a component that implements one or more encoder or decoder processes, according to embodiments, may be combined or replaced by a component set forth elsewhere in the description of aspects of the present description, with a component including one or more functions set forth elsewhere in the description of aspects of the present description, or with a component that implements one or more processes set forth elsewhere place in the description of aspects of the present description.

[00144] (6) In methods implemented by the encoder or decoder, according to embodiments, any of the processes included in the method may be replaced or combined with a process presented elsewhere in the description of aspects of the present description, or with any corresponding or equivalent process.

[00145] (7) One or more processes included in the methods implemented by the encoder or the decoder, according to the embodiments, may be combined with a process presented elsewhere in the description of aspects of the present description.

[00146] (8) The implementation of the processes and/or configurations presented in the description of the aspects of the present description is not limited to the encoder or the decoder, according to the embodiments. For example, the processes and/or settings may be implemented in a device used for a purpose other than the moving image encoder or the moving image decoder in the embodiments.

Mode 1 Encoder

[00147] First, an encoder, according to this embodiment, will be described. Figure 1 is a block diagram illustrating a functional configuration of the encoder 100 in accordance with this embodiment. Encoder 100 is a video encoder that encodes a video in units of a block.

[00148] As illustrated in Figure 1, the encoder 100 is an apparatus that encodes an image into units of a block, and includes a divider 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, a inverse quantizer 112, an inverse transformer 114, an adder 116, a block memory 118, an in-loop filter 120, a frame memory 122, an intrapredictor 124, an interpredictor 126, and a prediction controller 128.

[00149] Encoder 100 is implemented as, for example, a generic processor and memory. In this case, when a software program, stored in memory, is executed by the processor, the processor functions as a divider 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, a inverse transformer 114, an adder 116, a circuit filter 120, an intraforecaster 124, an interforecaster 126, and a prediction controller 128. Alternatively, the encoder 100 may be implemented as one or more dedicated electronic circuits corresponding to the divider 102, subtractor 104 , transformer 106, quantizer 108, entropy encoder 110, inverse quantizer 112, inverse transformer 114, adder 116, circuit filter 120, intrapredictor 124, interpredictor 126 and prediction controller 128.

[00150] Hereinafter, a general flow of processes performed by encoder 100 is described, and then each of the constituent elements included in encoder 100 will be described.

General Coding Process Flow

[00151] Figure 2 is a flowchart indicating an example of a general coding process performed by encoder 100.

[00152] First, the divider 102 of the encoder 100 divides each of the images included in an input image, which is a video, into a plurality of blocks having a fixed size (128 x 128 pixels) (Step Sa_1). The splitter 102 then selects a splitting pattern for the fixed-size block (also referred to as a block format) (Step Sa_2). In other words, the divider 102 further divides the fixed-size block into a plurality of blocks that form the selected division pattern. The encoder 100 performs, for each of the plurality of blocks, Steps Sa_3 to Sa_9 for the block (which is a current block to be encoded).

[00153] In other words, a prediction processor, which includes all or part of the intraforecaster 124, interforecaster 126, and prediction controller 128 generates a prediction signal (also referred to as a prediction block) from the current block to be encoded (also referred to as a prediction block) referred to as a current block) (Step Sa_3).

[00154] Next, subtractor 104 generates a difference between the current block and a prediction block as a prediction residual (also referred to as a difference block) (Step Sa_4).

[00155] Next, the transformer 106 transforms the difference block and the quantizer 108 quantizes the result, to generate a plurality of quantized coefficients (Step Sa_5). It should be noted that the block having the plurality of quantized coefficients is also referred to as a coefficient block.

[00156] Next, the entropy coder 110 encodes (specifically, entropy codes) the coefficient block and a prediction parameter related to generating a prediction signal to generate a coded signal (Step Sa_6). It should be noted that the encoded signal is also referred to as a coded bit stream, a compressed bit stream, or a sequence.

[00157] Next, the inverse quantizer 112 performs the inverse quantization of the coefficient block, and the inverse transformer 114 performs the inverse transformation of the result, to restore a plurality of prediction residuals (i.e., a difference block) (Step Sa_7 ).

[00158] Next, adder 116 adds the prediction block to the restored difference block to reconstruct the current block as a reconstructed image (also referred to as a reconstructed block or a decoded image block) (Step Sa_8). In this way, the reconstructed image is generated.

[00159] When the reconstructed image is generated, the filter circuit 120 performs filtering of the reconstructed image as necessary (Step Sa_9).

[00160] Encoder 100 then determines whether encoding of the entire image has finished (Step Sa_10). When it is determined that encoding has not yet been completed (Not in Step Sa_10), the processes in Step Sa_2 are executed repeatedly.

[00161] Although encoder 100 selects a division pattern for a fixed-size block, and encodes each block according to the division pattern in the example described above, it should be noted that each block may be encoded according to a corresponding pattern among a plurality of division patterns. In this case, the encoder 100 may evaluate a cost for each of the plurality of division patterns and, for example, may select the encoded signal obtainable by encoding in accordance with the division pattern that results in the lowest cost as a signal. encoded that is finally sent.

[00162] Alternatively, the processes in Steps Sa_1 to Sa_10 may be performed sequentially by encoder 100, or two or more of the processes may be performed in parallel or may be reordered.

Divider

[00163] Splitter 102 divides each of the images included in an input video into a plurality of blocks, and sends each block to subtractor 104. For example, splitter 102 first divides an image into blocks of a fixed size (e.g. example, 128 x 128). The fixed-size block is also referred to as a coding tree unit (CTU). The splitter 102 then divides each of the fixed-size blocks into blocks of variable sizes (e.g., 64 x 64 or smaller), based on quadtree and/or recursive binary tree block splitting. In other words, divider 102 selects a division pattern. The variable-size block is also referred to as a coding unit (CU), a prediction unit (PU), or a transformation unit (TU). It should be noted that in many types of assembly examples, there is no need to differentiate between CU, PU, and TU; all or some of the blocks in an image can be processed in units of a CU, a PU, or a TU.

[00164] Figure 3 is a diagram illustrating an example of block division according to this embodiment. In Figure 3, the solid lines represent block boundaries of the blocks divided by quadtree block split, and the dashed lines represent the block boundaries of blocks divided by binary tree block split.

[00165] Here, block 10 is a square block having 128 x 128 pixels (128 x 128 block). This 128 x 128 10 block is first divided into four 64 x 64 square blocks (quadtree block division).

[00166] The top left 64 x 64 block is further vertically divided into two 32 x 64 rectangular blocks, and the left 32 x 64 block is further vertically divided into two 16 x 64 rectangular blocks (tree block division binary). As a result of this, the top left 64 x 64 block is divided into two 16 x 64 blocks 11 and 12 and one 32 x 64 block 13.

[00167] The upper right 64 x 64 block is horizontally divided into two 64 x 32 rectangular blocks 14 and 15 (binary tree block division).

[00168] The lower left 64 x 64 block is first divided into four 32 x 32 square blocks (quadtree block division). The upper left block and the lower right block, among the four 32 x 32 blocks, are further divided. The top left block of 32 x 32 is vertically divided into two rectangular blocks of 16 x 32, and the right block of 16 x 32 is additionally divided horizontally into two blocks of 16 x 16 (binary tree block division). The bottom right 32 x 32 block is divided horizontally into two 32 x 16 blocks (binary tree block splitting). As a result of this, the bottom left block of 64 x 64 is divided into one block of 16 x 32 16, two blocks of 16 x 16 17 and 18, two blocks of 32 x 32 19 and 20, and two blocks of 32 x 16 21 and 22.

[00169] The lower right block of 64 x 64 23 is not divided.

[00170] As described above, in Figure 3, block 10 is divided into thirteen blocks of variable size 11 to 23 based on recursive binary tree and quadtree block division. This type of splitting is also referred to as quadtree plus binary tree (QTBT) splitting.

[00171] It should be noted that in Figure 3, a block is divided into four or two blocks (quadtree or binary tree block splitting), but the splitting is not limited to these examples. For example, a block can be divided into three blocks (ternary block division). Splitting including such ternary block splitting is also referred to as multi-type tree (MBT) splitting.

Image Structure: Slice/Tile

[00172] An image can be configured in units of one or more slices or tiles in order to decode the image in parallel. The image configured in units of one or more slices or tiles can be configured by divider 102.

[00173] Slices are basic coding units included in an image. An image may include, for example, one or more slices. Additionally, a slice includes one or more successive coding tree units (CTU).

[00174] Figure 4A is a diagram illustrating an example of a slice configuration. For example, an image includes 11 x 8 CTUs and is divided into four slices (slices 1 to 4). Slice 1 includes sixteen CTUs, slice 2 includes twenty-one CTUs, slice 3 includes twenty-nine CTUs, and slice 4 includes twenty-two CTUs. Here, each CTU in the image belongs to one of the slices. The format of each slice is a format obtainable by horizontally dividing the image. A boundary of each slice need not coincide with an edge of the image, and may coincide with any of the boundaries between the CTUs in the image. The processing order of the CTUs in a slice (an encoding order or a decoding order) is, for example, a raster scanning order. A slice includes header information and encoded data. The characteristics of the slice can be described in the header information. Characteristics include a CTU address of a higher CTU in the slice, a slice type, etc.

[00175] A tile is a unit of a rectangular region included in an image. Each of the tiles can be assigned a number referred to as TileId in the raster scanning order.

[00176] Figure 4B is a diagram indicating an example of a tile configuration. For example, an image includes 11 x 8 CTUs and is divided into four tiles of rectangular regions (tiles 1 through 4). When tiles are used, the processing order of the CTUs is changed from the processing order, in which case no tiles are used. When no tiles are used, the CTUs in an image are processed in raster scan order. When tiles are used, at least one CTU in each of the tiles is processed in raster scan order. For example, as illustrated in Figure 4B, the processing order of the CTUs included in tile 1 is the order starting from the left side of the first column of tile 1 towards the right end of the first column of tile 1 and then starts from the left end of the second column of tile 1 towards the right end of the second column of tile 1.

[00177] It should be noted that a tile may include one or more slices, and a slice may include one or more tiles.

Subtractor

[00178] Subtractor 104 subtracts a prediction signal (prediction sample that is recorded from prediction controller 128 indicated below) from an original signal (original sample) in units of a block recorded from divisor 102 and divided by the divisor 102. In other words, the subtractor 104 calculates the prediction errors (also referred to as residuals) of a block to be encoded (hereinafter also referred to as a current block). The subtractor 104 then sends the calculated prediction errors (residuals) to the transformer 106.

[00179] The original signal is a signal that was recorded in encoder 100 and represents one image of each image included in a video (e.g., one luminescence signal and two chrominance signals). Hereinafter, a signal representing an image is also referred to as a sample.

Transformer

[00180] Transformer 106 transforms spatial domain prediction errors into frequency domain transformation coefficients, and sends the transformation coefficients to quantizer 108. More specifically, transformer 106 applies, for example, a discrete cosine transformation preset (DCT) or discrete sine transform (DST) on prediction errors in spatial domain.

[00181] It should be noted that the transformer 106 can adaptively select a transformation type from a plurality of transformation types, and transform prediction errors in transformation coefficients by utilizing a transformation base function that corresponds to the selected transformation type. This type of transformation is also referred to as explicit multi-core transformation (EMT) or adaptive multiple transformation (AMT).

[00182] Transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I and DST-VII. Figure 5A is a graph indicating the transformation basis functions for each transformation type. In Figure 5A, N indicates the number of input pixels. For example, the selection of a transformation type from the plurality of transformation types may depend on a prediction type (one of intraforecast and interforecast) and may depend on an intraforecast mode.

[00183] Information indicating whether such an EMT or AMT applies (referred to as, for example, an EMT indicator or an AMT indicator) and information indicating the type of transformation selected is normally signaled at the CU level. It should be noted that signaling of such information does not necessarily need to be performed at the CU level, and may be performed at another level (e.g., at the bit sequence level, image level, slice level, tile or CTU level).

[00184] Additionally, the transformer 106 can transform the transformation coefficients (transformation result) again. Such new transformation is also referred to as adaptive secondary transformation (AST) or non-separable secondary transformation (NSST). For example, the transformer 106 performs the new transformation in units of a sub-block (e.g., the 4 x 4 sub-block) included in a transformation coefficient block that corresponds to an intraprediction error. Information indicating whether NSST applies and information relating to a transformation matrix for use in NSST is normally signaled at the CU level. It should be noted that signaling of such information does not necessarily need to be performed at the CU level, and may be performed at another level (e.g., at the sequence level, image level, slice level, tile level or CTU level).

[00185] Transformer 106 may employ a separable transformation and a non-separable transformation. A separable transformation is a method in which a transformation is performed multiple times by separately performing a transformation for each of several directions according to the number of input dimensions. A non-separable transformation is a method of performing a collective transformation in which two or more dimensions in the multidimensional inputs are collectively considered a unit dimension.

[00186] In an example of a non-separable transformation, when an input is a 4 x 4 block, the 4 x 4 block is considered a single set including sixteen elements, and the transformation applies a 16 x 16 transformation matrix to the whole.

[00187] In another example of a non-separable transformation, a 4 x 4 input block is considered a single set including sixteen elements and then a transformation (hypercube givens transformation) in which the givens revolution, which is performed in the set several times, can be performed.

[00188] In the transformation in transformer 106, the types of bases to be transformed in the frequency domain, according to the regions in a CU, can be switched. Examples include spatially varying transforms (SVT). In SVT, as illustrated in Figure 5B, the CUs are divided into two equal regions, horizontally or vertically, and only one of the regions is transformed into the frequency domain. A transformation base type can be configured for each region. For example, DST7 and DST8 are used. In this example, only one of these two regions in the CU is transformed, and the other is not transformed. However, both of these regions can be transformed. Additionally, the division method is not limited to dividing into two equal regions, and can be more flexible. For example, the CU may be divided into four equal regions, or the information indicating the division may be separately encoded and signaled in the same way as the CU division. It should be noted that SVT is also referred to as sub-block transformation (SBT). Quantizer

[00189] The quantizer 108 quantizes the transformation coefficients sent from the transformer 106. More specifically, the quantizer 108 digitizes, in a determined digitization order, the transformation coefficients of the current block, and quantizes the digitized transformation coefficients based on in the quantization parameters (QP) corresponding to the transformation coefficients. The quantizer 108 then sends the quantized transformation coefficients (hereinafter also referred to as quantized coefficients) of the current block to the entropy encoder 110 and the inverse quantizer 112.

[00190] A determined digitization order is an order of quantization/inverse quantization transformation coefficients. For example, a given scan order is defined as ascending frequency order (from low to high frequency) or descending frequency order (from high to low frequency).

[00191] A quantization parameter (QP) is a parameter that defines a quantization step (quantization width). For example, when the quantization parameter value increases, the quantization step also increases. In other words, when the quantization parameter value increases, the quantization error increases.

[00192] Additionally, a quantization matrix can be used for quantization. For example, various types of quantization matrices can be used corresponding to frequency transform sizes, such as 4 x 4 and 8 x 8, prediction modes, such as intraprediction and interprediction, and pixel components, such as as pixel components of luminescence and chrominance. It should be noted that quantization means scan values sampled at predetermined intervals corresponding to predetermined levels. In this technical field, quantization can be represented as other expressions, such as rounding and scaling.

[00193] Methods using quantization matrices include a method using a quantization matrix that has been determined directly on the encoder side and a method using a quantization matrix that has been configured as a standard (standard matrix). On the encoder side, a quantization matrix suitable for the characteristics of an image can be determined by directly configuring a quantization matrix. This case, however, has a disadvantage of increasing a coding amount to encode the quantization matrix.

[00194] There is a method of quantization of a high-frequency coefficient and a low-frequency coefficient without using a quantization matrix. It should be noted that this method is equivalent to a method using a quantization matrix (flat matrix) whose coefficients have the same value.

[00195] The quantization matrix can be specified using, for example, a sequence parameter set (SPS) or an image parameter set (PPS). SPS includes a parameter that is used for a sequence, and PPS includes a parameter that is used for an image. Each of SPS and PPS can simply be referred to as a set of parameters. Entropy Encoder

[00196] The entropy encoder 110 generates a coded signal (coded bit sequence) based on the quantized coefficients that have been recorded from the quantizer 108. More specifically, the entropy encoder 110, for example, binarizes the quantized coefficients, and Arithmetically encodes the binary signal, and sends a compressed bit sequence. Inverse Quantizer

[00197] The inverse quantizer 112 inversely quantizes the quantized coefficients that were recorded from the quantizer 108. More specifically, the inverse quantizer 112 inversely quantizes, in a determined digitization order, the quantized coefficients of the current block. The inverse quantizer 112 then sends the inverse quantized transformation coefficients of the current block to the inverse transformer 114. Inverse Transformer

[00198] The inverse transformer 114 restores the (residual) prediction errors by inversely transforming the transformation coefficients that were recorded from the inverse quantizer 112. More specifically, the inverse transformer 114 restores the prediction errors of the current block by applying an inverse transformation that corresponds to the transformation applied by transformer 106 on the transformation coefficients. The inverse transformer 114 then sends the restored prediction errors to the adder 116.

[00199] It should be noted that since information is lost in quantization, the restored prediction errors do not match the prediction errors calculated by subtractor 104. In other words, the restored prediction errors typically include quantization errors.

Adder

[00200] Adder 116 reconstructs the current block by adding prediction errors that were recorded from inverse transformer 114 and prediction samples that were recorded from prediction controller 128. Adder 116 then sends the reconstructed block for block memory 118 and loop filter 120. A reconstructed block is also referred to as a local decoded block.

Block Memory

[00201] Block memory 118 is, for example, the store for storing blocks in an image to be encoded (hereinafter referred to as a current image), which is referred to in intraprediction. More specifically, block memory 118 stores the reconstructed blocks sent from adder 116.

Frame Memory

[00202] Frame memory 122 is, for example, the store for storing reference images for use in interforecasting, and is also referred to as a temporary frame store. More specifically, frame memory 122 stores the reconstructed blocks filtered by circuit filter 120.

Circuit Filter

[00203] Loop filter 120 applies a loop filter to the blocks reconstructed by adder 116, and sends the filtered reconstructed blocks to frame memory 122. A loop filter is a filter used in a coding circuit (in-loop filter ), and includes, for example, a deblocking filter (DF or DBF), an adaptive sample offset (SAO), and an adaptive loop filter (ALF).

[00204] In an ALF, a mean square error filter to remove compression artifacts is applied. For example, a filter, selected from a plurality of filters based on the direction and activity of local gradients, is applied to each of the 2 x 2 sub-blocks in the current block.

[00205] More specifically, each sub-block (e.g., each 2 x 2 sub-block) is categorized into one of a plurality of classes (e.g., 15 or 25 classes). Sub-block classification is based on gradient directionality and activity. For example, the classification index C (e.g., C = 5D + A) is derived based on gradient directionality D (e.g., 0 to 2 or 0 to 4) and gradient activity A (e.g., 0 to 4). Then, based on the classification index C, each sub-block is categorized into one of a plurality of classes.

[00206] For example, the directionality of the gradient D is calculated by comparing the gradients of a plurality of directions (e.g., the horizontal, vertical, and two diagonal directions). Furthermore, for example, gradient activity A is calculated by adding gradients from a plurality of directions and quantizing the result of the addition.

[00207] The filter to be used for each sub-block is determined among the plurality of filters based on the result of such categorization.

[00208] The filter format to be used in an ALF, for example, is a circular symmetric filter format. Figures 6A to 6C illustrate examples of filter formats used in ALFs. Figure 6a illustrates a 5 x 5 diamond-shaped filter, Figure 6B illustrates a 7 x 7 diamond-shaped filter, and Figure 6C illustrates a 9 x 9 diamond-shaped filter. Filter format is typically signaled at the image level. It should be noted that signaling such information, indicating the filter format, does not necessarily need to be performed at the image level, and can be performed at another level (e.g., at the sequence level, slice level, tile, CTU level or CU level).

[00209] The ON or OFF of ALF is determined, for example, at the image level or CU level. For example, the decision of whether to apply ALF to luminescence can be made at the image level. Information indicating ALF ON or OFF is normally signaled at the image level or CU level. It should be noted that signaling information indicating the ON or OFF of ALF does not necessarily need to be performed at the image level or CU level, and can be performed at another level (e.g., at the sequence level, slice, tile level or CTU level).

[00210] The set of coefficients for the plurality of selectable filters (e.g., from fifteen to twenty-five filters) is typically signaled at the image level. It should be noted that coefficient set signaling does not necessarily need to be performed at the image level, and can be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level , CU level or sub-block level).

Circuit Filter > Unblock Filter

[00211] In a deblocking filter, the circuit filter 120 performs a filtering process at a block boundary in a reconstructed image so as to reduce the distortion that occurs at the block boundary.

[00212] Figure 7 is a block diagram illustrating an example of a specific configuration of circuit filter 120 that functions as an unblocking filter.

[00213] Circuit filter 120 includes a limit determiner 1201; a filter determiner 1203; a filtering executor 1205; a process determiner 1208; a filter characteristic determiner 1207; and switches 1202, 1204 and 1206.

[00214] Boundary determiner 1201 determines whether a pixel to be filtered for deblocking (i.e., a current pixel) is present at the edge of a block boundary. The limit determiner 1201 then sends the determination result to the switch 1202 and the processing determiner 1208.

[00215] In the case in which the boundary determiner 1201 has determined that a current pixel is present around a block boundary, the switch 1202 sends an unfiltered image to the switch 1204. In the opposite case, in which the boundary determiner 1201 has determined that no current pixels are present around a block boundary, switch 1202 sends an unfiltered image to switch 1206.

[00216] Filter determiner 1203 determines whether to perform deblocking filtering of the current pixel, based on the pixel value of at least one surrounding pixel located around the current pixel. The filter determiner 1203 then sends the determination result to the switch 1204 and the processing determiner 1208.

[00217] In the case in which the filter determiner 1203 has determined to perform the current pixel deblocking filtering, the switch 1204 sends the unfiltered image obtained through the switch 1202 to the filtering executor 1205. In the opposite case, in which the filter determiner 1203 has determined not to perform current pixel deblocking filtering, switch 1204 sends the unfiltered image obtained through switch 1202 to switch 1206.

[00218] When obtaining the unfiltered image through switches 1202 and 1204, the filtering executor 1205 performs, for the current pixel, the deblocking filtering with the filter characteristic determined by the filter characteristic determinator 1207. The deblocking executor filtering 1205 then sends the filtered pixel to switch 1206.

[00219] Under the control of the processing determiner 1208, the switch 1206 selectively outputs a pixel that has not been deblocking-filtered and a pixel that has been deblocking-filtered by the filtering executor 1205.

[00220] The processing determiner 1208 controls the switch 1206 based on the results of the determinations performed by the threshold determiner 1201 and filter determiner 1203. In other words, the processing determiner 1208 causes the switch 1206 to output the pixel that was deblocking filtered when the threshold determiner 1201 has determined that the current pixel is present around the block boundary and the filter determiner 1203 has determined to perform deblocking filtering of the current pixel. Additionally, in addition to the above case, the processing determiner 1208 causes the switch 1206 to output the pixel that has not been filtered by deblocking. A filtered image is sent from switch 1206 by repeatedly sending a pixel in this way.

[00221] Figure 8 is a diagram indicating an example of an unblocking filter having a symmetric filtering characteristic with respect to a block boundary.

[00222] In a deblocking filter process, one of two deblocking filters having different characteristics, that is, a strong filter and a weak filter, is selected using pixel values and quantization parameters. In the case of the strong filter, the pixels p0 to p2 and the pixels q0 to q2 are present across a block boundary as illustrated in Figure 8, the pixel values of the respective pixel q0 to q2 are changed to the pixel values q' 0 to q'2 by performing the computations according to the expressions below.

[00223] It should be noted that, in the above expressions, p0 to p2 and q0 to q2 are the pixel values of the respective pixels p0 to p2 and pixels q0 to q2. Additionally, q3 is the pixel value of the neighboring pixel q3 located on the opposite side of the pixel q2 with respect to the block boundary. Additionally, on the right side of each of the expressions, the coefficients that are multiplied by respective pixel values of the pixels to be used for deblocking filtering are filter coefficients.

[00224] Additionally, in deblocking filtering, fixing can be performed so that the calculated pixel values do not change through a threshold value. In the fixing process, the pixel values calculated according to the above expressions are fixed to a value obtained according to "a pre-computation pixel value +/- 2 x a threshold value" using the threshold value determined based in a quantization parameter. This way, excessive smoothing can be avoided.

[00225] Figure 9 is a diagram to illustrate a block boundary in which an unblocking filter process is performed. Figure 10 is a diagram indicating examples of Bs values.

[00226] The block boundary at which the deblocking filter process is performed is, for example, a boundary between prediction units (PU) having 8 x 8 pixel blocks, as illustrated in Figure 9, or a boundary between transformation units (TU). The deblocking filter process is performed on four-row or four-column units. First, the threshold intensity values (Bs) are determined as indicated in Figure 10 for the P block and Q block illustrated in Figure 9.

[00227] According to the Bs values in Figure 10, whether the filter processes of unblocking the block boundaries belonging to the same image using different intensities are carried out is determined. The deblocking filter process for a chrominance signal is performed when a Bs value is equal to 2. The deblocking filter process for a luminescence signal is performed when a Bs value is equal to 1 or more and a given condition is satisfied. It should be noted that the conditions for determining Bs values are not limited to those indicated in Figure 10, and a Bs value can be determined based on another parameter.

Forecast Processor (Intraforecaster, interforecaster, forecast controller)

[00228] Figure 11 is a diagram illustrating an example of a process performed by the prediction processor of the encoder 100. It should be noted that the prediction processor includes all or part of the following constituent elements: intrapredictor 124; interforecaster 126; and forecast controller 128.

[00229] The prediction processor generates a prediction image of a current block (Step Sb_1). This prediction image is also referred to as a prediction signal or a prediction block. It should be noted that the forecast signal is, for example, an intraforecast signal or an interforecast signal. Specifically, the prediction processor generates the prediction image of the current block using a reconstructed image that has already been obtained by generating a prediction block, generating a difference block, generating a coefficient block, restoring a difference, and generation of a decoded image block.

[00230] The reconstructed image can be, for example, an image in a reference image, or an image of a block encoded in a current image that is the image including the current block. The block encoded in the current image is, for example, a neighboring block of the current block.

[00231] Figure 12 is a diagram illustrating another example of a process performed by the encoder prediction processor 100.

[00232] The prediction processor generates a prediction image using a first method (Step Sc_1a), generates a prediction image using a second method (Step Sc_1b), and generates a prediction image using a third method (Step Sc_1c). The first method, the second method and the third method may be mutually different methods for generating a prediction image. Each of the first through third methods may be an interforecast method, an intraforecast method, or another forecast method. The reconstructed image described above can be used in these prediction methods.

[00233] Next, the prediction processor selects any one of a plurality of prediction methods generated in Steps Sc_1a, Sc_1b and Sc_1c (Step Sc_2). The selection of the forecast image, that is, the selection of a method or a mode for obtaining a final forecast image, can be carried out by calculating a cost for each of the generated forecast images and based on the cost. Alternatively, the selection of the prediction image can be done based on a parameter that is used in an encoding process. The encoder 100 may transform information for identifying a selected prediction image, a method, or a mode into a coded signal (also referred to as a coded bit sequence). The information may be, for example, an indicator or similar. In this way, the decoder is able to generate a prediction image in accordance with the method or mode selected based on the information in encoder 100. It should be noted that in the example illustrated in Figure 12, the prediction processor selects any of the forecast images after the forecast images are generated using the respective methods. However, the prediction processor may select a method or mode based on a parameter for use in the coding process described above prior to generating the prediction images, and may generate a prediction image in accordance with the selected method or mode. .

[00234] For example, the first method and the second method may be intraforecast and interforecast, respectively, and the prediction processor may select a final prediction image for a current block from the prediction images generated according to the prediction methods. forecast.

[00235] Figure 13 is a diagram illustrating another example of a process performed by the encoder prediction processor 100.

[00236] First, the prediction processor generates a prediction image using the intraforecast (Step Sd_1a), and generates a prediction image using the interforecast Step Sd_1b). It should be noted that the forecast image generated by intraforecast is also referred to as an intraforecast image, and the forecast image generated by interforecast is also referred to as an interforecast image.

[00237] Next, the prediction processor evaluates each of the intraforecast image and the interforecast image (Step Sd_2). A cost can be used in the evaluation. In other words, the prediction processor calculates the cost C for each of the intraforecast image and the interforecast image. The cost C is calculated according to an expression from an R-D optimization model, for example, C = D + À x R. In this expression, D indicates a coding distortion of a prediction image and is represented as, e.g. a sum of the absolute differences between the pixel value of a current block and the pixel value of a prediction image. Additionally, R indicates a predicted coding amount of a prediction image, specifically, the coding amount required to encode motion information to generate a prediction image, etc. Additionally, À indicates, for example, a multiplier according to the Lagrange multiplier method.

[00238] The prediction processor then selects the prediction image for which the lowest cost C was calculated from among the intraforecast image and the interforecast image, as the final prediction image for the current block (Step Sd_3). In other words, the prediction method or prediction image generation mode for the current block is selected.

Intraforecaster

[00239] The intraforecaster 124 generates a prediction signal (intraprediction signal) by performing intraprediction (also referred to as intraframe prediction) of the current block by reference to a block or blocks in the current image and stored in block memory 118. More specifically , the intrapredictor 124 generates an intraprediction signal by performing the intraprediction by reference to samples (e.g., luminescence and/or chrominance values) of a block or blocks neighboring the current block, and then sends the intraprediction signal to the prediction controller 128.

[00240] For example, the intraforecaster 124 performs intraforecast by using one mode among a plurality of intraforecast modes that have been predefined. Intraforecast modes include one or more non-directional forecast modes and a plurality of directional forecast modes.

[00241] The one or more non-directional prediction modes include, for example, the flat prediction mode and the DC prediction mode defined in H.265/High Efficiency Video Coding (HEVC) standard.

[00242] The plurality of directional prediction modes includes, for example, the thirty-three directional prediction modes defined in the H.265/HEVC standard. It should be noted that the plurality of directional forecast modes may additionally include thirty-two directional forecast modes in addition to the thirty-three directional forecast modes (for a total of sixty-five directional forecast modes). Figure 14 is a diagram illustrating sixty-seven total intraforecast modes used in intraforecast (two non-directional forecast modes and sixty-five directional forecast modes). The solid arrows represent the thirty-three directions defined in the H.265/HEVC standard, and the dashed arrows represent the additional thirty-two directions (the two non-directional prediction modes are not illustrated in Figure 14).

[00243] In various types of assembly examples, a luminescence block may be referred to as an intraprediction of a chrominance block. In other words, a chrominance component of the current block can be predicted based on a luminescence component of the current block. Such an intraforecast is also referred to as a cross-component linear model (CCLM) forecast. The intraprediction mode for a chrominance block in which such luminescence block is referred to (also referred to as, for example, a CCLM mode) can be added as one of the intraprediction modes to the chrominance blocks.

[00244] The intrapredictor 124 may correct the intrapredicted pixel values based on the horizontal/vertical reference pixel gradients. The intraforecast accompanied by this type of correction is also referred to as a position-dependent intraforecast combination (PDPC). Information indicating whether PDPC applies (referred to as, for example, a PDPC indicator) is typically signaled at the CU level. It should be noted that signaling of such information does not necessarily need to be performed at the CU level, and may be performed at another level (e.g., at the sequence level, image level, slice level, tile level or CTU level).

Interforecaster

[00245] The interpredictor 126 generates a prediction signal (interprediction signal) by performing interprediction (also referred to as interframe prediction) of the current block by reference to a block or blocks in a reference image, which is different from the current image and is stored in frame memory 122. Interprediction is performed in units of a current block or a current sub-block (e.g., a 4 x 4 block) in the current block. For example, interpredictor 126 performs motion estimation on a reference image for the current block or current sub-block, and discovers a reference block or sub-block that best matches the current block or sub-block. chain. Interpredictor 126 then obtains motion information (e.g., a motion vector) that compensates for a movement or change from the reference block or subblock to the current block or subblock. The inter-predictor 126 generates an inter-prediction signal of the current block or sub-block by performing motion compensation (or motion prediction) based on the motion information. The interforecaster 126 sends the generated interforecast signal to the forecast controller 128.

[00246] Motion information used in motion compensation can be signaled as interprediction signals in various ways. For example, a motion vector can be signaled. As another example, the difference between a motion vector and a motion vector predictor can be signaled.

Basic Interforecast Flow

[00247] Figure 15 is a flowchart illustrating a basic interforecast processing flow.

[00248] First, the interforecaster 126 generates a prediction signal (Steps Se_1 to Se_3). Next, subtractor 104 generates the difference between a current block and a prediction image as a prediction residual (Step Se_4).

[00249] Here, in generating the prediction image, the interforecaster 126 generates the prediction image by determining a motion vector (MV) of the current block (Steps Se_1 and Se_2) and motion compensation (Step Se_3). Additionally, in determining an MV, the interforecaster 126 determines the MV by selecting a motion vector candidate (MV candidate) (Step Se_1) and deriving an MV (Step Se_2). The selection of the MV candidate is done, for example, by selecting at least one MV candidate from a list of MV candidates. Alternatively, in deriving an MV, the interpredictor 126 may additionally select at least one MV candidate from the at least one MV candidate, and determine the at least one selected MV candidate as the MV for the current block. Alternatively, interpredictor 126 may determine the MV for the current block by performing the estimate on a reference image region specified by each of the at least one selected MV candidate. It should be noted that the estimation on a reference image region can be referred to as the motion estimation.

[00250] Additionally, although Steps Se_1 to Se_3 are performed by interpredictor 126 in the example described above, a process that is, for example, Step Se_1, Step Se_2 or similar can be performed by another constituent element included in encoder 100.

Motion Vector Derivation Flow

[00251] Figure 16 is a flowchart illustrating an example of deriving motion vectors.

[00252] Interpredictor 126 derives an MV from a current block in a motion information encoding mode (e.g., an MV). In this case, for example, motion information is encoded as a prediction parameter and is signaled. In other words, the encoded motion information is included in a coded signal (also referred to as a coded bit sequence).

[00253] Alternatively, interpredictor 126 derives a MV in a mode in which motion information is not encoded. In this case, no motion information is included in an encoded signal.

[00254] Here, the MV derivation modes include a normal intermode, a mixing mode, a FRUC mode, an affine mode, etc. which will be described later. The modes in which motion information is encoded, among the modes, include the normal intermode, the mixing mode, the affine mode (specifically, an affine intermode and an affine mixing mode), etc. It should be noted that the motion information may include not only an MV, but also motion vector predictor selection information which will be described later. Modes in which no motion information is encoded include FRUC mode, etc. The interpredictor 126 selects a mode to derive a MV of the current block from the modes, and derives the MV of the current block using the selected mode.

[00255] Figure 17 is a flowchart illustrating another example of deriving motion vectors.

[00256] Interpredictor 126 derives an MV from a current block in a mode in which an MV difference is encoded. In this case, for example, the MV difference is encoded as a prediction parameter, and is signaled. In other words, the coded MV difference is included in a coded signal. The MV difference is the difference between the MV of the current block and the MV predictor.

[00257] Alternatively, interpredictor 126 derives an MV in a mode in which no MV difference is encoded. In this case, no coded MV differences are included in a coded signal.

[00258] Here, as described above, the MV derivation modes include normal intermode, mixing mode, FRUC mode, affine mode, etc., which will be described later. The modes in which an MV difference is encoded among the modes include the normal intermode, the affine mode (specifically, the affine intermode), etc. Modes in which no MV difference is encoded include FRUC mode, mixing mode, affine mode (specifically, affine mixing mode), etc. The interpredictor 126 selects a mode for deriving an MV of the current block from the plurality of modes, and derives the MV of the current block using the selected mode.

Motion Vector Derivation Flow

[00259] Figure 18 is a flowchart illustrating another example of deriving motion vectors. MV derivation modes that are interprediction modes include a plurality of modes and are roughly divided into modes in which an MV difference is encoded and modes in which no motion difference is encoded. Modes in which no MV difference is encoded include mixing mode, FRUC mode, affine mode (specifically, affine mixing mode), etc. These modes will be described in detail later. Simply put, mixing mode is a way to derive an MV from a current block by selecting a motion vector from a coded surrounding block, and FRUC mode is a way to derive an MV from a current block by performing the estimation between coded regions. Affine mode is a way to derive, as an MV from a current block, a motion vector of each of a plurality of sub-blocks included in the current block, assuming affine transformation.

[00260] More specifically, when the interforecast mode information indicates 0 (0 in Sf_1), the interforecaster 126 derives a motion vector using the mixing mode (Sf_2). When the interforecast mode information indicates 1 (1 in Sf_1), the interforecaster 126 derives a motion vector using the FRUC mode (Sf_3). When the interforecast mode information indicates 2 (2 in Sf_1), the interforecaster 126 derives a motion vector using the affine mode (specifically, the affine mixing mode) (Sf_4). When the interforecast mode information indicates 3 (3 in Sf_1), the interforecaster 126 derives a motion vector using a mode in which an MV difference is encoded (e.g., a normal intermode (Sf_5)).

MV Derivation > Normal Intermode

[00261] Normal intermode is an interprediction mode for deriving an MV from a current block by discovering a block similar to the image of the current block from a reference image region specified by an MV candidate. In this normal intermode, a MV difference is encoded.

[00262] Figure 19 is a flowchart illustrating an example of interprediction in normal inter-mode.

[00263] First, the interpredictor 126 obtains a plurality of MV candidates for a current block based on information, such as the MVs of a plurality of temporally or spatially encoded blocks, surrounding the current block (Step Sg_1). In other words, interforecaster 126 generates a list of MV candidates.

[00264] Next, interpredictor 126 extracts N (an integer equal to 2 or more) MV candidates from the plurality of MV candidates obtained in Step Sg_1, as motion vector predictor candidates (also referred to as motion vector candidates). MV predictor), according to a predetermined priority order (Step Sg_2). It should be noted that the priority order is determined in advance for each of the N MV candidates.

[00265] Next, the interpredictor 126 selects a motion vector predictor candidate from N motion vector predictor candidates, such as the motion vector predictor (also referred to as an MV predictor) of the current block ( Step Sg_3). At this time, the interpredictor 126 encodes, in a sequence, the motion vector predictor selection information to identify the selected motion vector predictor. It should be noted that the sequence is a coded signal or a coded bit sequence as described above.

[00266] Next, interpredictor 126 derives an MV from a current block by reference to a coded reference image (Step Sg_4). At this time, the interpredictor 126 additionally encodes, in sequence, the difference value between the derived MV and the motion vector predictor as an MV difference. It is noted that the encoded reference image is an image including a plurality of blocks that have been reconstructed after being encoded.

[00267] Finally, the interpredictor 126 generates a prediction image for the current block by performing motion compensation of the current block using the derived MV and the encoded reference image (Step Sg_5). It should be noted that the forecast image is an interforecast signal as described above.

[00268] Additionally, information indicating the interforecast mode (normal intermode in the example above) used to generate the forecast image is, for example, encoded as a forecast parameter.

[00269] It should be noted that the list of MV candidates can also be used as a list for use in another mode. Additionally, processes related to the list of MV candidates can be applied to processes related to the list for use in another mode. Processes related to the MV candidate list include, for example, extracting or selecting a MV candidate from the MV candidate list, reordering the MV candidates, or deleting a MV candidate.

MV Derivation > Mixing Mode

[00270] The mixing mode is an interprediction mode for selecting an MV candidate from a list of MV candidates as an MV of a current block, thereby deriving the MV.

[00271] Figure 20 is a flowchart illustrating an example of interprediction in mixing mode.

[00272] First, the interpredictor 126 obtains a plurality of MV candidates for a current block based on information, such as MVs among a plurality of temporally or spatially encoded blocks, that surround the current block (Step Sh_1). In other words, interforecaster 126 generates a list of MV candidates.

[00273] Next, the interpredictor 126 selects a MV candidate from the plurality of MV candidates obtained in Step Sh_1, thus deriving an MV from the current block (Step Sh_2). At this time, the interpredictor 126 encodes, in a sequence, the MV selection information to identify the selected MV candidate.

[00274] Finally, the interpredictor 126 generates a prediction image for the current block by performing motion compensation of the current block using the derived MV and the encoded reference image (Step Sh_3).

[00275] Additionally, information indicating the interforecast mode (mixing mode in the example above), used to generate the forecast image and included in the encoded signal is, for example, encoded as a forecast parameter.

[00276] Figure 21 is a diagram to illustrate an example of a process of deriving a motion vector from a current image in blending mode.

[00277] First, a MV candidate list, in which MV predictor candidates are registered, is generated. Examples of MV predictor candidates include spatially neighboring MV predictors that are MVs of a plurality of spatially located coded blocks surrounding a current block; temporally neighboring MV predictors, which are MVs of surrounding blocks onto which the position of a current block in a coded reference image is projected; combined MV predictors which are MVs generated by combining the MV value of a spatially neighboring MV predictor and the MV of a temporally neighboring MV predictor; and a zero MV predictor which is an MV having a value of zero.

[00278] Next, an MV predictor is selected from a plurality of MV predictors registered in a list of MV predictors, and the selected MV predictor is determined as the MV of a current block.

[00279] Additionally, the variable length encoder describes and encodes, in a sequence, merge_idx which is a signal indicating which MV predictor was selected.

[00280] It should be noted that the MV predictors recorded in the list of MV predictors described in Figure 21 are examples. The number of MV predictors may be different from the number of MV predictors in the diagram, the MV predictor list may be configured such that some of the MV predictor types in the diagram may not be included, or that one or more MV predictors, in addition to the types of MV predictors in the diagram, are included.

[00281] A final MV can be determined by performing a dynamic motion vector update (DMVR) process to be described later using the MV of the current block derived in mixing mode.

[00282] It should be noted that the MV predictor candidates are the MV candidates described above, and the list of MV predictors is the list of MV candidates described above. It should be noted that the list of MV candidates can be referred to as a candidate list. Additionally, merge_idx is the VM selection information.

MV Derivation > FRUC Mode

[00283] Motion information can be derived on the decoder side without being signaled from the encoder side. It should be noted that, as described above, the mixing mode defined in the H.265/HEVC standard can be used. Additionally, for example, motion information may be derived by performing motion estimation on the decoder side. At this point, on the decoder side, motion estimation is performed without using any pixel value in a current block.

[00284] Here, a way of performing motion estimation on the decoder side is described. The mode of performing motion estimation on the decoder side may be referred to as a pattern-matched motion vector derivation (PMMVD) mode or a frame rate upconversion (FRUC) mode.

[00285] An example of a FRUC process is illustrated in Figure 22. First, a list of a plurality of candidates, each having a motion vector (MV) predictor (that is, a list of MV candidates that can also be used as a mix list), is generated by reference to a motion vector in a coded block that is spatially or temporally neighboring a current block (Step Si_1). Next, a best MV candidate is selected from the plurality of MV candidates registered in the MV candidate list (Step Si_2). For example, the evaluation values of the respective MV candidates included in the MV candidate list are calculated, and an MV candidate is selected based on the evaluation values. Based on the selected motion vector candidates, a motion vector for the current block is then derived (Step Si_4). More specifically, for example, the selected motion vector candidate (best MV candidate) is directly derived as the motion vector for the current block. Additionally, for example, the motion vector for the current block can be derived using pattern matching in a region surrounding a position in a reference image, where the position in the reference image corresponds to the candidate motion vector. selected. In other words, estimation using the combination of pattern and evaluation values can be performed in the region surrounding the best MV candidate, and when there is an MV that results in a better evaluation value, the best MV candidate can be updated to the MV that results in the best evaluation value, and the updated MV can be determined as the final MV for the current block. A configuration in which no such process for updating the best MV candidate to the MV having a better evaluation value is performed is also possible.

[00286] Finally, the interpredictor 126 generates a prediction image for the current block by performing motion compensation of the current block using the derived MV and the coded reference image (Step Si_5).

[00287] A similar process can be carried out on units of a sub-block.

[00288] Evaluation values can be calculated according to various types of methods. For example, a comparison is made between an image reconstructed in a region in a reference image corresponding to a motion vector and an image reconstructed in a given region (the region may be, for example, a region in another reference image or a region in a neighboring block of a current image, as indicated below). The difference between the pixel values of the two reconstructed images can be used for an evaluation value of the motion vectors. It should be noted that an appraisal value can be calculated using information beyond the difference value.

[00289] In the following, the pattern combination is described in detail. First, a MV candidate, included in a list of MV candidates (e.g., a mixing list), is selected as an estimation starting point by pattern matching. As the pattern combination, a first pattern combination or a second pattern combination is used. The first pattern combination and the second pattern combination are also referred to as bilateral combination and template combination, respectively. MV Derivation > FRUC >Bilateral Combination

[00290] In the first pattern combination, the pattern combination is performed between two blocks along a motion trajectory of a current block, which are two blocks in two different reference images. Accordingly, in the first pattern combination, a region in another reference image, along the movement path of the current block, is used as a determined region to calculate the evaluation value of the candidate described above.

[00291] Figure 23 is a diagram to illustrate an example of the first pattern combination (bilateral combination) between the two blocks in the two reference images along the main trajectory. As illustrated in Figure 23, in the first pattern combination, two motion vectors (MV0, MV1) are derived by estimating a pair that best matches the pairs in the two blocks in two different reference images (Ref0, Ref1), which are the two blocks along the movement path of the current block (Cur block). More specifically, a difference between the reconstructed image at a specified location in the first encoded reference image (Ref0) specified by a MV candidate, and the reconstructed image at a specified location in the second encoded reference image (Ref1), specified by a Symmetrical MV obtained by scaling the MV candidate in a display time interval is derived from the current block, and an evaluation value is calculated using the obtained difference value. It is excellent to select, as the final MV, the MV candidate that results in the best evaluation value among the plurality of MV candidates.

[00292] Considering a continuous motion trajectory, the motion vectors (MV0, MV1) that specify the two reference blocks are proportional to the temporal distances (TD0, TD1) between the current image (Cur Pic) and the two reference images (Ref0, Ref1). For example, when the current image is temporally located between the two reference images and the temporal distances from the current image to the two respective reference images are equal to each other, the mirror-symmetric bidirectional motion vectors are derived from the first combination of standard.

MV Derivation > FRUC >Template Combination

[00293] In the second pattern combination (template combination), the pattern combination is performed between a block in a reference image and a template in the current image (the template is a neighboring block of the current block in the current image (the block neighbor is, for example, a top and/or left neighbor block)). Accordingly, in the second pattern combination, the block neighboring the current block in the current image is used as the determined region to calculate the evaluation value of the candidate described above.

[00294] Figure 24 is a diagram to illustrate an example of pattern combination (template combination) between a template and a current image and a block in a reference image. As illustrated in Figure 24, in the second pattern combination, the motion vector of the current block (Cur block) is derived by estimating, in the reference image (Ref0), the block that best matches the neighboring block of the current block in the current image. (Cur Pic). More specifically, it is excellent that the difference between a reconstructed image in a coded region that is neighboring either the left side or top, and a reconstructed image that is in a corresponding region in the coded reference image (Ref0) and is specified by a MV candidate are derived, an evaluation value is calculated using the obtained difference value, and the MV candidate that results in the best evaluation value among a plurality of MV candidates is selected as the best MV candidate.

[00295] Such information indicating whether the FRUC mode applies (referred to, for example, as a FRUC indicator) may be signaled at the CU level. Additionally, when the FRUC mode is applied (e.g., when a FRUC indicator is true), information indicating an applicable pattern combination method (the first pattern combination or the second pattern combination) may be signaled at the ASS. It should be noted that signaling of such information does not necessarily need to be performed at the CU level, and may be performed at another level (e.g., at the sequence level, at the image level, at the slice level, at the tile, at the CTU level or at the sub-block level).

MV Derivation > Affine Model

[00296] Next, the affine way to derive a motion vector in units of a sub-block, based on the motion vectors among a plurality of neighboring blocks, is described. This mode is also referred to as an affine motion compensation prediction mode.

[00297] Figure 25A is a diagram to illustrate an example of deriving a motion vector of each sub-block based on the motion vectors of a plurality of neighboring blocks. In Figure 25A, the current block includes sixteen 4 x 4 sub-blocks. Here, the motion vector V0 at a control point in the upper left corner in the current block is derived based on a motion vector from a neighboring block, and, similarly, the motion vector V1 at a control point in the upper right corner in the current block is derived based on a motion vector from a neighboring sub-block. The two motion vectors v0 and v1 are designed according to an expression (1A) given below, and the motion vectors (vx, vy) for the respective sub-blocks in the current block are derived. Mathematics 1

[00298] Here, x and y indicate the horizontal position and vertical position of the sub-block, respectively, and w indicates a predetermined weighting coefficient.

[00299] Such information indicating affine mode (e.g., referred to as an affine indicator) may be signaled at the CU level. It should be noted that the signaling of information indicating the related mode does not necessarily need to be performed at the CU level, and can be performed at another level (e.g., at the sequence level, at the image level, at the slice level, tile level, CTU level or sub-block level).

[00300] Additionally, the affine mode may include multiple modes for different methods for deriving motion vectors at the upper left and upper right control points. For example, the affine mode includes two modes which are the affine intermode (also referred to as an affine normal intermode) and the affine mixing mode.

MV Derivation > Affine Model

[00301] Figure 25B is a diagram to illustrate an example of deriving a motion vector of each sub-block in affine mode in which three control points are used. In Figure 25B, the current block includes sixteen 4 x 4 blocks. Here, the motion vector V0 at the upper left control point for the current block is derived based on a motion vector from a neighboring block and, from Likewise, the motion vector V1 at the top-right control point for the current block is derived based on a motion vector from a neighboring block, and the motion vector V2 at the bottom-left control point for the Current block is derived based on a motion vector of a neighboring block. Three motion vectors v0, v1 and v2 are designed according to an expression (1B) given below, and the motion vectors (vx, vy) for the respective sub-blocks in the current block are derived. Mathematics 2

[00302] Here, x and y indicate the horizontal position and vertical position of the center of the sub-block, respectively, w indicates the width of the current block, and h indicates the height of the current block.

[00303] Affine modes, in which different numbers of control points (e.g., two and three control points) are used, can be switched and signaled at the CU level. It should be noted that information indicating the number of control points in the affine mode used at the CU level may be signaled at another level (e.g., at the sequence level, image level, slice level, tile level, CTU level or sub-block level).

[00304] Additionally, such an affine mode, in which three control points are used, may include different modes for deriving the motion vectors at the upper left, upper right and lower left control points. For example, affine modes include two modes which are the affine intermode (also referred to as the affine normal intermode) and the affine mixing mode.

MV Derivation > Affine Mixing Mode

[00305] Figures 26A, 26B and 26C are conceptual diagrams to illustrate the end mixing mode.

[00306] As illustrated in Figure 26A, in affine mixing mode, for example, motion vector predictors at respective control points of a current block are calculated based on a plurality of motion vectors corresponding to the coded blocks, according to the affine mode among the coded block A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) which are neighbors of the block chain. More specifically, coded block A (left), block B (top), block C (top right), block D (bottom left), and block E (top left) are scanned in the order listed, and the first effective block coded according to the affine mode is identified. Motion vector predictors at the control points of the current block are calculated based on a plurality of motion vectors corresponding to the identified block.

[00307] For example, as illustrated in Figure 26B, when block A, which is a neighbor on the left side of the current block, was coded according to an affine mode in which two control points are used, the motion vectors v3 and v4, designed at the top-left corner position and the top-right corner position of the encoded block, including block A, are derived. The motion vector predictor v0 at the top left corner control point of the current block, and the motion vector predictor v1 at the top right corner control point of the current block are then calculated from the derived motion vectors. v3 and v4.

[00308] For example, as illustrated in Figure 26C, when block A, which is neighboring on the left side of the current block, was coded according to an affine mode in which three control points are used, the motion vectors v3, v4 and v5 designed at the upper left corner position, the upper right corner position, and the lower left corner position of the coded block, including block A, are derived. The motion vector predictor v0 at the top-left corner control point of the current block, the motion vector predictor v1 at the top-right corner control point of the current block, and the motion vector predictor v2 at the control point from the bottom left corner of the current block are then calculated from the derived motion vectors v3, v4, and v5.

[00309] It should be noted that this method of deriving motion vector predictors can be used to derive the motion vector predictors of the respective control points of the current block in Step Sj_i in Figure 29 described later.

[00310] Figure 27 is a flowchart illustrating an example of the affine mixing mode.

[00311] In affine mixing mode, first, the interpredictor 126 derives the MV predictors from respective control points of a current block (Step Sk_1). The control points are the top-left corner point of the current block and a top-right corner point of the current block, as illustrated in Figure 25A, or a top-left corner point of the current block, a top-right corner point of the current block. current, and a lower left corner point of the current block, as illustrated in Figure 25B.

[00312] In other words, as illustrated in Figure 26A, the interpredictor 126 checks coded block A (left), block B (top), block C (top right), block D (bottom left), and the block E (top left) in the order listed, and identifies the first effective block encoded according to the affine mode.

[00313] When block A is identified and block A has two control points, as illustrated in Figure 26B, the interpredictor 126 calculates the motion vector v0 at the upper left corner control point of the current block and the motion vector v1 at the upper right control point of the current block, from the motion vectors v3 and v4 at the upper left and upper right corner of the encoded block, including block A. For example, the interpredictor 126 calculates the movement v0 at the control point in the upper left corner of the current block and the movement vector v1 at the control point in the upper right corner of the current block by the projection of the movement vectors v3 and v4 in the upper left corner and in the upper right corner of the block encoded in the current block.

[00314] Alternatively, when block A is identified and block A has three control points, as illustrated in Figure 26C, interpredictor 126 calculates the motion vector v0 at the upper left corner control point of the current block, the vector of motion vector v1 at the upper right corner control point of the current block, and the motion vector v2 at the lower left corner control point of the current block of the motion vectors v3, v4 and v5 in the upper left corner, the upper right corner , and the lower left corner of the coded block including block A. For example, the interpredictor 126 calculates the motion vector v0 at the upper left corner control point of the current block, the motion vector v1 at the upper left corner control point right of the current block, and the motion vector v2 at the bottom-left corner control point of the current block by projecting motion vectors v3, v4, and v5 onto the top-left corner, top-right corner, and bottom-left corner of the encoded block in the current block.

[00315] Next, the interpredictor 126 performs motion compensation for each of a plurality of sub-blocks included in the current block. In other words, the interpredictor 126 calculates, for each of the plurality of subblocks, a motion vector of the subblock as an affine MV, by using (i) two motion vector predictors v0 and v1 and the expression (1A) described above or (ii) three motion vector predictors v0, v1 and v2 and the expression (1B) described above (Step Sk_2). The interpredictor 126 then performs motion compensation of the sub-blocks using three affine MVs and coded reference images (Step Sk_3). As a result of this, motion compensation of the current block is performed to generate a prediction image of the current block.

MV Derivation > Affine Intermode

[00316] Figure 28A is a diagram to illustrate an affine intermode in which two control points are used.

[00317] In the affine intermode, as illustrated in Figure 28A, a motion vector, selected from motion vectors of coded block A, block B, and block C, which are neighbors of the current block, is used as the predictor of motion vector v0 at the control point in the upper left corner of the current block. Similarly, the motion vector selected from the encoded D-block and E-block motion vectors, which are neighbors of the current block, is used as the motion vector predictor v1 at the upper-right corner control point of the current block.

[00318] Figure 28B is a diagram to illustrate a related intermode in which three control points are used.

[00319] In the affine intermode, as illustrated in Figure 28B, a motion vector selected from the coded block A, block B, and block C motion vectors, which are neighbors of the current block, is used as the motion vector predictor. movement v0 at the upper left corner control point of the current block. Similarly, a motion vector selected from the encoded D-block and E-block motion vectors, which are neighbors of the current block, is used as the motion vector predictor v1 at the upper-right corner control point of the block. chain. Additionally, a motion vector selected from the encoded F-block and G-block motion vectors, which are neighbors of the current block, is used as the motion vector predictor v2 at the bottom-left corner control point of the current block. .

[00320] Figure 29 is a flowchart illustrating an example of a related intermode.

[00321] In the affine intermode, first, the interpredictor 126 derives the MV predictors (v0, v1) or (v0, v1, v2) from the respective two or three control points of a current block (Step Sj_1). The control points are a top-left corner point of the current block and a top-right corner point of the current block, as illustrated in Figure 25A, or a top-left corner point of the current block, a top-right corner point of the current block. current, and a lower left corner point of the current block, as illustrated in Figure 25B.

[00322] In other words, the interpredictor 126 derives the motion vector predictors (v0, v1) or (v0, v1, v2) of the respective two or three control points of the current block by selecting motion vectors from either of the blocks between the coded blocks in the vicinity of the respective control points of the current block illustrated in either of Figures 28A or 28B. At this time, the interpredictor 126 encodes, in a sequence, the motion vector predictor selection information, to identify the two selected motion vectors.

[00323] For example, the interpredictor 126 may determine, using a cost evaluation or the like, the block from which a motion vector, such as a motion vector predictor at a control point, is selected from among the coded blocks neighbors of the current block, and can describe, in a sequence of bits, an indicator that indicates which motion vector predictor was selected.

[00324] Next, interpredictor 126 performs motion estimation (Step Sj_3 and SJ_4) while updating a motion vector predictor selected or derived in Step Sj_1 (Step Sj_2). In other words, the interpredictor 126 calculates, as an affine MV, a motion vector from each of the sub-blocks that corresponds to an updated motion vector predictor, using expression (1A) or expression (1B) described above (Step Sj_3). The interforecaster 126 then performs motion compensation of the sub-blocks using these related MVs and the encoded reference images (Step Sj_4). As a result of this, for example, the interpredictor 126 determines the motion vector predictor that results in the lowest cost as a motion vector at a control point in a motion estimation circuit (Step Sj_5). At this time, the interpredictor 126 additionally encodes, in sequence, the difference value between the determined MV and the motion vector predictor as an MV difference.

[00325] Finally, the interpredictor 126 generates a prediction image for the current block by performing motion compensation of the current block using the determined MV and the coded reference image (Step Sj_6).

Derivation of MV > Affine Intermode

[00326] When affine modes, in which different numbers of control points (e.g., two and three control points) are used, can be switched and signaled at the CU level, the number of control points in a coded block and the number of control points in a current block may be different from each other. Figures 30A and 30B are conceptual diagrams to illustrate methods for deriving motion vector predictors at control points when the number of control points in a coded block and the number of control points in a current block are different from each other. other.

[00327] For example, as illustrated in Figure 30A, when a current block has three control points in the upper left corner, the upper right corner and the lower left corner, and block A, which is neighboring on the left side of the current block , were coded according to an affine mode, in which the two control points are used, the motion vectors v3 and v4, projected at the top left corner position and the top right corner position in the coded block, including block A , are derivatives. The motion vector predictor v0, at the top left corner control point of the current block, and the motion vector predictor v1 at the top right corner control point of the current block are then calculated from the motion vectors. derivatives v3 and v4. Additionally, the motion vector predictor v2 at the bottom left control point is calculated from the derived motion vectors v0 and v1.

[00328] For example, as illustrated in Figure 30B, when a current block has two control points in the upper left corner and the upper right corner, and block A, which is a neighbor on the left side of the current block, has been coded accordingly with affine mode, in which three control points are used, the motion vectors v3, v4 and v5, projected at the top left corner position, the top right corner position and the bottom left corner position in the coded block, including block A, are derivatives. The motion vector predictor v0 at the top left corner control point of the current block and the motion vector predictor v1 at the top right corner control point of the current block are then calculated from the derived motion vectors v3 , v4 and v5.

[00329] It should be noted that this method of deriving the motion vector predictors can be used to derive the motion vector predictors of the respective control points of the current block in Step Sj_1 in Figure 29.

MV Derivation > DMVR

[00330] Figure 31A is a diagram illustrating a relationship between mixing mode and DMVR.

[00331] Interpredictor 126 derives a motion vector of a current block according to the mixing mode (Step Sl_1). Next, interpredictor 126 determines whether to estimate a motion vector, that is, motion estimation (Step Sl_2). Here, when the motion estimate is determined not to be performed (not in Step Sl_2), the interpredictor 126 determines the motion vector derived in Step Sl_1 as the final motion vector for the current block (Step Sl_4). In other words, in this case, the motion vector of the current block is determined according to the mixing mode.

[00332] When determining whether to perform the motion estimate in Step Sl_1 (Yes in Step Sl_2), the interpredictor 126 derives the final motion vector for the current block by estimating a surrounding region of the reference image specified by the motion vector derived in Step Sl_1 (Step Sl_3). In other words, in this case, the motion vector of the current block is determined according to DMVR.

[00333] Figure 31B is a conceptual diagram to illustrate an example of a DMVR process for determining an MV.

[00334] First, (e.g., in mixing mode) the best MVP, which has been configured for the current block, is determined to be a MV candidate. A reference pixel is identified from a first reference image (L0), which is an image encoded in the L0 direction, according to a MV candidate (L0). Similarly, a reference pixel is identified from a second reference image (L1), which is an image encoded in the L1 direction, according to a MV candidate (L1). A template is generated by calculating an average of these reference pixels.

[00335] Next, each of the MV candidate surrounding regions of the first reference image (L0) and the second reference image (L1) is estimated, and the MV that results in the lowest cost is determined to be the final MV . It should be noted that the cost value can be calculated, for example, using a difference value between each of the pixel values in the template and a corresponding pixel value in the estimation region, the MV candidate values, etc.

[00336] It should be noted that the processes, configurations and operations described here are basically common between the encoder and a decoder to be described later.

[00337] Exactly the same processes described here do not always need to be carried out. Any process to allow derivation of the final MV by estimation in the surrounding regions of MV candidates can be used.

Motion Compensation > BIO/OBMC

[00338] Motion compensation involves a mode of generating a prediction image, and correcting the prediction image. The mode is, for example, BIO and OBMC to be described later.

[00339] Figure 32 is a flowchart illustrating an example of generating a prediction image.

[00340] The interforecaster 126 generates a prediction image (Step Sm_1), and corrects the prediction image according to any of the modes described above (Step Sm_2).

[00341] Figure 33 is a flowchart illustrating another example of generating a prediction image.

[00342] Interpredictor 126 determines a motion vector of a current block (Step Sn_1). Next, the interforecaster 126 generates a prediction image (Step Sn_2), and determines whether to perform a correction process (Step Sn_3). Here, when determining whether to perform a correction process (Yes in Step Sn_3), the interforecaster 126 generates the final prediction image by correcting the prediction image (Step Sn_4). When determining not to perform a correction process (Not in Step Sn_3), the interforecaster 126 sends the prediction image as the final prediction image without correction of the prediction image (Step Sn_5).

[00343] Additionally, motion compensation involves a way of correcting a luminescence of a prediction image when generating the prediction image. The mode is, for example, LIC to be described later.

[00344] Figure 34 is a flowchart illustrating another example of generating a prediction image.

[00345] Interpredictor 126 derives a motion vector from a current block (Step So_1). Next, the interforecaster 126 determines whether to perform a luminescence correction process (Step So_2). Here, when determining whether to perform a luminescence correction process (Yes in Step So_2), the interforecaster 126 generates the prediction image while performing a luminescence correction process (Step So_3). In other words, the prediction image is generated using LIC. When determining not to perform a luminescence correction process (Not in step So_2), the interforecaster 126 generates a prediction image by performing normal motion compensation without performing a luminescence correction process (Step So_4).

Motion Compensation > OBMC

[00346] It should be noted that an interprediction signal can be generated using the movement information for a neighboring block in addition to the movement information for the current block obtained from the movement estimate. More specifically, the interprediction signal can be generated in units of a sub-block in the current block by performing a weighted addition of a prediction signal based on the motion information obtained from the motion estimation (in the reference image). and a prediction signal based on movement information for a neighboring block (in the current image). Such interprediction (motion compensation) is also referred to as overlapping block motion compensation (OBMC).

[00347] In OBMC mode, information indicating a sub-block size for OBMC (referred to, for example, as an OBMC block size) may be signaled at the sequence level. Additionally, information indicating whether OBMC mode applies (referred to, for example, as an OBMC indicator) may be signaled at the CU level. It should be noted that signaling of such information does not necessarily need to be performed at the sequence level and CU level and can be performed at another level (e.g., at the image level, slice level, tile level, of CTU, or sub-block level).

[00348] The OBMC mode will be described in greater detail. Figures 35 and 36 are a flowchart and conceptual diagram to illustrate outlines of a predictive image correction process performed by an OBMC process.

[00349] First, as illustrated in Figure 36, a prediction image (Pred) is obtained through normal motion compensation using a motion vector (MV) designated for the processing target block (current). In Figure 36, the "MV" arrow points to a reference image and indicates what the current block of the current image refers to in order to obtain a prediction image.

[00350] Next, a prediction image (Pred_L) is obtained by applying a motion vector (MV_L) that has already been derived for the neighboring coded block on the left side of the current block to the current block (reusing the motion vector for the current block). The motion vector (MV_L) is indicated by an arrow "MV_L" which indicates a reference image of a current block. A first correction of a forecast image is performed by superimposing two forecast images Pred and Pred_L. This provides a boundary mixing effect between neighboring blocks.

[00351] Likewise, a prediction image (Pred_U) is obtained by applying a motion vector (MV_U) that has already been derived from the coded block, the top neighbor of the current block, to the current block (reusing the code vector movement to the current block). The motion vector (MV_U) is indicated by an arrow "MV_U" indicating a reference image of a current block. A second correction of a forecast image is performed by overlaying the Pred_U forecast image onto the forecast images (e.g., Pred and Pred_L) where the first correction was performed. This provides a boundary mixing effect between neighboring blocks. The prediction image obtained by the second correction is one in which the boundary between neighboring blocks has been blended (smoothed) and is thus the final prediction image of the current block.

[00352] Although the example above is a method of correcting two paths using the left and top neighboring blocks, it should be noted that the correction method can be the method of correcting three or more paths using also the block right neighbor and/or the lower neighbor block.

[00353] It is noted that the region in which such overlay is performed may only be part of a region close to a block boundary, rather than the pixel region of the entire block.

[00354] It should be noted that the prediction image correction process according to OBMC, for obtaining a prediction image Pred from a reference image, by superimposing the additional prediction image Pred_L and Pred_U, has been described above. However, when a prediction image is corrected based on a plurality of reference images, a similar process can be applied to each of the plurality of reference images. In such a case, after the corrected prediction images are obtained from the respective reference images, by performing OBMC image correction based on the plurality of reference images, the obtained corrected prediction images are further superimposed to obtain the final prediction image.

[00355] It should be noted that, in OBMC, the unit of a current block may be the unit of a prediction block or the unit of a sub-block obtained by further division of the prediction block.

[00356] An example of a method for determining whether an OBMC process applies is a method for using an obmc_flag which is a signal that indicates whether an OBMC process applies. As a specific example, an encoder determines whether the current block belongs to a region having complicated motion. The encoder sets obmc_flag to a value of "1" when the block belongs to a region having complicated motion and applies an OBMC process when decoding, and sets obmc_flag to a value of "0" when the block does not belong to a region possessing complicated motion and encodes the block without applying an OBMC process. The decoder switches between applying and not applying an OBMC process by decoding the obmc_flag written in the sequence (e.g., a compressed sequence) and decoding the block by switching between applying and not applying the OBMC process, according to the value of indicator.

[00357] Interprediction 126 generates a rectangular prediction image for a current rectangular block in the example above. However, the interpredictor 126 may generate a plurality of prediction images, each having a different shape from a rectangle to the current rectangular block, and may combine the plurality of prediction images to generate the final rectangular prediction image. The shape other than a rectangle could be, for example, a triangle.

[00358] Figure 37 is a diagram to illustrate the generation of two triangular prediction images.

[00359] Interforecaster 126 generates a triangular prediction image by performing motion compensation of a first partition having a triangular shape in a current block by using a first MV of the first partition to generate a triangular prediction image. Likewise, the interforecaster 126 generates a triangular prediction image by performing motion compensation of a second partition having a triangular shape in a current block by using a second MV of the second partition to generate a triangular prediction image. The interforecaster 126 then generates a prediction image having the same rectangular shape as the rectangular shape of the current block by combining these prediction images.

[00360] It should be noted that, although the first partition and the second partition are triangles in the example illustrated in Figure 37, the first partition and the second partition may be trapezoids, or other shapes different from each other. Additionally, although the current block includes two partitions in the example illustrated in Figure 37, the current block may include three or more partitions.

[00361] Additionally, the first partition and the second partition may overlap each other. In other words, the first partition and the second partition can include the same pixel region. In this case, a prediction image for a current block can be generated using a prediction image in the first partition and a prediction image in the second partition.

[00362] Additionally, despite an example, in which a prediction image is generated for each of the two partitions using interprediction, a prediction image can be generated for at least one partition using intraprediction.

Motion Compensation > BIO

[00363] Next, a method of deriving a motion vector is described. First, a way to derive a motion vector based on a model, assuming uniform linear motion, will be described. This mode is also referred to as a bidirectional optical flow (BIO) mode.

[00364] Figure 38 is a diagram to illustrate a model assuming uniform linear motion. In Figure 38, (vx, vy) indicates a velocity vector and TO and T1 indicate temporal distances between a current image (Cur Pic) and two reference images (Ref0, Ref1). (MVx0, MVy0) indicate the motion vectors corresponding to the reference image Ref0, and (MVx1, MVy1) indicate motion vectors corresponding to the reference image Ref1.

[00365] Here, considering the uniform linear movement exhibited by the velocity vectors (vx, vy), (MVx0, MVy0) and are represented as

[00366] Here, I(k) indicates a motion compensated luminescence value of the reference image k (k = 0, 1). This optical flow equation illustrates that the sum of (i) the time derived from the luminescence value, (ii) the product of the horizontal velocity and horizontal component of the spatial gradient of a reference image, and (iii) the product of the vertical velocity and vertical component of the spatial gradient of a reference image is equal to zero. A motion vector of each block obtained from, for example, a mix list can be fixed in units of a pixel, based on a combination of the optical flow equation and Hermite interpolation.

[00367] It should be noted that a motion vector may be derived on the decoder side using a method other than deriving a motion vector based on a model assuming uniform linear motion. For example, a motion vector can be derived in units of a sub-block based on the motion vectors of neighboring blocks.

Motion Compensation > LIC

[00368] In the following, an example of a mode, in which a prediction image is generated by using a local illumination compensation (LIC) process, will be described.

[00369] Figure 39 is a diagram to illustrate an example of a predictive image generation method using a luminescence correction process performed by a LIC process.

[00370] First, a MV is derived from a coded reference image, and a reference image corresponding to the current block is obtained.

[00371] Next, information indicating how the luminescence value changed between the reference image and the current image is extracted from the current block. This extraction is performed based on the luminescence pixel values for the encoded left neighboring reference region (surrounding reference region) and the encoded upper neighboring reference region (surrounding reference region), and the luminescence pixel value at position corresponding in the reference image specified by the derived MV. A luminescence correction parameter is calculated by using information that indicates how the luminescence value has changed.

[00372] The prediction image for the current block is generated by performing a luminescence correction process in which the luminescence correction parameter is applied to the reference image in the reference image specified by the MV.

[00373] It should also be noted that the shape of the surrounding reference region illustrated in Figure 39 is just an example; the surrounding reference region may have a different shape.

[00374] Furthermore, although the process in which a prediction image is generated from a singular reference image has been described herein, cases in which a prediction image is generated from a plurality of reference images may be described in the same way. The prediction image can be generated after carrying out a luminescence correction process of the reference images obtained from the reference images in the same way as described above.

[00375] An example of a method of determining whether a LIC process applies is a method for using a lic_flag, which is a signal that indicates whether the LIC process applies. As a specific example, the encoder determines whether the current block belongs to a region that has a change in luminescence. The encoder sets lic_flag to a value equal to "1" when the block belongs to a region having a luminescence change and applies an LIC process when encoding, and sets lic_flag to a value equal to "0" when the block does not belong to a region having a change in luminescence and encodes the current block without applying an LIC process. The decoder can decode lic_flag written in the sequence and decode the current block by switching between applying and not applying a LIC process according to the indicator value.

[00376] An example of a different method of determining whether a LIC process applies is a method of determining whether a LIC process has been applied to a surrounding block. In a specific example, when the mixing mode is used in the current block, whether a LIC process has been applied in encoding the surrounding coded block selected by deriving the MV in the mixing mode process will be determined. According to the result, coding is performed by switching between applying and non-applying an LIC process. It should be noted that, also in this example, the same processes are applied to the processes on the decoder side.

[00377] The luminescence correction process (LIC) described with reference to Figure 39 is described in detail below.

[00378] First, the interpredictor 126 derives a motion vector to obtain a reference image that corresponds to a current block to be encoded from a reference image that is a coded image.

[00379] Next, interpredictor 126 extracts information indicating how the luminescence value of the reference image was changed to the luminescence value of the current image, using the luminescence pixel value of a coded surrounding reference region, which is neighboring on the left side, or above, of the current block and the luminescence value at the corresponding position in the reference image specified by a motion vector, and calculates a luminescence correction parameter. For example, assume that the pixel luminescence value of a pixel determined in the surrounding reference region in the current image is p0, and that the pixel luminescence value of the pixel corresponding to the pixel determined in the surrounding reference region in the current image is reference is p1. The interprediction 126 calculates the coefficients A and B for optimization A x p1 + B = p0 as the luminescence correction parameter for a plurality of pixels in the surrounding reference region.

[00380] Next, the interpredictor 126 performs a luminescence correction process using the luminescence correction parameter for the reference image in the reference image specified by the motion vector, to generate a prediction image for the current block. For example, assume that the luminescence pixel value in the reference image is p2, and the luminescence-corrected luminescence pixel value of the prediction image is p3. The interforecaster 126 generates the prediction image after being subjected to the luminescence correction process by calculating A x p2 + B = p3 for each of the pixels in the reference image.

[00381] It should be noted that the shape of the surrounding reference region, illustrated in Figure 39, is an example; a different format in addition to the format of the surrounding reference region may be used. Additionally, part of the surrounding reference region illustrated in Figure 39 may be used. For example, a region having a given number of pixels extracted from each of a top neighboring pixel and a left neighboring pixel can be used as a surrounding reference region. Additionally, the surrounding reference region is not limited to a region that is neighboring the current block, and may be a region that is not neighboring the current block. In the example illustrated in Figure 39, the surrounding reference region in the reference image is a region specified by a motion vector in a current image, from a surrounding reference region in the current image. However, a region specified by another motion vector is also possible. For example, the other motion vector may be a motion vector in a surrounding reference region in the current image.

[00382] Although operations performed by encoder 100 have been described here, it should be noted that decoder 200 performs similar operations.

[00383] It should be noted that the LIC process can be applied not only to luminescence, but also to chrominance. At this time, a correction parameter can be derived individually for each of Y, Cb, and Cr, or a common correction parameter can be used for any of Y, Cb, and Cr.

[00384] Additionally, the LIC process can be applied to units of a sub-block. For example, a correction parameter can be derived using a surrounding reference region in a current sub-block and a surrounding reference region in a reference sub-block in a reference image specified by an MV of the sub-block. chain.

Forecast Controller

[00385] The interforecaster 128 selects one of an intraforecast signal (a signal sent from the intraforecaster 124) and an interforecaster signal (a signal sent from the interforecaster 126), and sends the selected signal to the subtractor 104 and adder 116 as a predictive signal.

[00386] As illustrated in Figure 1, in various types of assembly examples, the prediction controller 128 may send a prediction parameter that is recorded in the entropy encoder 110. The entropy encoder 110 may generate an encoded bit sequence ( or a sequence), based on the prediction parameter that is recorded from the prediction controller 128 and quantized coefficients that are recorded from the quantizer 108. The prediction parameter can be used in a decoder. The decoder may receive and decode the encoded bit sequence, and perform the same processes as the prediction processes performed by the intraforecaster 124, interforecaster 126, and prediction controller 128. The prediction parameter may include (i) a selection prediction signal (e.g., a motion vector, a forecast type, or a forecast mode used by the intraforecaster 124 or interforecaster 126), or (ii) an optional index, an indicator, or a value that is based on a forecasting process performed in each of the intraforecaster 124, interforecaster 126, and forecast controller 128, or indicating the forecasting process.

Encoder Assembly Example

[00387] Figure 40 is a block diagram illustrating an example assembly of encoder 100. Encoder 100 includes processor a1 and memory a2. For example, the plurality of constituent elements of the encoder 100 illustrated in Figure 1 are assembled in the processor a1 and memory a2 illustrated in Figure 40.

[00388] Processor a1 is the set of circuits that performs information processing and is accessible to memory a2. For example, the a1 processor is a dedicated or general circuitry that encodes a video. The processor a1 may be a processor such as a CPU. Additionally, the processor a1 may be an aggregate of a plurality of electronic circuits. Additionally, for example, the processor a1 may assume the roles of two or more constituent elements in addition to one constituent element for storing information from the plurality of constituent elements of the encoder 100 illustrated in Figure 1, etc.

[00389] Memory a2 is a dedicated or general memory for storing information that is used by processor a1 to encode a video. Memory a2 can be a set of electronic circuits, and can be connected to processor a1. Additionally, memory a2 can be included in processor a1. Additionally, memory a2 may be an aggregate of a plurality of electronic circuits. Additionally, memory a2 may be a magnetic disk, an optical disk, or the like, or it may be represented as a store, a recording medium, or the like. Additionally, memory a2 may be non-volatile memory or volatile memory.

[00390] For example, memory a2 can store a video to be encoded or a bit sequence corresponding to an encoded video. Additionally, memory a2 can store a program to cause processor a1 to encode a video.

[00391] Additionally, for example, memory a2 can assume the roles of two or more constituent elements to store information from the plurality of constituent elements of the encoder 100 illustrated in Figure 1, etc. Specifically, memory a2 can assume the roles of block memory 118 and frame memory 112 illustrated in Figure 1. More specifically, memory a2 can store a reconstructed block, a reconstructed image, etc.

[00392] It should be noted that, in the encoder 100, the entire plurality of constituent elements indicated in Figure 1, etc., may not be implemented and all the processes described above may not be carried out. Part of the constituent elements indicated in Figure 1, etc., may be included in another device, or part of the processes described above may be carried out by another device.

Decoder

[00393] Next, a decoder capable of decoding a coded signal (coded bit sequence) sent from the coder 100 described above will be described. Figure 41 is a block diagram illustrating a functional configuration of decoder 200 in accordance with this embodiment. Decoder 200 is a video decoder that decodes a video into units of a block.

[00394] As illustrated in Figure 41, decoder 200 includes entropy decoder 202, inverse quantizer 204, inverse transformer 206, adder 208, block memory 210, loop filter 212, frame memory 214 , the intraforecaster 216, the interforecaster 218 and the forecast controller 220.

[00395] Decoder 200 is implemented as, for example, a generic processor and memory. In this case, when a software program, stored in memory, is executed by the processor, the processor functions as the entropy decoder 202, inverse quantizer 204, inverse transformer 206, adder 208, loop filter 212, intrapredictor 216, interpredictor 218, and prediction controller 220. Alternatively, decoder 200 may be implemented as one or more dedicated electronic circuits corresponding to entropy decoder 202, inverse quantizer 204, inverse transformer 206, adder 208, circuit filter 212, intrapredictor 216, interpredictor 218, and controller forecast 220.

[00396] Hereinafter, a general flow of processes performed by decoder 200 is described and then each of the constituent elements included in decoder 200 will be described.

Decoding Process Overview

[00397] Figure 42 is a flowchart illustrating an example of a general decoding process performed by decoder 200.

[00398] First, the entropy decoder 202 of the decoder 200 identifies a division pattern of a block having a fixed size (128 x 128 pixels) (Step Sp_1_. This division pattern is a division pattern selected by the encoder 100. The decoder 200 then performs the processes of Step Sp_2 to Sp_6 for each of a plurality of blocks of the division pattern.

[00399] In other words, the entropy decoder 202 decodes (specifically, entropy decodes) the encoded quantized coefficients and a prediction parameter of a current block to be decoded (also referred to as a current block) (Step Sp_2).

[00400] Next, the inverse quantizer 204 performs the inverse quantization of the plurality of quantized coefficients, and the inverse transformer 206 performs the inverse transformation of the result, to restore a plurality of prediction residuals (i.e., a difference block) (Step Sp_3).

[00401] Next, the prediction processor, including all or part of intraforecaster 216, interforecaster 218, and prediction controller 220, generates a prediction signal (also referred to as a prediction block) from the current block (Step Sp_4).

[00402] Next, adder 208 adds the prediction block to the difference block to generate a reconstructed image (also referred to as a decoded image block) of the current block (Step Sp_5).

[00403] When the reconstructed image is generated, the circuit filter 212 performs filtering of the reconstructed image (Step Sp_6).

[00404] Decoder 200 then determines whether decoding of the entire image has finished (Step Sp_7). Upon determining that decoding has not yet been completed (Not at Step Sp_7), decoder 200 repeatedly executes processes beginning with Step Sp_1.

[00405] It should be noted that the processes of these Steps Sp_1 to Sp_7 can be carried out sequentially by decoder 200, or two or more processes can be carried out in parallel. The processing order of two or more of the processes can be modified.

Entropy Decoder

[00406] Entropy decoder 202 entropy decodes an encoded bit sequence. More specifically, for example, entropy decoder 202 arithmetically decodes a sequence of bits encoded into a binary signal. The entropy decoder 202 then debinarizes the binary signal. Thereby, the entropy decoder 202 sends the quantized coefficients of each block to the inverse quantizer 204. The entropy decoder 202 can send a prediction parameter included in a coded bit sequence (see Figure 1) to the intrapredictor 216, the interforecaster 218 and the prediction controller 220. The intraforecaster 216, the interforecaster 218, and the prediction controller 220 can perform the same prediction processes as those performed by the intraforecaster 124, the interforecaster 126, and the prediction controller 128 on the encoder side .

Inverse Quantizer

[00407] The inverse quantizer 204 inversely quantizes the quantized coefficients of a block to be decoder (hereinafter referred to as a current block) that are recorded from the decoder by entropy 202. More specifically, the inverse quantizer 204 inversely quantizes the quantized coefficients of the current block, based on the quantization parameters corresponding to the quantized coefficients. The inverse quantizer 204 then sends the inverse quantized transformation coefficients from the current block to the inverse transformer 206.

Inverse Transformer

[00408] The inverse transformer 206 restores prediction errors by inversely transforming the transformation coefficients that are recorded from the inverse quantizer 204.

[00409] For example, when information analyzed from a coded bit sequence indicates that EMT or AMT should be applied (e.g., when an AMT indicator is true), the inverse transformer 206 inversely transforms the transformation coefficients of the current block based on information indicating the type of transformation analyzed.

[00410] Furthermore, for example, when information, analyzed from a coded bit sequence, indicates that NSST should be applied, the inverse transformer 206 applies a secondary inverse transformation to the transformation coefficients.

Adder

[00411] Adder 208 reconstructs the current block by adding prediction errors that are recorded from inverse transformer 206 and prediction samples that are recorded from prediction controller 220. Adder 208 then sends the reconstructed block for block memory 210 and loop filter 212.

Block Memory

[00412] Block memory 210 is the store for storing the blocks in an image to be decoded (hereinafter referred to as a current image) and to be referred to in the intraprediction. More specifically, block memory 210 stores the reconstructed blocks sent from adder 208.

Circuit Filter

[00413] Circuit filter 212 applies a circuit filter to the blocks reconstructed by adder 208, and sends the filtered reconstructed blocks to frame memory 214, display device, etc.

[00414] When information indicating the ON or OFF of an ALF, analyzed from a coded bit sequence, indicates that an ALF is ON, a filter, among a plurality of filters, is selected based on the direction and activity of local gradients, and the selected filter is applied to the reconstructed block.

Frame Memory

[00415] Frame memory 214 is, for example, the store for storing reference images for use in interprediction, and is also referred to as a frame store. More specifically, frame memory 214 stores a reconstructed block filtered by circuit filter 212.

Forecast Processor (Intraforecaster, Interforecaster, Forecast Controller)

[00416] Figure 43 is a diagram illustrating an example of a process performed by a decoder prediction processor 200. It should be noted that the prediction processor includes all or part of the following constituent elements: intrapredictor 216; interforecaster 218 and forecast controller 220.

[00417] The prediction processor generates a prediction image of a current block (Step Sq_1). This prediction image is also referred to as a prediction signal or a prediction block. It should be noted that the forecast signal, for example, is an intraforecast signal or an interforecast signal. Specifically, the prediction processor generates the prediction image of the current block using a reconstructed image that has already been obtained by generating a prediction block, generating a difference block, generating a coefficient block, restoring a difference, and generation of a decoded image block.

[00418] The reconstructed image can be, for example, an image in a reference image, or an image of a block decoded in a current image which is the image that includes the current block. The block decoded in the current image is, for example, a neighboring block of the current block.

[00419] Figure 44 is a diagram illustrating another example of a process performed by the decoder prediction processor 200.

[00420] The prediction processor determines a method or mode of generating a prediction image (Step Sr_1). For example, the method or mode may be determined based on, for example, a prediction parameter, etc.

[00421] When determining a first method as a mode of generating a prediction image, the prediction processor generates a prediction image according to the first method (Step Sr_2a). When determining a second method as a generation mode and a prediction image, the prediction processor generates a prediction image according to the second method (Step Sr_2b). When determining a third method as a mode of generating a prediction image, the prediction processor generates a prediction image according to the third method (Step Sr_2c).

[00422] The first method, the second method, and the third method may be mutually different methods for generating a prediction image. Each of the first to third methods may be an interforecast method, an interforecast method, or another forecast method. The reconstructed image described above can be used in these prediction methods.

Intraforecaster

[00423] The intraforecaster 216 generates a prediction signal (intraprediction signal) by performing intraprediction by reference to a block or blocks in the current image stored in block memory 210, based on the intraprediction mode analyzed from the bit sequence encoded. More specifically, the intrapredictor 216 generates an intraprediction signal by performing the intraprediction by reference to samples (e.g., luminescence and/or chrominance values) of a neighboring block or blocks of the current block, and then sends the intraprediction signal to the prediction controller 220.

[00424] It should be noted that when an intraprediction mode, in which a luminescence block is referred to in the intraprediction of a chrominance block, is selected, the intrapredictor 216 can predict the chrominance component of the current block based on the chrominance component. luminescence of the current block.

[00425] Furthermore, when information analyzed from an encoded bit sequence indicates that PDPC should be applied, the intrapredictor 216 corrects the intrapredicted pixel values based on the horizontal/vertical reference pixel gradients.

Interforecaster

[00426] Interpredictor 218 predicts the current block by reference to a reference image stored in frame memory 214. Interprediction is performed in units of a current block or a subblock (e.g., a 4 x 4 block). in the current block. For example, the interpredictor 218 generates an interprediction signal of the current block or sub-block by performing motion compensation using motion information (e.g., a motion vector) analyzed from a coded bit sequence ( for example, a prediction parameter sent from the entropy decoder 202), and sends the interprediction signal to the prediction controller 220.

[00427] It should be noted that when the information analyzed from the encoded bit sequence indicates that the OBMC mode should be applied, the interpredictor 218 generates the interprediction signal using the movement information of a neighboring block in addition to the information of current block movement obtained from the movement estimate.

[00428] Furthermore, when the information analyzed from the encoded bit sequence indicates that the FRUC mode should be applied, the interpredictor 218 derives the motion information by performing motion estimation in accordance with the pattern matching method ( bilateral combination or template combination) analyzed from the encoded bit sequence. The interforecaster 218 then performs motion compensation (prediction) using the derived motion information.

[00429] Additionally, when BIO mode is to be applied, the interpredictor 218 derives a motion vector based on a model, assuming uniform linear motion. Furthermore, when information analyzed from the encoded bit sequence indicates that the affine motion compensation prediction mode should be applied, the interpredictor 218 derives a motion vector of each subblock based on the motion vectors of neighboring blocks. .

MV Derivation > Normal InterMode

[00430] When the information analyzed from a coded bit sequence indicates that the normal intermode should be applied, the interpredictor 218 derives an MV based on the information analyzed from the coded bit sequence and performs motion compensation (prediction ) using MV.

[00431] Figure 45 is a flowchart illustrating an example of interprediction in normal mode in decoder 200.

[00432] Interpredictor 218 of decoder 200 performs motion compensation for each block. At this time, first, the interpredictor 218 obtains a plurality of MV candidates for a current block based on information, such as MVs among a plurality of temporally or spatially decoded blocks surrounding the current block (Step Ss_1). In other words, interforecaster 218 generates a list of MV candidates.

[00433] Next, the interpredictor 218 extracts N (an integer equal to 2 or greater) MV candidates from the plurality of MV candidates obtained in Step Ss_1, as motion vector predictor candidates (also referred to as motion vector predictor candidates). MV predictor), according to a predetermined priority order (Step Ss_2). It should be noted that the priority order is determined in advance for each of the N MV predictor candidates.

[00434] Next, the interpredictor 218 decodes the motion vector predictor selection information from an input sequence (i.e., an encoded bit sequence) and selects a MV predictor candidate from the N candidates the MV predictor using the decoded motion vector predictor selection information, as a motion vector (also referred to as an MV predictor) of the current block (Step Ss_3).

[00435] Next, the interpredictor 218 decodes an MV difference from the input sequence, and derives an MV for a current block by adding a difference value, which is the decoded MV difference, and a selected motion vector predictor. (Step Ss_4).

[00436] Finally, the interpredictor 218 generates a prediction image for the current block by performing motion compensation of the current block using the derived MV and the decoded reference image (Step Ss_5).

Forecast Controller

[00437] The forecast controller 220 selects the intraforecast signal or the interforecast signal, and sends the selected forecast signal to the adder 208. As a whole, the settings, functions and processes of the forecast controller 220, intraforecaster 216 and interforecaster 218 on the decoder side may correspond to the configurations, functions and processes of the prediction controller 128, intraforecaster 124 and interforecaster 126 on the encoder side.

Decoder Assembly Example

[00438] Figure 46 is a block diagram illustrating an example assembly of decoder 200. Decoder 200 includes processor b1 and memory b2. For example, the plurality of constituent elements of the decoder 200, illustrated in Figure 41, is assembled in the processor b1 and memory b2 illustrated in Figure 46.

[00439] Processor b1 is a set of circuits that performs information processing and is accessible to memory b2. For example, the b1 processor is a dedicated or general set of electronic circuits that decodes a video (that is, an encoded sequence of bits). The processor b1 may be a processor such as a CPU. Additionally, the processor b1 may be an aggregate of a plurality of electronic circuits. Additionally, for example, the processor b1 may assume the roles of two or more constituent elements in addition to one constituent element for storing information from the plurality of constituent elements of the decoder 200 illustrated in Figure 41, etc.

[00440] Memory b2 is a dedicated or general memory for storing information that is used by processor b1 to decode an encoded bit sequence. Memory b2 can be the electronic circuitry, and can be connected to processor b1. Additionally, memory b2 can be included in the processor b1. Additionally, memory b2 may be an aggregate of a plurality of electronic circuits. Additionally, the memory b2 may be a magnetic disk, an optical disk, or the like, or it may be represented as a store, a recording medium, or the like. Additionally, memory b2 can be a non-volatile memory, or a volatile memory.

[00441] For example, memory b2 can store a video or a bit sequence. Additionally, memory b2 may store a program to cause processor b1 to decode an encoded bit sequence.

[00442] Additionally, for example, memory b2 can assume the roles of two or more constituent elements to store information among the plurality of constituent elements of the decoder 200 illustrated in Figure 41, etc. Specifically, memory b2 can assume the roles of block memory 210 and frame memory 214 illustrated in Figure 41. More specifically, memory b2 can store a reconstructed block, a reconstructed image, etc.

[00443] It should be noted that, in decoder 200, all of the plurality of constituent elements illustrated in Figure 41, etc., may not be implemented, and all of the processes described above may not be carried out. Part of the constituent elements indicated in Figure 41, etc., may be included in another device, or part of the processes described above may be carried out by another device.

Definitions of Terms

[00444] The respective terms can be defined as indicated below as examples.

[00445] An image is a set of luminescence samples in monochromatic format or a set of luminescence samples and two corresponding sets of chrominance samples in 4:2:0, 4:2:2 and 4:4 color format: 4. An image can be a frame or a field.

[00446] A frame is the composition of an upper field and a lower field, where sample rows 0, 2, 4... originate from the upper field and sample rows 1, 3, 5,... originate from the lower field.

[00447] A slice is an integer number of coding tree units contained in an independent slice segment and all subsequent dependent slice segments (if any), which precede the next independent slice segment (if any), among the same access unit.

[00448] A tile is a rectangular region of coding tree blocks within a particular column of tiles and a particular row of tiles in an image. A tile can be a rectangular region of the frame that must be able to be decoded and encoded independently, although circuit filtering across the edges of the tile can still be applied.

[00449] A block is a set of M x N samples (column M x row N), or a set M x N of transformation coefficients. A block can be a square or rectangular region of pixels including a Luminance matrix and two Chrominance matrices.

[00450] A coding tree unit (CTU) may be a coding tree block of luminescence samples of an image that has three sets of samples, and two corresponding coding tree blocks of chrominance samples. Alternatively, a CTU may be a coding tree block of samples from a monochrome image and an image that is encoded using three separate color planes and syntax structures used to encode the samples.

[00451] A super block may be a 64 x 64 pixel square block that consists of 1 or 2 mode information blocks or is recursively divided into four 32 x 32 blocks, which may be further divided.

Primary Aspect

[00452] Hereinafter, the encoder 100, the decoder 200, a coding method, and a decoding method, in accordance with the primary aspect of the present description, will be described.

[00453] Figure 47 is a flowchart illustrating an example of interforecasting, according to the first aspect. The following describes an example of the interprediction in decoder 200.

[00454] Interpredictor 218 in decoder 200 derives a base motion vector to be used in predicting a current block to be decoded; derives a different first motion vector from the base motion vector; derives a motion vector difference based on a difference between the base motion vector and the first motion vector; determines whether the motion vector difference is greater than a threshold; modifies the first motion vector when the motion vector difference is determined to be greater than the threshold, and does not modify the first motion vector when the motion vector difference is determined to be not greater than the threshold; and decodes the current block using the first modified motion vector or the first unmodified motion vector. For example, the base motion vector corresponds to the first set of pixels in the current block, and the first motion vector corresponds to the second set of pixels different from the first set of pixels in the current block. The interforecast will be more specifically described above with reference to the drawings. Note that the base motion vector and the first motion vector described below are each an example, and are not limited to the same.

[00455] First, in step S1001, interpredictor 218 derives the base motion vector from the first set of pixels in the current block. The processing in step S1001 will be more specifically described below with reference to Figure 48.

[00456] Figure 48 is a diagram illustrating an example of the current block. As illustrated in Figure 48, the current block may include subblocks (subblock 0 to subblock 5). Additionally, each of the sub-blocks can have a different first motion vector (MV0 to MV5). An example of the first set of pixels might be subblock 0. Here, the base motion vector is one of the first motion vectors in the current block. In this example, the base motion vector is MV0.

[00457] Another example of the first set of pixels could be an entire current block. In this example, the base motion vector is the average of the first motion vectors of all sub-blocks in the current block, that is, the average of MV0 to MV5.

[00458] Next, in step S1002, interpredictor 218 derives the first motion vector from the second set of pixels in the current block. The second set of pixels is different from the first set of pixels. An example of the second set of pixels might be subblock 2. In this example, the first motion vector of the second set of pixels is MV2.

[00459] Another example of the second set of pixels could be sub-block 1. In this example, the first motion vector is MV1.

[00460] Next, in step S1003, the interpredictor 218 derives the motion vector difference based on a difference between the base motion vector of the first set of pixels and the first motion vector of the second set of pixels. An example of the base motion vector value might be (-3, 4). An example of the first motion vector might be (16, 5). Accordingly, the motion vector difference between the first motion vector and the base motion vector is (-19, -1).

[00461] Next, in step S1004, interpredictor 218 determines whether the motion vector difference in step S1003 is greater than the threshold. The threshold is a set of first value (hereinafter referred to as the first threshold value) and the second value (hereinafter referred to as the second threshold value). An example of the limit might be (10, 20). An example of this determination process will be described below.

[00462] When (i) the absolute value of the horizontal component of the motion vector difference derived in step S1003 is greater than the first threshold value or (ii) the absolute value of the vertical component of the motion vector difference is greater than the second value threshold, the interpredictor 218 determines that the motion vector difference is greater than the threshold (Yes in step S1004). In other words, when at least one of the absolute values of the horizontal and vertical components of the motion vector difference is greater than the threshold, the motion vector difference is determined to be greater than the threshold. For example, when the absolute value of the horizontal component of the motion vector difference (-19, 01), illustrated as an example in step S1003, is compared with the first threshold value, |-19|>10 is obtained, and, therefore, the motion vector difference is determined to be greater than the threshold. In this case, the interpredictor 218 modifies the first motion vector. This modification will be described in detail in step S1006.

[00463] On the other hand, when (i) the absolute value of the horizontal component of the motion vector difference is not greater than the first threshold value and (ii) the absolute value of the vertical component of the motion vector difference is not greater than the second threshold value, interpredictor 218 determines that the motion vector difference is not greater than the threshold (not in step S1004). In other words, when both the absolute values of the horizontal and vertical components of the motion vector difference are less than or equal to the threshold, the motion vector difference is determined to be less than or equal to the threshold. When the motion vector difference is determined to be no greater than the threshold (not in step S1004), the interpredictor 218 does not modify the first motion vector (step S1005).

[00464] Next, when the motion vector difference is determined to be greater than the limit (yes in step S1004), in step S1006, the interpredictor 218 modifies the first motion vector using the value obtained by fixing the vector difference of movement. In doing so, the motion vector difference (19, -1) is fixed to (-10, -1). As described above, the absolute value |-19| of the horizontal component of the motion vector difference is greater than 10 which is the first limiting value of the limit (10, 20) and thus the absolute value of the horizontal component of the motion vector difference is fixed so that it is equal to the first limit value. Accordingly, the fixed motion vector difference is (-10, -1).

[00465] Subsequently, the interpredictor 218 modifies the first motion vector using the fixed motion vector difference and the base motion vector. More specifically, the first motion vector can be modified by adding the fixed motion vector difference to the base motion vector. For example, the first modified motion vector is (-3, 4)-(-10, -1) = (7, 5).

[00466] In step S1007, the interpredictor 218 decodes the second set of pixels using the first modified motion vector or the first original motion vector. For example, sub-block 2 is decoded using the first modified motion vector (7, 5).

[00467] Note that processing 1000 illustrated in Figure 47 may be encoder processing.

[00468] Processing 1000 can be applied to all subblocks in the current block. When processing 1000 is applied to all sub-blocks in the current block, the prediction mode for the current block can be affine mode. Additionally, when processing 1000 is applied to all sub-blocks in the current block, the prediction mode for the current block may be the Alternative Temporal Motion Vector Prediction (ATMVP) mode.

[00469] Note that the ATMVP mode is an example of the sub-block modes classified in the mixing mode. For example, in the coded reference image specified by the MV (MV0) of the lower left block neighboring the current block, a temporal MV reference block, which corresponds to the current block, is identified, and for each of the sub-blocks in the block current, an MV used in encoding a region in the temporal MV reference block, located together with the sub-block, is identified.

[00470] Additionally, the first motion vector modified in the specific sub-block can be used to update the first motion vector of another sub-block in the current block. The processing example will be described below.

[00471] For example, MV0 of sub-block 0 is identified as the base motion vector, MV2 of sub-block 2 is identified as the first motion vector, and the first modified motion vector is determined using the processing 1000. The first modified motion vector of sub-block 2 is represented as MV2' = (V2x', V2y'). Using the base motion vector MV0 and the first modified motion vector MV2', the first modified motion vector of sub-block i (i = 1, 3, 4 or 5), in addition to sub-block 2 in the current block, is calculated based on the following equations.

[00472] When MVi' is not greater than the limit, MVi is updated as MVi', and sub-block i is decoded using MVi. POSix and POSiy are horizontal and vertical positions of sub-block i, respectively.

[00473] In the example of Figure 48, they are as follows:

[00474] W and H are horizontal and vertical positions, for example, x-coordinate and y-coordinate in the lower left corner of sub-block 2, respectively.

[00475] The limit can be adaptive with respect to the size of the current block. For example, the limit increases with an increase in the current block size.

[00476] Additionally, the threshold can be adaptive with respect to the number of reference images. For example, the threshold decreases with an increase in the number of reference images.

[00477] The limit may be encoded in a header, such as the SPS header, the PPS header, or the Slice header.

[00478] The limit can be predetermined, and does not need to be encoded in a sequence.

[00479] The threshold may be determined to restrict the size of the memory access in a worst-case scenario when motion compensation (prediction) is performed on the current block in a prediction mode predetermined to be less than or equal to the size of memory access in a worst-case scenario, when motion compensation (prediction) is performed bidirectionally in the current block, or on an 8 x 8 pixel basis in another prediction mode. The default prediction mode is, for example, affine mode. The restriction example will be described below.

[00480] "Mem_base" represents the memory access size per block of 8 x 8 pixels in a worst-case situation, when motion compensation (Luminescence) is performed bidirectionally on the current block based on 8 x 8 pixels in a forecast mode in addition to the default forecast mode.

[00481] The memory access size, in a worst-case situation, when motion compensation (Luminescence) is performed on the current block (size: M x N), in the predetermined prediction mode, is represented as "Mem_CU", and the corresponding limit "Mem_th" is represented using "Mem_base" as follows:

[00482] Mem_CU is restricted to not be greater than Mem_th. The example of limit calculation will be described below.

[00483] Assuming that the size of the luminescence signal per pixel is 1 byte and the number of filter outputs for motion compensation is 8 outputs, (in byte) is true.

[00484] Assuming that the prediction mode for the current block is the affine mode and the current block is 64 x 64 pixels, (in byte) is true.

[00485] "H" and "V" are assumed to represent a limit value of the first limit component (first limit value) and a limit value of the second limit component (second limit value), respectively.

[00486] The worst situation, when prediction is performed in a normal intermode in addition to the affine mode, is the case in which the current block is divided into blocks of 8 x 8 pixels and motion compensation is performed bidirectionally on all the 8 x 8 pixel blocks. The referenceable memory range when performing the prediction in affine mode is determined so as not to exceed the memory access size in this worst case scenario. The referenceable memory range in affine mode is assumed to be calculated based on the following equation 1. H and V are calculated such that the memory access range in affine mode is less than or equal to Mem_th (e.g. 28800). Note that the affine mode here is described as an example of the case in which bidirectional prediction is performed.

[00487] Assuming H = V, H = V = 24 is obtained by solving equation [1].

[00488] Note that the above-threshold calculation process is also applied to the current block having a different size and a different total number of reference images. Figure 49 is a diagram illustrating the calculated limiting examples. Note that Figure 49 illustrates examples of H = V, but H > V or H < V is possible.

[00489] As illustrated in Figure 49, the limit differs between the case in which the current block is predicted unidirectionally (the number of reference frames = 1) and the case in which the current block is predicted bidirectionally (the number of reference frames = 2). In the example of Equation 1 above, the size of the current block to be predicted in affine mode is 64 x 64. When the current block is predicted bidirectionally, that is, two reference images are referred to, the first threshold value H and the second threshold value V are each equal to 24. For example, when the current block of size equal to 64 x 64 to be predicted in affine mode is predicted unidirectionally, that is, only one reference image is referred to , the first threshold value H and the second threshold value V are each equal to 49. Figure 49 is a table illustrating the thresholds of all possible block sizes in affine mode prediction, which are calculated using the Equation [1] above.

Technical Advantages in the Primary Aspect

[00490] In the primary aspect, in accordance with the present description, a process for deriving the first motion vector is introduced into the interprediction. As described above, the process in which the best first motion vector is selected, such that the variation of the first motion vectors in the current block is within the predetermined range, can reduce memory bandwidth in interprediction.

Additional

[00491] Additionally, the encoder 100 and the decoder 200, according to the present embodiment, can be used as an image encoder and an image decoder, or can be used as a video encoder or a video decoder, respectively .

[00492] Alternatively, the encoder 100 and the decoder 200 can be used as an entropy encoder and an entropy decoder, respectively. In other words, encoder 100 and decoder 200 can correspond to only entropy coding unit 110 and entropy decoding unit 202, respectively. The other components can be included in the other devices.

[00493] Additionally, at least part of the present embodiment can be used as an encoding method, a decoding method, an entropy encoding method, an entropy decoding method, or another method.

[00494] Additionally, in the present embodiment, each component can be configured by dedicated hardware, or can be implemented by executing a software program suitable for each component. Each component can be implemented by having a program executor, such as a CPU or processor, read and execute a software program stored on a recording medium, such as a hard disk or semiconductor memory.

[00495] In particular, the encoder 100 and the decoder 200 may each include a set of processing circuits and a store that is electrically connected to that set of processing circuits and accessible from that set of processing circuits. For example, the processing circuitry corresponds to processor a1 or b1, and the store corresponds to memory a2 or b2.

[00496] The processing circuitry includes at least one of dedicated hardware and a program executor, and performs processing using the store. Additionally, when the processing circuitry includes the program executor, the store stores a software program to be executed by the program executor.

[00497] Here, a software for implementing the encoder 100, the decoder 200, etc., in accordance with the present embodiment, is a program as follows.

[00498] For example, such a program may cause a computer to perform an encoding method to encode a video. The encoding method includes: deriving a base motion vector to be used in predicting a current block to be encoded; deriving a first motion vector different from the base motion vector; deriving a motion vector difference based on a difference between the base motion vector and the first motion vector; determine whether the motion vector difference is greater than a threshold; modifying the first motion vector when the motion vector difference is determined to be greater than the threshold, and not modifying the first motion vector when the motion vector difference is determined to be not greater than the threshold; and encoding the current block using the first modified motion vector or the first unmodified motion vector.

[00499] Additionally, for example, this program may use a computer to perform a decoding method for decoding a video. The decoding method includes deriving a base motion vector to be used in predicting a current block to be decoded; deriving a first motion vector different from the base motion vector; deriving a motion vector difference based on a difference between the base motion vector and the first motion vector; determine whether the motion vector difference is greater than a threshold; modifying the first motion vector when the motion vector difference is determined to be greater than the threshold and not modifying the first motion vector when the motion vector difference is determined to be not greater than the threshold; and decode the current block using the first modified motion vector or the first unmodified motion vector.

[00500] Additionally, as described above, each component can be a circuit. The circuits can be integrated into a single circuit as a whole, or they can be separated from each other. Additionally, each component can be implemented as a general-purpose circuit, or as a dedicated circuit.

[00501] Additionally, a process performed by a specific component can be performed by another component. Additionally, the order of processes can be changed, or multiple processes can be carried out in parallel. Additionally, an encoding device may include encoder 100 and decoder 200.

[00502] The ordinal numbers used in the illustration, such as first and second, can be renumbered as necessary. Additionally, the ordinal number may be newly assigned to a component, etc., or it may be deleted from a component, etc.

[00503] As described above, aspects of the encoder 100 and decoder 200 have been described based on the embodiment, but aspects of the encoder 100 and decoder 200 are not limited to that embodiment. Various modifications to the embodiment that may be devised by those skilled in the art, and shapes configured for combining the components in different embodiments, without departing from the spirit of the present invention, may be included within the scope of aspects of the encoder 100 and the decoder 200.

[00504] One or more of the aspects described here can be realized by combining at least part of other aspects in the present description. Additionally, one or more of the aspects described here can be realized by combining, with other aspects, part of the processes indicated in any of the flowcharts, according to the aspects, part of the configuration of any of the devices, part of the syntaxes, etc. .

Mode 2 Implementations and Applications

[00505] As described in each of the above embodiments, each functional or operational block can typically be realized as an MPU (micro processing unit) and memory, for example. Furthermore, the processes carried out by each of the functional blocks can be carried out as a program execution unit, such as a processor that reads and executes software (a program) recorded on a recording medium, such as ROM. The software can be distributed. Software can be written to a variety of recording media, such as semiconductor memory. Note that each functional block can also be realized as hardware (dedicated circuit).

[00506] The processing described in each of the modalities can be carried out through integrated processing using a single device (system), and, alternatively, it can be carried out through decentralized processing using a plurality of devices. Furthermore, the processor executing the program described above may be a single processor or a plurality of processors. In other words, integrated processing can be carried out and, alternatively, decentralized processing can be carried out.

[00507] The modalities of the present description are not limited to the illustrative modalities above; Various modifications may be made to the illustrative embodiments, the results of which are also included within the scope of the embodiments of the present description.

[00508] In the following, examples of application of the moving image encoding method (image coding method) and the moving image decoding method (image decoding method) described in each of the above embodiments will be described, in addition to several systems that implement the application examples. Such a system may be characterized as including an image encoder that employs the image encoding method, an image decoder that employs the image decoding method, or an image encoder-decoder that includes both the image encoder and the decoder. of image. Other settings of such a system can be modified on a case-by-case basis.

Usage Examples

[00509] Figure 50 illustrates a general configuration of the ex100 content delivery system suitable for implementing a content delivery service. The area in which the communication service is provided is divided into cells of desired sizes, and base stations ex106, ex107, ex108, ex109 and ex110, which are fixed wireless stations in the illustrated example, are located in respective cells.

[00510] In the content delivery system ex100, the devices including the computer ex111, the gaming device ex112, the camera ex113, the home appliance ex114, and the smartphone ex115 are connected to the Internet ex101 through the Internet service provider ex102 or ex104 communications network and ex106 to ex110 base stations. The ex100 content delivery system can combine and connect any combination of the above devices. In many implementations, devices may be connected directly or indirectly through a telephone network or near-field communication, rather than through the ex106 to ex110 base stations. Additionally, the streaming server ex103 can be connected to devices including the computer ex111, the gaming device ex112, the camera ex113, the home appliance ex114, and the smartphone ex115 via, for example, the Internet ex101. The ex103 broadcast server can also be connected, for example, to a terminal at a hotspot on the ex117 aircraft via the ex116 satellite.

[00511] Note that instead of base stations ex106 to ex110, wireless access points or hotspots can be used. The ex103 broadcast server can be connected to the ex104 communications network directly instead of through the ex101 Internet or the ex102 Internet service provider, and can be connected to the ex117 aircraft directly instead of through the ex116 satellite.

[00512] The ex113 camera is a device capable of capturing still images and video, just like a digital camera. The ex115 smartphone is a smartphone, cell phone, or personal handyphone system (PHS) device that can operate under the mobile communications system standards of 2G, 3G, 3.9G, and 4G systems in addition to the next-generation 5G system.

[00513] Home appliance ex114, for example, is a refrigerator or a device included in a domestic fuel cell operating system.

[00514] In the content delivery system ex100, a terminal including an image and/or video capture function may, for example, transmit live by connecting the transmission server ex103 through, for example, the base station ex106. When live broadcasting, a terminal (e.g., the computer ex111, the gaming device ex112, camera ex113, home appliance ex114, smartphone ex115, or a terminal on the aircraft ex117) may perform the coding processing described in the above embodiments in content of still image or video captured by a user through the terminal, may multiplex video data obtained by encoding and audio data obtained by encoding the audio corresponding to the video, and may transmit the obtained data to the transmission server ex103. In other words, the terminal functions as the image encoder, in accordance with an aspect of the present description.

[00515] The transmission server ex103 transmits the transmitted content data to the clients requesting the transmission. Client examples include the computer ex111, the gaming device ex112, the camera ex113, the household appliance ex114, the smartphone ex115, and terminals inside the aircraft ex117, which can decode the encoded data described above. Devices receiving transmitted data decode and reproduce the received data. In other words, the devices may each function as the image decoder in accordance with an aspect of the present disclosure.

Decentralized Processing

[00516] The transmission server ex103 can be realized as a plurality of servers or computers between which tasks such as processing, recording and transmitting data are divided. For example, the ex103 streaming server can be realized as a content delivery network (CDN) that streams content over a network connecting multiple edge servers located throughout the world. In a CDN, an edge server physically close to the client is dynamically assigned to the client. Content is temporarily stored and streamed to the edge server to reduce load times. In the case, for example, of some type of error or change in connectivity resulting from, for example, a spike in traffic, it is possible to transmit data stably at high speed, as it is possible to avoid the affected parts of the network , for example, by dividing processing among a plurality of edge servers, or switching transmission tasks to a different edge server and continuing transmission.

[00517] Decentralization is not limited to just dividing processing for transmission; Encoding of captured data can be split between and performed by terminals, server-side, or both. In one example, in typical coding, processing is performed in two circuits. The first circuit serves to detect how complicated the image is, frame by frame, or scene by scene, or detect the encoding load. The second circuit is for processing that maintains image quality and improves coding efficiency. For example, it is possible to reduce the processing load on the terminals and improve the quality and efficiency of content encoding by having the terminals perform the first encoding circuit and having the server side that received the content perform the second encoding circuit. codification. In such a case, after receiving a decoding request, it is possible for the encoded data, resulting from the first circuit performed by one terminal, to be received and reproduced on another terminal almost in real time. This makes it possible to perform smooth transmission in real time.

[00518] In another example, the camera ex113 or similar extracts a feature amount from an image, compresses data related to the feature amount as metadata, and transmits the compressed metadata to a server. For example, the server determines the significance of an object based on the feature quantity and changes the quantization precision accordingly to perform appropriate compression for the meaning (or content significance) of the image. The feature quantity data is particularly effective in improving the accuracy and efficiency of motion vector prediction during the second pass of compression performed by the server. Furthermore, encoding that has a relatively low processing load, such as variable-length encoding (VLC), can be handled by the terminal, and encoding that has a relatively high processing load, such as context-adaptive binary arithmetic encoding (CABAC), can be handled by the server.

[00519] In another example, there are cases in which a plurality of videos of approximately the same scene are captured by a plurality of terminals, for example, in a stadium, a shopping mall, or factory. In such a case, for example, encoding can be decentralized by dividing the processing tasks between the plurality of terminals that captured the videos and, if necessary, other terminals that did not capture the videos, and the server, on a per-unit basis. Units can be, for example, groups of images (GOP), images or tiles resulting from the division of an image. This makes it possible to reduce charging times and obtain transmission that is closer to real time.

[00520] Since the videos are of approximately the same scene, management and/or instructions can be performed by the server, so that videos captured by the terminals can be cross-referenced. Additionally, the server can receive the encoded data from the terminals, change the reference relationship between the data items, or correct or replace the images themselves, and then perform the encoding. This makes it possible to generate a sequence with increased quality and efficiency for individual data items.

[00521] Additionally, the server may transmit video data after performing transcoding to convert the video data encoding format. For example, the server can convert the encoding format from MPEG to VP (for example, VP9) and can convert H.264 to H.265.

[00522] In this way, encoding can be performed by a terminal or one or more servers. Accordingly, although the device, which performs the encryption, is referred to as a "server" or "terminal" in the following description, some or all of the processes performed by the server may be performed by the terminal, and, likewise, some or all processes performed by the terminal can be performed by the server. This also applies to decoding processes.

3D, multiple angles

[00523] There has been an increase in the use of images or videos combined from images or videos of different scenes captured simultaneously, or of the same scene captured from different angles, by a plurality of terminals, such as the ex113 camera and/or or the ex115 smartphone. Videos captured by the terminals are combined based on, for example, a relative position relationship obtained separately between the terminals, or regions in a video having combined feature points.

[00524] In addition to encoding two-dimensional moving images, the server may encode a still image based on scene analysis of a moving image, either automatically or at a user-specified time, and transmit the encoded still image to a terminal reception. Additionally, when the server can obtain the relative position relationship between video capture terminals, in addition to two-dimensional moving images, the server can generate three-dimensional geometry of a scene based on a video of the same scene captured from angles many different. The server may separately encode three-dimensional data generated from, for example, a point cloud and, based on a result of recognizing or tracking a person or object using three-dimensional data, may select or reconstruct and generate a video to be transmitted to a receiving terminal, from videos captured by a plurality of terminals.

[00525] This allows the user to enjoy a scene through the free selection of videos corresponding to the video capture terminals, and allows the user to enjoy the content obtained by extracting a video, at a point of view selected from the three-dimensional data reconstructed from a plurality of images or videos. Additionally, as with video, sound can be recorded from relatively different angles, and the server can multiplex audio from a specific angle or space with corresponding video, and transmit the multiplexed video and audio.

[00526] Recently, content, which is a composite of real world and a virtual world, such as virtual reality (VR) and augmented reality (AR) content, has become popular. In the case of VR images, the server can create images from the points of view of both the left and right eyes, and perform encoding that tolerates reference between images from two points of view, such as separate sequences without reference. When images are decoded as separate sequences, the sequences can be synchronized when played back, so as to recreate a virtual three-dimensional space according to the user's point of view.

[00527] In the case of AR images, the server overlays virtual object information existing in a virtual space on camera information that represents a space in the real world, based on a three-dimensional position or movement from the user's perspective. The decoder can obtain or store virtual object information and three-dimensional data, generate two-dimensional images based on the movement of the user's perspective, and then generate overlapping data by continuously connecting the images. Alternatively, the decoder may transmit, to the server, movement from the user's perspective in addition to a request for virtual object information. The server can generate superimposed data based on three-dimensional data stored in the server according to the received motion, and encode and transmit the generated superimposed data to the decoder. Note that the overlaid data includes, in addition to the RGB values, an α value indicating transparency, and the server sets the α value for sections beyond the object generated from the three-dimensional data, for example, 0, and can perform encoding while these sections are transparent. Alternatively, the server can set the background to a given RGB value, such as a chrominance key, and generate data in which areas beyond the object are set as the background.

[00528] Decoding of similarly transmitted data can be performed by the client (i.e., the terminals), on the server side, or divided between them. In an example, a terminal may transmit a receive request to a server, the requested content may be received and decoded by another terminal, and a decoded signal may be transmitted to a device having a display. It is possible to reproduce data with high image quality by decentralizing processing and appropriate selection of content regardless of the processing capacity of the communications terminal itself. In another example, while a TV, for example, is receiving image data that is large in size, a region of an image, such as a tile obtained by splitting the image, may be decoded and displayed on a personal terminal or computer terminals. a TV viewer or viewers. This allows viewers to share a broad view in addition to each viewer checking out their designated area, or inspecting a region in closer detail.

[00529] In situations in which a plurality of wireless connections are possible across near, intermediate and distant distances, internally or externally, it may be possible to continuously receive content using a transmission system standard, such as MPEG- DASH. The user can switch between real-time data while freely selecting a decoder or display device that includes the user terminal, internally or externally arranged monitors, etc. Furthermore, using, for example, information about the user's position, decoding can be performed while switching which terminal handles the decoding and which terminal handles the display of content. This makes it possible to map and display information, while the user is in motion en route to a destination, on the wall of a nearby building in which a device, capable of displaying content, is embedded, or on part of the floor. Furthermore, it is also possible to switch the bit rate of received data based on accessibility to the encoded data on a network, such as when the encoded data is temporarily stored on a server quickly accessible from the receiving terminal, or when the data encoded files are copied to an edge server in a content delivery service.

Scalable Coding

[00530] Content switching will be described with reference to a scalable sequence, illustrated in Figure 51, which is compression encoded by implementing the moving image coding method described in the above embodiments. The server may have a configuration in which content is switched while making use of the temporal and/or spatial scaling capability of a sequence, which is achieved by dividing and encoding the layers, as illustrated in Figure 51. Note that there may be a plurality of individual sequences that have the same content but different quality. In other words, by determining which layer to decode based on internal factors, such as processing capacity on the decoder side, and external factors, such as communication bandwidth, the decoder side can freely switch between low-end content and resolution and high-resolution content, while decoding. For example, in a case in which the user wishes to continue watching, for example at home on a device such as an Internet-connected TV, a video that the user has previously been watching on the ex115 smartphone while on the move, the device can simply decode the same transmission up to a different layer, which reduces the load on the server side.

[00531] Additionally, in addition to the configuration described above, in which scalability is achieved as a result of the images being encoded by layer, with the enhancement layer being above the base layer, the enhancement layer may include metadata based , for example, in the statistical information in the image. The decoder side can generate high-quality image content by performing super-resolution imaging on an image in the base layer, based on metadata. Super-resolution imaging can improve the SN ratio while maintaining resolution and/or increasing resolution. Metadata includes information to identify a linear or nonlinear filter coefficient, as used in super resolution processing, or information that identifies a parameter value in filter processing, machine learning, or mean squares method used in super resolution processing. resolution.

[00532] Alternatively, a configuration may be provided, in which an image is divided, for example, into tiles, according to, for example, the meaning of an object in the image. On the decoder side, only a partial region is decoded by selecting a tile to decode. Additionally, by storing an object attribute (person, car, ball, etc.) and an object position in the video (coordinates in identical images) as metadata, the decoder side can identify the position of a desired object based on the metadata and determine which tile or tiles include that object. For example, as illustrated in Figure 52, metadata can be stored using a data storage structure other than pixel data, such as a SEI (supplemental enhancement information) message in HEVC. This metadata indicates, for example, the position, size, or color of the main object.

[00533] Metadata can be stored in units of a plurality of images, such as broadcast, sequence, or random access units. The decoder side can obtain, for example, the time at which a specific person appears in the video, and by matching the time information with the image unit information, can identify an image in which the object is present, and can determine the position of the object in the image.

Web Page Optimization

[00534] Figure 53 illustrates an example of a screen displaying a web page on the ex111 computer, for example. Figure 54 illustrates an example of a screen displaying a web page on the ex115 smartphone, for example. As illustrated in Figure 53 and Figure 54, a web page may include a plurality of image links that are links to image content, and the appearance of the web page differs depending on the device used to view the web page. When a plurality of image links can be displayed on the screen until the user explicitly selects an image link, or until the image link is in the approximate center of the screen or the entire image link fits on the screen, the display apparatus (decoder) may display, as image links, static images included in the content or images I; can display video such as an animated gif using a plurality of still images or I-images; or it can receive just the base layer and decode and display the video.

[00535] When an image link is selected by the user, the display apparatus performs decoding while providing the highest priority to the base layer. Note that if there is information in the HTML code of the web page indicating that the content is scalable, the display device can decode up to the enhancement layer. Additionally, in order to ensure real-time playback, before a selection is made or when bandwidth is seriously limited, the display apparatus may reduce the delay between the time at which the front image is decoded and the time at which the front image is decoded. which decoded image is displayed (that is, the delay between the start of content decoding to the display of content) by decoding and displaying only forward reference images (I-image, P-image, forward reference B-image) . Additionally, the display apparatus may purposely ignore the reference relationship between the images, and approximately decode all B and P images as forward reference images, and then perform normal decoding as the number of images received increases over time.

Autonomous Activation

[00536] When transmitting and receiving still images or video data, such as two- or three-dimensional map information for the autonomous drive or assisted drive of an automobile, the receiving terminal may receive, in addition to image data belonging to one or more layers, information, for example, about the weather or buildings on the road as metadata, and associate the metadata with the image data through decoding. Note that metadata can be designated by layer and alternatively can simply be multiplexed with image data.

[00537] In such a case, since the automobile, drone, aircraft, etc., containing the receiving terminal is mobile, the receiving terminal can continuously receive and perform decoding while switching between base stations between base stations ex106 to ex110, by transmitting information that indicates the position of the receiving terminal. Furthermore, according to the selection made by the user, the user situation and/or connection bandwidth, the receiving terminal can dynamically select to what extent metadata is received, or to what extent map information, e.g. , is updated.

[00538] In the ex100 content delivery system, the client can receive, decode and reproduce, in real time, the encoded information transmitted by the user.

Transmission of Individual Content

[00539] In the ex100 content delivery system, in addition to high image quality, long content distributed by a video distribution entity, low image quality unicast or multicast transmission, and short content from an individual are also possible. Such content from individuals is more likely to increase further in terms of popularity. The server may first perform editing processing on the content before encoding processing in order to refine the individual content. This can be achieved using the following configuration, for example.

[00540] In real time, while capturing video or image content, or after the content has been captured and accumulated, the server performs recognition processing based on the raw data or encoded data, such as capture error processing , scene search processing, meaning analysis, and/or object detection processing. Then, based on the result of the recognition processing, the server - upon request or automatically - edits the content, examples of which include correction, such as focus correction and/or motion blur; removing low priority scenes, such as scenes that are low in brightness compared to other images, or out of focus; object edge adjustment, and color tone adjustment. The server encodes the edited data based on the edit result. It is known that excessively long videos tend to receive fewer views. Accordingly, in order to keep content within a specific duration that scales with the length of the original video, the server may, in addition to the low priority scenes described above, automatically remove scenes with low motion, based on a result of processing. Alternatively, the server may generate and encode a summary of the video based on a result of an analysis of the meaning of a scene.

[00541] There may be cases in which individual content may include content that infringes a copyright, moral right, image right, etc. Such a case may result in an unfavorable situation for the creator, such as when content is shared beyond the creator's intended scope. Accordingly, before encoding, the server can, for example, edit images in order to blur the faces of people on the outskirts of the scene or blur the interior of a house, for example. Additionally, the server can be configured to recognize the faces of people other than a person recorded in the images to be encoded, and when such faces appear in an image, it can apply a mosaic filter, for example, to the person's face. Alternatively, as pre- or post-processing for encoding, the user may specify, for copyright reasons, a region of an image including a person or a region of the background to be processed. The server may process the specified region, for example, replacing the region with a different image, or smearing the region. If the region includes a person, the person can be tracked in the moving image, and the person's head region can be replaced with another image as the person moves.

[00542] Since there is a demand for real-time viewing of content produced by individuals, which tends to be small in data size, the decoder first receives the base layer as the highest priority, and performs decoding and playback, although this may differ depending on bandwidth. When content is played two or more times, such as when the decoder receives the enhancement layer during decoding and playback from the base layer, and creates a playback cycle, the decoder can play a high image quality video including the enhancement layer. If the stream is encoded using such scalable encoding, the video may be of low quality when in an unselected state or at the beginning of the video, but may provide an experience in which the image quality of the stream progressively increases in an intelligent manner. This is not limited to just scalable coding; The same experience can be provided by configuring a single stream from a low-quality stream played for the first time and a second encoded stream using the first stream as a reference.

Other Implementation and Application Examples

[00543] Encoding and decoding can be performed by LSI (Large Scale Integration Circuit Set) ex500 (see Figure 50), which is typically included in each terminal. LSI ex500 can be configured from a single chip or a plurality of chips. The software for encoding and decoding the moving images may be integrated into some type of recording medium (such as CD-ROM, floppy disk, or hard disk) that is readable, for example, by the ex111 computer, and the encoding and decoding can be performed using software. Additionally, when the ex115 smartphone is equipped with a camera, video data obtained by the camera can be transmitted. In this case, the video data is encoded by LSI ex500 included in the smartphone ex115.

[00544] Note that LSI ex500 can be configured to download and activate an application. In such a case, the terminal first determines whether it is compatible with the scheme used to encode the content, or whether it is capable of performing a specific service. When the terminal does not support the encoding scheme of the content, or when the terminal is not capable of performing a specific service, the terminal first downloads a codec or application software and then obtains and plays the content.

[00545] In addition to the example of content delivery system ex100 utilizing the Internet ex101, at least the moving image encoder (picture encoder) or moving image decoder (picture decoder) described in the above embodiments may be implemented in a digital broadcast system. The same encoding processing and decoding processing can be applied to transmit and receive broadcast radio waves overlaid with multiplexed audio and video data using, for example, a satellite, although this is aimed at multicasting, whereas the Unicasting is easier with the ex100 content delivery system.

Hardware Configuration

[00546] Figure 55 illustrates additional details of the ex115 smartphone illustrated in Figure 50. Figure 56 illustrates an example configuration of the ex115 smartphone. The EX115 smartphone includes the EX450 antenna for transmitting and receiving radio waves to and from the EX100 base station, the EX465 camera capable of capturing video and still images, and the EX458 monitor that displays decoded data such as video captured by the EX465 camera and video received by the ex450 antenna. The ex115 smartphone additionally includes the ex466 user interface, such as a touch panel, an ex457 audio output unit, such as a speaker for outputting speech or other audio, ex456 audio input unit, such as a microphone for audio input, ex467 memory capable of storing decoded data such as captured video or still images, recorded audio, received video or still images, and mail in addition to decoded data, and the ex464 partition which is an interface for SIM ex468 , to authorize access to a network and various data. Note that external memory can be used instead of ex467 memory.

[00547] The main controller ex460, which deeply controls the display ex458 and the user interface ex466, the power supply circuit ex461, the user interface input controller ex462, the video signal processor ex455, the EX463 camera interface, EX459 display controller, EX452 modulator/demodulator, EX453 multiplexer/demultiplexer, EX454 audio signal processor, EX464 partition, and EX467 memory are connected via the EX470 bus.

[00548] When the user turns on the power button of the ex461 power supply circuit, the ex115 smartphone is energized to an operational state, and each component is supplied with power from a battery pack.

[00549] The smartphone ex115 performs processing, for example, call and data transmission, based on the control carried out by the main controller ex460, which includes a CPU, ROM and RAM. When making calls, an audio signal recorded by the ex456 audio input unit is converted into a digital audio signal by the ex454 audio signal processor, to which spread spectrum processing is applied by the ex452 modulator/demodulator and conversion from digital to analog, and frequency conversion processing is applied by the EX451 transmitter/receiver, and the resulting signal is transmitted through the EX450 antenna. The received data is amplified, frequency converted, and converted from analog to digital, the reverse spread spectrum is processed by the ex452 modulator/demodulator, converted into an analog audio signal by the ex454 audio signal processor, and then sent to from the ex457 audio output unit. In data transmission mode, text, still image or video data is transmitted by the EX460 main controller through the EX462 interface input controller based on the operation of the EX466 user interface of the main body, for example. Similar transmit and receive processing is performed. In data transmission mode, when sending a video, still image or video and audio, the video signal processor ex455 encodes by compression, through the moving image coding method described in the above embodiments, a video signal stored in the ex467 memory or a video signal recorded from the ex465 camera, and transmits the encoded video data to the ex453 multiplexer/demultiplexer. The ex454 audio signal processor encodes an audio signal recorded by the ex456 audio input unit while the ex465 camera is capturing a video or still image, and transmits the encoded audio data to the ex453 multiplexer/demultiplexer. The ex453 multiplexer/demultiplexer multiplexes the encoded video data and the encoded audio data using a given scheme, modulates and converts the data using the ex452 modulator/demodulator (modulator/demodulator circuit) and ex451 transmitter/receiver, and transmits the result through of the ex450 antenna.

[00550] When video attached to an e-mail message or a chat, or a video linked from a web page, is received, for example, in order to decode the multiplexed data received through the ex450 antenna, the multiplexer /ex453 demultiplexer demultiplexes the multiplexed data to divide the multiplexed data into a video data bit sequence and an audio data bit sequence, supplies the encoded video data to the ex455 video signal processor through the ex470 synchronized bus , and supplies the encoded audio data to the ex454 audio signal processor via the ex470 synchronized bus. The video signal processor ex455 decodes the video signal using a moving picture decoding method corresponding to the moving picture coding method described in the above embodiments, and the video or a still image included in the connected moving picture file is displayed on the ex458 monitor via the ex459 display controller. The EX454 audio signal processor decodes the audio signal and outputs the audio from the EX457 audio output unit. As real-time streaming is becoming increasingly popular, there may be cases where audio playback may be socially inappropriate depending on the user's environment. Accordingly, as a starting value, a configuration in which only video data is played, that is, the audio signal is not played, may be preferable; audio can be synchronized and played only when input, such as when the user clicks on video data, is received.

[00551] Although the ex115 smartphone was used in the example above, three other implementations are conceivable: a transceiver terminal including both an encoder and a decoder; a transmitting terminal including only an encoder; and a receiving terminal including a decoder only. In the description of the digital broadcast system, an example is provided in which multiplexed data obtained as a result of video data being multiplexed with audio data is received or transmitted. The multiplexed data, however, may be video data multiplexed with data other than audio data, such as video-related text data. Additionally, video data itself, rather than multiplexed data, can be received or transmitted.

[00552] Although the ex460 main controller, including a CPU, is described as controlling the encoding or decoding processes, various terminals often include GPUs. Accordingly, a configuration is acceptable in which a large area is processed at once, making use of the performance capability of the GPU through memory shared by the CPU and GPU, or memory including an address that is managed so as to allow utilization common by the CPU and GPU. This makes it possible to shorten the encoding time, maintaining the real-time nature of the transmission, and reducing delay. In particular, processing regarding motion estimation, deblocking filtering, sample adaptive shift (SAO), and transformation/quantization can be effectively performed by the GPU rather than the CPU on image units, e.g., all of a turn.

Industrial Applicability

[00553] The present description is applicable to a television receiver, a digital video recorder, an automobile navigation, a mobile telephone, a digital camera, a digital video camera, a teleconferencing system, to an electronic mirror, etc. Reference Marks in Drawings 100 encoder 102 divider 104 subtractor 106 transformer 108 quantizer 110 entropy encoder 112,204 inverse quantizer 114,206 inverse transformer 116,208 adder 118,210 block memory 120,212 circuit filter 122,214 frame memory 124,216 intraforecaster 126,218 interforecaster 128,220 forecast controller 200 decoder 202 entropy decoder 1000 processing 1201 threshold determiner 1202,1204, 1206 switch 1203 filter determiner 1025 filtering executor 1207 filter characteristic determiner 1208 processing determiner a1, b1 processor b1, b2 memory

Claims (2)

1. Encoder that encodes a video, the encoder characterized by the fact that it comprises: a set of circuits; and a memory coupled to the circuitry, wherein a prediction mode for a current block to be encoded is an affine mode, and in operation, the circuitry: derives a base motion vector to be used in prediction of the current block ; derives a first motion vector different from the base motion vector; derives a motion vector difference based on a difference between the base motion vector and the first motion vector; determines whether the motion vector difference is greater than a threshold; sets a second motion vector to a first value when the motion vector difference is determined to be greater than threshold, and sets the second motion vector to a second value being different from the first value when the motion vector difference is determined to be no greater than limit; and encodes the current block using the second motion vector, where the threshold differs between a case in which the current block is unidirectionally predicted and a case in which the current block is bidirectionally predicted. 2. Decoder that decodes a video, characterized by the fact that it comprises: set of circuits; and memory coupled to the circuitry, wherein a prediction mode for a current block to be decoded is an affine mode, and in operation, the circuitry: derives a base motion vector to be used in predicting the current block; derives a first motion vector different from the base motion vector; derives a motion vector difference based on a difference between the base motion vector and the first motion vector; determines whether the motion vector difference is greater than a threshold; sets a second motion vector to a first value when the motion vector difference is determined to be greater than the threshold, and sets the second motion vector to a second value being different from the first value when the motion vector difference is determined to be no greater than the limit; and decodes the current block using the second motion vector, where the threshold differs between a case in which the current block is unidirectionally predicted and a case in which the current block is bidirectionally predicted.