RU2810304C2 - Encoder, decoder, encoding method and decoding method - Google Patents

Abstract

FIELD: means for image encoding.

SUBSTANCE: total sum of the absolute values of the sums of the horizontal gradient values, respectively, for pairs of relative pixel positions is retrieved, as a first parameter. The total sum of the absolute values of the sums of the vertical gradient values, respectively, for pairs of relative pixel positions is retrieved, as a second parameter. The total sum of horizontally related pixel difference values, respectively, for pairs of relative pixel positions is retrieved, as a third parameter. The total sum of vertically related pixel difference values, respectively, for pairs of relative pixel positions is retrieved, as a fourth parameter. The total sum of vertically related sums of horizontal gradient values, respectively, for pairs of relative pixel positions is retrieved, as a fifth parameter. Prediction image is generated, which is to be used to encode the current block using the first, second, third, fourth and fifth parameters.

EFFECT: increase of efficiency of image encoding.

1 cl, 72 dwg

Description

Field of technology to which the invention relates

[0001] The present disclosure relates to video encoding and relates, for example, to systems, components, methods, etc. when encoding and decoding video.

State of the art

[0002] Video encoding technology is evolving from H.261 and MPEG-1 to H.264/AVC (Advanced Video Coding standard), MPEG-LA, H.265/HEVC (High Efficiency Video Coding standard) and H.266/VVC ( universal video codec). Because of such advancements, there is always a need to improve and optimize video encoding technology to handle digital video data, the volume of which continues to increase in various types of application cases.

[0003] It should be noted that H.265/HEVC is associated with one example of a legacy standard associated with the video encoding technology described above.

List of bibliographic references

Patent documents

[0004]

[NPL 1] H.265 (ISO/IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding)

The essence of the invention

Technical problem

[0005] Regarding the encoding scheme, as described above, proposals for new schemes are desirable for increasing encoding efficiency, increasing image quality, reducing processing volume, reducing circuit sizes, proper selection, etc. element that represents a reference frame, reference block, etc. or operations.

[0006] The present disclosure provides, for example, a configuration or method that can provide at least one of increased coding efficiency, increased image quality, reduced processing volume, reduced circuit size, appropriate selection, etc. element or operation. It should be noted that the present disclosure may cover possible configurations or methods that may provide advantages other than the above advantages.

The solution of the problem

[0007] For example, an encoder according to one aspect of the present disclosure is an encoder including circuitry and a memory device coupled to the circuitry. In operation, the circuit: retrieves the absolute value of the sum of horizontal gradient values, which indicates the absolute value of the sum of the horizontal gradient value at a relative pixel position in the first range and the horizontal gradient value at a relative pixel position in the second range, for each of a plurality of pairs of relative pixel positions in the first range and a relative pixel position in a second range of a plurality of pairs of relative pixel positions, the first range including a first reference block for the current block, the second range including a second reference block for the current block, the plurality of pairs of relative pixel positions being defined typically and relative to the first range and the second range; retrieves, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieves the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieves, as a second parameter, the total sum of the plurality of absolute values of the sums of the vertical gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; retrieves, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates the difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieves a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieves, as a third parameter, the total sum of the horizontally related pixel difference values of the set retrieved, respectively, for the plurality of pairs of relative pixel positions; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieves a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieves, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values retrieved, respectively, for a plurality of pairs of relative pixel positions; inverts or stores the plus or minus sign of the sum of the horizontal gradient values retrieved for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieves the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieves, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values retrieved, respectively, for the plurality of pairs of relative pixel positions; and generates a prediction image to be used to encode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0008] Some implementations of embodiments according to the present disclosure may: improve encoding efficiency; simplify the encoding/decoding process; increase encoding and/or decoding speed; and efficiently select appropriate components/operations to be used for encoding and decoding, for example, appropriate filters, block sizes, motion vectors, reference frames, reference blocks, etc.

[0009] Additional benefits and advantages according to one or more aspects of the present disclosure will become apparent from the description of the invention and the drawings. These benefits and advantages may be obtained through the features described in some embodiments, in the description of the invention and in the drawings. However, all features do not necessarily have to be provided in order to obtain one or more benefits and/or effects.

[0010] It should be noted that these general or specific aspects may be implemented as a system, method, integrated circuit, computer program, recording medium, or any combination thereof.

Advantages of the invention

[0011] A configuration or method according to an aspect of the present disclosure may provide at least one of increased encoding efficiency, increased image quality, reduced processing volume, reduced circuit size, appropriate selection, etc. element or operation. It should be noted that the configuration or method according to an aspect of the present disclosure may provide advantages other than those described above.

Brief description of drawings

[0012]

[Fig. 1] Fig. 1 is a block diagram illustrating a functional configuration of an encoder according to an embodiment.

[Fig. 2] Fig. 2 is a flowchart indicating one example of a complete decoding process performed by the encoder.

[Fig. 3] Fig. 3 is a conceptual diagram illustrating one example of blocking.

[Fig. 4A] FIG. 4A is a conceptual diagram illustrating one example of a slice configuration.

[Fig. 4B] FIG. 4B is a conceptual diagram illustrating one example of a tile configuration.

[Fig. 5A] FIG. 5A is a diagram indicating basic transformation functions for various types of transformation.

[Fig. 5B] FIG. 5B is a conceptual diagram illustrating exemplary spatially varying transforms (SVT).

[Fig. 6A] FIG. 6A is a conceptual diagram illustrating one example of a filter shape used in an adaptive loop filter (ALF).

[Fig. 6B] FIG. 6B is a conceptual diagram illustrating another example of a filter shape used in the ALF.

[Fig. 6C] FIG. 6C is a conceptual diagram illustrating another example of a filter shape used in the ALF.

[Fig. 7] Fig. 7 is a block diagram indicating one example of a specific configuration of a loop filter that functions as a deblocking filter (DBF).

[Fig. 8] Fig. 8 is a conceptual diagram indicating an example of a deblocking filter having a symmetrical filtering characteristic with respect to a block boundary.

[Fig. 9] Fig. 9 is a conceptual diagram for illustrating a block boundary on which a filtering process to remove blocking is performed.

[Fig. 10] Fig. 10 is a conceptual diagram indicating examples of Bs values.

[Fig. 11] Fig. 11 is a flowchart illustrating one example of a process performed by an encoder prediction processor.

[Fig. 12] Fig. 12 is a flowchart illustrating another example of a process performed by an encoder prediction processor.

[Fig. 13] Fig. 13 is a flowchart illustrating another example of a process performed by an encoder prediction processor.

[Fig. 14] Fig. 14 is a conceptual diagram illustrating sixty-seven intra prediction modes used in intra prediction in the embodiment.

[Fig. 15] Fig. 15 is a flowchart illustrating an exemplary basic flow of inter-prediction processing.

[Fig. 16] Fig. 16 is a flowchart illustrating one example of extracting motion vectors.

[Fig. 17] Fig. 17 is a flowchart illustrating another example of extracting motion vectors.

[Fig. 18] Fig. 18 is a flowchart illustrating another example of extracting motion vectors.

[Fig. 19] Fig. 19 is a flowchart illustrating an example of external prediction in a normal external mode.

[Fig. 20] Fig. 20 is a flowchart illustrating an example of inter-prediction in the combining mode.

[Fig. 21] Fig. 21 is a conceptual diagram for illustrating one example of a motion vector extraction process in a fusion mode.

[Fig. 22] Fig. 22 is a flowchart illustrating one example of a frame rate upconversion (FRUC) process.

[Fig. 23] Fig. 23 is a conceptual diagram for illustrating one example of pattern matching (bilateral matching) between two blocks along a motion path.

[Fig. 24] Fig. 24 is a conceptual diagram for illustrating one example of pattern matching (pattern matching) between a pattern in the current frame and a block in the reference frame.

[Fig. 25A] FIG. 25A is a conceptual diagram for illustrating one example of extracting the motion vector of each sub-block based on the motion vectors of a plurality of neighboring blocks.

[Fig. 25B] FIG. 25B is a conceptual diagram for illustrating one example of extracting the motion vector of each sub-block in an affine mode in which three control points are used.

[Fig. 26A] FIG. 26A is a conceptual diagram for illustrating an affine combining mode.

[Fig. 26B] FIG. 26B is a conceptual diagram for illustrating an affine combining mode in which two control points are used.

[Fig. 26C] FIG. 26C is a conceptual diagram for illustrating an affine combining mode that uses three control points.

[Fig. 27] Fig. 27 is a flowchart illustrating one example of a process in the affine combining mode.

[Fig. 28A] FIG. 28A is a conceptual diagram for illustrating an affine outer mode in which two control points are used.

[Fig. 28B] FIG. 28B is a conceptual diagram for illustrating an affine outer mode in which three control points are used.

[Fig. 29] Fig. 29 is a flowchart illustrating one example of an affine external mode process.

[Fig. 30A] FIG. 30A is a conceptual diagram for illustrating an affine outer mode in which the current block has three control points and the adjacent block has two control points.

[Fig. 30B] FIG. 30B is a conceptual diagram for illustrating an affine outer mode in which the current block has two control points and the adjacent block has three control points.

[Fig. 31A] FIG. 31A is a flowchart illustrating a process in the combining mode including decoder motion vector detail (DMVR).

[Fig. 31B] FIG. 31B is a conceptual diagram for illustrating one example of a DMVR process.

[Fig. 32] Fig. 32 is a flowchart illustrating one example of predictive image generation.

[Fig. 33] Fig. 33 is a flowchart illustrating another example of predictive image generation.

[Fig. 34] Fig. 34 is a flowchart illustrating another example of predictive image generation.

[Fig. 35] Fig. 35 is a flowchart illustrating one example of a prediction image correction process performed by an overlapping block motion compensation (OBMC) process.

[Fig. 36] Fig. 36 is a conceptual diagram for illustrating one example of a prediction image correction process performed by an OBMC process.

[Fig. 37] Fig. 37 is a conceptual diagram for illustrating the generation of two triangular projection images.

[Fig. 38] Fig. 38 is a conceptual diagram to illustrate the model under the condition of uniform linear motion.

[Fig. 39] Fig. 39 is a conceptual diagram for illustrating one example of a method for generating predictive images using a luminance correction process performed by a local illuminance compensation (LIC) process.

[Fig. 40] Fig. 40 is a block diagram illustrating an installation example of an encoder.

[Fig. 41] FIG. 41 is a block diagram illustrating a functional configuration of a decoder according to the embodiment.

[Fig. 42] FIG. 42 is a flowchart illustrating one example of a complete decoding process performed by the decoder.

[Fig. 43] Fig. 43 is a flowchart illustrating one example of a process performed by a decoder prediction processor.

[Fig. 44] Fig. 44 is a flowchart illustrating another example of a process performed by a decoder prediction processor.

[Fig. 45] Fig. 45 is a flowchart illustrating an example of inter-prediction in normal inter-mode in a decoder.

[Fig. 46] Fig. 46 is a block diagram illustrating an installation example of a video decoder.

[Fig. 47] Fig. 47 is a flowchart indicating one example of external prediction according to BIO.

[Fig. 48] Fig. 48 is a diagram illustrating one example of a functional configuration of an external prediction module that performs external prediction according to a BIO.

[Fig. 49] Fig. 49 is a flowchart indicating a first specific example of a BIO-based decoding process according to the embodiment.

[Fig. 50] Fig. 50 is a conceptual diagram illustrating an example of calculating horizontal gradient values according to an embodiment.

[Fig. 51] Fig. 51 is a conceptual diagram illustrating an example of calculating vertical gradient values according to the embodiment.

[Fig. 52] Fig. 52 is a flowchart indicating a third specific example of a BIO-based decoding process according to the embodiment.

[Fig. 53] Fig. 53 is a flowchart indicating a fourth specific example of a BIO-based decoding process according to the embodiment.

[Fig. 54] Fig. 54 is a flowchart indicating an operation performed by the encoder according to the embodiment.

[Fig. 55] Fig. 55 is a flowchart indicating an operation performed by the decoder according to the embodiment.

[Fig. 56] Fig. 56 is a block diagram illustrating a general configuration of a content delivery system for implementing a content distribution service.

[Fig. 57] Fig. 57 is a conceptual diagram illustrating one example of a coding structure in scalable coding.

[Fig. 58] Fig. 58 is a conceptual diagram illustrating one example of a coding structure in scalable coding.

[Fig. 59] Fig. 59 is a conceptual diagram illustrating an example of a web page display screen.

[Fig. 60] Fig. 60 is a conceptual diagram illustrating an example of a web page display screen.

[Fig. 61] FIG. 61 is a block diagram illustrating one example of a smartphone.

[Fig. 62] Fig. 62 is a block diagram illustrating an example configuration of a smartphone.

Detailed Description of Embodiments

[0013] In recent years, video encoding using bidirectional optical flow has been studied. Bidirectional optical flow is also called "BIO" or "BDOF".

[0014] For example, in bidirectional optical flow, a prediction image is generated based on the optical flow equation. More specifically, in bidirectional optical flow, a prediction image having a prediction value adjusted for each pixel is generated using a parameter extracted based on the pixel value of the reference image for each block and the gradient value of the reference image for the block. The use of bidirectional optical flow provides a high probability that a highly accurate predictive image is generated.

[0015] For example, the encoder encodes a difference image between the prediction image and the original image. The decoder then decodes the difference image and adds the difference image and the prediction image to generate a reconstructed image. Using a high-precision predictive image can reduce the amount of difference image codes. In other words, using bidirectional optical flow provides a high probability that the code size of the resulting video is reduced.

[0016] On the other hand, a parameter for use in bidirectional optical flow is extracted based on the pixel value and the gradient value at each pixel position in the reference image. For this reason, the calculation performed for each pixel in the parameter extraction to be used in bidirectional optical flow may increase the computational amount, which may increase the size of the circuits.

[0017] In view of this, for example, an encoder according to an aspect of the present disclosure is an encoder including circuitry; and a memory device connected to the circuit. In operation, the circuit: retrieves the absolute value of the sum of horizontal gradient values, which indicates the absolute value of the sum of the horizontal gradient value at a relative pixel position in the first range and the horizontal gradient value at a relative pixel position in the second range, for each of a plurality of pairs of relative pixel positions in the first range and a relative pixel position in a second range of a plurality of pairs of relative pixel positions, wherein the first range includes a first reference block for the current block, wherein the second range includes a second reference block for the current block, wherein the plurality of relative pixel position pairs are defined typically and relative to the first range and the second range; retrieves, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieves the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieves, as a second parameter, the total sum of the plurality of absolute values of the sums of the vertical gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; retrieves, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates the difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieves a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieves, as a third parameter, the total sum of the horizontally related pixel difference values of the set retrieved, respectively, for the plurality of pairs of relative pixel positions; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieves a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieves, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values retrieved, respectively, for a plurality of pairs of relative pixel positions; inverts or stores the plus or minus sign of the sum of the horizontal gradient values retrieved for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieves the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieves, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values retrieved, respectively, for the plurality of pairs of relative pixel positions; and generates a prediction image to be used to encode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0018] Thus, significant multiplication, which requires a large amount of computation in the calculation performed for each pixel position, can be reduced, which allows many parameters to be extracted to form a predictive image with a small amount of computation. Accordingly, the amount of processing during encoding may be reduced. In addition, the prediction image may be suitably generated based on a plurality of parameters including a parameter associated with horizontal gradient values, a parameter associated with vertical gradient values, and a parameter associated with both horizontal gradient values and vertical gradient values. .

[0019] For example, the circuit extracts: a first parameter according to expression (11.1) described below: a second parameter according to expression (11.2) described below: a third parameter according to expression (11.3) described below: a fourth parameter according to expression (11.4) described below; and the fifth parameter according to expression (11.5) described below. In expressions: Ω denotes a set of multiple pairs of relative pixel positions; [i, j] denotes each of the relative pixel positions included in the plurality of relative pixel position pairs; for each of a plurality of pairs of relative pixel positions, I _x ⁰ denotes a horizontal gradient value at a relative pixel position in a first range, and I _x ¹ denotes a horizontal gradient value at a relative pixel position in a second range; for each of a plurality of pairs of relative pixel positions, I _y ⁰ denotes a vertical gradient value at a relative pixel position in a first range, and I _y ¹ denotes a vertical gradient value at a relative pixel position in a second range; I ⁰ denotes a pixel value at a relative pixel position in the first range; I ¹ denotes a pixel value at a relative pixel position in the second range; abs (I _x ¹ + I _x ⁰ ) denotes the absolute value of I _x ¹ +I _x ⁰ ; sign (I _x ¹ +I _x ⁰ ) denotes the plus or minus sign I _x ¹ +I _x ⁰ ; abs (I _y ¹ +I _y ⁰ ) denotes the absolute value of I _y ¹ +I _y ⁰ ; and sign (I _y ¹ +I _y ⁰ ) denotes the plus or minus sign of I _y ¹ +I _y ⁰ .

[0020] Thus, multiple parameters can be extracted with little computation using pixel values, horizontal gradient values, and vertical gradient values.

[0021] In addition, for example, the circuit: retrieves the sixth parameter by dividing the third parameter by the first parameter; extracts the seventh parameter by subtracting the product of the fifth parameter and the sixth parameter from the fourth parameter and dividing the result of the subtraction by the second parameter; and generates a forecast image using the sixth parameter and the seventh parameter.

[0022] Thus, the plurality of parameters can be suitably narrowed down to two parameters corresponding to the horizontal direction and the vertical direction. The parameter associated with the horizontal gradient values may be appropriately included in the parameter corresponding to the horizontal direction. A parameter associated with the vertical gradient values, a parameter associated with both the horizontal gradient values and the vertical gradient values, and a parameter corresponding to the horizontal direction may be suitably included in the parameter corresponding to the vertical direction. The prediction image can be suitably generated using two parameters.

[0023] In addition, for example, the circuit extracts: a sixth parameter according to expression (10.8) described below; and the seventh parameter according to expression (10.9) described below. In expressions: sG _x denotes the first parameter; sG _y denotes the second parameter; sG _x dI denotes the third parameter; sG _y dI denotes the fourth parameter; sG _x G _y denotes the fifth parameter; u denotes the sixth parameter; and Bits is a function that returns a value obtained by rounding up the binary logarithm of the argument to an integer.

[0024] In this way, two parameters corresponding to the horizontal direction and the vertical direction can be extracted with a small amount of computation.

[0025] In addition, for example, the circuit generates a prediction image by retrieving a prediction pixel value at the position of a current pixel included in the current block using: a first pixel value at a first pixel position corresponding to the position of the current pixel in the first reference block; the first horizontal gradient value at the first pixel position; the first vertical gradient value at the first pixel position; a second pixel value at a second pixel position corresponding to the position of the current pixel in the second reference block; a second horizontal gradient value at a second pixel position; a second vertical gradient value at a second pixel position; sixth parameter; and the seventh parameter.

[0026] This provides a possibility that the prediction image is generated using, for example, two parameters corresponding to the horizontal direction and the vertical direction, and the two parameters corresponding to the horizontal direction and the vertical direction are properly reflected in the prediction image.

[0027] In addition, for example, the circuit extracts the predicted pixel value by dividing by 2 the sum of the first pixel value, the second pixel value, the first correction value, and the second correction value. The first correction value corresponds to the product of the difference between the first horizontal gradient value and the second horizontal gradient value and the sixth parameter, and the second correction value corresponds to the product of the difference between the first vertical gradient value and the second vertical gradient value and the seventh parameter.

[0028] Thus, a prediction image can be suitably generated using, for example, two parameters corresponding to a horizontal direction and a vertical direction.

[0029] In addition, for example, the circuit extracts: the predicted pixel value according to expression (10.10). In the expression: I ⁰ denotes the first pixel value; I ¹ denotes the second pixel value; u denotes the sixth parameter; I _x ⁰ denotes the first value of the horizontal gradient; I _x ¹ denotes the second value of the horizontal gradient; v denotes the seventh parameter; I _y ⁰ denotes the first value of the vertical gradient; and I _y ¹ denotes the second value of the vertical gradient.

[0030] Thus, the prediction image can be suitably generated according to an expression associated with, for example, two parameters corresponding to the horizontal direction and the vertical direction.

[0031] Moreover, for example, a decoder according to an aspect of the present disclosure is a decoder including a circuit and a memory device coupled to the circuit. In operation, the circuit: retrieves the absolute value of the sum of horizontal gradient values, which indicates the absolute value of the sum of the horizontal gradient value at a relative pixel position in the first range and the horizontal gradient value at a relative pixel position in the second range, for each of a plurality of pairs of relative pixel positions in the first range and a relative pixel position in a second range of a plurality of pairs of relative pixel positions, wherein the first range includes a first reference block for the current block, wherein the second range includes a second reference block for the current block, wherein the plurality of relative pixel position pairs are defined typically and relative to the first range and the second range; retrieves, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieves the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieves, as a second parameter, the total sum of the plurality of absolute values of the sums of the vertical gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; retrieves, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates the difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieves a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieves, as a third parameter, the total sum of the horizontally related pixel difference values of the set retrieved, respectively, for the plurality of pairs of relative pixel positions; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieves a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieves, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values retrieved, respectively, for a plurality of pairs of relative pixel positions; inverts or stores the plus or minus sign of the sum of the horizontal gradient values retrieved for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieves the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieves, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values retrieved, respectively, for the plurality of pairs of relative pixel positions; and generates a prediction image to be used to decode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0032] Thus, significant multiplication, which requires a large amount of computation in the calculation performed for each pixel position, can be reduced, which allows many parameters to be extracted to form a predictive image with a small amount of calculation. Accordingly, the processing amount of decoding may be reduced. In addition, the prediction image may be suitably generated based on a plurality of parameters including a parameter associated with horizontal gradient values, a parameter associated with vertical gradient values, and a parameter associated with both horizontal gradient values and vertical gradient values. .

[0033] In addition, for example, the circuit extracts: a first parameter according to expression (11.1) described below: a second parameter according to expression (11.2) described below: a third parameter according to expression (11.3) described below: a fourth parameter according to expression (11.4) ), described below; and the fifth parameter according to expression (11.5) described below. In expressions: Ω denotes a set of multiple pairs of relative pixel positions; [i, j] denotes each of the relative pixel positions included in the plurality of relative pixel position pairs; for each of a plurality of pairs of relative pixel positions, I _x ⁰ denotes a horizontal gradient value at a relative pixel position in a first range, and I _x ¹ denotes a horizontal gradient value at a relative pixel position in a second range; for each of a plurality of pairs of relative pixel positions, I _y ⁰ denotes a vertical gradient value at a relative pixel position in a first range, and I _y ¹ denotes a vertical gradient value at a relative pixel position in a second range; I ⁰ denotes a pixel value at a relative pixel position in the first range; I ¹ denotes a pixel value at a relative pixel position in the second range; abs (I _x ¹ +I _x ⁰ ) denotes the absolute value of I _x ¹ +I _x ⁰ ; sign (I _x ¹ +I _x ⁰ ) denotes the plus or minus sign I _x ¹ +I _x ⁰ ; abs (I _y ¹ +I _y ⁰ ) denotes the absolute value of I _y ¹ +I _y ⁰ ; and sign (I _y ¹ +I _y ⁰ ) denotes the plus or minus sign of I _y ¹ +I _y ⁰ .

[0034] Thus, multiple parameters can be extracted with little computation using pixel values, horizontal gradient values, and vertical gradient values.

[0035] In addition, for example, the circuit: retrieves the sixth parameter by dividing the third parameter by the first parameter; extracts the seventh parameter by subtracting the product of the fifth parameter and the sixth parameter from the fourth parameter and dividing the result of the subtraction by the second parameter; and generates a forecast image using the sixth parameter and the seventh parameter.

[0036] Thus, a plurality of parameters can be properly integrated into two parameters corresponding to the horizontal direction and the vertical direction. The parameter associated with the horizontal gradient values may be suitably included in the parameter corresponding to the horizontal direction. A parameter associated with the vertical gradient values, a parameter associated with both the horizontal gradient values and the vertical gradient values, and a parameter corresponding to the horizontal direction may be suitably included in the parameter corresponding to the vertical direction. The prediction image can be suitably generated using two parameters.

[0037] In addition, for example, the circuit extracts: a sixth parameter according to expression (10.8) described below; and the seventh parameter according to expression (10.9) described below. In expressions: sG _x denotes the first parameter; sG _y denotes the second parameter; sG _x dI denotes the third parameter; sG _y dI denotes the fourth parameter; sG _x G _y denotes the fifth parameter; u denotes the sixth parameter; and Bits is a function that returns a value obtained by rounding up the binary logarithm of the argument to an integer.

[0038] In this way, two parameters corresponding to the horizontal direction and the vertical direction can be extracted with a small amount of computation.

[0039] In addition, for example, the circuit generates a prediction image by retrieving a prediction pixel value at a position of a current pixel included in a current block using: a first pixel value at a first pixel position corresponding to a position of a current pixel in a first reference block; the first horizontal gradient value at the first pixel position; the first vertical gradient value at the first pixel position; a second pixel value at a second pixel position corresponding to the position of the current pixel in the second reference block; a second horizontal gradient value at a second pixel position; a second vertical gradient value at a second pixel position; sixth parameter; and the seventh parameter.

[0040] This provides a possibility that the prediction image is generated using, for example, two parameters corresponding to the horizontal direction and the vertical direction, and the two parameters corresponding to the horizontal direction and the vertical direction are properly reflected in the prediction image.

[0041] In addition, for example, the circuit extracts the predicted pixel value by dividing by 2 the sum of the first pixel value, the second pixel value, the first correction value, and the second correction value. The first correction value corresponds to the product of the difference between the first horizontal gradient value and the second horizontal gradient value and the sixth parameter, and the second correction value corresponds to the product of the difference between the first vertical gradient value and the second vertical gradient value and the seventh parameter.

[0042] Thus, a prediction image can be suitably generated using, for example, two parameters corresponding to a horizontal direction and a vertical direction.

[0043] In addition, for example, the circuit extracts: the predicted pixel value according to expression (10.10). In the expression: I ⁰ denotes the first pixel value; I ¹ denotes the second pixel value; u denotes the sixth parameter; I _x ⁰ denotes the first value of the horizontal gradient; I _x ¹ denotes the second value of the horizontal gradient; v denotes the seventh parameter; I _y ⁰ denotes the first value of the vertical gradient; and I _y ¹ denotes the second value of the vertical gradient.

[0044] Thus, the prediction image can be suitably generated according to an expression associated with, for example, two parameters corresponding to the horizontal direction and the vertical direction.

[0045] In addition, for example, the encoding method according to an aspect of the present disclosure is an encoding method including: retrieving an absolute value of the sum of horizontal gradient values, which indicates the absolute value of the sum of the horizontal gradient value at a relative pixel position in the first range and the horizontal gradient value at a relative pixel position in a second range, for each of a plurality of pairs of relative pixel positions in the first range and a relative pixel position in a second range of the plurality of pairs of relative pixel positions, wherein the first range includes a first reference block for the current block, wherein the second range includes including a second reference block for the current block, wherein a plurality of pairs of relative pixel positions are determined generally and relatively for the first range and the second range; retrieving, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieving the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieving, as a second parameter, the total sum of a plurality of absolute values of the sums of vertical gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieving, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates a difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverting or storing the plus or minus sign of the pixel difference value extracted for each of the plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieving a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieving, as a third parameter, the total sum of the horizontally related pixel difference values of the set extracted, respectively, for the plurality of pairs of relative pixel positions; inverting or storing a plus or minus sign of the pixel difference value extracted for each of the plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieving a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieving, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values extracted, respectively, for a plurality of pairs of relative pixel positions; inverting or storing the plus or minus sign of the sum of the horizontal gradient values extracted for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieving the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieving, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; and generating a prediction image to be used to encode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0046] Thus, significant multiplication, which requires a large amount of computation in the calculation performed for each pixel position, can be reduced, which allows many parameters to be extracted to form a predictive image with a small amount of calculation. Accordingly, the amount of processing during encoding may be reduced. In addition, the prediction image may be suitably generated based on a plurality of parameters including a parameter associated with horizontal gradient values, a parameter associated with vertical gradient values, and a parameter associated with both horizontal gradient values and vertical gradient values. .

[0047] In addition, for example, a decoding method according to an aspect of the present disclosure is a decoding method including: retrieving an absolute value of a sum of horizontal gradient values, which indicates an absolute value of the sum of a horizontal gradient value at a relative pixel position in a first range and a horizontal gradient value at a relative pixel position in a second range, for each of a plurality of pairs of relative pixel positions in the first range and a relative pixel position in a second range of the plurality of pairs of relative pixel positions, wherein the first range includes a first reference block for the current block, wherein the second range includes including a second reference block for the current block, wherein a plurality of pairs of relative pixel positions are determined generally and relatively for the first range and the second range; retrieving, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieving the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieving, as a second parameter, the total sum of a plurality of absolute values of the sums of vertical gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieving, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates a difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverting or storing the plus or minus sign of the pixel difference value extracted for each of the plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieving a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieving, as a third parameter, the total sum of the horizontally related pixel difference values of the set extracted, respectively, for the plurality of pairs of relative pixel positions; inverting or storing a plus or minus sign of the pixel difference value extracted for each of the plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieving a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieving, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values extracted, respectively, for a plurality of pairs of relative pixel positions; inverting or storing the plus or minus sign of the sum of the horizontal gradient values extracted for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieving the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieving, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; and generating a prediction image to be used to decode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0048] Thus, significant multiplication, which requires a large amount of computation in the calculation performed for each pixel position, can be reduced, which allows many parameters to be extracted to form a predictive image with a small amount of calculation. Accordingly, the processing amount of decoding may be reduced. In addition, the prediction image may be suitably generated based on a plurality of parameters including a parameter associated with horizontal gradient values, a parameter associated with vertical gradient values, and a parameter associated with both horizontal gradient values and vertical gradient values. .

[0049] Alternatively, the encoder according to an aspect of the present disclosure is an encoder that includes a partitioner, an intra prediction module, an inter prediction module, a transformer, a quantizer, and an entropy encoder.

[0050] The splitter splits the current frame to be encoded included in the video into a plurality of blocks. The intra prediction module performs intra prediction to generate a prediction image for the current block to be encoded in the current frame using a reference image in the current frame. The external prediction module performs external prediction to generate a prediction image for the current block to be encoded using a reference image in a reference frame different from the current frame.

[0051] The converter converts the prediction error signal between (i) either the prediction image generated by the intra prediction module or the prediction image generated by the inter prediction module and (ii) the current block image to generate a current block transform coefficient signal. The quantizer quantizes the signal of the conversion coefficients. The entropy encoder encodes the signal with quantized transform coefficients.

[0052] In addition, for example, the external prediction module: retrieves the absolute value of the sum of the horizontal gradient values, which indicates the absolute value of the sum of the horizontal gradient value at the relative pixel position in the first range and the horizontal gradient value at the relative pixel position in the second range, for each of a plurality of pairs of relative pixel positions in a first range and a relative pixel position in a second range of the plurality of pairs of relative pixel positions, the first range including a first reference block for the current block, the second range including a second reference block for the current block, the plurality of pairs of relative pixel positions are determined generally and relative to the first range and the second range; retrieves, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieves the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieves, as a second parameter, the total sum of the plurality of absolute values of the sums of the vertical gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; retrieves, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates the difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieves a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieves, as a third parameter, the total sum of the horizontally related pixel difference values of the set retrieved, respectively, for the plurality of pairs of relative pixel positions; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieves a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieves, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values retrieved, respectively, for a plurality of pairs of relative pixel positions; inverts or stores the plus or minus sign of the sum of the horizontal gradient values retrieved for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieves the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; extracts, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; and generates a prediction image to be used to encode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0053] Alternatively, for example, a decoder according to an aspect of the present disclosure includes an entropy decoder, an inverse quantizer, an inverse converter, an intra prediction module, an inter prediction module, and an adder (reconstruction module).

[0054] The entropy decoder decodes the quantized transform coefficient signal of the current block to be decoded in the current frame to be decoded into video. An inverse quantizer inversely quantizes the signal of the quantized transform coefficients. The inverse converter inversely converts the transform coefficient signal to obtain a prediction error signal of the current block.

[0055] The intra prediction module performs intra prediction to generate a prediction image for the current block to be decoded using a reference image in the current frame to be decoded. The external prediction module performs external prediction to generate a prediction image for the current block to be decoded using a reference image in a reference frame different from the current frame. The adder adds the prediction error signal and either the prediction image generated by the internal prediction module or the prediction image generated by the external prediction module to reconstruct the image of the current block.

[0056] In addition, for example, the external prediction module: retrieves the absolute value of the sum of the horizontal gradient values, which indicates the absolute value of the sum of the horizontal gradient value at the relative pixel position in the first range and the horizontal gradient value at the relative pixel position in the second range, for each of a plurality of pairs of relative pixel positions in a first range and a relative pixel position in a second range of the plurality of pairs of relative pixel positions, the first range including a first reference block for the current block, the second range including a second reference block for the current block, the plurality of pairs of relative pixel positions are determined generally and relative to the first range and the second range; retrieves, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieves the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieves, as a second parameter, the total sum of the plurality of absolute values of the sums of the vertical gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; retrieves, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates the difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieves a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieves, as a third parameter, the total sum of the horizontally related pixel difference values of the set retrieved, respectively, for the plurality of pairs of relative pixel positions; inverts or preserves the plus or minus sign of the pixel difference value retrieved for each of a plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieves a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieves, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values retrieved, respectively, for a plurality of pairs of relative pixel positions; inverts or stores the plus or minus sign of the sum of the horizontal gradient values retrieved for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieves the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; extracts, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; and generates a prediction image to be used to decode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0057] Additionally, these general or specific aspects may be implemented using a system, device, method, integrated circuit, computer program, or non-volatile computer readable recording medium such as a CD-ROM, or any combination of systems, devices, methods, integrated circuits , computer programs or computer-readable recording media.

[0058] Embodiments are described hereinafter with reference to the drawings. It should be noted that the embodiments described below show a general or specific example. The numerical values, shapes, materials, components, arrangement and connection of components, steps, relationship and order of steps, etc., indicated in the following embodiments are merely examples and are therefore not intended to limit the scope of the claims.

[0059] Embodiments of the encoder and decoder are described below. The embodiments are examples of an encoder and decoder to which the processes and/or configurations presented in the description of aspects of the present disclosure are applicable. The processes and/or configurations may also be implemented in an encoder and decoder different from the encoder and decoder of the embodiments. For example, with respect to the processes and/or configurations applied to the embodiments, any of the following may be implemented:

[0060] (1) Any of the components of the encoder or decoder according to the embodiments presented in the description of aspects of the present disclosure may be replaced or combined with another component presented anywhere in the description of aspects of the present disclosure.

[0061] (2) In the encoder or decoder according to the embodiments, discretionary changes may be made to functions or processes performed by one or more components of the encoder or decoder, such as adding, replacing, deleting, etc. functions or processes. For example, any function or process may be substituted for or combined with another function or process presented anywhere in the description of aspects of this disclosure.

[0062] (3) In the methods implemented by the encoder or decoder according to the embodiments, discretionary changes such as adding, replacing and deleting one or more processes included in the method may be made. For example, any process in the method may be replaced by or combined with another process presented anywhere in the description of aspects of the present disclosure.

[0063] (4) One or more components included in an encoder or decoder according to embodiments may be combined with a component presented anywhere in the description of aspects of the present disclosure may be combined with a component including one or more functions presented in anywhere in the description of aspects of the present disclosure, and may be combined with a component that implements one or more of the processes implemented by the component presented in the description of aspects of the present disclosure.

[0064] (5) A component that includes one or more encoder or decoder functions according to embodiments, or a component that implements one or more encoder or decoder processes according to embodiments, may be combined or replaced by a component presented anywhere in the description aspects of this disclosure, a component that includes one or more of the functions presented anywhere in the description of aspects of this disclosure, or a component that implements one or more processes presented anywhere in the description of aspects of this disclosure.

[0065] (6) In methods implemented by an encoder or decoder according to embodiments, any of the processes included in the method may be replaced or combined with a process presented anywhere in the description of aspects of this disclosure, or with any corresponding or equivalent process .

[0066] (7) One or more processes included in the methods implemented by the encoder or decoder according to the embodiments may be combined with a process presented anywhere in the description of aspects of the present disclosure.

[0067] (8) The implementation of the processes and/or configurations presented in the description of aspects of the present disclosure is not limited to the encoder or decoder according to the embodiments. For example, the processes and/or configurations may be implemented in an apparatus used for a purpose other than the moving frame encoder or moving frame decoder disclosed in the embodiments.

[0068] Encoder

First, the encoder according to the embodiment is described. Fig. 1 is a block diagram illustrating a functional configuration of the encoder 100 according to the embodiment. Encoder 100 is a video encoder that encodes video in units of blocks.

[0069] As illustrated in FIG. 1, the encoder 100 is a device that encodes an image in units of blocks, and includes a partitioner 102, a subtractor 104, a converter 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse converter 114, an adder 116, and a memory 118. blocks, a loop filter 120, a frame memory 122, an internal prediction module 124, an external prediction module 126, and a prediction controller 128.

[0070] Encoder 100 is implemented, for example, as a general processor and storage device. In this case, when the software program stored in the memory device is executed by the processor, the processor functions as a partitioner 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116 , a loop filter 120, an internal prediction module 124, an external prediction module 126, and a prediction controller 128. Alternatively, encoder 100 may be implemented as one or more dedicated electronic circuits corresponding to partitioner 102, subtractor 104, transformer 106, quantizer 108, entropy encoder 110, inverse quantizer 112, inverse converter 114, adder 116, loop filter 120, module 124 internal prediction module 126 external prediction and controller 128 prediction.

[0071] Hereinafter, this document describes the complete flow of processes performed by the encoder 100, and thereafter describes each of the constituent elements included in the encoder 100.

[0072] General flow of the encoding process

Fig. 2 is a flowchart indicating one example of a complete decoding process performed by encoder 100.

[0073] First, the partitioning unit 102 of the encoder 100 partitions each of the frames included in the input image, which is a video, into a plurality of blocks having a fixed size (eg, 128×128 pixels) (step Sa_1). The partitioning module 102 then selects a partitioning pattern for a fixed-size block (also called “block shape”) (step Sa_2). In other words, the partitioning module 102 further partitions the fixed-size block into a plurality of blocks that form a selected partitioning pattern. Encoder 100 performs, for each of a plurality of blocks, steps Sa_3-Sa_9 for the block (which is the current block to be encoded).

[0074] In other words, the prediction processor, which includes all or part of the internal prediction module 124, the external prediction module 126, and the prediction controller 128, generates a prediction signal (also called a “prediction block”) of the current block to be encoded (also called "current block") (stage Sa_3).

[0075] Next, the subtractor 104 generates the difference between the current block and the prediction block as a prediction remainder (also called a “difference block”) (step Sa_4).

[0076] Next, the converter 106 converts the difference block, and the quantizer 108 quantizes the result to generate a plurality of quantized coefficients (step Sa_5). It should be noted that a block having a plurality of quantized coefficients is also called a "coefficient block".

[0077] Next, the entropy encoder 110 encodes (specifically, entropy encodes) a block of coefficients and a prediction parameter associated with generating a prediction signal to generate a coded signal (step Sa_6). It should be noted that the encoded signal is also called a "coded bitstream", "compressed bitstream" or "stream".

[0078] Next, the inverse quantizer 112 inversely quantizes the coefficient block, and the inverse transformer 114 inversely transforms the result to recover a plurality of prediction residuals (ie, a difference block) (step Sa_7).

[0079] Next, the adder 116 adds the prediction block with the reconstructed difference block to restore the current block as a reconstructed image (also called a “recovered block” or a “decoded image block”) (step Sa_8). Thus, a reconstructed image is formed.

[0080] When the reconstructed image is generated, the loop filter 120 filters the reconstructed image as necessary (step Sa_9).

[0081] The encoder 100 then determines whether or not encoding of the entire frame is complete (step Sa_10). When determining that encoding is not yet complete ("No" in step Sa_10), the processes from step Sa_2 are executed repeatedly.

[0082] Although the encoder 100 selects one partitioning pattern for a fixed-size block and encodes each block according to the partitioning pattern in the above example, it should be noted that each block may be encoded according to a corresponding one of a plurality of partitioning patterns. In this case, the encoder 100 may estimate costs for each of a plurality of partitioning patterns and, for example, may select a coded signal obtained by encoding according to the partitioning pattern that results in the lowest cost as the coded signal that is output.

[0083] As illustrated, the processes in steps Sa_1-Sa_10 are executed sequentially by the encoder 100. Alternatively, two or more of the processes may be executed in parallel, the processes may be reordered, etc.

[0084] Partitioner

The partitioner 102 splits each of the frames included in the input video into a plurality of blocks and outputs each block to the subtractor 104. For example, the partitioner 102 first partitions the frame into blocks of a fixed size (eg, 128×128). Other fixed block sizes may be used. A fixed size block is also called a "coding tree unit (CTU)". Partitioner 102 then partitions each fixed-size block into variable-size blocks (eg, 64×64 or smaller) based on recursive quadtree and/or binary block partitioning. In other words, the partitioning module 102 selects a partitioning pattern. The variable size block is also called a "coding unit (CU)", "prediction unit (PU)" or "transformation unit (TU)". It should be noted that, in various kinds of processing examples, there is no need to distinguish between CU, PU and TU; all or some of the blocks in a frame can be processed in CU, PU or TU units.

[0085] FIG. 3 is a conceptual diagram illustrating one example of blocking according to an embodiment. In fig. 3, solid lines represent block boundaries for blocks divided by quadtree blocking, and dotted lines represent block boundaries for blocks divided by binary tree blocking.

[0086] Here, block 10 is a square block having 128×128 pixels (128×128 block). This 10 128x128 block is first split into four 64x64 square blocks (quadtree blocking).

[0087] The top left 64x64 block is further vertically split into two 32x64 rectangular blocks, and the left 32x64 block is further vertically split into two 16x64 rectangular blocks (binary tree chunking). As a result, the top left 64x64 block is split into two 16x64 blocks 11 and 12 and one 32x64 block 13.

[0088] The top right 64×64 block is horizontally split into two 64×32 rectangular blocks 14 and 15 (binary tree blocking).

[0089] The bottom left 64x64 block is first split into four 32x32 square blocks (quadtree blocking). The top left block and bottom right block of four 32x32 blocks are further broken down. The top left 32x32 block is vertically split into two 16x32 rectangular blocks, and the right 16x32 block is further split horizontally into two 16x16 blocks (binary tree blocking). The bottom right 32x32 block is horizontally split into two 32x16 blocks (binary tree blocking). As a result, the bottom left 64x64 block is split into a 16x32 block, two 16x16 blocks 17 and 18, two 32x32 blocks 19 and 20, and two 32x16 blocks 21 and 22.

[0090] The bottom right block 23 64×64 is not broken.

[0091] As described above, FIG. 3, block 10 is divided into thirteen blocks 11-23 of variable size based on recursive quadtree and binary tree blocking. This type of partitioning is also called "quadtree plus binary tree (QTBT) partitioning."

[0092] It should be noted that, in FIG. 3, one block is split into four or two blocks (quadtree or binary tree chunking), but splitting is not limited to these examples. For example, one block can be split into three blocks (ternary block splitting). Partitioning that includes such ternary block partitioning is also called "multi-type tree (MBT) partitioning."

[0093] Frame structure: slice/tile

A frame may be configured in units of one or more slices or tiles to decode the frame in parallel. A frame configured in units of one or more slices or tiles may be configured by the partitioning unit 102.

[0094] Slices are basic coding units included in a frame. A frame may include, for example, one or more slices. In addition, a slice includes one or more consecutive coding tree units (CTUs).

[0095] FIG. 4A is a conceptual diagram illustrating one example of a slice configuration. For example, a frame includes an 11x8 CTU and is divided into four slices (slices 1-4). Slice 1 includes sixteen CTUs, slice 2 includes twenty-one CTUs, slice 3 includes twenty-nine CTUs, and slice 4 includes twenty-two CTUs. Here, each CTU in the frame belongs to one of the slices. The shape of each slice is the shape obtained by dividing the frame horizontally. The boundary of each slice does not have to coincide with the end of the image and can coincide with any of the boundaries between CTUs in the image. The processing order of CTUs in a slice (encoding order or decoding order), for example, is the raster scan order. The slice includes header information and encoded data. The attributes of a slice may be described in header information. Signs include the CTU address of the top CTU in the slice, slice type, etc.

[0096] A tile is a unit of rectangular area included in a frame. Each of the tiles can be assigned a number called a "TileId" in raster scan order.

[0097] FIG. 4B is a conceptual diagram indicating an example of a tile configuration. For example, a frame includes an 11x8 CTU and is divided into four rectangular region tiles (tiles 1-4). When tiles are used, the CTU processing order changes relative to the processing order if tiles are not used. When tiles are not used, the CTUs in the frame are processed in raster scan order. When tiles are used, at least one CTU in each tile is processed in raster scan order. For example, as illustrated in FIG. 4B, the processing order of CTUs included in tile 1 is an order that starts from the left end of the first line of tile 1 to the right end of the first line of tile 1 and then starts from the left end of the second line of tile 1 to the right end of the second line of tile 1.

[0098] It should be noted that one tile may include one or more slices, and one slice may include one or more tiles.

[0099] Subtractor

The subtractor 104 subtracts the prediction signal (the prediction sample that is input from the prediction controller 128 specified below) from the original signal (the original sample), in units of the block input from the partitioning unit 102 and split by the partitioning unit 102. In other words, the subtractor 104 calculates the prediction errors (also called “residuals”) of the block to be encoded (hereinafter also called the “current block”). Subtractor 104 then outputs the calculated prediction errors (residuals) to converter 106.

[0100] The original signal is a signal that is input to the encoder 100 and represents an image of each frame included in the video (eg, a luma signal and two chrominance signals). Hereinafter in this document, a signal representing an image is also called a “sample”.

[0101] Converter

Converter 106 converts spatial domain prediction errors into frequency domain transform coefficients and outputs the transform coefficients to quantizer 108. More specifically, transformer 106 applies, for example, a specified discrete cosine transform (DCT) or discrete sine transform (DST) to the spatial domain prediction errors. areas. The specified DCT or DST can be preset.

[0102] It should be noted that the transformer 106 can adaptively select a transform type from a plurality of transform types and convert prediction errors into transform coefficients by using a transform basis function corresponding to the selected transform type. This kind of transformation is also called "explicit multiple base transformation (EMT)" or "adaptive multiple transformation (AMT)".

[0103] Conversion types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I and DST-VII. Fig. 5A is a diagram indicating basic transformation functions for example transformation types. In fig. 5A, N indicates the number of input pixels. For example, the selection of a transformation type from a plurality of transformation types may depend on the prediction type (one of intra-prediction and outer prediction) and may depend on the intra-prediction mode.

[0104] Information indicating whether or not such EMT or AMT should be applied (called, for example, "EMT flag" or "AMT flag"), and information indicating the selected type of conversion are normally signaled at the CU level . It should be noted that the signaling of this information does not necessarily have to be performed at the CU level and may be performed at another level (eg, bit sequence level, frame level, slice level, tile level, or CTU level).

[0105] In addition, the converter 106 can re-convert the conversion coefficients (conversion result). This re-transformation is also called "adaptive secondary transform (AST)" or "non-separable secondary transform (NSST)". For example, converter 106 performs reconversion in units of a subblock (eg, a 4x4 subblock) included in a block of transform coefficients corresponding to the intra prediction error. Information indicating whether or not to apply NSST and information associated with a transformation matrix to be used in NSST are normally signaled at the CU level. It should be noted that signaling of this information does not necessarily have to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, or CTU level).

[0106] The converter 106 may use a separable transform and an inseparable transform. Separable transformation is a method in which the transformation is performed multiple times by separately performing the transformation for each of a certain number of directions according to the number of dimensions of the inputs. An inseparable transformation is a method of performing a collective transformation in which two or more dimensions in multidimensional inputs are collectively treated as one dimension.

[0107] In one example of a non-separable transformation, when the input is a 4x4 block, the 4x4 block is treated as a single array including sixteen elements, and the transformation applies a 16x16 transformation matrix to the array.

[0108] In another example of a non-separable transformation, a 4x4 input block is treated as a single array including sixteen elements, and a transformation (hypercubic Givens transform) can then be performed in which a Givens rotation is performed on the array multiple times.

[0109] When converting in the converter 106, the types of bases to be converted to the frequency domain according to the domains in the CU can be switched. Examples include spatially varying transforms (SVT). In SVT, as illustrated in FIG. 5B, the CUs are divided into two equal regions horizontally or vertically, and only one of the regions is converted to the frequency domain. The basic transformation type can be specified for each area. For example, DST7 and DST8 are used. In this example, only one of the two areas in the CU is converted, and the other is not converted. However, both of these two areas can transform. In addition, the partitioning method is not limited to partitioning into two equal areas and can be more flexible. For example, the CU may be partitioned into four equal areas, or information indicating the partition may be encoded separately and signaled identically to the CU partition. It should be noted that SVT is also called "sub-block transform (SBT)".

[0110] Quantizer

Quantizer 108 quantizes transform coefficients output from transformer 106. More specifically, quantizer 108 scans, in a specific scanning order, transform coefficients of the current block and quantizes the scanned transform coefficients based on quantization parameters (QPs) corresponding to the transform coefficients. Quantizer 108 then outputs quantized transform coefficients (hereinafter also referred to as “quantized coefficients”) of the current block to entropy encoder 110 and inverse quantizer 112. A specific scan order may be predetermined.

[0111] The determined scanning order is the order for quantizing/inverse quantizing the transform coefficients. For example, a particular scan order may be specified as ascending frequency order (low to high frequency) or descending frequency order (high to low frequency).

[0112] The quantization parameter (QP) is a parameter that specifies the quantization step (quantization width). For example, when the value of the quantization parameter increases, the quantization step size also increases. In other words, when the value of the quantization parameter increases, the quantization error increases.

[0113] In addition, the quantization matrix can be used for quantization. For example, several kinds of quantization matrices may be used, respectively, to perform frequency conversion for sizes such as 4×4 and 8×8, prediction modes such as intra-prediction and outer-prediction, and pixel components such as luminance and pixel components. color signal. It should be noted that quantization means digitizing values sampled at certain intervals, respectively, into certain levels. In this art, quantization may be referred to using other expressions such as rounding and scaling, and may use rounding and scaling. Certain intervals and levels can be predefined.

[0114] Methods using quantization matrices include a method using a quantization matrix that is directly set on the encoder side, and a method using a quantization matrix that is set as a default value (default matrix). On the encoder side, a quantization matrix suitable for image features can be specified by directly specifying the quantization matrix. However, this case has the disadvantage of increasing the encoding volume for encoding the quantization matrix.

[0115] A method is provided for quantizing a high frequency coefficient and a low frequency coefficient without using a quantization matrix. It should be noted that this method is equivalent to the method using a quantization matrix (flat matrix), the coefficients of which have identical values.

[0116] The quantization matrix may be specified using, for example, a sequence parameter set (SPS) or a frame parameter set (PPS). SPS includes a parameter that is used for a sequence, and PPS includes a parameter that is used for a frame. Each of the SPS and PPS may be referred to simply as a "parameter set".

[0117] Entropy encoder

Entropy encoder 110 generates a coded signal (coded bit stream) based on the quantized coefficients that are input from quantizer 108. More specifically, entropy encoder 110, for example, binarizes the quantized coefficients and arithmetically encodes the binary signal and outputs a compressed bit stream or sequence.

[0118] Inverse Quantizer

The inverse quantizer 112 inversely quantizes the quantized coefficients that are input from the quantizer 108. More specifically, the inverse quantizer 112 inversely quantizes, in a specific scan order, the quantized coefficients of the current block. The inverse quantizer 112 then outputs the inversely quantized transform coefficients of the current block to the inverse quantizer 114. A specific scan order may be predetermined.

[0119] Inverter

The inverse transformer 114 recovers prediction errors (residuals) by inversely transforming the transform coefficients that are input from the inverse quantizer 112. More specifically, the inverse transformer 114 recovers the prediction errors of the current block by applying an inverse transform corresponding to the transform applied by the transformer 106 to the transform coefficients. Inverter 114 then outputs the recovered prediction errors to adder 116.

[0120] It should be noted that since information is lost in quantization, the reconstructed prediction errors are not the same as the prediction errors calculated by the subtractor 104. In other words, the reconstructed prediction errors normally include quantization errors.

[0121] Totalizer

The adder 116 reconstructs the current block by summing the prediction errors that are input from the inverter 114 and the prediction samples that are input from the prediction controller 128 . Adder 116 then outputs the reconstructed block to block memory 118 and loop filter 120. The reconstructed block is also referred to as a “local decoded block.”

[0122] Block memory

The block memory 118, for example, is a storage device for storing blocks in a frame to be encoded (hereinafter referred to as a “current frame”) referred to in intra prediction. More specifically, block storage 118 stores reconstructed blocks output from adder 116.

[0123] Frame memory

The frame memory 122, for example, is a storage device for storing reference frames for use in external prediction, and is also called a “frame buffer.” More specifically, frame memory 122 stores reconstructed blocks filtered by loop filter 120.

[0124] Loop filter

Loop filter 120 applies a loop filter to the blocks reconstructed by the adder 116 and outputs the filtered reconstructed blocks to frame memory 122. A loop filter is a filter used in a coding loop (in-loop filter) and includes, for example, a deblocking filter (DF or DBF), sampled adaptive offset (SAO), and adaptive loop filter (ALF).

[0125] In ALF, a least squares error filter is applied to remove compression artifacts. For example, one filter, selected from a plurality of filters based on the direction and activity of local gradients, is applied to each of the 2x2 subblocks in the current block.

[0126] More specifically, first, each subblock (eg, each 2x2 subblock) is classified into one of a plurality of classes (eg, fifteen or twenty-five classes). The subblock classification is based on the directionality and activity of the gradients. For example, a classification index C (eg, C=5D+A) is extracted based on the directionality of the D gradients (eg, 0-2 or 0-4) and the activity of the A gradients (eg, 0-4). Then, based on the classification index C, each sub-block is classified into one of a variety of classes.

[0127] For example, the directionality D of the gradients is calculated by comparing the gradients of multiple directions (eg, horizontal, vertical, and two diagonal directions). In addition, for example, the activity A of the gradients is calculated by summing the gradients of multiple directions and quantizing the result of the summation.

[0128] The filter to be used for each sub-block is determined from the plurality of filters based on the result of such classification.

[0129] The filter shape to be used in the ALF, for example, is a circular symmetrical filter shape. Fig. 6A-6C illustrate examples of filter shapes used in ALF. Fig. 6A illustrates a 5x5 diamond shape filter, FIG. 6B illustrates a 7x7 diamond shape filter, and FIG. 6C illustrates a 9x9 diamond shaped filter. Information indicating the shape of the filter is normally carried in frame-level signaling. It should be noted that the signaling of this information indicating the shape of the filter need not be performed at the frame level and may be performed at another level (eg, sequence level, slice level, tile level, CTU level, or CU level).

[0130] Enabling or disabling ALF is determined, for example, at the frame level or at the CU level. For example, a decision regarding whether or not to apply ALF to a luma signal may be made at the CU level, and a decision regarding whether or not to apply ALF to a chrominance signal may be made at the frame level. Information indicating whether ALF is enabled or disabled is normally carried in signaling at the frame level or at the CU level. It should be noted that the signaling of information indicating the enabling or disabling of ALF need not be performed at the frame level or the CU level and may be performed at another level (eg, sequence level, slice level, tile level, or CTU level).

[0131] A set of coefficients for a plurality of selectable filters (eg, fifteen or up to twenty-five filters) is normally signaled at the frame level. It should be noted that signaling of the set of coefficients need not be performed at the frame level and may be performed at another level (eg, sequence level, slice level, tile level, CTU level, CU level, or subblock level).

[0132] Loop Filter > Deblocking Filter

In the deblocking filter, the edge filter 120 performs a filtering process on a block boundary in the reconstructed image so as to reduce distortion that occurs at the block boundary.

[0133] FIG. 7 is a block diagram illustrating one example of a particular configuration of a loop filter 120 that functions as a deblocking filter.

[0134] The loop filter 120 includes: a boundary detection module 1201; filter determination module 1203; filtering execution module 1205; processing determination module 1208; filter characteristics determination module 1207; and switches 1202, 1204 and 1206.

[0135] Boundary determination module 1201 determines whether or not a pixel to be filtered for deblocking (ie, the current pixel) is present near a block boundary. The boundary determination module 1201 then outputs the determination result to the switch 1202 and the processing determination module 1208 .

[0136] In the case where the edge detection module 1201 determines that the current pixel is present near the block boundary, the switch 1202 outputs an unfiltered image to the switch 1204. In the opposite case, in which the edge detection module 1201 determines that the current pixel is not present near the boundary block, switch 1202 outputs the unfiltered image to switch 1206.

[0137] The filter determination unit 1203 determines whether or not to perform filtering to deblock the current pixel based on the pixel value of at least one surrounding pixel located near the current pixel. The filter determination unit 1203 then outputs the determination result to the switch 1204 and the processing determination unit 1208.

[0138] In case it is determined that the filter determination unit 1203 performs filtering to remove blockiness of the current pixel, the switch 1204 outputs the unfiltered image obtained through the switch 1202 to the filtering execution unit 1205. In the opposite case, in which it is determined that the filter determination module 1203 does not perform filtering to deblock the current pixel, the switch 1204 outputs the unfiltered image obtained through the switch 1202 to the switch 1206.

[0139] Upon receiving an unfiltered image via switches 1202 and 1204, filtering execution unit 1205 performs, for the current pixel, deblocking filtering with the filter characteristic determined by filter characteristic determining unit 1207. The filtering module 1205 then outputs the filtered pixel to the switch 1206.

[0140] Under the control of the processing determination unit 1208, the switch 1206 selectively outputs a pixel that is not filtered for deblocking and a pixel that is filtered for deblocking by the filtering execution unit 1205.

[0141] The processing determination module 1208 controls the switch 1206 based on the results of the determinations made by the boundary determination module 1201 and the filtering determination module 1203 . In other words, the processing determination module 1208 instructs the switch 1206 to output a pixel that is filtered to deblock when the edge determination module 1201 determines that the current pixel is present near a block boundary, and it is determined that the filter determination module 1203 performs filtering to deblock the current pixel. In addition, in addition to the above-described case, the processing determination unit 1208 instructs the switch 1206 to output a pixel that is not filtered for deblocking. The filtered image is output from the switch 1206 by repeating the pixel output in this manner.

[0142] FIG. 8 is a conceptual diagram indicating an example of a deblocking filter having a symmetrical filtering characteristic with respect to a block boundary.

[0143] In the deblocking filtering process, one of two deblocking filters having different characteristics, i.e. strong filter and weak filter, selected using pixel values and quantization parameters. In the case of a strong filter, pixels p0-p2 and pixels q0-q2 are present at the block boundary, as illustrated in FIG. 8, the pixel values of the corresponding pixel q0-q2 are changed to the pixel values q'0-q'2 by performing, for example, calculation according to the following expressions.

[0144] q'0=(p1+2xp0+2xq0+2xq1+q2+4)/8

q'1=(p0+q0+q1+q2+2)/4

q'2=(p0+q0+q1+3xq2+2xq3+4)/8

[0145] It should be noted that, in the above expressions, p0-p2 and q0-q2 represent the pixel values of the corresponding pixels p0-p2 and pixels q0-q2. In addition, q3 is the pixel value of the neighboring pixel q3 located on the opposite side of pixel q2 relative to the block boundary. In addition, on the right side of each of the expressions, coefficients that are multiplied by the corresponding pixel values of the pixels to be used for filtering to remove blocking are filtering coefficients.

[0146] Additionally, when filtering to remove blocking, clipping may be performed such that the calculated pixel values are not set above a threshold value. In the clipping process, the pixel values calculated according to the above expressions are clipped to the value obtained according to "calculated pixel value ± 2 x threshold value" using a threshold value determined based on the quantization parameter. In this way, excessive smoothing can be prevented.

[0147] FIG. 9 is a conceptual diagram for illustrating a block boundary on which a filtering process to remove blocking is performed. Fig. 10 is a conceptual diagram indicating examples of Bs values.

[0148] A block boundary on which the deblocking filtering process is performed, for example, is a boundary between prediction units (PUs) having 8x8 pixel blocks, as illustrated in FIG. 9, or the boundary between conversion units (TU). The deblocking filtering process can be performed in units of four rows or four columns. First, the boundary intensity values (Bs) are determined as indicated in FIG. 10 for the P block and Q block illustrated in FIG. 9.

[0149] According to the Bs values in FIG. 10, it is determined whether or not filtering processes for deblocking the boundaries of blocks belonging to an identical image should be performed using different intensities. The deblocking filtering process for the chroma signal is performed when the Bs value is 2. The deblocking filtering process for the luma signal is performed when the Bs value is 1 or more and a certain condition is satisfied. A certain condition can be predefined. It should be noted that the conditions for determining the Bs values are not limited to those indicated in FIG. 10, and the Bs value may be determined based on another parameter.

[0150] Prediction Processor (Internal Prediction Module, External Prediction Module, Prediction Controller)

Fig. 11 is a flowchart illustrating one example of a process performed by the prediction processor of the encoder 100. It should be noted that the prediction processor includes all or part of the following constituent elements: an intra prediction module 124; external prediction module 126; and a prediction controller 128.

[0151] The prediction processor generates a prediction image of the current block (step Sb_1). This prediction image is also called a "prediction signal" or a "prediction block". It should be noted that the prediction signal is, for example, an internal prediction signal or an external prediction signal. Specifically, the prediction processor generates a prediction image of the current block using a reconstructed image that has already been obtained through prediction block generation, difference block generation, coefficient block generation, difference block reconstruction, and decoded image block generation.

[0152] The reconstructed image, for example, may be an image in a reference frame or an image of a coded block in a current frame, which is a frame including the current block. The coded block in the current frame, for example, is a block adjacent to the current block.

[0153] FIG. 12 is a flowchart illustrating another example of a process performed by the encoder prediction processor 100.

[0154] The prediction processor generates a prediction image using the first method (step Sc_1a), generates a prediction image using the second method (step Sc_1b), and generates a prediction image using the third method (step Sc_1c). The first method, the second method and the third method may be mutually different methods for generating a predictive image. Each of the first to third methods may be an external forecasting method, an internal forecasting method, or another forecasting method. The above-described reconstructed image can be used in these prediction methods.

[0155] Next, the prediction processor selects any of the plurality of prediction methods generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). Predictive image selection, which is the selection of a method or mode for obtaining the final predictive image, may be performed by calculating a cost for each of the generated predictive images and based on the cost. Alternatively, the selection of the prediction image may be based on a parameter that is used in the encoding process. Encoder 100 may convert information to identify a selected prediction image, method, or mode into a coded signal (also called a "coded bitstream"). The information may, for example, be a flag or the like. Thus, the decoder is capable of generating a prediction image according to a method or mode selected based on information in the encoder 100. It should be noted that, in the example illustrated in FIG. 12, the prediction processor selects any of the prediction images after the prediction images are generated using appropriate methods. However, the prediction processor may select a method or mode based on a parameter to be used in the above-described encoding process before generating prediction images, and may generate a prediction image according to the selected method or mode.

[0156] For example, the first method and the second method may be internal prediction and external prediction, respectively, and the prediction processor may select the final prediction image for the current block from prediction images generated according to the prediction methods.

[0157] FIG. 13 is a flowchart illustrating another example of a process performed by the encoder prediction processor 100.

[0158] First, the prediction processor generates a prediction image using internal prediction (step Sd_1a) and generates a prediction image using external prediction (step Sd_1b). It should be noted that the prediction image generated by internal prediction is also called "internal prediction image", and the prediction image generated by external prediction is also called "external prediction image".

[0159] Next, the prediction processor evaluates each of the inner prediction image and the outer prediction image (step Sd_2). Costs can be used in estimation. In other words, the prediction processor calculates a cost C for each of the inner prediction image and the outer prediction image. Cost C can be calculated according to the expression of the R-D optimization model, for example, C=D+λ*R. In this expression, D indicates the distortion when encoding the prediction image and is represented, for example, as the sum of the absolute differences between the pixel value of the current block and the pixel value of the prediction image. In addition, R indicates a predicted encoding amount of the prediction image, namely, an encoding amount required to encode motion information to generate the prediction image, etc. In addition, λ indicates, for example, a multiplier according to the Lagrange multiplier method.

[0160] The prediction processor then selects the prediction image for which the least cost C is calculated from the inner prediction image and the outer prediction image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode is selected to generate a prediction image for the current block.

[0161] Internal prediction module

The intra prediction module 124 generates a prediction signal (intra prediction signal) by performing intra prediction (also called “intra prediction”) of the current block by reference to the block or blocks in the current frame stored in the block memory 118. More specifically, the intra prediction module 124 generates an internal prediction signal by performing intra prediction by reference to samples (eg, luminance value and/or chrominance value) of the block or blocks adjacent to the current block, and then outputs the internal prediction signal to the prediction controller 128 .

[0162] For example, the intra prediction unit 124 performs intra prediction by using one mode of a plurality of intra prediction modes that are set. The intra prediction modes include one or more non-directional prediction modes and a plurality of directed prediction modes. Preset modes can be preset.

[0163] The one or more non-directional prediction modes include, for example, a planar prediction mode and a DC prediction mode defined in the H.265/High Efficiency Video Coding (HEVC) standard.

[0164] The plurality of directional prediction modes includes, for example, the thirty-three directional prediction modes defined in the H.265/HEVC standard. It should be noted that the plurality of directional prediction modes may further include thirty-two directional prediction modes in addition to the thirty-three directional prediction modes (for a total of sixty-five directional prediction modes). Fig. 14 is a conceptual diagram illustrating sixty-seven intra-prediction modes in total that can be used in intra-prediction (two non-directional prediction modes and sixty-five directional prediction modes). The solid arrows represent the thirty-three directions defined in the H.265/HEVC standard, and the dotted arrows represent the additional thirty-two directions (the two non-directional prediction modes are not illustrated in FIG. 14).

[0165] In various kinds of processing examples, a luma signal block may be referenced in internal prediction of a chrominance signal block. In other words, the chrominance signal component of the current block can be predicted based on the luminance signal component of the current block. This internal forecasting is also called “Cross Component Linear Model (CCLM) forecasting.” An intra-prediction mode for a chrominance block in which such a luma block is referenced (also called, for example, a “CCLM mode”) may be added as one of the intra-prediction modes for the chrominance blocks.

[0166] The internal prediction module 124 may adjust the internally predicted pixel values based on horizontal/vertical reference pixel gradients. The internal prediction followed by this kind of correction is also called "position-dependent combination with internal prediction (PDPC)". Information indicating whether or not to use PDPC (called, for example, a "PDPC flag") is normally signaled at the CU level. It should be noted that signaling of this information does not necessarily have to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, or CTU level).

[0167] External Prediction Module

The inter-prediction unit 126 generates a prediction signal (external prediction signal) by performing inter-prediction (also called “inter-frame prediction”) of the current block by reference to a block or blocks in a reference frame that is different from the current frame and stored in the frame memory 122 . External prediction is performed in units of the current block or the current subblock (eg, a 4x4 block) within the current block. For example, the external prediction module 126 performs motion estimation in a reference frame for the current block or current sub-block and learns the reference block or sub-block that has the best match with the current block or current sub-block. The external prediction module 126 then receives motion information (eg, a motion vector) that compensates for the motion or change from the reference block or subblock to the current block or subblock. The external prediction unit 126 generates an external prediction signal of the current block or sub-block by performing motion compensation (or motion prediction) based on the motion information. The external prediction module 126 outputs the generated external prediction signal to the prediction controller 128 .

[0168] Motion information used in motion compensation may be signaled as external predictive signals in various forms. For example, the motion vector may be signaled. In other words, the difference between the motion vector and the motion vector predictor may be signaled.

[0169] Basic External Forecasting Flow

Fig. 15 is a flowchart illustrating an exemplary basic flow of external prediction processing.

[0170] First, the external prediction module 126 generates a prediction signal (steps Se_1 to Se_3). Next, the subtractor 104 generates the difference between the current block and the prediction image as a prediction residual (step Se_4).

[0171] Here, when generating a prediction image, the external prediction unit 126 generates a prediction image by determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and motion compensation (step Se_3). In addition, when determining the MV, the external prediction unit 126 determines the MV by selecting a motion vector candidate (MV candidate) (step Se_1) and extracting the MV (step Se_2). The selection of a possible MV variant is carried out, for example, by selecting at least one possible MV variant from a list of possible MV variants. Alternatively, when extracting the MV, the external prediction module 126 may further select at least one MV candidate from the at least one MV candidate and determine the selected at least one MV candidate as the MV for the current block. Alternatively, the inter-prediction module 126 may determine the MV for the current block by performing an estimate in a reference frame region indicated by each of the selected at least one MV candidate. It should be noted that the estimation in the reference frame region may be referred to as "motion estimation".

[0172] In addition, although steps Se_1 to Se_3 are performed by the external prediction unit 126 in the above example, the process which is, for example, step Se_1, step Se_2, etc. may be performed by another component included in the encoder 100.

[0173] Motion Vector Extraction Flow

Fig. 16 is a flowchart illustrating one example of extracting motion vectors.

[0174] The external prediction unit 126 extracts the MV of the current block in the mode for encoding motion information (eg, MV). In this case, for example, motion information is encoded as a prediction parameter and signaled. In other words, the encoded motion information is included in a coded signal (also called a "coded bit stream").

[0175] Alternatively, the external prediction module 126 extracts the MV in a mode in which motion information is not encoded. In this case, motion information is not included in the encoded signal.

[0176] Here, MV extraction modes may include normal external mode, merge mode, FRUC mode, affine mode, etc., which are described below. The modes in which motion information is encoded include the normal outer mode, the combining mode, the affine mode (specifically, the affine outer mode and the affine combining mode), etc. It should be noted that the motion information may include not only MV, but also motion vector predictor selection information, which is described below. Modes in which motion information is not encoded include FRUC mode, etc. The external prediction unit 126 selects a mode for extracting the MV of the current block from the modes, and extracts the MV of the current block using the selected mode.

[0177] FIG. 17 is a flowchart illustrating another example of extracting motion vectors.

[0178] The external prediction unit 126 extracts the MV of the current block in a mode in which the MV difference is encoded. In this case, for example, the MV difference is encoded as a prediction parameter and signaled. In other words, the encoded MV difference is included in the encoded signal. The MV difference is the difference between the MV of the current block and the MV predictor.

[0179] Alternatively, the inter prediction module 126 extracts the MV in a mode in which the MV difference is not encoded. In this case, the encoded MV difference is not included in the encoded signal.

[0180] Here, as described above, MV extraction modes include normal external mode, merge mode, FRUC mode, affine mode, etc., which are described below. Modes in which the MV difference is encoded include normal outer mode, affine mode (in particular, affine outer mode), etc. Modes in which the MV difference is not encoded include FRUC mode, concatenated mode, affine mode (specifically, affine concatenated mode), etc. The external prediction unit 126 selects a mode for extracting the MV of the current block from a plurality of modes, and extracts the MV of the current block using the selected mode.

[0181] Motion Vector Extraction Flow

Fig. 18 is a flowchart illustrating another example of extracting motion vectors. MV extraction modes, which are inter-prediction modes, include a variety of modes and are roughly divided into modes in which the MV difference is encoded and modes in which the motion vector difference is not encoded. Modes in which the MV difference is not encoded include merge mode, FRUC mode, affine mode (specifically, affine pool mode), etc. These modes are described in detail below. Simply, the merging mode is a mode for extracting the MV of a current block by selecting a motion vector from a coded surrounding block, and the FRUC mode is a mode for extracting the MV of a current block by performing estimation between coded regions. The affine mode is a mode for extracting, as the MV of the current block, the motion vector of each of a plurality of sub-blocks included in the current block, subject to an affine transformation.

[0182] More specifically, as illustrated, when the inter prediction mode information indicates 0 (0 on Sf_1), the inter prediction unit 126 extracts the motion vector using the combining mode (Sf_2). When the inter prediction mode information indicates 1 (1 to Sf_1), the inter prediction unit 126 extracts the motion vector using the FRUC mode (Sf_3). When the inter prediction mode information indicates 2 (2 to Sf_1), the inter prediction unit 126 extracts the motion vector using an affine mode (specifically, an affine combining mode) (Sf_4). When the inter prediction mode information indicates 3 (3 to Sf_1), the inter prediction unit 126 extracts the motion vector using the mode in which the MV difference is encoded (eg, normal outer mode (Sf_5)).

[0183] MV extract > normal external mode

The normal outer mode is an outer prediction mode for extracting the MV of the current block based on a block similar to the image of the current block from the reference frame area indicated by the MV candidate. In this normal external mode, the MV difference is encoded.

[0184] FIG. 19 is a flowchart illustrating an example of external prediction in a normal external mode.

[0185] First, the inter-prediction module 126 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sg_1). In other words, the external prediction module 126 generates a list of possible MV options.

[0186] Next, the external prediction module 126 extracts N (an integer of 2 or more) MV candidates from the plurality of MV candidates obtained in step Sg_1 as motion vector predictor candidates (also called “MV candidates”). predictors") according to a certain order of priorities (stage Sg_2). It should be noted that the priority order can be determined in advance for each of the N possible MV options.

[0187] Next, the external prediction unit 126 selects one motion vector predictor candidate from the motion vector predictor candidates N as the motion vector predictor (also called “MV predictor”) of the current block (step Sg_3). At this time, the external prediction unit 126 encodes, in the stream, motion vector predictor selection information to identify the selected motion vector predictor. It should be noted that the stream is a coded signal or a coded bit stream as described above.

[0188] Next, the inter prediction unit 126 extracts the MV of the current block by reference to the encoded reference frame (step Sg_4). At this time, the external prediction unit 126 further encodes, in the stream, a difference value between the extracted MV and the motion vector predictor as an MV difference. It should be noted that an encoded reference frame is a frame including a plurality of blocks that are reconstructed after encoding.

[0189] Finally, the external prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Sg_5). It should be noted that the prediction image is an external prediction signal, as described above.

[0190] In addition, information indicating the external prediction mode (normal external mode in the above example) used to generate a prediction image, for example, is encoded as a prediction parameter.

[0191] It should be noted that the list of possible MV options can also be used as a list for use in another mode. In addition, processes associated with a list of possible MV options can be applied to processes associated with a list for use in another mode. Processes associated with a list of MV candidates include, for example, retrieving or selecting a candidate MV from a list of candidate MVs, reordering the candidate MVs, or removing a candidate MV.

[0192] MV extract > merge mode

The merging mode is an external prediction mode for selecting a candidate MV from a list of candidate MVs as the MV of the current block, thereby extracting the MV.

[0193] FIG. 20 is a flowchart illustrating an example of inter-prediction in the combining mode.

[0194] First, the inter-prediction module 126 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sh_1). In other words, the external prediction module 126 generates a list of possible MV options.

[0195] Next, the external prediction module 126 selects one candidate MV from the plurality of candidate MVs obtained in step Sh_1, thereby extracting the MV of the current block (step Sh_2). At this time, the external prediction module 126 encodes, in the stream, MV selection information to identify the selected MV candidate.

[0196] Finally, the external prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Sh_3).

[0197] In addition, information indicating the inter prediction mode (combining mode in the above example) used to generate the prediction image and included in the encoded signal, for example, is encoded as a prediction parameter.

[0198] FIG. 21 is a conceptual diagram for illustrating one example of a process for extracting motion vectors of a current frame in a fusion mode.

[0199] First, a list of possible MV variants, in which possible variants of the MV predictors are registered, is generated. Examples of possible MV predictor options include: spatially adjacent MV predictors, which are the MVs of a plurality of coded blocks located spatially surrounding the current block; temporally adjacent MV predictors, which are the MVs of surrounding blocks onto which the position of the current block in the encoded reference frame is projected; combined MV predictors, which are MVs formed by combining the MV value of a spatially adjacent MV predictor and the MV of a temporally adjacent MV predictor; and a null MV predictor, which is an MV having a value of zero.

[0200] Next, one MV predictor is selected from the plurality of MV predictors registered in the MV predictor list, and the selected MV predictor is determined as the MV of the current block.

[0201] In addition, the variable length encoder describes and encodes, in the stream, merge_idx, which is a signal indicating which MV predictor is selected.

[0202] It should be noted that the MV predictors registered in the MV predictor list described in FIG. 21 are examples. The number of MV predictors may differ from the number of MV predictors in the diagram, the list of MV predictors may be configured such that some types of MV predictors in the diagram may not be included, or such that one or more MV predictors other than types of MV predictors in the diagram are included.

[0203] The final MV may be determined by performing a decoder motion vector refinement (DMVR) process, which is described below, using the MV of the current block extracted in the combining mode.

[0204] It should be noted that the MV predictor candidates are the MV candidate candidates described above, and the MV predictor list is the list of MV candidate candidates described above. It should be noted that the list of possible MV options may be referred to as a "candidate list". In addition, merge_idx represents MV selection information.

[0205] MV extraction > FRUC mode

Motion information can be extracted at the decoder side without being signaled from the encoder side. It should be noted that, as described above, the aggregation mode specified in the H.265/HEVC standard can be used. In addition, for example, motion information can be extracted by performing motion estimation on the decoder side. In an embodiment, on the decoder side, motion estimation is performed without using the pixel values in the current block.

[0206] A mode for performing motion estimation on the decoder side is described here. The mode for performing motion estimation on the decoder side may be called “pattern matching based motion vector extraction (PMMVD) mode” or “frame rate upconversion (FRUC) mode”.

[0207] One example of a FRUC process in flowchart form is illustrated in FIG. 22. First, a list of multiple candidates having a motion vector (MV) predictor (i.e., a list of MV candidates that can also be used as a merge list) is generated by reference to the motion vector in the coded block, which is spatially or temporally adjacent to the current block (step Si_1). Then, the best possible MV option is selected from the set of possible MV options registered in the list of possible MV options (step Si_2). For example, the evaluation values of the corresponding MV candidates included in the MV candidate list are calculated, and one MV candidate is selected based on the evaluation values. Based on the selected motion vector candidates, the motion vector for the current block is then extracted (step Si_4). More specifically, for example, the selected motion vector candidate (the best possible MV candidate) is retrieved directly as the motion vector for the current block. In addition, for example, a motion vector for the current block may be retrieved using pattern matching in the surrounding region of a position in a reference frame, wherein the position in the reference frame corresponds to a selected motion vector candidate. In other words, scoring using template matching and scoring values can be performed on the surrounding region of the best possible MV, and when there is an MV that results in a better scoring value, the best possible MV can be updated to the MV that results in the best estimate value, and the updated MV can be determined as the final MV for the current block. It is also possible to have a configuration in which no such process is performed to update the best possible MV to the MV having the best score value.

[0208] Finally, the external prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Si_5).

[0209] A similar process may be performed in subblock units.

[0210] The evaluation values may be calculated according to various kinds of methods. For example, a comparison is made between a reconstructed image in a region in a reference frame corresponding to a motion vector and a reconstructed image in a specific region (the region, for example, may be a region in another reference frame or a region in an adjacent block of the current frame, as defined below). The specific area can be pre-defined.

[0211] The difference between the pixel values of the two reconstructed images can be used for the motion vector estimation value. It should be noted that the score value may be calculated using information other than the difference value.

[0212] An example of pattern matching is described in detail next. First, one MV candidate included in the list of MV candidate candidates (eg, a merge list) is selected as the starting evaluation point through template matching. For example, as a pattern match, a first pattern match or a second pattern match may be used. The first pattern matching and the second pattern matching are also called "bilateral matching" and "pattern matching", respectively.

[0213] MV extraction > FRUC > bilateral matching

In the first pattern matching, pattern matching is performed between two blocks along the current block's motion path, which are two blocks in different two reference frames. Accordingly, in the first template matching, a region in another reference frame along the motion path of the current block is used as a determined region for calculating the evaluation value of the above-described candidate. The specific area can be pre-defined.

[0214] FIG. 23 is a conceptual diagram for illustrating one example of first template matching (bilateral matching) between two blocks in two reference frames along a motion path. As illustrated in FIG. 23, in the first template matching, two motion vectors (MV ₀ , MV ₁ ) are extracted by evaluating the pair that has the best match from among the pairs in two blocks in two different reference frames (Ref ₀ , Ref ₁ ), which represent two blocks along the trajectory of the current block (Cur block). More specifically, the difference between the reconstructed image at a specified location in the first coded reference frame (Ref ₀ ) indicated by the MV option, and the reconstructed image at the specified location in the second coded reference frame (Ref ₁ ) indicated by the symmetric MV, obtained by scaling the MV candidate in the display time interval is retrieved for the current block, and a score value is calculated using the resulting difference value. One can choose, as the final MV, a possible MV option that results in the best evaluation value from the set of possible MV options and which is likely to lead to good results.

[0215] Assuming a continuous motion path, the motion vectors (MV ₀ , MV ₁ ) indicating the two reference blocks are proportional to the time distances (TD ₀ , TD ₁ ) between the current frame (Cur Pic) and the two reference frames (Ref ₀ ,Ref ₁ ). For example, when the current frame is temporarily between two reference frames, and the temporal distances from the current frame to the corresponding two reference frames are equal to each other, mirror-symmetric bidirectional motion vectors are extracted in the first template matching.

[0216] MV extraction > FRUC > reference matching

In the second pattern matching (reference matching), pattern matching is performed between a block in the reference frame and a reference in the current frame (the reference is the block adjacent to the current block in the current frame (the adjacent block, for example, represents the top and/or left adjacent block(s)). Therefore, in the second pattern matching, a block adjacent to the current block in the current frame is used as a predetermined region for the above-described calculation of the candidate evaluation value.

[0217] FIG. 24 is a conceptual diagram for illustrating one example of pattern matching (pattern matching) between a pattern in the current frame and a block in the reference frame. As illustrated in FIG. 24, in the second template matching, the motion vector of the current block (Cur block) is extracted by estimating, in the reference frame (Ref ₀ ), the block that has the best match to the block adjacent to the current block in the current frame (Cur Pic). More specifically, it is possible that the difference between a reconstructed image in a coded region that is adjacent both to the left and above, or to the left or above, and a reconstructed image that is in a corresponding region in a coded reference frame (Ref ₀ ) and is indicated by candidate MV option, the score value is calculated using the value of the resulting difference, and the candidate MV option that results in the best score value from the plurality of possible MV options is selected as the best possible MV option.

[0218] This information indicating whether or not to use FRUC mode (called, for example, a “FRUC flag”) may be signaled at the CU level. In addition, when the FRUC mode is applied (eg, when the FRUC flag is true), information indicating the applicable pattern matching method (first pattern matching or second pattern matching) may be signaled at the CU level. It should be noted that signaling of this information does not necessarily need to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, CTU level, or subblock level).

[0219] MV extraction > affine mode

Next, an affine mode for extracting a motion vector in subblock units based on the motion vectors of multiple neighboring blocks is described. This mode is also called "motion compensated affine prediction mode".

[0220] FIG. 25A is a conceptual diagram for illustrating one example of extracting the motion vector of each sub-block based on the motion vectors of a plurality of neighboring blocks. In fig. 25A, the current block includes sixteen 4x4 subblocks. Here, the motion vector V ₀ at the upper left corner control point in the current block is extracted based on the motion vector of the adjacent block, and similarly, the motion vector V ₁ at the upper right corner control point in the current block is extracted based on the motion vector of the adjacent sub-block. The two motion vectors v ₀ and v ₁ may be projected according to the expression (1A) below, and the motion vectors (v _x , v _y ) for the corresponding sub-blocks in the current block can be extracted.

[0221] Mathematical expression 1

[0222] Here, x and y indicate the horizontal position and vertical position of the sub-block, respectively, and w indicates a certain weighting factor. The specific weighting factor may be predetermined.

[0223] This information indicating the affine mode (eg, called an "affine flag") may be signaled at the CU level. It should be noted that signaling information indicating the affine mode need not be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, CTU level, or subblock level).

[0224] In addition, the affine mode may include multiple modes for different methods for extracting motion vectors at the top left and top right corner control points. For example, the affine mode includes two modes, which are the affine outer mode (also called "affine normal outer mode") and the affine union mode.

[0225] MV extraction > affine mode

Fig. 25B is a conceptual diagram for illustrating one example of extracting the motion vector of each sub-block in an affine mode in which three control points are used. In fig. 25B, the current block includes sixteen 4×4 blocks. Here, the motion vector V ₀ at the upper left corner control point for the current block is extracted based on the motion vector of the adjacent block, and similarly, the motion vector V ₁ at the upper right corner control point for the current block is extracted based on the motion vector of the adjacent block, and the vector V The ₂ motion at the bottom left corner control point for the current block is extracted based on the motion vector of the adjacent block. The three motion vectors v ₀ , v ₁ and v ₂ can be projected according to expression (1B) below, and the motion vectors (v _x , v _y ) for the corresponding sub-blocks in the current block can be extracted.

[0226] Mathematical expression 2

[0227] Here, x and y indicate the horizontal position and vertical position of the center of the sub-block, respectively, w indicates the width of the current block, and h indicates the height of the current block.

[0228] Affine modes that use different numbers of control points (eg, two and three control points) can be switched and signaled at the CU level. It should be noted that information indicating the number of control points in the affine mode used at the CU level may be signaled at another level (eg, sequence level, frame level, slice level, tile level, CTU level, or subblock level).

[0229] In addition, such an affine mode that uses three control points may include various methods for extracting motion vectors at the top left, top right, and bottom left corner control points. For example, affine modes include two modes, which are the affine outer mode (also called "affine normal outer mode") and the affine union mode.

[0230] MV extract > affine join mode

Fig. 26A, FIG. 26B and FIG. 26C are conceptual diagrams to illustrate the affine combining mode.

[0231] As illustrated in FIG. 26A, in the affine combining mode, for example, motion vector predictors at the corresponding control points of the current block are calculated based on a plurality of motion vectors corresponding to blocks coded according to the affine mode from coded block A (left), block B (top), block C ( top right), block D (bottom left) and block E (top left) that are adjacent to the current block. More specifically, the encoded block A (left), block B (upper), block C (upper right), block D (lower left) and block E (upper left) are checked in the listed order, and the first effective block encoded according to the affine mode , is identified. Motion vector predictors at control points of the current block are calculated based on the set of motion vectors corresponding to the identified block.

[0232] For example, as illustrated in FIG. 26B, when block A, which is adjacent to the left of the current block, is encoded according to an affine mode that uses two control points, motion vectors v ₃ and v ₄ , projected at the upper left corner position and upper right corner position of the encoded block including block A itself is extracted. The motion vector predictor v ₀ at the top left corner control point of the current block and the motion vector predictor v ₁ at the top right corner control point of the current block are then calculated from the extracted motion vectors v ₃ and v ₄ .

[0233] For example, as illustrated in FIG. 26C, when block A, which is adjacent to the left of the current block, is encoded according to an affine mode that uses three control points, motion vectors v ₃ , v ₄ and v ₅ projected at the upper left corner position, upper right corner position and lower the left corner position of the coded block including block A is retrieved. The motion vector predictor v ₀ at the top left corner control point of the current block, the motion vector predictor v ₁ at the top right corner control point of the current block, and the motion vector predictor v ₂ at the bottom left corner control point of the current block are then calculated from the extracted vectors v ₃ , v ₄ and v ₅ movements.

[0234] It should be noted that this method for extracting motion vector predictors can be used to extract motion vector predictors of the corresponding control points of the current block in step Sj_1 in FIG. 29, described below.

[0235] FIG. 27 is a flowchart illustrating one example of the affine combining mode.

[0236] In the affine merging mode, as illustrated first, the external prediction module 126 extracts the MV predictors of the corresponding control points of the current block (step Sk_1). The control points are the top left corner point of the current block and the top right corner point of the current block, as illustrated in FIG. 25A, or the top left corner point of the current block, the top right corner point of the current block, and the bottom left corner point of the current block, as illustrated in FIG. 25B.

[0237] In other words, as illustrated in FIG. 26A, the external prediction module 126 examines the coded block A (left), block B (top), block C (top right), block D (bottom left), and block E (top left) in the listed order and identifies the first effective block coded according to the affine regime.

[0238] When block A is identified and block A has two control points, as illustrated in FIG. 26B, the external prediction module 126 calculates the motion vector v ₀ at the top left corner control point of the current block and the motion vector v ₁ at the top right corner control point of the current block from the motion vectors v ₃ and v ₄ at the top left corner and the top right corner of the encoded block. including block A. For example, the external prediction module 126 calculates the motion vector v ₀ at the top left corner control point of the current block and the motion vector v ₁ at the top right corner control point of the current block by projecting the motion vectors v ₃ and v ₄ at the top left corner and upper right corner of the encoded block to the current block.

[0239] Alternatively, when block A is identified and block A has three control points, as illustrated in FIG. 26C, the external prediction module 126 calculates the motion vector v ₀ at the top left corner control point of the current block, the motion vector v ₁ at the top right corner control point of the current block, and the motion vector v ₂ at the bottom left corner control point of the current block from the vectors v ₃ . v ₄ and v ₅ movements in the upper left corner, upper right corner and lower left corner of the coded block including block A. For example, the external prediction module 126 calculates the movement vector v ₀ at the upper left corner control point of the current block, vector v ₁ motion at the top right corner control point of the current block and the motion vector v ₂ at the bottom left corner control point of the current block by projecting the motion vectors v ₃ , v ₄ and v ₅ at the top left corner, top right corner and bottom left corner of the encoded block onto the current one block.

[0240] Next, the inter-prediction module 126 performs motion compensation of each of the plurality of sub-blocks included in the current block. In other words, the external prediction module 126 calculates, for each of the plurality of sub-blocks, the motion vector of the sub-block as an affine MV, by using either (i) the two motion vector predictors v ₀ and v ₁ and the expression (1A) described above, or (ii) ) three predictors v ₀ , v ₁ and v ₂ motion vectors and expression (1B) described above (stage Sk_2). The inter-prediction module 126 then performs motion compensation of the sub-blocks using these affine MVs and coded reference frames (step Sk_3). As a result, motion compensation of the current block is performed in order to generate a predictive image of the current block.

[0241] MV Extract > Affine External Mode

Fig. 28A is a conceptual diagram for illustrating an affine outer mode in which two control points are used.

[0242] In affine external mode, as illustrated in FIG. 28A, a motion vector selected from the motion vectors of coded block A, block B, and block C that are adjacent to the current block is used as the motion vector predictor v ₀ at the upper left corner control point of the current block. Likewise, a motion vector selected from the motion vectors of a coded block D and a block E that are adjacent to the current block is used as a motion vector predictor v ₁ at the top right corner control point of the current block.

[0243] FIG. 28B is a conceptual diagram for illustrating an affine outer mode in which three control points are used.

[0244] In affine external mode, as illustrated in FIG. 28B, a motion vector selected from the motion vectors of coded block A, block B, and block C that are adjacent to the current block is used as the motion vector predictor v ₀ at the upper left corner control point of the current block. Likewise, a motion vector selected from the motion vectors of a coded block D and a block E that are adjacent to the current block is used as a motion vector predictor v ₁ at the top right corner control point of the current block. In addition, a motion vector selected from the motion vectors of a coded block F and a block G that are adjacent to the current block is used as a motion vector predictor v ₂ at the lower left corner control point of the current block.

[0245] FIG. 29 is a flowchart illustrating one example of an affine outer mode.

[0246] In the affine external mode, as illustrated, first, the external prediction module 126 extracts the MV predictors (v ₀ , v ₁ ) or (v ₀ , v ₁ , v ₂ ) of the corresponding two or three control points of the current block (step Sj_1 ). The control points are the top left corner point of the current block and the top right corner point of the current block, as illustrated in FIG. 25A, or the top left corner point of the current block, the top right corner point of the current block, and the bottom left corner point of the current block, as illustrated in FIG. 25B.

[0247] In other words, the external prediction module 126 extracts the (v ₀ , v ₁ ) or (v ₀ , v ₁ , v ₂ ) motion vector predictors of the corresponding two or three control points of the current block by selecting the motion vectors of any of the blocks from among those coded blocks near the corresponding control points of the current block illustrated in any of FIG. 28A or FIG. 28B. At this time, the external prediction unit 126 encodes, in the stream, motion vector predictor selection information for identifying the selected two motion vectors.

[0248] For example, the external prediction unit 126 may determine, using a cost estimate or the like, a block from which a motion vector is selected as a motion vector predictor at a control point from among coded blocks adjacent to the current block, and may describe, in a bit stream, a flag indicating which motion vector predictor is selected.

[0249] Next, the external prediction unit 126 performs motion estimation (step Sj_3 and Sj_4) while updating the motion vector predictor selected or extracted in step Sj_1 (step Sj_2). In other words, the external prediction unit 126 calculates, as an affine MV, the motion vector of each of the sub-blocks that corresponds to the updated motion vector predictor using either expression (1A) or expression (1B) described above (step Sj_3). The inter-prediction module 126 then performs motion compensation of the sub-blocks using these affine MVs and coded reference frames (step Sj_4). As a result, for example, the external prediction unit 126 determines the motion vector predictor that results in the least cost as the motion vector at the control point in the motion estimation loop (step Sj_5). At this time, the external prediction unit 126 further encodes, in the stream, a difference value between the determined MV and the motion vector predictor as an MV difference.

[0250] Finally, the external prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the determined MV and the encoded reference frame (step Sj_6).

[0251] MV Extract > Affine External Mode

When affine modes that use different numbers of control points (for example, two and three control points) can be switched and signaled at the CU level, the number of control points in the coded block and the number of control points in the current block may differ from each other . Fig. 30A and FIG. 30B are conceptual diagrams for illustrating methods for extracting motion vector predictors at control points when the number of control points in a coded block and the number of control points in a current block are different from each other.

[0252] For example, as illustrated in FIG. 30A, when the current block has three control points in the upper left corner, upper right corner and lower left corner, and block A, which is adjacent to the left of the current block, is encoded according to the affine mode in which two control points, vectors v ₃ and v ₄ movements projected at the upper left corner position and the upper right corner position in the coded block including block A are extracted. The motion vector predictor v ₀ at the top left corner control point of the current block and the motion vector predictor v ₁ at the top right corner control point of the current block are then calculated from the extracted motion vectors v ₃ and v ₄ . In addition, the motion vector predictor v ₂ at the lower left corner control point is calculated from the extracted motion vectors v ₀ and v ₁ .

[0253] For example, as illustrated in FIG. 30B, when the current block has two control points at the upper left corner and the upper right corner, and block A, which is adjacent to the left of the current block, is encoded according to the affine mode in which three control points, vectors v ₃ , v ₄ and v are used ₅ , movements projected at the upper left corner position, upper right corner position and lower left corner position in the coded block including block A are extracted. The motion vector predictor v ₀ at the top left corner control point of the current block and the motion vector predictor v ₁ at the top right corner control point of the current block are then calculated from the extracted motion vectors v ₃ , v ₄ and v ₅ .

[0254] It should be noted that this method for extracting motion vector predictors can be used to extract motion vector predictors of the corresponding control points of the current block in step Sj_1 in FIG. 29.

[0255] MV extract > DMVR

Fig. 31A is a flowchart illustrating the relationship between the combining mode and the DMVR.

[0256] The external prediction unit 126 extracts the motion vector of the current block according to the combining mode (step Sl_1). Next, the external prediction unit 126 determines whether or not to perform motion vector estimation, i.e. motion assessment (stage Sl_2). Here, when determining not to perform motion estimation (“No” in step Sl_2), the external prediction unit 126 determines the motion vector extracted in step Sl_1 as the final motion vector for the current block (step Sl_4). In other words, in this case, the motion vector of the current block is determined according to the merging mode.

[0257] When determining to perform motion estimation in step Sl_1 (Yes in step Sl_2), the external prediction unit 126 extracts the final motion vector for the current block by estimating the surrounding area of the reference frame indicated by the motion vector extracted in step Sl_1 (step Sl_3 ). In other words, in this case, the motion vector of the current block is determined according to the DMVR.

[0258] FIG. 31B is a conceptual diagram for illustrating one example of a DMVR process for determining MV.

[0259] First (eg, in merge mode), the best MVP that is specified as the current block is determined as a candidate MV. The reference pixel is identified from the first reference frame (L0), which is a coded frame in the L0 direction according to the MV candidate (L0). Likewise, a reference pixel is identified from a second reference frame (L1), which is a coded frame in the L1 direction according to the MV candidate (L1). The reference is formed by calculating the average of these reference pixels.

[0260] Next, each of the surrounding regions of candidate MVs of the first reference frame (L0) and the second reference frame (L1) is evaluated, and the MV that results in the lowest cost is determined as the final MV. It should be noted that the cost value may be calculated, for example, using the difference value between each of the pixel values in the reference and the corresponding one of the pixel values in the evaluation area, the values of the MV candidate values, etc.

[0261] It should be noted that the processes, configurations and operations described here are typically substantially common between the encoder and decoder, which are described below.

[0262] The completely identical exemplary processes described herein do not necessarily have to be performed. Any process to enable the extraction of the final MV by evaluating the surrounding areas of candidate MVs can be used.

[0263] Motion compensation > BIO/OBMC

Motion compensation includes a mode for generating a predictive image and correcting a predictive image. The mode, for example, is BIO and OBMC, which are described below.

[0264] FIG. 32 is a flowchart illustrating one example of predictive image generation.

[0265] The external prediction module 126 generates a prediction image (step Sm_1) and corrects the prediction image, for example, according to any of the modes described above (step Sm_2).

[0266] FIG. 33 is a flowchart illustrating another example of predictive image generation.

[0267] The external prediction unit 126 determines the motion vector of the current block (step Sn_1). Next, the external prediction unit 126 generates a prediction image (step Sn_2) and determines whether or not to perform the correction process (step Sn_3). Here, upon determining to carry out the correction process (“Yes” in step Sn_3), the external prediction unit 126 generates the final prediction image by correcting the prediction image (step Sn_4). When determining not to perform the correction process (“No” in step Sn_3), the external prediction unit 126 outputs the prediction image as the final prediction image without correcting the prediction image (step Sn_5).

[0268] In addition, motion compensation includes a mode for correcting the brightness of the projection image when generating the projection image. A mode, for example, is LIC, which is described below.

[0269] FIG. 34 is a flowchart illustrating another example of predictive image generation.

[0270] The external prediction unit 126 extracts the motion vector of the current block (step So_1). Next, the external prediction unit 126 determines whether or not to perform the brightness correction process (step So_2). Here, when determining to carry out the brightness correction process ("Yes" in step So_2), the external prediction unit 126 generates a prediction image when performing the brightness correction process (step So_3). In other words, the forecast image is generated using LIC. When determining not to perform the brightness correction process ("No" in step So_2), the external prediction unit 126 generates a prediction image by performing normal motion compensation without performing the brightness correction process (step So_4).

[0271] Motion compensation > OBMC

It should be noted that the external prediction signal may be generated using motion information for an adjacent block in addition to motion information for the current block obtained from the motion estimate. More specifically, the external prediction signal may be generated in units of sub-blocks in the current block by performing weighting summation of the prediction signal based on the motion information obtained from the motion estimate (in the reference frame) and the prediction signal based on the motion information for the adjacent block (in the current frame ). This external prediction (motion compensation) is also called "overlapping block motion compensation (OBMC)".

[0272] In OBMC mode, information indicating the subblock size for the OBMC (referred to, for example, as "OBMC block size") may be signaled at the sequence level. In addition, information indicating whether or not to use the OBMC mode (called, for example, an “OBMC flag”) is signaled at the CU level. It should be noted that the signaling of this information need not necessarily be performed at the sequence level and the CU level and may be performed at another level (eg, frame level, slice level, tile level, CTU level, or subblock level).

[0273] Examples of the OBMC mode are described in more detail below. Fig. 35 and 36 are a flowchart and a conceptual diagram for illustrating a brief view of the prediction image correction process performed by the OBMC process.

[0274] First, as illustrated in FIG. 36, a prediction image (Pred) is obtained through normal motion compensation using a motion vector (MV) assigned to the target (current) processing unit. In fig. 36, the arrow "MV" points to the reference frame and indicates what the current block of the current frame is referenced to obtain the prediction image.

[0275] Next, the prediction image (Pred_L) is obtained by applying the motion vector (MV_L) that is already extracted for the coded block adjacent to the left of the current block to the current block (reusing the motion vector for the current block). The motion vector (MV_L) is indicated by an arrow "MV_L" indicating a reference frame from the current block. The first correction of the prediction image is performed by overlapping two prediction images Pred and Pred_L. This provides the advantage of blending the boundary between adjacent blocks.

[0276] Similarly, a prediction image (Pred_U) is obtained by applying a motion vector (MV_U) that is already extracted for a coded block adjacent above the current block to the current block (reusing the motion vector for the current block). The motion vector (MV_U) is indicated by an arrow "MV_U" indicating a reference frame from the current block. The second correction to the prediction image is performed by overlapping the prediction image Pred_U with the prediction images (eg, Pred and Pred_L) for which the first correction was performed. This provides the advantage of blending the boundary between adjacent blocks. The prediction image obtained by the second correction is a prediction image in which the boundary between adjacent blocks is blended (smoothed) and therefore represents the final prediction image of the current block.

[0277] Although the above example is a two-path correction method using a left and an upper adjacent block, it should be noted that the correction method may be a three or more path correction method using also a right adjacent block and/or a lower adjacent block.

[0278] It should be noted that the area in which such overlap is performed may be only a portion of the area near the block boundary instead of the pixel area of the entire block.

[0279] It should be noted that the above describes a process for correcting prediction images according to OBMC to obtain one prediction image Pred from one reference frame by overlapping the additional prediction image Pred_L and Pred_U. However, when the prediction image is corrected based on a plurality of reference pictures, a similar process may be applied to each of the plurality of reference frames. In such a case, after corrected prediction images are obtained from the corresponding reference frames by performing OBMC image correction based on a plurality of reference frames, the obtained corrected prediction images are further overlapped to obtain a final prediction image.

[0280] It should be noted that, in OBMC, a unit of a current block may be a unit of a prediction block or a unit of a subblock obtained by further splitting a prediction block.

[0281] One example of a method for determining whether or not to apply an OBMC process is a method for using obmc_flag, which is a signal indicating whether or not to apply an OBMC process. As one specific example, the encoder determines whether or not the current block belongs to a region having complex motion. The encoder sets obmc_flag to a value of "1" when the block belongs to a region having complex motion and applies the OBMC process when encoding, and sets obmc_flag to a value of "0" when the block does not belong to a region having complex motion and encodes the block without applying OBMC -process. The decoder switches between applying and not applying the OBMC process by decoding the obmc_flag recorded in the stream (eg, a compressed sequence) and decoding the block by switching between applying and not applying the OBMC process according to the value of the flag.

[0282] The external prediction unit 126 generates one rectangular prediction image for the rectangular current block in the above example. However, the external prediction unit 126 can generate a plurality of prediction images having a shape other than a rectangle for the rectangular current block, and can combine the plurality of prediction images to generate a final rectangular prediction image. A shape other than a rectangle may, for example, be a triangle.

[0283] FIG. 37 is a conceptual diagram for illustrating the generation of two triangular projection images.

[0284] The external prediction unit 126 generates a triangular prediction image by performing motion compensation of the first segment having a triangular shape in the current block by using the first MV of the first segment to generate the triangular prediction image. Likewise, the external prediction unit 126 generates a triangular prediction image by performing motion compensation of the second segment having a triangular shape in the current block by using the second MV of the second segment to generate the triangular prediction image. The external prediction module 126 then generates a prediction image having a rectangular shape identical to the rectangular shape of the current block by combining these prediction images.

[0285] It should be noted that although the first segment and the second segment are triangles in the example illustrated in FIG. 37, the first segment and the second segment may be trapezoidal or other shapes different from each other. In addition, although the current block includes two segments in the example illustrated in FIG. 37, the current block may include three or more segments.

[0286] In addition, the first segment and the second segment may overlap each other. In other words, the first segment and the second segment may include an identical pixel region. In this case, the prediction image for the current block may be generated using the prediction image in the first segment and the prediction image in the second segment.

[0287] In addition, although an example in which a prediction image is generated for each of two segments using inter-prediction, a prediction image may be generated for at least one segment using intra-prediction.

[0288] Motion compensation > BIO

Next, a method for extracting a motion vector is described. First, a mode for model-based motion vector extraction under the condition of uniform straight-line motion is described in the following. This mode is also called "Bidirectional Optical Flow (BIO) mode".

[0289] FIG. 38 is a conceptual diagram to illustrate the model under the condition of uniform linear motion. In fig. 38, (v _x , v _y ) indicates the velocity vector, and τ ₀ and τ ₁ indicate the time distances between the current frame (Cur Pic) and two reference frames (Ref ₀ , Ref ₁ ). (MVx ₀ , MVy ₀ ) indicate motion vectors corresponding to the reference frame Ref ₀ and (MVx ₁ , MVy ₁ ) indicate motion vectors corresponding to the reference frame Ref ₀ .

[0290] Here, under the assumption of relatively uniform linear motion demonstrated by the velocity vectors (v _x , v _y ), (MVx ₀ , MVy ₀ ) and (MVx ₁ , MVy ₁ ), are represented as (v _x τ ₀ , v _y τ ₀ ) and (-v _x τ ₁ , -v _y τ ₁ ), respectively, and the following optical flow equation can be used:

[0291] Mathematical expression 3

[0292] Here, I(k) indicates the value of the motion-compensated luminance signal of the reference frame k (k=0, 1). This optical flow equation indicates that the sum of (i) the time derivative of the luminance signal value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is zero. The motion vector of each block, obtained from, for example, a merge list, can be adjusted in pixel units based on a combination of the optical flow equation and Hermitian interpolation.

[0293] It should be noted that the motion vector can be extracted at the decoder side using a method other than model-based motion vector extraction under the condition of uniform straight-line motion. For example, a motion vector may be extracted in sub-block units based on the motion vectors of neighboring blocks.

[0294] Motion compensation > LIC

Next, an example of a mode in which a prediction image (prediction) is generated by using a local illuminance compensation (LIC) process is described.

[0295] FIG. 39 is a conceptual diagram for illustrating one example of a method for generating predictive images using a brightness correction process performed by an LIC process.

[0296] First, the MV is extracted from the encoded reference frame, and a reference picture corresponding to the current block is obtained.

[0297] Next, information indicating how the luminance signal value is changed between the reference frame and the current frame is retrieved for the current block. This extraction is performed based on the luminance signal pixel values for the encoded left adjacent reference region (surrounding reference region) and the encoded top adjacent reference region (surrounding reference region) and the pixel value of the luminance signal at the corresponding position in the reference frame indicated by the extracted MV. The luminance correction parameter is calculated by using information indicating how the luminance signal value is changed.

[0298] The prediction image for the current block is generated by performing a luminance correction process in which a luminance correction parameter is applied to a reference image in a reference frame indicated by the MV.

[0299] It should be noted that the shape of the surrounding support region illustrated in FIG. 39 is just one example; the surrounding support area may have a different shape.

[0300] In addition, although a process in which a prediction image is generated from a single reference frame is described here, cases in which a prediction image is generated from a plurality of reference frames may be similarly described. The prediction image can be generated after performing the brightness correction process of the reference images obtained from the reference frames, identical to that described above.

[0301] One example of a method for determining whether or not to apply the LIC process is a method for using a lic_flag, which is a signal indicating whether or not to apply the LIC process. As one specific example, the encoder determines whether or not the current block belongs to a region having a brightness change. The encoder sets lic_flag to a value of "1" when the block belongs to a region having a luminance change and applies the LIC process when encoding, and sets lic_flag to a value of "0" when the block does not belong to a region having a luminance change and encodes the current block without applying LIC process. The decoder can decode the lic_flag written in the stream and decode the current block by switching between applying and not applying the LIC process according to the value of the flag.

[0302] One example of another method for determining whether or not to apply an LIC process is a method for determining according to whether or not an LIC process is applied to a surrounding block. In one specific example, when the merge mode is used in the current block, it is determined whether or not the LIC process is applied when encoding the surrounding coded block selected after MV extraction in the merge mode process. According to the result, encoding is performed by switching between applying and not applying the LIC process. It should be noted that also in this example, identical processes are applied in the processes on the decoder side.

[0303] The following describes in detail an embodiment of the luminance correction (LIC) process described with reference to FIG. 39.

[0304] First, the inter prediction unit 126 extracts a motion vector to obtain a reference picture corresponding to the current block to be encoded from a reference frame that is a coded frame.

[0305] Next, the inter prediction module 126 extracts information indicating how the luminance signal value of the reference frame is changed to the luminance signal value of the current frame, using the pixel luminance value of the encoded surrounding reference region that is adjacent to the left of or above the current block, and a luminance signal value at a corresponding position in the reference frame indicated by the motion vector, and calculates a luminance correction parameter. For example, it is assumed that the pixel value of the luminance signal of a given pixel in the surrounding reference region in the current frame is p0, and that the pixel value of the luminance signal of the pixel corresponding to a given pixel in the surrounding reference region in the reference frame is p1. The external prediction unit 126 calculates coefficients A and B to optimize A*p1+B=p0 as a luminance correction parameter for a plurality of pixels in the surrounding reference region.

[0306] Next, the external prediction unit 126 performs a luminance correction process using a luminance correction parameter for the reference image in the reference frame indicated by the motion vector to generate a prediction image for the current block. For example, it is assumed that the pixel value of the luminance signal in the reference image is p2, and that the luminance-corrected pixel value of the luminance signal of the prediction image is p3. The external prediction unit 126 generates a prediction image after undergoing the brightness correction process by calculating A*p2+B=p3 for each of the pixels in the reference image.

[0307] It should be noted that the shape of the surrounding support region illustrated in FIG. 39 is one example; a different shape from the surrounding support area may be used. In addition, part of the surrounding support region illustrated in FIG. 39. For example, a region having a certain number of pixels extracted from each of the top adjacent pixel and the left adjacent pixel can be used as a surrounding reference region. A certain number of pixels may be predetermined.

[0308] In addition, the surrounding reference area is not limited to an area that is adjacent to the current block, and may be an area that is not adjacent to the current block. In the example illustrated in FIG. 39, the surrounding reference area in the reference frame is an area indicated by the motion vector in the current frame from the surrounding reference area in the current frame. However, it is also possible for an area indicated by another motion vector. For example, the other motion vector may be a motion vector in the surrounding reference region in the current frame.

[0309] Although the operations performed by the encoder 100 are described here, it should be noted that the decoder 200 typically performs similar operations.

[0310] It should be noted that the LIC process can be applied not only to the luma signal, but also to the chrominance signal. At this time, a correction parameter can be extracted separately for each of Y, Cb and Cr, or a common correction parameter can be used for any of Y, Cb and Cr.

[0311] In addition, the LIC process may be applied in subblock units. For example, the correction parameter may be extracted using a surrounding reference area in the current sub-block and a surrounding reference area in the reference sub-block in the reference frame indicated by the MV of the current sub-block.

[0312] Prediction Controller

The external prediction unit 128 selects one of the internal prediction signal (the signal output from the internal prediction unit 124) and the external prediction signal (the signal output from the external prediction unit 126) and outputs the selected signal to the subtractor 104 and the adder 116 as a prediction signal.

[0313] As illustrated in FIG. 1, in various kinds of encoder examples, the prediction controller 128 may output a prediction parameter that is input to the entropy encoder 110. The entropy encoder 110 may generate a coded bit stream (or sequence) based on the prediction parameter that is input from the prediction controller 128 and the quantized coefficients that are input from quantizer 108. The prediction parameter may be used in the decoder. The decoder can receive and decode the encoded bit stream and perform processes identical to the prediction processes performed by the intra prediction module 124, the external prediction module 126, and the prediction controller 128. The prediction parameter may include (i) a predictive selection signal (e.g., a motion vector, prediction type, or prediction mode used by intraprediction module 124 or interprediction module 126) or (ii) an optional index, flag, or value that is based on a prediction process performed in each of the internal prediction module 124, the external prediction module 126, and the prediction controller 128, or which indicates the prediction process.

[0314] Encoder installation example

Fig. 40 is a block diagram illustrating an installation example of the encoder 100. The encoder 100 includes a processor a1 and a memory a2. For example, the plurality of constituent elements of the encoder 100 illustrated in FIG. 1 are mounted in the processor a1 and memory a2 illustrated in FIG. 40.

[0315] Processor a1 is a circuit that performs information processing and is available to storage device a2. For example, processor a1 is a specialized or general electronic circuit that encodes video. Processor a1 may be a processor such as a CPU. In addition, processor a1 may be a collection of multiple electronic circuits. In addition, for example, processor a1 may serve as two or more constituent elements of the plurality of constituent elements of the encoder 100 illustrated in FIG. 1, etc.

[0316] Storage device a2 is a dedicated or general storage device for storing information that is used by processor a1 to encode video. The storage device a2 may be an electronic circuit and may be coupled to the processor a1. In addition, the storage device a2 may be included in the processor a1. In addition, the storage device a2 may be a collection of a plurality of electronic circuits. In addition, the storage device a2 may be a magnetic disk, an optical disk or the like. or may be represented as a storage device, recording medium, or the like. In addition, the storage device a2 may be a non-volatile storage device or a volatile storage device.

[0317] For example, storage device a2 may store video to be encoded or a bit stream corresponding to the encoded video. In addition, the storage device a2 may store a program for instructing the processor a1 to encode the video.

[0318] In addition, for example, the storage device a2 may serve as two or more constituent elements for storing information from a plurality of constituent elements of the encoder 100 illustrated in FIG. 1, etc. For example, memory a2 may serve as block memory 118 and frame memory 122 illustrated in FIG. 1. More specifically, the storage device a2 can store a reconstructed block, a reconstructed frame, etc.

[0319] It should be noted that, in the encoder 100, all of the plurality of constituent elements indicated in FIG. 1, etc. may not be implemented, and all the processes described above may not be performed. Some of the constituent elements shown in FIG. 1, etc., may be included in another device, or part of the processes described above may be performed by another device.

[0320] Decoder

The following describes a decoder capable of decoding a coded signal (coded bit stream) output, for example, from the encoder 100 described above. Fig. 41 is a block diagram illustrating a functional configuration of the decoder 200 according to the embodiment. Decoder 200 is a video decoder that decodes video in units of blocks.

[0321] As illustrated in FIG. 41, the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse converter 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an internal prediction module 216, an external prediction module 218, and a prediction controller 220.

[0322] Decoder 200 is implemented, for example, as a general processor and storage device. In this case, when the software program stored in the memory device is executed by the processor, the processor functions as an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a loop filter 212, an internal prediction module 216, an external prediction module 218. prediction and prediction controller 220. Alternatively, the decoder 200 may be implemented as one or more application-specific circuits corresponding to an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a loop filter 212, an internal prediction module 216, an external prediction module 218, and a prediction controller 220.

[0323] Hereinafter, this document describes the complete flow of processes performed by the decoder 200, and thereafter describes each of the constituent elements included in the decoder 200.

[0324] General flow of decoding process

Fig. 42 is a flowchart illustrating one example of a complete decoding process performed by decoder 200.

[0325] First, the entropy decoder 202 of the decoder 200 identifies a block partitioning pattern having a fixed size (eg, 128×128 pixels) (step Sp_1). This split pattern is a split pattern selected by the encoder 100. The decoder 200 then performs the processes of step Sp_2-Sp_6 for each of the plurality of split pattern blocks.

[0326] In other words, the entropy decoder 202 decodes (specifically, entropy decodes) the encoded quantized coefficients and the prediction parameter of the current block to be decoded (also called “current block”) (step Sp_2).

[0327] Next, the inverse quantizer 204 inversely quantizes the plurality of quantized coefficients, and the inverse transformer 206 inversely transforms the result to recover the plurality of prediction residuals (ie, the difference block) (step Sp_3).

[0328] Next, the prediction processor including all or part of the internal prediction module 216, the external prediction module 218, and the prediction controller 220 generates a prediction signal (also called a “prediction block”) of the current block (step Sp_4).

[0329] Next, the adder 208 adds the prediction block with the difference block to generate a reconstructed image (also called a “decoded image block”) of the current block (step Sp_5).

[0330] When the reconstructed image is generated, the contour filter 212 performs filtering on the reconstructed image (step Sp_6).

[0331] The decoder 200 then determines whether or not decoding of the entire frame is complete (step Sp_7). When determining that decoding is not yet completed (“No” in step Sp_7), the decoder 200 repeatedly executes processes starting from step Sp_1.

[0332] As illustrated, the processes of steps Sp_1 to Sp_7 are executed sequentially by the decoder 200. Alternatively, two or more of the processes may be executed in parallel, the processing order of two or more of the processes may be modified, etc.

[0333] Entropy decoder

Entropy decoder 202 entropy decodes the encoded bitstream. More specifically, for example, entropy decoder 202 arithmetically decodes the encoded bit stream into a binary signal. Entropy decoder 202 then debinarizes the binary signal. Consequently, the entropy decoder 202 outputs the quantized coefficients of each block to the inverse quantizer 204. The entropy decoder 202 may output a prediction parameter included in the coded bit stream (see FIG. 1) to the intra prediction module 216, the inter prediction module 218, and the controller 220. forecasting. The internal prediction module 216, the external prediction module 218, and the prediction controller 220 in the embodiment are capable of performing prediction processes identical to those performed by the internal prediction module 124, the external prediction module 126, and the encoder-side prediction controller 128.

[0334] Inverse Quantizer

The inverse quantizer 204 inversely quantizes the quantized coefficients of the block to be decoded (hereinafter referred to as the “current block”), which are inputs from the entropy decoder 202. More specifically, the inverse quantizer 204 inversely quantizes the quantized coefficients of the current block based on the quantization parameters corresponding to the quantized coefficients. The inverse quantizer 204 then outputs the inversely quantized transform coefficients of the current block to the inverse quantizer 206.

[0335] Inverter

The inverse transformer 206 recovers the prediction errors by inversely transforming the transform coefficients that are the inputs from the inverse quantizer 204.

[0336] For example, when information parsed from the encoded bitstream indicates that EMT or AMT should be applied (eg, when the AMT flag is true), the inverter 206 inverts the transform coefficients of the current block based on the information syntactically indicating parsed conversion type.

[0337] In addition, for example, when information parsed from the encoded bitstream indicates that NSST should be applied, inverter 206 applies a secondary inverse transform to the transform coefficients.

[0338] Totalizer

The adder 208 reconstructs the current block by summing the prediction errors, which are the inputs from the inverter 206, and the prediction samples, which are the inputs from the prediction controller 220. The adder 208 then outputs the reconstructed block to a block memory 210 and a loop filter 212.

[0339] Block memory

The block memory 210 is a storage device for storing blocks in a frame to be decoded (hereinafter referred to as a "current frame") and to be referenced in intra prediction. More specifically, block memory 210 stores reconstructed blocks output from adder 208.

[0340] Loop filter

The loop filter 212 applies the loop filter to the blocks reconstructed by the adder 208 and outputs the filtered reconstructed blocks to the frame memory 214, display device, etc.

[0341] When information indicating ALF on or off parsed from the coded bitstream indicates that ALF is on, one filter from the plurality of filters is selected based on the direction and activity of the local gradients, and the selected filter is applied to the reconstructed block.

[0342] Frame memory

The frame memory 214, for example, is a storage device for storing reference frames for use in external prediction, and is also called a “frame buffer.” More specifically, frame memory 214 stores the reconstructed block filtered by loop filter 212.

[0343] Prediction Processor (Internal Prediction Module, External Prediction Module, Prediction Controller)

Fig. 43 is a flowchart illustrating one example of a process performed by the prediction processor of the decoder 200. It should be noted that the prediction processor includes all or part of the following constituent elements: an intra prediction unit 216; external prediction module 218; and a prediction controller 220.

[0344] The prediction processor generates a prediction image of the current block (step Sq_1). This prediction image is also called a "prediction signal" or a "prediction block". It should be noted that the prediction signal is, for example, an internal prediction signal or an external prediction signal. Specifically, the prediction processor generates a prediction image of the current block using a reconstructed image that has already been obtained through prediction block generation, difference block generation, coefficient block generation, difference block reconstruction, and decoded image block generation.

[0345] The reconstructed image, for example, may be an image in a reference frame or an image of a decoded block in a current frame, which is a frame including the current block. The decoded block in the current frame, for example, is a block adjacent to the current block.

[0346] FIG. 44 is a flowchart illustrating another example of a process performed by the decoder prediction processor 200.

[0347] The prediction processor determines a method or mode for generating a prediction image (step Sr_1). For example, the method or mode may be determined, for example, based on a prediction parameter, etc.

[0348] When determining the first method as the mode for generating a prediction image, the prediction processor generates a prediction image according to the first method (step Sr_2a). When determining the second method as the mode for generating the prediction image, the prediction processor generates the prediction image according to the second method (step Sr_2b). When determining the third method as the mode for generating the prediction image, the prediction processor generates the prediction image according to the third method (step Sr_2c).

[0349] The first method, the second method and the third method may be mutually different methods for generating a predictive image. Each of the first to third methods may be an external forecasting method, an internal forecasting method, or another forecasting method. The above-described reconstructed image can be used in these prediction methods.

[0350] Internal prediction module

The intra prediction unit 216 generates a prediction signal (intra prediction signal) by performing intra prediction by reference to a block or blocks in the current frame stored in the block memory 210 based on the intra prediction mode parsed from the encoded bit stream. More specifically, the intra prediction module 216 generates an internal prediction signal by performing intra prediction by reference to samples (eg, luma value and/or chrominance value) of the block or blocks adjacent to the current block, and then outputs the internal prediction signal to the prediction controller 220 .

[0351] It should be noted that when an intra prediction mode in which a luma block is referenced in intra prediction of a chrominance block is selected, the intra prediction unit 216 can predict the chrominance signal component of the current block based on the luma signal component of the current block.

[0352] In addition, when information parsed from the coded bitstream indicates that PDPC should be applied, intra prediction module 216 adjusts internally predicted pixel values based on horizontal/vertical reference pixel gradients.

[0353] External Prediction Module

The external prediction module 218 predicts the current block by reference to a reference frame stored in the frame memory 214. External prediction is performed in units of the current block or subblock (eg, 4x4 block) within the current block. For example, the external prediction module 218 generates an external prediction signal of the current block or sub-block by performing motion compensation by using motion information (eg, a motion vector) parsed from the encoded bitstream (eg, a prediction parameter output from the entropy decoder 202) and outputs an external prediction signal to the prediction controller 220.

[0354] It should be noted that when information parsed from the encoded bit stream indicates that the OBMC mode is to be applied, the external prediction unit 218 generates an external prediction signal using the motion information of the adjacent block in addition to the motion information of the current block. obtained from the motion estimate.

[0355] In addition, when information parsed from the encoded bitstream indicates the use of the FRUC mode, the external prediction module 218 extracts the motion information by performing motion estimation in accordance with a template matching method (bilateral matching or template matching) syntactically parsed from the encoded bit stream. The external prediction module 218 then performs motion compensation (prediction) using the extracted motion information.

[0356] In addition, when the BIO mode is to be applied, the external prediction module 218 extracts a motion vector based on the model under the condition of uniform straight-line motion. In addition, when information parsed from the encoded bitstream indicates that the motion-compensated affine prediction mode should be applied, the inter-prediction module 218 extracts the motion vector of each sub-block based on the motion vectors of neighboring blocks.

[0357] MV extract > normal external mode

When the information parsed from the coded bitstream indicates that the normal outer mode should be applied, the outer prediction unit 218 extracts the MV based on the information parsed from the coded bitstream and performs motion compensation (prediction) using the MV.

[0358] FIG. 45 is a flowchart illustrating an example of inter-prediction in normal inter-mode in the decoder 200.

[0359] The external prediction module 218 of the decoder 200 performs motion compensation for each block. The external prediction module 218 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Ss_1). In other words, the external prediction module 218 generates a list of possible MV options.

[0360] Next, the external prediction module 218 extracts N (an integer of 2 or more) MV candidates from the plurality of MV candidates obtained in step Ss_1 as motion vector predictor candidates (also called “MV candidates”). predictors") according to a certain order of priorities (stage Ss_2). It should be noted that the priority order can be determined in advance for each of the N possible MV predictor options.

[0361] Next, the external prediction unit 218 decodes the motion vector predictor selection information from the input stream (i.e., the coded bit stream) and selects one MV predictor candidate from the N candidate MV predictor candidates using the decoded motion vector predictor selection information. , as the motion vector (also called "MV predictor") of the current block (step Ss_3).

[0362] Next, the external prediction unit 218 decodes the MV difference from the input stream and extracts the MV for the current block by summing the difference value, which is the decoded MV difference, and the selected motion vector predictor (step Ss_4).

[0363] Finally, the external prediction unit 218 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the decoded reference frame (step Ss_5).

[0364] Prediction Controller

The prediction controller 220 selects an internal prediction signal or an external prediction signal and outputs the selected prediction signal to the adder 208. In general, the configurations, functions and processes of the prediction controller 220, the internal prediction module 216 and the external prediction module 218 on the decoder side may correspond to the configurations, functions and processes of the prediction controller 128, the internal prediction module 124, and the external prediction module 126 on the encoder side.

[0365] Decoder installation example

Fig. 46 is a block diagram illustrating an installation example of the decoder 200. The decoder 200 includes a processor b1 and a memory b2. For example, the plurality of constituent elements of the decoder 200 illustrated in FIG. 41 are mounted in the processor b1 and memory b2 illustrated in FIG. 46.

[0366] Processor b1 is a circuit that performs information processing and is available to storage device b2. For example, processor b1 is a specialized or general electronic circuit that decodes video (ie, an encoded bit stream). Processor b1 may be a processor such as a CPU. In addition, processor b1 may be a collection of multiple electronic circuits. In addition, for example, processor b1 may serve as two or more constituent elements of the plurality of constituent elements of the decoder 200 illustrated in FIG. 41, etc.

[0367] Memory b2 is a dedicated or general memory for storing information that is used by processor b1 to decode the encoded bit stream. Memory b2 may be an electronic circuit and may be coupled to processor b1. In addition, memory b2 may be included in processor b1. In addition, the storage device b2 may be a collection of a plurality of electronic circuits. In addition, the storage device b2 may be a magnetic disk, an optical disk or the like. or may be represented as a storage device, recording medium, or the like. In addition, the storage device b2 may be a non-volatile storage device or a volatile storage device.

[0368] For example, storage device b2 may store video or a bit stream. In addition, memory b2 may store a program for instructing processor b1 to decode the encoded bit stream.

[0369] In addition, for example, the storage device b2 may serve as two or more constituent elements for storing information from a plurality of constituent elements of the decoder 200 illustrated in FIG. 41, etc. In particular, memory b2 may serve as block memory 210 and frame memory 214 illustrated in FIG. 41. More specifically, the storage device b2 can store a reconstructed block, a reconstructed frame, etc.

[0370] It should be noted that, in the decoder 200, all of the plurality of constituent elements illustrated in FIG. 41, etc. may not be implemented, and all the processes described above may not be performed. Some of the constituent elements shown in FIG. 41, etc., may be included in another device, or part of the processes described above may be performed by another device.

[0371] Definitions of terms

The corresponding terms may be specified as below as examples.

[0372] A frame is an array of luma samples in monochrome, or an array of luma samples and two corresponding arrays of chrominance samples in 4:2:0, 4:2:2, and 4:4:4 color formats. A frame can be either a frame or a field.

[0373] A frame is a composition of a top field and a bottom field, with lines 0, 2, 4,..., samples being initiated from the top field, and lines 1, 3, 5,..., samples being initiated from the bottom field.

[0374] A slice is an integer number of coding tree units contained in one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) in an identical access unit.

[0375] A tile is a rectangular area of coding tree blocks in a particular column of tiles and a particular row of tiles in a frame. A tile can be a rectangular area of the frame that should be able to be decoded and encoded independently, although edge filtering can still be applied at the edges of the tiles.

[0376] The block is an MXN array of (M columns by N rows) samples or an MXN array of transform coefficients. A block may be a square or rectangular area of pixels including one luma matrix and two chroma matrices.

[0377] A coding tree unit (CTU) may be a frame luma sample coding tree block that has three sample arrays, or two corresponding chrominance sample coding tree blocks. Alternatively, the CTU may be a sample encoding tree block of one of a monochrome frame and a frame that is encoded using three separate color planes and syntax structures used to encode the samples.

[0378] A superblock may be a 64x64 pixel square block that consists of either 1 or 2 mode information blocks or is recursively segmented into four 32x32 blocks, which themselves can be further segmented.

[0379] Description of the BDOF(BIO) process structure

The structure of the BDOF(BIO) process is described again with reference to FIG. 38, 47 and 48.

[0380] FIG. 47 is a flowchart indicating one example of external prediction according to BIO. In addition, FIG. 48 is a diagram illustrating one example of a functional configuration of the external prediction unit 126 that performs external prediction according to the BIO.

[0381] As illustrated in FIG. 48, the external prediction unit 126 includes, for example, a memory 126a, an interpolated image extracting unit 126b, a gradient image extracting unit 126c, an optical flow extracting unit 126d, a correction value extracting unit 126e, and a prediction image corrector 126f. It should be noted that the memory 126a may be a frame memory 122.

[0382] The external prediction unit 126 extracts two motion vectors (M0, M1) using two reference frames (Ref0, Ref1) different from the frame (Cur Pic) including the current block. The external prediction unit 126 then extracts the prediction image for the current block using two motion vectors (M0, M1) (step Sy_1). It should be noted that the motion vector M0 is a motion vector (MVx0, MVy0) corresponding to the reference frame Ref0, and the motion vector M1 is a motion vector (MVx1, MVy1) corresponding to the reference frame Ref0.

[0383] Next, the interpolated image extracting unit 126b extracts the interpolated image I ⁰ for the current block using the motion vector M0 and the reference frame L0 with reference to the storage device 126a. In addition, the interpolated image extracting unit 126b extracts the interpolated image I ¹ for the current block using the motion vector M1 and the reference frame L1 with reference to the storage device 126a (step Sy_2). Here, the interpolated image I ⁰ is an image included in the reference frame Ref0 and is to be extracted for the current block, and the interpolated image I ¹ is an image included in the reference frame Ref0 and is to be extracted for the current block. Each of the interpolated image I ⁰ and the interpolated image I ¹ may have a size identical to the size of the current block. Alternatively, each of the interpolated image I ⁰ and the interpolated image I ¹ may be an image larger than the current block to properly extract the gradient image, which is described below. In addition, the interpolated image I ⁰ and the interpolated image I ¹ may include motion vectors (M0, M1), reference frames (L0, L1), and a prediction image extracted by applying a motion compensation filter.

[0384] In addition, the gradient image extracting unit 126c extracts the gradient images (I _x ⁰ , I _x ¹ , I _y ⁰ , I _y ¹ ) of the current block from the interpolated image I ⁰ and the interpolated image I ¹ (step Sy_3). It should be noted that the gradient images in the horizontal direction are (I _x ⁰ , I _x ¹ ), and the gradient images in the vertical direction are (I _y ⁰ , I _y ¹ ). The gradient image extraction module 126c may extract gradient images, for example, by applying a gradient filter to the interpolated images. The gradient images may be gradient images, each of which indicates the amount of spatial variation of a pixel value along a horizontal direction or a vertical direction.

[0385] Next, the optical flow extractor 126d extracts, for each sub-block of the current block, the optical flow (vx, vy), which is the above-described velocity vector, using the interpolated images (I ⁰ , I ¹ ) and gradient images (I _x ⁰ , I _x ¹ , I _y ⁰ , I _y ¹ ) (stage Sy_4). Optical flow is a coefficient for correcting the amount of spatial pixel movement and may be called a "local estimated motion value", "corrected motion vector" or "corrected weight vector". As one example, the subunit may be a 4x4 pixel sub-CU. It should be noted that the optical flow may be extracted in units other than subblock units, such as pixel units.

[0386] Next, the external prediction module 126 corrects the prediction image for the current block using the optical flow (vx, vy). For example, the correction value extracting unit 126e extracts a correction value for a pixel value included in the current block using optical flow (vx, vy) (step Sy_5). The prediction image corrector 126f can then correct the prediction image for the current block using the correction value (step Sy_6). It should be noted that the correction value may be retrieved in units of a pixel, or may be retrieved in units of a plurality of pixels or in units of sub-blocks.

[0387] It should be noted that the BIO process flow is not limited to the process disclosed in FIG. 47. Only part of the processes disclosed in FIG. 47 may be executed, or another process may be added or used as a replacement, or the processes may be executed in a different processing order.

[0388] For example, BDOF(BIO) may be defined as a process for generating a corrected prediction image using the gradient of a reference block. Alternatively, BDOF can be defined as a process for generating a prediction image using the gradients of two reference blocks. Alternatively, BDOF may be defined as a process for generating a predictive image using a gradient in a mode other than an affine mode. Alternatively, BDOF may be specified as an optical flow process in a mode other than the affine mode. Here, gradient means the spatial gradient of a pixel value.

[0389] First concrete example of BIO

Next, a first specific example of a BIO-based decoding process will be described with reference to FIG. 49, 50 and 51. For example, decoder 200 calculates a BIO parameter from two reference blocks that are subject to motion compensation, and decodes the current block using the calculated BIO parameter. The BIO parameter is a parameter corresponding to the optical flow described above. The BIO parameter is also referred to as "local estimated motion value", "adjusted motion vector", "adjusted MV value", "adjusted weighted motion vector" or "BIO correction value".

[0390] FIG. 49 is a flowchart indicating a first specific example of a BIO-based decoding process. Fig. 50 is a conceptual diagram illustrating an example of calculating horizontal gradient values that represent gradient values in the horizontal direction. Fig. 51 is a conceptual diagram illustrating an example of calculating vertical gradient values that represent gradient values in the vertical direction.

[0391] First, the decoder 200 calculates a first sum for the current block using the horizontal gradient value of the first reference block and the horizontal gradient value of the second reference block (S1001). The current block may be a sub-block of the current coding unit (current CU), as illustrated in FIG. 50 and 51.

[0392] The first reference block is a block to be referenced when decoding the current block, and is specified in the L0 reference frame by the current block or the first motion vector of the current CU. The second reference block is a block to be referenced when decoding the current block, and is specified in the L1 reference frame by the current block or the second motion vector of the current CU.

[0393] Essentially, the L0 reference frame and the L1 reference frame are two different reference frames, and the first reference block and the second reference block are two different reference blocks. In addition, here, the first reference block and the second reference block used here are adjusted, respectively, with sub-pixel precision using a motion compensation filter and have a size identical to the size of the current block.

[0394] When the current block is a sub-block in the current coding unit, each of the first reference block and the second reference block may be a sub-block of the reference block in the current coding unit.

[0395] In other words, the plurality of pixel values in the reference frame L0 in FIG. 50 and 51 may be a plurality of pixel values in a block specified as a reference block in the current encoding unit for a reference frame L0. Similarly, the plurality of pixel values in the reference frame L1 in FIG. 50 and 51 may be a plurality of pixel values in a block specified as a reference block in the current coding unit for the L1 reference frame.

[0396] The decoder 200 calculates the first sum using the horizontal gradient value of the above first reference block and the horizontal gradient value of the above second reference block. Decoder 200 may calculate the first sum using not only the horizontal gradient value of the first reference block and the horizontal gradient value of the second reference block, but also the horizontal gradient value of the adjacent zone that borders the first reference block and the horizontal gradient value of the adjacent zone that borders second support block. Expressions (3.1) and (3.2) below indicate an example of a method for the process of calculating the second amount.

[0397] Mathematical expression 4

[0398] Mathematical expression 5

[0399] Here, x denotes multiplication, and + denotes addition. In addition, the sign indicates a plus or minus sign. Specifically, the sign denotes a as a plus value and -1 as a minus value. In particular, the sign means, for example, the following.

[0400] Mathematical expression 6

[0401] As a result, sign (G _x [i, j]) x G _x [i, j] becomes G _x [i, j] without changing sign when G _x [i, j] is a signed value "plus" and becomes -G _x [i, j] when G _x [i, j] is a value with a minus sign. Thus, the calculation is equivalent to retrieving the absolute value (abs) of G _x [i, j]).

[0402] In expression (3.1), I _x ⁰ denotes the horizontal gradient value in the first reference block in the L0 reference frame, and I _x ¹ denotes the horizontal gradient value in the second reference block in the L1 reference frame.

[0403] An example of a horizontal gradient filter for obtaining horizontal gradient values is a three-tap filter having a set of filter coefficients [-1, 0, 1]. The horizontal gradient value in the first reference block is calculated by applying a horizontal gradient filter to a plurality of reference pixels in the first reference block. The horizontal gradient value in the second reference block is calculated by applying a horizontal gradient filter to a plurality of reference pixels in the second reference block.

[0404] In the example illustrated in FIG. 50, the horizontal gradient value I _x ⁰ of a pixel located at position [3, 2] in the first reference block is calculated as the matrix product of [-1, 0, 1] ^T [2, 3, 5], and the resulting value is 3. The horizontal gradient value I _x ¹ of the pixel located at position [3, 2] in the second reference block is calculated as the matrix product [-1, 0, 1] ^T [5, 3, 2], and the resulting value is -2. It should be noted that [a, b, c] denotes a matrix having three rows and one column.

[0405] In expression (3.2), sG _x denotes the first sum and is calculated as the sum of the absolute values of G _x for the window, denoted as Ω. The window size Ω can be identical to the size of the current block. Alternatively, the window size Ω may be larger than the current block size. In the second case, the value of G _x at the position adjacent to the current block is included in the process of calculating the first sum.

[0406] As with the first sum, which is the sum of the horizontal gradient values, the decoder 200 calculates a second sum for the current block using the vertical gradient value of the first reference block and the vertical gradient value of the second reference block (S1002). Expressions (3.3) and (3.4) below indicate an example of a method for the process of calculating the second amount.

[0407] Mathematical expression 7

[0408] Mathematical expression 8

[0409] In expression (3.3), I _y ⁰ denotes the vertical gradient value in the first reference block in the L0 reference frame, and I _y ¹ denotes the vertical gradient value in the second reference block in the L1 reference frame.

[0410] An example of a vertical gradient filter for obtaining vertical gradient values is a three-tap filter having a set of filter coefficients [-1, 0, 1]. The vertical gradient value in the first reference block is calculated by applying a vertical gradient filter to a plurality of reference pixels in the first reference block. The vertical gradient value in the second reference block is calculated by applying a vertical gradient filter to a plurality of reference pixels in the second reference block.

[0411] In the example illustrated in FIG. 51, the vertical gradient value I _y ⁰ of a pixel located at position [3, 2] in the first reference block is calculated as the matrix product of [-1, 0, 1] ^T [2, 3, 5], and the resulting value is 3. The vertical gradient value I _y ¹ of the pixel located at position [3, 2] in the second reference block is calculated as the matrix product [-1, 0, 1] ^T [5, 3, 2], and the resulting value is -2.

[0412] In expression (3.4), sG _y denotes the second sum and is calculated as the sum of the absolute values of G _y for the window denoted as Ω. When the window size Ω is larger than the value of the current block, the value G _y at the position adjacent to the current block is included in the process of calculating the second sum.

[0413] Next, the decoder 200 makes a judgment as to whether or not the first amount is greater than the second amount (S1003). When making such a judgment that the first sum is greater than the second sum ("Yes" in S1003), the decoder 200 determines the BIO parameter for the current block without using the vertical gradient value at all (S1004). Expressions (3.5)-(3.9) indicate an example of the calculation process for determining the BIO parameter in this case. According to these expressions, the BIO parameter, denoted by u, is calculated using the horizontal gradient values.

[0414] Mathematical expression 9

[0415] Mathematical expression 10

[0416] Mathematical expression 11

[0417] Mathematical expression 12

[0418] Mathematical expression 13

[0419] Here, "-" denotes subtraction, and ">>" denotes shift calculation. For example, "a>>b" means that a is shifted to the right by b bits. In addition, Bits, BIOShift and Clip respectively indicate the following. It should be noted that in the description below, ceil and floor denote decimal rounding up and decimal rounding down, respectively.

[0420] Mathematical expression 14

[0421] According to expression (3.5), sG _x dI is calculated as the sum of the products of the differences between I0 _{i, j} and I1 _{i, j} and sign (Gx[i, j]) for the window Ω. Here, I0 _{i, j} denotes the pixel value at position [i, j] in the first reference block in the reference frame L0, and I1 _{i, j} denotes the pixel value at position [i, j] in the second reference block in the reference frame L1. I0 _{i, j} and I1 _{i, j} can be represented simply as I ⁰ and I ¹ . The BIO parameter u is calculated using sG _x dI and sG _x according to expressions (3.6)-(3.9).

[0422] When making such an assessment that the first sum does not exceed the second sum ("No" in S1003), the decoder 200 determines the BIO parameter u for the current block without using the horizontal gradient value at all (S1005). Expressions (3.10)-(3.14) indicate an example of the calculation process for determining the BIO parameter u in this case. Expressions (3.10)-(3.14) are essentially identical to expressions (3.5)-(3.9) except that the BIO parameter u is calculated using the vertical gradient values in expression (3.10)-(3.14).

[0423] Mathematical expression 15

[0424] Mathematical expression 16

[0425] Mathematical expression 17

[0426] Mathematical expression 18

[0427] Mathematical expression 19

[0428] According to expression (3.10), sG _y dI is calculated as the sum of the products of the differences between I0 _{i, j} and I1 _{i, j} and sign (G _y [i, j]) for the window Ω. The BIO parameter u is calculated using sG _y dI and sG _y according to expressions (3.11)-(3.14).

[0429] The decoder 200 then decodes the current block using the BIO parameter u (S1006). Specifically, the decoder 200 generates a prediction sample using the BIO parameter u and decodes the current block using the prediction sample. Expressions (3.15) and (3.16) indicate an example of the calculation process for generating a predictive sample.

[0430] Mathematical expression 20

[0431] Mathematical expression 21

[0432] When it is determined that the first amount is greater than the second amount (“Yes” at S1003), expression (3.15) is used. When it is determined that the first amount does not exceed the second amount ("No" in S1003), expression (3.16) is used.

[0433] The decoder 200 may repeat the processes (S1001-S1006) for all sub-blocks in each of the current CUs.

[0434] Decoder 200 allows the accuracy of the predictive sample in the current block to be improved using BIO. The decoder 200 further allows for a reduction in computational increase because the decoder 200 uses only one of the horizontal gradient values and the vertical gradient values when calculating the BIO parameter.

[0435] The above expressions are examples. Expressions for calculating the BIO parameter are not limited to the above expressions. For example, the plus or minus sign included in each expression may vary, or an expression equivalent to any of the above expressions may be used. In particular, the following expression (4.1) can be used as an expression corresponding to expressions (3.1) and (3.2) described above.

[0436] Mathematical expression 22

[0437] In addition, for example, as an expression corresponding to expression (3.5) described above, the following expression (4.2) can be used.

[0438] Mathematical expression 23

[0439] For example, as an expression corresponding to expression (3.15) described above, the following expression (4.3) can be used.

[0440] Mathematical expression 24

[0441] Expressions (3.6)-(3.9) actually indicate division and as such can be represented as expression (4.4).

[0442] Mathematical expression 25

[0443] Expressions (4.1)-(4.4) are practically identical to expressions (3.1), (3.2), (3.5)-(3.9) and (3.15) described above.

[0444] Similarly, in particular, as an expression corresponding to expressions (3,3) and (3.4) described above, the following expression (4.5) can be used.

[0445] Mathematical expression 26

[0446] For example, as an expression corresponding to expression (3.10) described above, the following expression (4.6) can be used.

[0447] Mathematical expression 27

[0448] For example, as an expression corresponding to expression (3.16) described above, the following expression (4.7) can be used.

[0449] Mathematical expression 28

[0450] Expressions (3.11)-(3.14) actually indicate division and as such can be represented as expression (4.8).

[0451] Mathematical expression 29

[0452] Expressions (4.5)-(4.8) are practically identical to expressions (3.3), (3.4), (3.10)-(3.14) and (3.16) described above.

[0453] In the above process, a horizontal gradient value or a vertical gradient value is used based on a comparison between the first sum and the second sum. However, the flowcharts of the decoding process are not limited to the flowcharts described above. Whether a horizontal gradient value or a vertical gradient value should be used may be determined in advance by another encoding parameter or the like. The BIO parameter may be extracted using the horizontal gradient values, and the BIO parameter may be extracted using the vertical gradient values, without comparing the first sum and the second sum. Alternatively, only one of the first sum and the second sum may be calculated.

[0454] Even without comparing the first sum and the second sum, the decoder 200 allows the reduction of significant multiplication, which requires a large amount of computation in the calculations performed on the corresponding pixel positions, and the extraction of multiple parameters to form a predictive image with a small amount of computation. In particular, although the plus or minus signs in expressions (3.2), (3.4), (3.5), (3.10), (4.1), (4.2), (4.5), (4.6), etc. change, no significant multiplication is used. Accordingly, the number of significant multiplications in BIO processing can be significantly reduced.

[0455] In other words, decoder 200 allows the calculation of sG _x , sG _x dI, sG _y and sG _y dI with a small amount of computation. Accordingly, the decoder 200 allows for a reduction in decoding processing.

[0456] Although the decoding process is described in the above description, identical processes included in the decoding process may also be applied in the encoding process. In other words, the decoding in the above description can be read as encoding.

[0457] It should be noted that the computational expressions described herein are examples. Computational expressions may be partially modified, parts of expressions may be deleted, or parts may be added to expressions, provided that the resulting expressions perform similar processing. For example, one or more of the plurality of expressions associated with the first amount, the second amount, or the BIO parameter may be replaced by one or more expressions in other specific examples, or may be replaced by another one or more expressions that are different from the expressions in other specific examples.

[0458] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0459] Second specific example BIO

Next, a second specific example of a BIO-based decoding process is described. For example, similar to the first specific example, the decoder 200 calculates a BIO parameter from two reference blocks that are subject to motion compensation, and decodes the current block using the calculated BIO parameter.

[0460] In this specific example, the prediction sample for decoding the current block is generated according to expressions (5.1)-(5.8) specified below.

[0461] Mathematical expression 30

[0462] Mathematical expression 31

[0463] Mathematical expression 32

[0464] Mathematical expression 33

[0465] Mathematical expression 34

[0466] Mathematical expression 35

[0467] Mathematical expression 36

[0468] Mathematical expression 37

[0469] S ₁ in expression (5.1) in this specific example corresponds to sG _x in expression (3.2) in the first specific example. In addition, S ₂ in expression (5.2) in this specific example corresponds to sG _y in expression (3.4) in the first specific example. In addition, S ₃ in expression (5.3) in this specific example corresponds to sG _x dI in expression (3.5) in the first specific example. In addition, S ₄ in expression (5.4) in this particular example corresponds to sG _y dI in expression (3.10) in this particular example.

[0470] In addition, v _x and v _y in expressions (5.5)-(5.7) in this specific example correspond to the BIO parameter and correspond to u in expressions (3.9) and (3.14) in the first specific example.

[0471] In the first specific example, no change is made to the plus or minus sign, and significant multiplication is not performed in the calculation of sG _x , sG _y , sG _x dI and sG _y dI . In the specific example, significant multiplication is performed in the calculation of S ₁ , S ₂ , S ₃ and S ₄ . Thus, although the amount of computation increases, the predictive sample is formed with greater accuracy. On the other hand, the amount of computation is reduced in the first case study.

[0472] It should be noted that the computational expressions described herein are examples. Computational expressions may be partially modified, parts of expressions may be deleted, or parts may be added to expressions, provided that the resulting expressions perform similar processing. For example, one or more of the plurality of expressions associated with the first amount, the second amount, or the BIO parameter may be replaced by one or more expressions in other specific examples, or may be replaced by another one or more expressions that are different from the expressions in other specific examples.

[0473] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0474] Third specific example BIO

Next, a third specific example of a BIO-based decoding process will be described with reference to FIG. 52. For example, similar to the first specific example, decoder 200 calculates at least one BIO parameter from two reference blocks that are subject to motion compensation, and decodes the current block using the at least one calculated BIO parameter.

[0475] FIG. 52 is a flowchart indicating a third specific example of a BIO-based decoding process.

[0476] First, the decoder 200 calculates a first sum for the current block using the horizontal gradient value of the first reference block and the horizontal gradient value of the second reference block (S2001). The first sum calculation process (S2001) in the specific example may be identical to the first sum calculation process (S1001) in the first specific example. Specifically, the decoder 200 calculates sG _x as the first sum according to expressions (6.1) and (6.2), which are identical to expressions (3.1) and (3.2) in the first specific example.

[0477] Mathematical expression 38

[0478] Mathematical expression 39

[0479] In addition, the decoder 200 calculates a second sum for the current block using the vertical gradient value of the first reference block and the vertical gradient value of the second reference block (S2002). The second sum calculation process (S2002) in the specific example may be identical to the second sum calculation process (S1002) in the first specific example. Specifically, the decoder 200 calculates sG _y as the second sum according to expressions (6.3) and (6.4), which are identical to expressions (3.3) and (3.4) in the first specific example.

[0480] Mathematical expression 40

[0481] Mathematical expression 41

[0482] Next, the decoder 200 makes a judgment as to whether or not at least one of the following conditions is satisfied: the first sum and the second sum are greater than the first value; and both the first amount and the second amount are less than the second value (S2003). The first value may be greater than the second value, less than the second value, or identical to the second value. The first value and the second value may be represented as a first threshold value and a second threshold value, respectively.

[0483] For example, when the first value=100 and the second value=100 are satisfied, the results of making an evaluation based on the first amount and the second amount are as follows.

[0484] First amount=300 and second amount=50: The evaluation result is "false".

First Sum=50 and Second Sum=50: The result of the evaluation is true.

First sum=300 and second sum=300: The result of the evaluation is "true".

First amount=50 and second amount=300: The evaluation result is "false".

[0485] When the evaluation result is true (Yes in S2003), the decoder 200 determines two BIO parameters for the current block using the horizontal gradient values and the vertical gradient values (S2004). Expressions (6.5)-(6.8) indicate an example of the calculation process for determining the two BIO parameters in this case. According to these expressions, two BIO parameters, denoted as u and v, are calculated using the horizontal gradient values and the vertical gradient values.

[0486] Mathematical expression 42

[0487] Mathematical expression 43

[0488] Mathematical expression 44

[0489] Mathematical expression 45

[0490] Expressions (6.5) and (6.6) are identical to expressions (3.5) and (3.10) in the first case study, respectively. The BIO parameter u is extracted using sG _x dI and sG _x according to expression (6.7). The BIO parameter v is calculated using sG _y dI and sG _y according to expression (6.8). The BIO parameters u and v can be calculated using expressions (6.9) and (6.10) below, respectively, instead of using expressions (6.7) and (6.8), respectively.

[0491] Mathematical expression 46

[0492] Mathematical expression 47

[0493] When the evaluation result is false (No in S2003), the decoder 200 determines one BIO parameter for the current block without using either the horizontal gradient value or the vertical gradient value (S2005). An example of the calculation process for determining one BIO parameter in this case is shown below.

[0494] For example, decoder 200 makes a judgment as to whether or not the first amount is greater than the second amount. In case the first sum is greater than the second sum, the decoder 200 calculates only "u" as the BIO parameter according to expressions (6.5) and (6.7) (or (6.9)). In other words, in this case, the decoder 200 determines the BIO parameter for the current block without using the vertical gradient values.

[0495] In the opposite case, in which the first sum does not exceed the second sum, the decoder 200 calculates only "v" as the BIO parameter according to expressions (6.6) and (6.8) (or (6.10)). In other words, in this case, the decoder 200 determines the BIO parameter for the current block without using horizontal gradient values.

[0496] Alternatively, for example, decoder 200 may make a judgment regarding whether the first sum of the first value is greater than or not and the second sum of the second value is less than or not.

[0497] When the first sum is greater than the first value and the second sum is less than the second value, the decoder 200 can then determine only "u" as the BIO parameter for the current block without using the vertical gradient values. When the first sum is not greater than the first value and the second sum is not less than the second value, the decoder 200 can then determine only "v" as the BIO parameter for the current block without using the horizontal gradient values.

[0498] Finally, decoder 200 then decodes the current block using at least one BIO parameter (S2006). Specifically, decoder 200 generates a predictive sample using at least one of the BIO parameters u and v and decodes the current block using the predictive sample. Expressions (6.11)-(6.13) indicate an example of the calculation process for generating a predictive sample.

[0499] Mathematical expression 48

[0500] Mathematical expression 49

[0501] Mathematical expression 50

[0502] When the result of judging based on the first sum, the second sum, the first value and the second value (S2003) is true, expression (6.11) is used. In the case where only "u" is calculated when the evaluation result (S2003) is "false", expression (6.12) is used. In the case where only "v" is calculated when the evaluation result (S2003) is "false", expression (6.13) is used.

[0503] The decoder 200 may repeat the processes (S2001-S2006) for all sub-blocks in each of the current CUs.

[0504] In addition, an evaluation process different from the evaluation process (S2003) described above may be used. For example, decoder 200 may evaluate whether the first sum and the second sum are greater than or equal to the first value.

[0505] In the event that the first sum and the second sum are greater than the first value, the decoder 200 can then determine two BIO parameters for the current block using the horizontal gradient values and the vertical gradient values. In the event that at least one of the first sum and the second sum is not greater than the first value, the decoder 200 may determine one BIO parameter for the current block without using the horizontal gradient value and the vertical gradient value at all.

[0506] Alternatively, for example, decoder 200 may make a judgment regarding whether the first sum of the first value is greater than or not and the second sum of the second value is less than or not.

[0507] In the event that the first sum is greater than the first value, and the second sum is greater than the second value, the decoder 200 can then determine two BIO parameters for the current block using the horizontal gradient values and the vertical gradient values. In the event that the first sum is not greater than the first value, or the second sum is not greater than the second value, the decoder 200 can then determine a single BIO parameter for the current block without using the horizontal gradient value or any vertical gradient value at all.

[0508] Alternatively, for example, decoder 200 may determine whether or not at least one of the following conditions is satisfied: the first sum is greater than the first value, and the second sum is greater than the second value; and the first sum is less than the third value, and the second sum is less than the fourth value.

[0509] When at least one of the two conditions is satisfied, decoder 200 can then determine two BIO parameters for the current block using the horizontal gradient values and the vertical gradient values. When either of the two conditions is not satisfied, decoder 200 may determine one BIO parameter for the current block without using either any horizontal gradient value or any vertical gradient value.

[0510] Decoder 200 allows the accuracy of the predictive sample in the current block to be improved using BIO. In addition, the decoder 200 may use only one of the horizontal gradient values and the vertical gradient values based on the condition. Thus, the decoder 200 allows the increase in computation to be reduced.

[0511] The above expressions are examples, and the expressions for calculating the BIO parameter are not limited to the above expressions. For example, the plus or minus sign included in each expression may vary, or an expression equivalent to any of the above expressions may be used. In particular, the following expression (7.1) can be used as an expression corresponding to expressions (6.1) and (6.2) described above.

[0512] Mathematical expression 51

[0513] In addition, for example, as an expression corresponding to expression (6.5) described above, the following expression (7.2) can be used.

[0514] Mathematical expression 52

[0515] For example, as an expression corresponding to expression (6.12) described above, the following expression (7.3) can be used.

[0516] Mathematical expression 53

[0517] Expressions (6.7) and (6.9) actually indicate division and as such can be represented as the following expression (7.4).

[0518] Mathematical expression 54

[0519] Expressions (7.1)-(7.4) are practically identical to expressions (6.1), (6.2), (6.5), (6.7), (6.9) and (6.12) described above.

[0520] Similarly, in particular, as an expression corresponding to expressions (6.3) and (6.4) described above, the following expression (7.5) can be used.

[0521] Mathematical expression 55

[0522] In addition, for example, as an expression corresponding to expression (6.6) described above, the following expression (7.6) can be used.

[0523] Mathematical expression 56

[0524] For example, as an expression corresponding to expression (6.13) described above, the following expression (7.7) can be used.

[0525] Mathematical expression 57

[0526] Expressions (6.8) and (6.10) actually indicate division and as such can be represented as the following expression (7.8).

[0527] Mathematical expression 58

[0528] Expressions (7.5)-(7.8) are practically identical to expressions (6.3), (6.4), (6.6), (6.8), (6.10) and (6.13) described above. For example, as an expression corresponding to expression (6.11) described above, the following expression (7.9) can be used. The following expression (7.9) is almost identical to expression (6.11) described above.

[0529] Mathematical expression 59

[0530] In addition, in the above process flow, at least one of the horizontal gradient values and the vertical gradient value is used based on the first sum and the second sum in the above process flow, the process flow of the decoding process is not limited to the above-described process flow. It is also advantageous that whether or not to use horizontal gradient values, vertical gradient values, or both horizontal gradient values and vertical gradient values can be specified by another encoding parameter and the like.

[0531] Independent of the first sum and the second sum, the at least one BIO parameter may be retrieved using horizontal gradient values, vertical gradient values, or both horizontal gradient values and vertical gradient values.

[0532] Regardless of the first sum and the second sum, according to the expressions described above, the decoder 200 is capable of reducing the significant multiplication that requires a large amount of computation in the calculations performed on the corresponding pixel positions, and extracting a plurality of parameters to form a predictive image with a small amount of computation . In particular, although the plus or minus signs in expressions (6.2), (6.4), (6.5), (6.6), (7.1), (7.2), (7.5), (7.6), etc. change, no significant multiplication is used. Accordingly, the number of significant multiplications in BIO processing can be significantly reduced.

[0533] In other words, decoder 200 allows the calculation of sG _x , sG _x dI, sG _y and sG _y dI with a small amount of computation. Accordingly, the decoder 200 allows for a reduction in decoding processing. Specifically, regardless of the first sum and the second sum, the decoder 200 may extract at least one BIO parameter using the horizontal gradient values and the vertical gradient values. Thus, the decoder 200 allows the prediction image to be properly generated using both the horizontal gradient value and the vertical gradient value while reducing the amount of decoding processing.

[0534] Although the decoding process is described in the above description, identical processes included in the decoding process may also be applied in the encoding process. In other words, the decoding in the above description can be read as encoding.

[0535] It should be noted that expressions for calculating the BIO parameter can be replaced by other expressions, provided that other expressions are used to calculate the BIO parameter using either horizontal gradient values or vertical gradient values.

[0536] For example, in the third specific example, expressions for calculating the BIO parameter You are not limited to expressions (6.7), (6.9), (7.4), etc., and other expressions may be provided provided that other expressions are used in order to calculate the BIO parameter u based on horizontal gradient values and without a basis on vertical gradient values. In addition, expressions for calculating v BIO parameter are not limited to expressions (6.8), (6.10), (7.8), etc., and other expressions may be provided provided that other expressions are used to calculate v BIO -parameter based on vertical gradient values and without basis on horizontal gradient values.

[0537] In addition, in the first specific example and the third specific example, the first sum corresponds to the horizontal gradient values, and the second sum corresponds to the vertical gradient values. However, the order may be reversed. In other words, the first sum may correspond to the vertical gradient values, and the second sum may correspond to the horizontal gradient values.

[0538] It should be noted that the computational expressions described herein are examples. Computational expressions may be partially modified, parts of expressions may be deleted, or parts may be added to expressions, provided that the resulting expressions perform similar processing. For example, one or more of the plurality of expressions associated with the first amount, the second amount, or the BIO parameter may be replaced by one or more expressions in other specific examples, or may be replaced by another one or more expressions that are different from the expressions in other specific examples.

[0539] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0540] Fourth BIO Case Study

The BIO parameter calculation processes and prediction image generation processes in the first specific example, the second specific example and the third specific example are examples, and other calculation processes and other generation processes may be applied. For example, the process indicated in the flowchart in FIG. 53 may apply.

[0541] FIG. 53 is a flowchart indicating a fourth specific example of a BIO-based decoding process. In many of the specific examples described above, decoder 200 switches methods for extracting a predictive sample based on the absolute value of the first sum and the absolute value of the second sum, as indicated in FIG. 49 and 52. By comparison, in this particular example, the decoder 200 always calculates the optical flow component in the vertical direction and the horizontal direction to extract a predictive sample. This creates the possibility of further increasing the accuracy of forecasting.

[0542] Specifically, in the example of FIG. 53, similar to the example in FIG. 49 and the example in FIG. 52, the decoder 200 calculates a first sum for the current block using the horizontal gradient value of the first reference block and the horizontal gradient value of the second reference block (S2001). In addition, the decoder 200 calculates a second sum for the current block using the vertical gradient value of the first reference block and the vertical gradient value of the second reference block (S2002).

[0543] In the example of FIG. 53, the decoder 200 then determines the BIO parameter for the current block using both the horizontal gradient values and the vertical gradient values regardless of the absolute value of the first sum and the absolute value of the second sum (S2004). The operation for determining the BIO parameter for the current block using both horizontal gradient values and vertical gradient values may be identical to the operation in the third specific example. For example, the decoder 200 may use expressions (6.1)-(6.10) described in the third specific example as computational expressions for determining the BIO parameter.

[0544] The decoder 200 then decodes the current block using the BIO parameter (S2006). For example, decoder 200 generates a predictive sample using two BIO parameters u and v. At this time, the decoder 200 can extract the prediction sample according to expression (6.11), etc. Decoder 200 may use the expressions described in the first specific example or other expressions. Decoder 200 then decodes the current block using the predictive sample.

[0545] In addition, the sign (x) appearing in expressions (6.2), (6.4), (6.5), (6.6), etc. may be specified by the binary expression (a) described above, or may be specified through the following expression (b).

[0546] Mathematical expression 60

[0547] In expression (a), the sign function returns a value indicating whether the argument provided to the sign function is a plus sign value or a minus sign value. In expression (b), the sign function returns a value indicating whether the argument provided to the sign function is a plus sign value, a minus sign value, or 0.

[0548] In the original expressions for extracting optical flows, sign (G _x [i, j]) and sign (G _y [i, j]) in expressions (6.2), (6.4), (6.5) and (6.6) represent are G _x [i, j] and G _y [i, j], respectively. Accordingly, for example, G _x [i, j] x (I0 _{i, j} -I1 _{i, j} )=0 is calculated as an intermediate value when G _x [i, j]=0 is satisfied and G _y [i, j] x (I0 _{i, j} -I1 _{i, j} )=0 is calculated as an intermediate value when G _y [i, j]=0 is satisfied.

[0549] However, in simplified expressions for extracting optical flows, any corresponding intermediate value can be obtained when sign(0)=1 is satisfied. For example, in expression (6.5), sign (G _x [i, j]), x (I0 _{i, j} -I1 _{i, j} )=(I0 _{i, j} -I1 _{i, j} ) is calculated as an intermediate value when G _x [i, j]=0 is satisfied. For example, in expression (6.6), sign (G _y [i, j]), x (I0 _{i, j} -I1 _{i, j} )=(I0 _{i, j} -I1 _{i, j} ) is calculated as an intermediate value when G _y [i, j]=0 is satisfied. In other words, the intermediate value is not 0, and the value other than 0 remains.

[0550] When determining the sign in expression (b) described above, sign (G _x [i, j]) x (I0 _{i, j} -I1 _{i, j} )=(I0 _{i, j} -I1 _{i, j} ) is calculated as an intermediate value when G _x [i, j]=0 is satisfied, and sign (G _y [i, j]) x (I0 _{i, j} -I1 _{i, j} )=(I0 _{i, j} -I1 _{i, j} ) is calculated as an intermediate value when G _y [i, j]=0 is satisfied. Accordingly, in these cases, identical intermediate values with the original expressions for extracting optical flows are calculated.

[0551] Thus, the sign specified by expression (b) described above has a value closer to the value obtained in the original optical flow expression than the sign specified by expression (a) described above. Accordingly, this creates the possibility of further increasing the accuracy of further increasing the accuracy of prediction.

[0552] It should be noted that the above modifications may be combined in their entirety in the present disclosure. For example, the definition of sign (x) in the first specific example may be replaced by expression (b) described above, or the definition of sign (x) in the third specific example may be replaced by expression (b) described above.

[0553] Although the decoding process is described in the above description, identical processes included in the decoding process may also be applied in the encoding process. In other words, the decoding in the above description can be read as encoding.

[0554] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, for example, a filter having a set of filter coefficients [1 , 0, -1], can be used instead.

[0555] Expression (b) is one example of a sign function that returns a value indicating whether the argument indicates a plus sign, a minus value, or 0. A sign function that provides a value indicating that indicates the argument plus value, minus value, or 0 can be represented by another expression, which can take on a value depending on whether the argument specifies a plus value, minus value, or 0 of numbers of three possible values.

[0556] Fifth BIO Case Study

Next, a fifth specific example of a BIO-based decoding process is described. In the fifth specific example, the optical flow component is always calculated in the vertical direction and horizontal direction such that the prediction sample is calculated in the fourth specific example. The following computational expressions are the computational expressions in the fifth case study.

[0557] Mathematical expression 61

[0558] Mathematical expression 62

[0559] Mathematical expression 63

[0560] Mathematical expression 64

[0561] Mathematical expression 65

[0562] Mathematical expression 66

[0563] Mathematical expression 67

[0564] Mathematical expression 68

[0565] Mathematical expression 69

[0566] Mathematical expression 70

[0567] Expressions (8.1)-(8.6), (8.8) and (8.10) are identical to expressions (6.1)-(6.6), (6.9) and (6.11) in the third specific example. In a specific example, expression (8.7) is added, and expression (6.10) is replaced by expression (8.9). In other words, the BIO parameter "v" in the Y direction is extracted through a calculation process that depends on the BIO parameter "u" in the X direction and the correlation parameter "sG _x G _y " which indicates the correlation between the gradient in the direction along the X-axis and a gradient in the Y-axis direction. This allows the BIO parameter to be extracted with greater accuracy, which increases the likelihood of being able to improve encoding efficiency. It should be noted that sG _x G _y can be represented as a third sum.

[0568] In addition, the expression (b) described in the fourth specific example can be used as the sign (x). As given in expression (b), there is a possibility that it is possible to further improve the encoding efficiency by the sign (x) corresponding to the three values. In addition, the expression (a) described in the first specific example can be used as the sign (x). This simplifies the expression compared to the case in which the sign (x) corresponds to three values, which generates the probability of being able to increase encoding efficiency while reducing processing load.

[0569] It should be noted that the computational expressions described herein are examples. Computational expressions may be partially modified, parts of expressions may be deleted, or parts may be added to expressions, provided that the resulting expressions perform similar processing. For example, one or more of the plurality of expressions associated with a first amount, a second amount, a third amount, or a BIO parameter may be replaced by one or more expressions in other specific examples, or may be replaced by another one or more expressions that are different from the expressions in other specific examples. .

[0570] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0571] The computational expression may not only be applied to the flowchart of FIG. 53 in the fourth specific example, but also to the flowchart in FIG. 49 in the first specific example, to the flowchart in FIG. 52 in the third specific example and to other flowcharts.

[0572] Although the decoding process is described in the above description, identical processes included in the decoding process may also be applied in the encoding process. In other words, the decoding in the above description can be read as encoding.

[0573] Sixth BIO Case Study

Next, a sixth specific example of a BIO-based decoding process is described. In the sixth specific example, the optical flow component is always calculated in the vertical direction and horizontal direction such that the prediction sample is calculated in the fourth specific example, and so on. The following computational expressions are the computational expressions in the sixth case study.

[0574] Mathematical expression 71

[0575] Mathematical expression 72

[0576] Mathematical expression 73

[0577] Mathematical expression 74

[0578] Mathematical expression 75

[0579] Mathematical expression 76

[0580] Mathematical expression 77

[0581] Mathematical expression 78

[0582] Mathematical expression 79

[0583] Expressions (9.1), (9.3) and (9.7)-(9.9) are identical to expressions (6.1), (6.3) and (6.9)-(6.11) in the third specific example. In this particular example, expressions (6.2) and (6.4)-(6.6) are replaced by expressions (9.2) and (9.4)-(9.6), respectively.

[0584] For example, in expressions (6.2) and (6.4)-(6.6), sign (G _x [i, j]) and sign (G _y [i, j]), is used instead of G _x [i, j] and G _y [i, j], and the essential multiplication is removed. On the contrary, in expressions (9.2) and (9.4)-(9.6), sign (G _x [i, j]) and sign (G _y [i, j]) are not used, and the values of G _x [i, j] and G _y [i, j] are directly used. In other words, essential multiplication is used.

[0585] This increases the amount of computational processing. However, this allows the BIO parameter to be extracted with greater accuracy, which increases the likelihood of being able to improve encoding efficiency.

[0586] It should be noted that the computational expressions described herein are examples. Computational expressions may be partially modified, parts of expressions may be deleted, or parts may be added to expressions, provided that the resulting expressions perform similar processing. For example, one or more of the plurality of expressions associated with a first amount, a second amount, a third amount, or a BIO parameter may be replaced by one or more expressions in other specific examples, or may be replaced by another one or more expressions that are different from the expressions in other specific examples. .

[0587] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0588] The computational expression corresponding to this specific example may be applied not only to the flowchart in FIG. 53 in the fourth specific example, but also to the flowchart in FIG. 49 in the first specific example, to the flowchart in FIG. 52 in the third specific example and to other flowcharts.

[0589] Although the decoding process is described in the above description, identical processes included in the decoding process may also be applied in the encoding process. In other words, the decoding in the above description can be read as encoding.

[0590] Seventh BIO Case Study

Next, a seventh specific example of a BIO-based decoding process is described. In the seventh specific example, the optical flow component is always calculated in the vertical direction and horizontal direction such that the prediction sample is calculated in the fourth specific example, and so on. The following computational expressions are the computational expressions in the seventh case study.

[0591] Mathematical expression 80

[0592] Mathematical expression 81

[0593] Mathematical expression 82

[0594] Mathematical expression 83

[0595] Mathematical expression 84

[0596] Mathematical expression 85

[0597] Mathematical expression 86

[0598] Mathematical expression 87

[0599] Mathematical expression 88

[0600] Mathematical expression 89

[0601] Expressions (10.1), (10.3) and (10.5)-(10.10) are identical to expressions (8.1), (8.3) and (8.5)-(8.10) in the fifth specific example. In this particular example, expressions (8.2) and (8.4) are replaced by expressions (10.2) and (10.4), respectively. Specifically, the abs function is used instead of the sign function to convert a plus or minus sign.

[0602] The results obtained by using the abs functions similar to expressions (10.2) and (10.4) are identical to the results obtained by converting the plus or minus functions using the sign functions similar to expressions (8.2) and (8.4) . In other words, this particular function is almost identical to the fifth specific example.

[0603] In the fifth specific example, the product of the plus or minus sign of G _x and G _x is used, and the product of the plus or minus sign of G _y and G _y is used. In this particular example, parts of the product are replaced with absolute values. This allows low volume processing to be achieved.

[0604] For example, the plus or minus sign included in each expression may be changed as necessary, or an expression equivalent to any of the above expressions may be used. In particular, the following expression (11.1)-(11.5) can be used as expressions corresponding to expressions (10.1)-(10.7) described above.

[0605] Mathematical expression 90

[0606] Mathematical expression 91

[0607] Mathematical expression 92

[0608] Mathematical expression 93

[0609] Mathematical expression 94

[0610] Expressions (10.8) and (10.9) actually indicate division and as such can be represented as the following expressions (11.6) and (11.7).

[0611] Mathematical expression 95

[0612] Mathematical expression 96

[0613] It should be noted that the computational expressions described herein are examples. Computational expressions may be partially modified, parts of expressions may be deleted, or parts may be added to expressions, provided that the resulting expressions perform similar processing. For example, one or more of the plurality of expressions associated with a first amount, a second amount, a third amount, or a BIO parameter may be replaced by one or more expressions in other specific examples, or may be replaced by another one or more expressions that are different from the expressions in other specific examples. .

[0614] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0615] The computational expression corresponding to this specific example may be applied not only to the flowchart in FIG. 53 in the fourth specific example, but also to the flowchart in FIG. 49 in the first specific example, to the flowchart in FIG. 52 in the third specific example and to other flowcharts.

[0616] Although the decoding process is described in the above description, identical processes included in the decoding process may also be applied in the encoding process. In other words, the decoding in the above description can be read as encoding.

[0617] Eighth BIO Case Study

Next, an eighth specific example of a BIO-based decoding process is described. In the eighth specific example, the optical flow component is always calculated in the vertical direction and horizontal direction such that the prediction sample is calculated in the fourth specific example, and so on. The following computational expressions are the computational expressions in the eighth case study.

[0618] Mathematical expression 97

[0619] Mathematical expression 98

[0620] Mathematical expression 99

[0621] Mathematical expression 100

[0622] Mathematical expression 101

[0623] Mathematical expression 102

[0624] Mathematical expression 103

[0625] Mathematical expression 104

[0626] Mathematical expression 105

[0627] Mathematical expression 106

[0628] Expressions (12.1), (12.3) and (12.8)-(12.10) are identical to expressions (8.1), (8.3) and (8.8)-(8.10) in the fifth specific example.

[0629] On the other hand, regarding expressions (8.2) and (8.4)-(8.7): values indicating the plus or minus signs of G _x and G _y are always allowed to be equal to 1; and expressions (8.2) and (8.4)-(8.7) are replaced by expressions (12.2) and (12.4)-(12.7), respectively. This is based on the assumption that both the absolute values of the pixel value gradients and the plus or minus signs are constant in the low band Ω.

[0630] Since the plus or minus signs are assumed to be constant, the process of calculating the plus or minus signs of G _x and G _y for each pair of pixels may be reduced. In addition, the computational expression sG _x and the computational expression sG _x G _y are equal to each other. This provides further reduction in processing.

[0631] It should be noted that the computational expressions described herein are examples. Computational expressions may be partially modified, parts of expressions may be deleted, or parts may be added to expressions, provided that the resulting expressions perform similar processing. For example, one or more of the plurality of expressions associated with a first amount, a second amount, a third amount, or a BIO parameter may be replaced by one or more expressions in other specific examples, or may be replaced by another one or more expressions that are different from the expressions in other specific examples. .

[0632] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0633] The computational expression may not only be applied to the flowchart of FIG. 53 in the fourth specific example, but also to the flowchart in FIG. 49 in the first specific example, to the flowchart in FIG. 52 in the third specific example and to other flowcharts.

[0634] Although the decoding process is described in the above description, identical processes included in the decoding process may also be applied in the encoding process. In other words, the decoding in the above description can be read as encoding.

[0635] Ninth BIO Case Study

Next, a ninth specific example of a BIO-based decoding process is described. In the ninth specific example, the optical flow component is always calculated in the vertical direction and horizontal direction such that the prediction sample is calculated in the fourth specific example, and so on. The following computational expressions are the computational expressions in the ninth case study.

[0636] Mathematical expression 107

[0637] Mathematical expression 108

[0638] Mathematical expression 109

[0639] Mathematical expression 110

[0640] Mathematical expression 111

[0641] Mathematical expression 112

[0642] Mathematical expression 113

[0643] Mathematical expression 114

[0644] Mathematical expression 115

[0645] Expressions (13.1)-(13.9) are identical to expressions (12.1)-(12.6) and (12.8)-(12.10) in the eighth specific example. Based on the equality of the computational expression sG _x and the computational expression sG _x G _y in the eighth specific example, the cross-correlation extraction process in the third sum is removed in this specific example. To extract v, sG _x is used.

[0646] It should be noted that the computational expressions described herein are examples. Computational expressions may be partially modified, parts of expressions may be deleted, or parts may be added to expressions, provided that the resulting expressions perform similar processing. For example, one or more of the plurality of expressions associated with the first amount, the second amount, or the BIO parameter may be replaced by one or more expressions in other specific examples, or may be replaced by another one or more expressions that are different from the expressions in other specific examples.

[0647] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0648] The computational expression corresponding to this specific example may be applied not only to the flowchart in FIG. 53 in the fourth specific example, but also to the flowchart in FIG. 49 in the first specific example, to the flowchart in FIG. 52 in the third specific example and to other flowcharts.

[0649] Although the decoding process is described in the above description, identical processes included in the decoding process may also be applied in the encoding process. In other words, the decoding in the above description can be read as encoding.

[0650] Tenth BIO Specific Example

This specific example indicates an example of variation that is applicable to other specific examples. For example, a lookup table could be used for division in each specific example. For example, a 6-bit lookup table having 64 entries (divSigTable) can be used. Expressions (8.8) and (8.9) in the fifth specific example can be replaced by the following expressions (14.1) and (14.2), respectively. Expressions of the same form in other specific examples may be replaced in an identical manner.

[0651] Mathematical expression 116

[0652] Mathematical expression 117

[0653] Upper6digits in expressions (14.1) and (14.2) described above represents the following.

[0654] Mathematical expression 118

[0655] In addition, a gradient image is obtained by calculating the gradient values of a plurality of pixels. The gradient values may be extracted, for example, by applying a gradient filter to a plurality of pixels. Increasing the number of gradient filter taps can improve the accuracy of gradient values, improve the accuracy of the prediction image, and improve coding efficiency.

[0656] On the other hand, since processing is performed for each pair of pixel positions, the amount of calculation increases when the number of pairs of pixel positions to be processed is large. Thus, a two-tap filter can be used as a gradient filter. In other words, the gradient value may be the difference value between two pixels located above or below, or to the right or left of the target pair of pixels for which the gradient value is to be calculated. Alternatively, the gradient value may be the difference value between each of the target pair of pixels for which the gradient value is to be calculated and an adjacent pixel that borders the target pixel (an adjacent pixel is, in particular, a pixel located above or below or to the right or to the left of the target pixel). This may make it possible to reduce the amount of processing compared to the case in which the number of taps is large.

[0657] It should be noted that the plurality of pixels for which gradient values are to be calculated may be a plurality of integer pixels or may include subpixels.

[0658] In addition, although a filter having a set of filter coefficients [-1, 0, 1] is used as an example of a filter for calculating a horizontal gradient value and a vertical gradient value in the above description, a filter having a set of filter coefficients [1, 0 , -1] can be used instead.

[0659] Representative examples of configurations and processing

Below are descriptions of typical examples of configurations and processes performed by the encoder 100 and decoder 200 described above. These representative examples mainly correspond to the fifth specific example and the seventh specific example described above.

[0660] FIG. 54 is a flowchart indicating an operation performed by the encoder 100. For example, the encoder 100 includes circuitry and a memory device connected to the circuitry. The circuit and memory included in the encoder 100 may correspond to the processor a1 and memory a2 illustrated in FIG. 40. The encoder circuit 100 performs the following steps in operation.

[0661] For example, encoder circuit 100 may extract, for each of a plurality of pairs of relative pixel positions, an absolute value of the sum of horizontal gradient values, which indicates the absolute value of the sum of the horizontal gradient value at the relative pixel position in the first range and the horizontal gradient value at the relative pixel position in the second range (S3101).

[0662] Here, the first range includes a first reference block in the current block, and the second range includes a second reference block in the current block. Each of a plurality of pairs of relative pixel positions is defined generally and relative to both the first range and the second range, and each pair of relative pixel positions consists of a pixel position in the first range and a pixel position in the second range.

[0663] In addition, a pixel position determined generally and relatively for both the first range and the second range means a pixel position determined for both the first range and the second range in a relatively identical manner. For example, when one absolute value of the sum of the horizontal gradient values is calculated, the horizontal gradient values at pixel positions that are relatively identical in the first range and the second range are used. More specifically, for example, one absolute value of the sum of the horizontal gradient values is retrieved using the horizontal gradient value at the top-left pixel position in the first range and the horizontal gradient value at the top-left pixel position in the second range.

[0664] The encoder circuit 100 then extracts, as a first parameter, the total sum of the plurality of absolute values of the sums of the horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions (S3102).

[0665] In addition, the encoder circuit 100 extracts, for each of the plurality of pairs of relative pixel positions, an absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range (S3103).

[0666] The encoder circuit 100 then extracts, as a second parameter, the total sum of the plurality of absolute values of the sums of the vertical gradient values extracted, respectively, for the plurality of pairs of relative pixel positions (S3104).

[0667] In addition, the encoder circuit 100 extracts, for each of the plurality of relative pixel position pairs, a pixel difference value that indicates the difference between the pixel value at the relative pixel position in the first range and the pixel value at the relative pixel position in the second range (S3105) . For example, at this time, the encoder circuit 100 extracts a pixel difference value with a plus or minus sign by subtracting one of a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range in each of a plurality of pairs relative pixel positions from another.

[0668] The encoder circuit 100 then inverts or stores the plus or minus sign of the pixel difference value extracted for a pair of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, and extracts the horizontally related pixel difference value (S3106). Here, the sum of the horizontal gradient values at each pair of relative pixel positions indicates the sum of the horizontal gradient value at the relative pixel position in the first range and the horizontal gradient value at the relative pixel position in the second range. A horizontally associated pixel difference value indicates a pixel difference value having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values.

[0669] The encoder circuit 100 then extracts, as a third parameter, the total sum of the plurality of horizontally related pixel difference values extracted, respectively, for the plurality of relative position pairs (S3107).

[0670] In addition, the encoder circuit 100 then inverts or stores the plus or minus sign of the pixel difference value extracted for each pair of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and extracts vertically -associated pixel difference value (S3108). Here, the sum of the vertical gradient values at a pair of relative pixel positions indicates the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range. A vertically associated pixel difference value indicates a pixel difference value having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values.

[0671] The encoder circuit 100 then extracts, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values extracted, respectively, for a plurality of pairs of relative pixel positions (S3109).

[0672] In addition, the encoder circuit 100 inverts or stores the plus or minus sign of the sum of the vertical gradient values extracted for a pair of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and extracts the vertical gradient values. the associated sum of the horizontal gradient values (S3110). Here, the vertically related sum of the horizontal gradient values indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values.

[0673] The encoder circuit 100 then extracts, as a fifth parameter, the total sum of the plurality of vertically related sums of horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions (S3111).

[0674] The encoder circuit 100 then generates a prediction image to be used to encode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter (S3112).

[0675] In this way, significant multiplication, which requires a large amount of computation in the calculation performed for each pixel position, can be reduced, allowing multiple parameters to be extracted to form a predictive image with a small amount of computation. Accordingly, the amount of processing during encoding may be reduced. In addition, the prediction image may be suitably generated based on a plurality of parameters including a parameter associated with horizontal gradient values, a parameter associated with vertical gradient values, and a parameter associated with both horizontal gradient values and vertical gradient values. .

[0676] In addition, for example, the encoder circuit 100 may extract the first parameter according to expression (11.1) described above and the second parameter according to expression (11.2) described above. In addition, for example, the encoder circuit 100 may extract a third parameter according to expression (11.3) described above and a fourth parameter according to expression (11.4) described above. In addition, for example, the encoder circuit 100 may extract the fifth parameter according to expression (11.5) described above.

[0677] Here, Ω denotes a set of relative pixel positions, and [i, j] denotes each of the relative pixel positions. For each of a plurality of pairs of relative pixel positions, I _x ⁰ denotes a horizontal gradient value at a relative pixel position in a first range, and I _x ¹ denotes a horizontal gradient value at a relative pixel position in a second range. For each of a plurality of pairs of relative pixel positions, I _y ⁰ denotes a vertical gradient value at a relative pixel position in a first range, and I _y ¹ denotes a vertical gradient value at a relative pixel position in a second range.

[0678] For each of a plurality of pairs of relative pixel positions, I ⁰ denotes a pixel value at a relative pixel position in a first range, and I ¹ denotes a pixel value at a relative pixel position in a second range. In addition, abs (I _x ¹ +I _x ⁰ ) denotes the absolute value of I _x ¹ +I _x ⁰ , and sign (I _x ¹ +I _x ⁰ ) denotes the plus or minus sign of I _x ¹ +I _x ⁰ . In addition, abs (I _y ¹ +I _y ⁰ ) denotes the absolute value of I _y ¹ +I _y ⁰ , and sign (I _y ¹ +I _y ⁰ ) denotes the plus or minus sign of I _y ¹ +I _y ⁰ .

[0679] Thus, multiple parameters can be extracted with little computation using pixel values, horizontal gradient values, and vertical gradient values.

[0680] In addition, for example, the encoder circuit 100 may extract the sixth parameter by dividing the third parameter by the first parameter. In addition, the encoder circuit 100 may extract the seventh parameter by subtracting the product of the fifth parameter and the sixth parameter from the fourth parameter and dividing the result of the subtraction by the second parameter. The encoder circuit 100 can then generate a prediction image using the sixth parameter and the seventh parameter.

[0681] Thus, the plurality of parameters can be suitably narrowed down to two parameters corresponding to the horizontal direction and the vertical direction. The parameter associated with the horizontal gradient values may be appropriately included in the parameter corresponding to the horizontal direction. A parameter associated with the vertical gradient values, a parameter associated with both the horizontal gradient values and the vertical gradient values, and a parameter associated with the horizontal gradient values may be suitably included in the parameter corresponding to the vertical direction. The prediction image can be suitably generated using two parameters.

[0682] In addition, for example, the encoder circuit 100 may extract the sixth parameter according to expression (10.8) described above. In addition, for example, the encoder circuit 100 may extract the seventh parameter according to expression (10.9) described above.

[0683] Here, sG _x denotes a first parameter, sG _y denotes a second parameter, sG _x dI denotes a third parameter, sG _y dI denotes a fourth parameter, sG _x G _y denotes a fifth parameter, and u denotes a sixth parameter. Bits is a function that returns a value obtained by rounding up the binary logarithm of the argument to an integer.

[0684] In this way, two parameters corresponding to the horizontal direction and the vertical direction can be extracted with a small amount of computation.

[0685] In addition, the encoder circuit 100 may generate a prediction image by extracting a prediction pixel value using: a first pixel value; the first value of the horizontal gradient; the first value of the vertical gradient; second pixel value; second horizontal gradient value; second vertical gradient value; sixth parameter; and the seventh parameter.

[0686] Here, the predicted pixel value is the predicted pixel value at the position of the current pixel included in the current block. The first pixel value is the first pixel value at a first pixel position corresponding to the position of the current pixel in the first reference block. The first horizontal gradient value is the horizontal gradient value at the first pixel position. The first vertical gradient value is the vertical gradient value at the first pixel position. The second pixel value is a pixel value at a second pixel position corresponding to the position of the current pixel in the second reference block. The second horizontal gradient value is the horizontal gradient value at the second pixel position. The second vertical gradient value is the vertical gradient value at the second pixel position.

[0687] This provides a possibility that the prediction image is generated using, for example, two parameters corresponding to the horizontal direction and the vertical direction, and the two parameters corresponding to the horizontal direction and the vertical direction are properly reflected in the prediction image.

[0688] In addition, for example, the encoder circuit 100 may extract the predicted pixel value by dividing by 2 the sum of the first pixel value, the second pixel value, the first correction value, and the second correction value. Here, the first correction value corresponds to the product of the difference between the first horizontal gradient value and the second horizontal gradient value and the sixth parameter, and the second correction value corresponds to the product of the difference between the first vertical gradient value and the second vertical gradient value and the seventh parameter.

[0689] Thus, a prediction image can be suitably generated using, for example, two parameters corresponding to a horizontal direction and a vertical direction.

[0690] In addition, for example, the encoder circuit 100 may extract the predicted pixel value according to expression (10.10) described above. Here, I ⁰ denotes the first pixel value, I ¹ denotes the second pixel value, u denotes the sixth parameter, I _x ⁰ denotes the first horizontal gradient value, I _x ¹ denotes the second horizontal gradient value, v denotes the seventh parameter, I _y ⁰ denotes the first value vertical gradient, and I _y ¹ denotes the second value of the vertical gradient. Thus, the prediction image can be suitably generated according to an expression associated with, for example, two parameters corresponding to a horizontal direction and a vertical direction.

[0691] In addition, for example, the encoder circuit 100 may extract a parameter in the bidirectional optical flow using a first parameter, a second parameter, a third parameter, a fourth parameter, and a fifth parameter, and generate a prediction image using at least one of the parameters in the bidirectional optical flow and bidirectional optical flow. Thus, the encoder 100 allows the prediction image to be properly generated.

[0692] The at least one parameter in the bidirectional optical flow may be one of the sixth parameter and the seventh parameter described above.

[0693] It should be noted that the external prediction module 126 of the encoder 100 can perform the operation described above as a circuit of the encoder 100.

[0694] FIG. 55 is a flowchart indicating an operation performed by the decoder 200. For example, the decoder 200 includes circuitry and a memory device connected to the circuitry. The circuit and memory included in the decoder 200 may correspond to the processor b1 and memory b2 illustrated in FIG. 46. The decoder circuit 200 does the following in operation.

[0695] For example, decoder circuit 200 may calculate, for each of a plurality of pairs of relative pixel positions, an absolute value of the sum of horizontal gradient values, which indicates the absolute value of the sum of the horizontal gradient value at the relative pixel position in the first range and the horizontal gradient value at the relative pixel position in the second range (S3201).

[0696] Here, the first range includes a first reference block in the current block, and the second range includes a second reference block in the current block. Each of a plurality of pairs of relative pixel positions is defined generally and relative to both the first range and the second range, and each pair of relative pixel positions consists of a pixel position in the first range and a pixel position in the second range.

[0697] The decoder circuit 200 then extracts, as a first parameter, the total sum of the plurality of absolute values of the sums of the gradient values extracted, respectively, for the plurality of relative position pairs (S3202).

[0698] In addition, the decoder circuit 200 calculates, for each of the plurality of pairs of relative pixel positions, an absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range (S3203).

[0699] The decoder circuit 200 then extracts, as a second parameter, the total sum of the plurality of absolute values of the sums of the gradient values extracted, respectively, for the plurality of relative position pairs (S3204).

[0700] In addition, the decoder circuit 200 extracts a pixel difference value that is the difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions (S3205) . For example, at this time, the decoder circuit 200 extracts a pixel difference value with a plus or minus sign by subtracting one of a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range in each of a plurality of pairs relative pixel positions from another.

[0701] The decoder circuit 200 then inverts or stores the plus or minus sign of the pixel difference value extracted for a pair of relative pixel positions according to the plus or minus sign of the sum of the horizontal gradient values, and extracts the horizontally related pixel difference value (S3206). Here, the sum of the horizontal gradient values at each pair of relative pixel positions indicates the sum of the horizontal gradient value at the relative pixel position in the first range and the horizontal gradient value at the relative pixel position in the second range. A horizontally associated pixel difference value indicates a pixel difference value having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values.

[0702] The decoder circuit 200 then extracts, as a third parameter, the total sum of the plurality of horizontally related pixel difference values extracted, respectively, for the plurality of relative position pairs (S3207).

[0703] In addition, the decoder circuit 200 then inverts or stores the plus or minus sign of the pixel difference value extracted for each pair of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and extracts vertically -associated pixel difference value (S3208). Here, the sum of the vertical gradient values at a pair of relative pixel positions indicates the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range. A vertically associated pixel difference value is a pixel difference value having a plus or minus sign inverted or stored according to the plus or minus sign of the sum of the vertical gradient values.

[0704] The decoder circuit 200 then extracts, as a fourth parameter, the total sum of the plurality of horizontally related pixel difference values extracted, respectively, for the plurality of relative position pairs (S3209).

[0705] In addition, the decoder circuit 200 inverts or stores the plus or minus sign of the sum of the horizontal gradient values extracted for a pair of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and extracts the vertical gradient values. the associated sum of the horizontal gradient values (S3210). Here, the vertically related sum of the horizontal gradient values indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values.

[0706] The decoder circuit 200 then extracts, as a fifth parameter, the total sum of the plurality of vertically related sums of horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions (S3211).

[0707] The decoder circuit 200 then generates a prediction image to be used to decode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter (S3212).

[0708] In this way, significant multiplication, which requires a large amount of computation in the calculation performed for each pixel position, can be reduced, allowing multiple parameters to be extracted to form a predictive image with a small amount of computation. Accordingly, the processing amount of decoding may be reduced. In addition, the prediction image may be suitably generated based on a plurality of parameters including a parameter associated with horizontal gradient values, a parameter associated with vertical gradient values, and a parameter associated with both horizontal gradient values and vertical gradient values. .

[0709] In addition, for example, the decoder circuit 200 may extract a second parameter according to expression (11.1) described above, and a second parameter according to expression (11.2) described above. In addition, for example, the decoder circuit 200 may extract a third parameter according to expression (11.3) described above, and a fourth parameter according to expression (11.4) described above. In addition, for example, the decoder circuit 200 may extract the fifth parameter according to expression (11.5) described above.

[0710] Here, Ω denotes a set of relative pixel positions, and [i, j] denotes each of the relative pixel positions. For each of a plurality of pairs of relative pixel positions, I _x ⁰ denotes a horizontal gradient value at a relative pixel position in a first range, and I _x ¹ denotes a horizontal gradient value at a relative pixel position in a second range. For each of a plurality of pairs of relative pixel positions, I _y ⁰ denotes a vertical gradient value at a relative pixel position in a first range, and I _y ¹ denotes a vertical gradient value at a relative pixel position in a second range.

[0711] For each of a plurality of pairs of relative pixel positions, I ⁰ denotes a pixel value at a relative pixel position in a first range, and I ¹ denotes a pixel value at a relative pixel position in a second range. In addition, abs (I _x ¹ +I _x ⁰ ) denotes the absolute value of I _x ¹ +I _x ⁰ , and sign (I _x ¹ +I _x ⁰ ) denotes the plus or minus sign of I _x ¹ +I _x ⁰ . In addition, abs (I _y ¹ +I _y ⁰ ) denotes the absolute value of I _y ¹ +I _y ⁰ , and sign (I _y ¹ +I _y ⁰ ) denotes the plus or minus sign of I _y ¹ +I _y ⁰ .

[0712] Thus, multiple parameters can be extracted with little computation using pixel values, horizontal gradient values, and vertical gradient values.

[0713] In addition, for example, the decoder circuit 200 may extract the sixth parameter by dividing the third parameter by the first parameter. In addition, the decoder circuit 200 may extract the seventh parameter by subtracting the product of the fifth parameter and the sixth parameter from the fourth parameter and dividing the subtraction result by the second parameter. The decoder circuit 200 can then generate a prediction image using the sixth parameter and the seventh parameter.

[0714] In this way, a plurality of parameters can be properly integrated into two parameters corresponding to the horizontal direction and the vertical direction. The parameter associated with the horizontal gradient values may be suitably included in the parameter corresponding to the horizontal direction. A parameter associated with the vertical gradient values, a parameter associated with both the horizontal gradient values and the vertical gradient values, and a parameter associated with the horizontal gradient values may be suitably included in the parameter corresponding to the vertical direction. The prediction image can be suitably generated using two parameters.

[0715] In addition, for example, the decoder circuit 200 may extract the sixth parameter according to expression (10.8) described above. In addition, for example, the decoder circuit 200 may extract the seventh parameter according to expression (10.9) described above.

[0716] Here, sG _x denotes a first parameter, sG _y denotes a second parameter, sG _x dI denotes a third parameter, sG _y dI denotes a fourth parameter, sG _x G _y denotes a fifth parameter, and u denotes a sixth parameter. Bits is a function that returns a value obtained by rounding up the binary logarithm of the argument to an integer.

[0717] In this way, two parameters corresponding to the horizontal direction and the vertical direction can be extracted with a small amount of computation.

[0718] In addition, the decoder circuit 200 may generate a prediction image by extracting a prediction pixel value using: a first pixel value; the first value of the horizontal gradient; the first value of the vertical gradient; second pixel value; second horizontal gradient value; second vertical gradient value; sixth parameter; and the seventh parameter.

[0719] Here, the predicted pixel value is the predicted pixel value at the position of the current pixel included in the current block. The first pixel value is the pixel value at the first pixel position corresponding to the position of the current pixel in the first reference block. The first horizontal gradient value is the horizontal gradient value at the first pixel position. The second vertical gradient value is the vertical gradient value at the second pixel position. The second pixel value is a pixel value at a second pixel position corresponding to the position of the current pixel in the second reference block. The second horizontal gradient value is the horizontal gradient value at the second pixel position. The second vertical gradient value is the vertical gradient value at the second pixel position.

[0720] This provides a possibility that the prediction image is generated using, for example, two parameters corresponding to the horizontal direction and the vertical direction, and the two parameters corresponding to the horizontal direction and the vertical direction are properly reflected in the prediction image.

[0721] In addition, for example, the decoder circuit 200 may extract the predicted pixel value by dividing by 2 the sum of the first pixel value, the second pixel value, the first correction value, and the second correction value. Here, the first correction value corresponds to the product of the difference between the first horizontal gradient value and the second horizontal gradient value and the sixth parameter, and the second correction value corresponds to the product of the difference between the first vertical gradient value and the second vertical gradient value and the seventh parameter.

[0722] Thus, a prediction image can be suitably generated using, for example, two parameters corresponding to a horizontal direction and a vertical direction.

[0723] In addition, for example, the decoder circuit 200 may extract the predicted pixel value according to expression (10.10) described above. Here, I ⁰ denotes the first pixel value, I ¹ denotes the second pixel value, u denotes the sixth parameter, I _x ⁰ denotes the first horizontal gradient value, I _x ¹ denotes the second horizontal gradient value, v denotes the seventh parameter, I _y ⁰ denotes the first value vertical gradient, and I _y ¹ denotes the second value of the vertical gradient. Thus, the prediction image can be suitably generated according to an expression associated with, for example, two parameters corresponding to a horizontal direction and a vertical direction.

[0724] In addition, for example, the decoder circuit 200 may extract at least one parameter in the bidirectional optical flow using a first parameter, a second parameter, a third parameter, a fourth parameter, and a fifth parameter, and generate a prediction image using at least one parameter in bidirectional optical flow and bidirectional optical flow. Thus, the decoder 200 allows the prediction image to be properly generated.

[0725] The at least one parameter in the bidirectional optical flow may be one of the sixth parameter and the seventh parameter described above.

[0726] It should be noted that the external prediction module 218 of the decoder 200 can perform the operation described above as a circuit of the decoder 200.

[0727] Other examples

The encoder 100 and decoder 200 in each of the above examples may be used as an image encoder and an image decoder, respectively, or may be used as a video encoder and video decoder, respectively.

[0728] Alternatively, encoder 100 and decoder 200 may each be used as a predictor. In other words, the encoder 100 and decoder 200 can only correspond to the inter-prediction module 126 and the inter-prediction module 218, respectively. Other constituent elements may be included in other devices.

[0729] In addition, at least a portion of each of the examples described above may be used as an encoding method or a decoding method, may be used as a prediction method, or may be used as another method.

[0730] In addition, each constituent element may be configured with dedicated hardware or may be implemented by executing a software program suitable for the constituent element. Each constituent element may be implemented by a program execution unit, such as a CPU, or a processor that reads and executes a software program recorded on a recording medium, such as a hard disk or a semiconductor storage device.

[0731] More specifically, encoder 100 and decoder 200 may each include processing circuitry and a data storage device that is electrically coupled to and accessible from the processing circuitry. For example, the processing circuit corresponds to a processor a1 or b1, and the data storage device corresponds to a storage device a2 or b2.

[0732] The processing circuitry includes at least one of dedicated hardware and a program execution unit and performs processing using a data storage device. In addition, the data storage device stores a program that is executed by the program execution unit when the processing circuit includes the program execution unit.

[0733] Here, the software that implements the encoder 100, decoder 200 and the like described above is the program listed below.

[0734] For example, a program may cause a computer to implement an encoding method including: retrieving an absolute value of a sum of horizontal gradient values that indicates the absolute value of the sum of a horizontal gradient value at a relative pixel position in a first range and a horizontal gradient value at a relative pixel position in a second range range, for each of a plurality of pairs of relative pixel positions in the first range and a relative pixel position in a second range of the plurality of pairs of relative pixel positions, the first range including a first reference block for the current block, the second range including a second reference block for the current block, wherein a plurality of pairs of relative pixel positions are determined generally and relative to the first range and the second range; retrieving, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieving the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieving, as a second parameter, the total sum of a plurality of absolute values of the sums of vertical gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; retrieving, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates a difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverting or storing the plus or minus sign of the pixel difference value extracted for each of the plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieving a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieving, as a third parameter, the total sum of the horizontally related pixel difference values of the set extracted, respectively, for the plurality of pairs of relative pixel positions; inverting or storing a plus or minus sign of the pixel difference value extracted for each of the plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieving a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieving, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values extracted, respectively, for a plurality of pairs of relative pixel positions; inverting or storing the plus or minus sign of the sum of the horizontal gradient values extracted for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieving the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieving, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; and generating a prediction image to be used to encode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0735] For example, the program instructs the computer to perform a decoding method including: retrieving an absolute value of a sum of horizontal gradient values that indicates the absolute value of the sum of a horizontal gradient value at a relative pixel position in a first range and a horizontal gradient value at a relative pixel position in a second range , for each of a plurality of pairs of relative pixel positions in a first range and a relative pixel position in a second range of the plurality of pairs of relative pixel positions, the first range including a first reference block for the current block, the second range including a second reference block for the current block, wherein a plurality of pairs of relative pixel positions are determined generally and relative to the first range and the second range; retrieving, as a first parameter, the total sum of a plurality of absolute values of the sums of horizontal gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieving the absolute value of the sum of the vertical gradient values, which indicates the absolute value of the sum of the vertical gradient value at the relative pixel position in the first range and the vertical gradient value at the relative pixel position in the second range, for each of the plurality of pairs of relative pixel positions; retrieving, as a second parameter, the total sum of a plurality of absolute values of the sums of vertical gradient values extracted, respectively, for a plurality of pairs of relative pixel positions; retrieving, for each of the plurality of pairs of relative pixel positions, a pixel difference value that indicates a difference between a pixel value at a relative pixel position in the first range and a pixel value at a relative pixel position in the second range; inverting or storing the plus or minus sign of the pixel difference value extracted for each of the plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the horizontal gradient values, which indicates the sum of the horizontal gradient value at the relative pixel position in the first range and a horizontal gradient value at a relative pixel position in the second range, and retrieving a horizontally associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the horizontally associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the horizontal gradient values; retrieving, as a third parameter, the total sum of the horizontally related pixel difference values of the set extracted, respectively, for the plurality of pairs of relative pixel positions; inverting or storing a plus or minus sign of the pixel difference value extracted for each of the plurality of pairs of relative pixel positions, according to the plus or minus sign of the sum of the vertical gradient values, which indicates the sum of the vertical gradient value at the relative pixel position in the first range and a vertical gradient value at a relative pixel position in the second range, and retrieving a vertically associated pixel difference value that indicates a pixel difference value to be retrieved for a pair of relative pixel positions, wherein the vertically associated pixel difference value has a plus sign or " minus", inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieving, as a fourth parameter, the total sum of a plurality of vertically related pixel difference values extracted, respectively, for a plurality of pairs of relative pixel positions; inverting or storing the plus or minus sign of the sum of the horizontal gradient values extracted for each of the plurality of pairs of relative pixel positions according to the plus or minus sign of the sum of the vertical gradient values, and retrieving the vertically related sum of the horizontal gradient values, which indicates the sum of the horizontal gradient values having a plus or minus sign, inverted or stored according to the plus or minus sign of the sum of the vertical gradient values; retrieving, as a fifth parameter, the total sum of the vertically related sums of the plurality of horizontal gradient values extracted, respectively, for the plurality of pairs of relative pixel positions; and generating a prediction image to be used to decode the current block using the first parameter, the second parameter, the third parameter, the fourth parameter and the fifth parameter.

[0736] In addition, each constituent element may be a circuit, as described above. The circuits may constitute a circuit as a whole or may be separate circuits. Alternatively, each constituent element may be implemented as a general processor or may be implemented as a dedicated processor.

[0737] In addition, a process that is performed by a particular component element may be performed by another component element. In addition, the processing order may be modified, or multiple processes may be executed in parallel. In addition, the encoder and decoder may include an encoder 100 and a decoder 200.

[0738] In addition, the ordinal numbers, for example, "first" and "second", used for explanation, can be changed as appropriate. A new ordinal may be added to the constituent element, or an ordinal attached to the constituent element may be removed.

[0739] Although aspects of the encoder 100 and decoder 200 are described based on many examples, aspects of the encoder 100 and decoder 200 are not limited to these examples. The scope of aspects of the encoder 100 and decoder 200 may include embodiments obtained by adding, to any of these embodiments, various types of modifications that should be obvious to those skilled in the art without departing from the scope of the present disclosure, and embodiments configurable by combining the constituent elements in various embodiments.

[0740] One or more aspects disclosed herein may be accomplished by combining at least a portion of the other aspects in the present disclosure. In addition, one or more aspects disclosed herein may be performed by combining, with other aspects, portions of the processes indicated in any of the flowcharts of the aspects, portions of the configuration of any of the devices, portions of syntaxes, etc. .

[0741] Implementations and Applications

As described in each of the above-described embodiments, each functional or operating unit can typically be implemented, for example, as an MPU (microprocessor) and a memory device. In addition, processes carried out by each of the functional blocks may be implemented as a program execution unit such as a processor that reads and executes software (program) recorded on a recording medium such as ROM. Software can be distributed. The software may be recorded on a variety of recording media, such as a semiconductor storage device. It should be noted that each functional block can also be implemented as hardware (specialized circuitry). Various combinations of hardware and software may be used.

[0742] In addition, the processing described in each of the embodiments may be implemented through integrated processing using a single device (system) and, alternatively, may be implemented through decentralized processing using multiple devices. Moreover, the processor that executes the above-described program may be a single processor or a plurality of processors. In other words, integrated processing may be performed, and alternatively, decentralized processing may be performed.

[0743] Embodiments of the present disclosure are not limited to the above exemplary embodiments; Various modifications may be made to the exemplary embodiments, the effects of which are also included within the scope of the embodiments of the present disclosure.

[0744] The following describes application examples of the moving frame encoding method (image encoding method) and the moving frame decoding method (image decoding method) described in each of the above embodiments, as well as various systems that implement the example applications. Such a system may be characterized as including an image encoder that uses an image encoding method, an image decoder that uses an image decoding method, and an image codec that includes both an image encoder and an image decoder. Other configurations of such a system can be modified for each individual case.

[0745] Examples of use

Fig. 56 illustrates a general configuration of a content provision system ex100 suitable for implementing a content distribution service. The area in which the communication service is provided is divided into cells of required sizes, and base stations ex106, ex107, ex108, ex109 and ex110, which are fixed wireless stations in the illustrated example, are located in the corresponding cells.

[0746] In the content delivery system ex100, devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 through an Internet service provider ex102 or a communications network ex104 and base stations ex106- ex110. The content delivery system ex100 can combine and connect any combination of the above devices. In various implementations, devices may directly or indirectly communicate with each other through the telephone network or short-range communications rather than through the ex106-ex110 base stations. Additionally, the streaming server ex103 can connect with devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114 and a smartphone ex115, for example, via the Internet ex101. The ex103 streaming server can also connect, for example, to a terminal on a public hotspot on an ex117 aircraft via an ex116 satellite.

[0747] It should be noted that instead of base stations ex106-ex110, wireless access points or public access points may be used. The streaming server ex103 can connect to the communications network ex104 directly rather than via the Internet ex101 or Internet service provider ex102, and can connect to the aircraft ex117 directly rather than via a satellite ex116.

[0748] The camera ex113 is a device capable of capturing still images and video, such as a digital camera. The ex115 smartphone is a smartphone, cellular phone or personal mobile phone system (PHS) phone that can operate according to mobile communication system standards for 2G, 3G, 3.9G and 4G systems, as well as the following 5G system generations.

[0749] Home appliance ex114, for example, is a refrigerator or device included in a home fuel cell cogeneration system.

[0750] In the content delivery system ex100, a terminal including an image and/or video capture function allows, for example, live streaming through a connection to a streaming server ex103, for example, through a base station ex106. When live streaming, a terminal (eg, a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or an airplane terminal ex117) may perform the encoding processing described in the above embodiments for still image content or video content captured by the user through the terminal, can multiplex video data obtained through encoding and audio data obtained through audio encoding corresponding to the video, and can transmit the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to one aspect of the present disclosure.

[0751] Streaming server ex103 streams transmitted content data to clients that request the stream. Examples of the client include a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and airplane terminals ex117 that are capable of decoding the above-described encoded data. Devices that receive streaming data can decode and play back the received data. In other words, the devices may function as an image decoder in accordance with one aspect of the present disclosure.

[0752] Decentralized processing

The ex103 streaming server can be implemented as a plurality of servers or computers that share tasks such as processing, recording, and streaming data. For example, the ex103 streaming server may be implemented as a content delivery network (CDN) that streams content over a network connecting multiple edge servers located around the world. In a CDN, an edge server physically near the client can be dynamically assigned to the client. Content is cached and streamed to the edge server to reduce loading times. In the event of, for example, a certain type of error or connection change, such as due to a spike in traffic, data can be streamed reliably at high speeds because affected parts of the network can be avoided, for example by dividing processing among multiple edge servers or switching streaming loads transfer to another edge server and continue streaming.

[0753] Decentralization is not limited to simply sharing processing for streaming; Encoding of the captured data can be shared between and performed via terminals, on the server side, or both. In one example, in typical encoding, processing is performed in two loops. The first loop serves to detect how complex the image is on a frame-by-frame or scene-by-scene basis, or to detect encoding load. The second circuit provides processing that preserves image quality and improves encoding efficiency. For example, it is possible to reduce the processing load of the terminals and improve the quality and efficiency of content encoding by causing the terminals to perform a first encoding loop and causing the server side that receives the content to perform a second encoding loop. In such a case, when a decoding request is received, the encoded data resulting from the first loop performed by one terminal can be received and reproduced at the other terminal in approximately real time. This allows for smooth, real-time streaming.

[0754] In another example, camera ex113, etc. extracts the feature count (number of features or characteristics) from the image, compresses the data associated with the feature count as metadata, and transmits the compressed metadata to the server. For example, the server determines the significance of an object based on the number of features and changes the quantization precision accordingly to perform compression appropriate for the semantic meaning (or content significance) of the image. These feature counts are particularly effective in improving the accuracy and efficiency of predicting motion vectors during a second pass of compression performed by the server. In addition, coding that has a relatively low processing load, such as variable length coding (VLC), can be processed by the terminal, and coding that has a relatively high processing load, such as context-adaptive binary arithmetic coding (CABAC), can be processed via the server.

[0755] In yet another example, cases arise in which multiple videos of an approximately identical scene are captured through multiple terminals, for example, in a stadium, a shopping mall, or a factory. In such a case, for example, encoding can be decentralized by dividing processing tasks among multiple terminals that capture video and, if necessary, other terminals that do not capture video, and a server on a per-unit basis. The units, for example, can be groups of frames (GOPs), frames, or tiles resulting from dividing a frame. This allows you to reduce loading times and achieve streaming that is closer to real time.

[0756] Since the videos have approximately the same scene, control and/or instructions can be performed by the server such that videos captured through the terminals can be cross-referenced. In addition, the server may receive encoded data from the terminals, change the reference relationship between data elements, or adjust or replace the frames themselves, and then perform encoding. This allows you to generate a stream with increased quality and efficiency for individual data elements.

[0757] In addition, the server may stream video data after performing transcoding to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (for example, VP9), can convert H.264 to H.265, etc.

[0758] Thus, encoding may be performed by a terminal or one or more servers. Accordingly, although the device that performs the encoding is referred to as a “server” or “terminal” in the following description, some or all of the processes performed by the server may be performed by the terminal, and similarly, some or all of the processes performed by the terminal may be performed by the server . This also applies to decoding processes.

[0759] 3D Multi-View Mode

There is an increase in the use of images or videos that are combined from images or videos of different scenes captured in parallel, or an identical scene captured from different angles through multiple terminals, such as the ex113 camera and/or the ex115 smartphone. Video captured by the terminals may be combined, for example, based on a separately obtained relative positional relationship between terminals or regions in the video having matching feature points.

[0760] In addition to encoding two-dimensional moving frames, the server may encode the still image based on scene analysis of the moving frame, either automatically or at a time specified by the user, and transmit the encoded still image to the receiving terminal. In addition, when the server can obtain the relative positional relationship between video capture terminals, in addition to two-dimensional moving frames, the server can generate three-dimensional scene geometry based on video of an identical scene captured from different angles. The server may separately encode 3D data generated from, for example, a point cloud, and, based on the result of recognizing or tracking a person or object using the 3D data, may select or reconstruct and generate videos to be transmitted to the receiving terminal from the videos captured through multiple terminals.

[0761] This allows the user to enjoy the scene by freely selecting videos corresponding to the video capture terminals, and allows the user to enjoy the content obtained by extracting video at a selected viewpoint from 3D data recovered from a plurality of images or video, video. Additionally, similar to video, audio can be recorded from relatively different angles, and the server can multiplex the audio from a particular angle or space with the corresponding video and transmit the multiplexed video and audio.

[0762] In recent years, content that is a composite of the real world and the virtual world, such as virtual reality (VR) and augmented reality (AR) style content, has also become popular. In the case of VR images, the server may create images from the left and right eye viewpoints and perform encoding that allows a link between the two viewpoint images, such as multi-view encoding (MVC), and alternatively, may encode the images as separate streams without a link . When images are decoded as separate streams, the streams can be synchronized during playback in such a way as to recreate the virtual 3D space according to the user's point of view.

[0763] In the case of AR images, the server may overlay virtual object information existing in virtual space with camera information representing real world space based on three-dimensional position or movement from the user's perspective. The decoder can receive or store virtual object information and 3D data, generate 2D images based on movement from the user's point of view, and then generate overlay data by transparently connecting the images. Alternatively, the decoder may transmit, to the server, motion from the user's point of view in addition to a request for virtual object information. The server may generate overlay data based on three-dimensional data stored in the server according to the received motion, and encode and stream the generated overlay data to the decoder. It should be noted that the overlay data typically includes, in addition to RGB values, an α value indicating transparency, and the server sets the α value for sections other than the object formed from the 3D data, for example, to 0, and may perform the encoding while these sections are transparent. Alternatively, the server may set the background to a specific RGB value, such as a chroma key, and generate data in which areas other than the object are set as the background. A specific RGB value can be predefined.

[0764] Decoding of similarly streamed data can be performed on the client (eg, terminals), on the server side, or shared between them. In one example, one terminal may transmit a receive request to a server, the requested content may be received and decoded by the other terminal, and the decoded signal may be transmitted to a device having a display. It is possible to reproduce data with high image quality by decentralizing processing and appropriately selecting content regardless of the processing characteristics of the communication terminal itself. In yet another example, while a television, for example, receives image data that is large in size, an area of a frame, such as a tile, obtained by dividing the frame can be decoded and displayed on the personal terminal or terminals of the viewer or viewers of the television. This allows viewers to share a view with large frames, as well as each viewer to check their assigned area or explore an area in more detail with a close-up view.

[0765] In situations in which multiple wireless connections are possible over short, medium and long distances, indoors or outdoors, it may be possible to transparently receive content using a streaming system standard such as MPEG DASH. The user can switch between real-time data by freely selecting a decoder or display device including a user terminal, indoor or outdoor displays, etc. In addition, using, for example, information regarding the user's position, decoding can be performed when switching which terminal processes decoding and which terminal processes content display. This allows information to be converted and displayed while the user is on the move on a route to a destination, on the wall of a nearby building into which a device capable of displaying the content is embedded, or on a piece of ground. In addition, it is also possible to switch the bit rate of the received data based on the reachability of the encoded data over the network, for example, when the encoded data is cached on a server readily accessible from the receiving terminal, or when the encoded data is copied to an edge server in a content delivery service.

[0766] Scalable Coding

In the following, content switching will be described with reference to the scalable stream illustrated in FIG. 57, which is compression encoded through an implementation of the moving frame encoding method described in the above embodiments. The server may be configured in which content is switched using temporal and/or spatial stream scalability, which is achieved through partitioning and layer encoding, as illustrated in FIG. 57. It should be noted that there may be multiple separate streams that have identical content but different quality. In other words, by determining which layer to decode based on internal factors such as processing characteristics of the decoder side and external factors such as communication link bandwidth, the decoder side can freely switch between low-resolution content and high-resolution content when decoding. For example, in a case in which the user wants to continue watching, for example, at home on a device such as an Internet-connected TV, a video that the user was previously watching on the ex115 smartphone on the move, the device can simply decode the identical stream down to a different layer, which reduces the load on the server side.

[0767] Additionally, in addition to the configuration described above in which scalability is achieved by encoding frames per layer with an enhancement layer above the base layer, the enhancement layer may include metadata, such as based on statistical information relative to the image. The decoder side can generate high image quality content by performing super-resolution imaging on a frame in the base layer based on the metadata. Super-resolution imaging can improve the SN ratio while maintaining resolution and/or increasing resolution. Metadata includes information to identify the linear or nonlinear filtering coefficient used in super-resolution processing, or information identifying the value of a parameter in filter, machine learning, or least squares processing used in super-resolution processing.

[0768] Alternatively, a configuration may be provided in which the frame is divided, for example, into tiles, for example, according to the semantic meaning of an object in the image. On the decoder side, only a partial region is decoded by selecting a tile to decode. Additionally, by storing the attribute of the object (person, car, ball, etc.) and the position of the object in the video (coordinates in identical images) as metadata, the decoder side can identify the position of the desired object based on the metadata and determine which tile or tiles include this object. For example, as illustrated in FIG. 58, metadata may be stored using a data storage structure other than pixel data, such as an SEI (Additional Enhancement Information) message in HEVC. This metadata indicates, for example, the position, size or color of the underlying object.

[0769] Metadata may be stored in multiple frame units, such as stream, sequence, or random access units. The decoder side can obtain, for example, the time at which a particular person appears in the video, and by matching the time information to the frame unit information, can identify a frame in which an object is present, and can determine the position of the object in the frame.

[0770] Web page optimization

Fig. 59, for example, illustrates an example of a web page display screen on a computer ex111. Fig. 60, for example, illustrates an example of a web page display screen on a smartphone ex115. As illustrated in FIG. 59 and fig. 60, a web page may include a plurality of image links that are links to image content, and the appearance of the web page may differ depending on the device used to view the web page. When multiple image links are viewable on the screen until the user explicitly selects the image link, either until the image link is approximated in the center of the screen, or the entire image link fits into the screen, display device (decoder) may display, as image links, still images included in the content, or I-frames, may display video, such as an animated GIF using multiple still images or I-frames, for example, or may only receive base layer and decode and display the video.

[0771] When an image reference is selected by the user, the display device performs decoding, for example, by providing the highest priority to the base layer. It should be noted that if there is information in the HTML code of the web page indicating that the content is scalable, the display device can decode down to the enhancement layer. Additionally, to ensure real-time playback, before a selection is made, or when bandwidth is severely limited, the display device may reduce the delay between the time at which the leading frame is decoded and the time at which the decoded frame is displayed (i.e., e. the delay from the start of decoding the content until the content is displayed) by decoding and displaying only direct reference frames (I-frame, P-frame, forward reference B-frame). Still further, the display device may deliberately ignore the reference relationship between frames and approximately decode all B and P frames as direct reference frames and then perform normal decoding as the number of frames received in time increases.

[0772] Autonomous driving

When transmitting and receiving still image or video data such as two- or three-dimensional map information for autonomous or assisted driving, the receiving terminal may receive, in addition to image data belonging to one or more layers, information such as regarding weather or roadworks as metadata, and associate the metadata with image data when decoding. It should be noted that the metadata may be assigned on a per-layer basis and, alternatively, may simply be multiplexed with the image data.

[0773] In such a case, since the vehicle, unmanned vehicle, aircraft, etc. containing the receiving terminal is mobile, the receiving terminal can transparently receive and perform decoding when switching between base stations among the base stations ex106 to ex110 by transmitting information , indicating the position of the receiving terminal. In addition, according to the selection made by the user, the user's situation and/or the connection bandwidth, the receiving terminal can dynamically select the extent to which metadata is received or the extent to which, for example, map information is updated.

[0774] In the content delivery system ex100, the client can receive, decode and reproduce, in real time, encoded information transmitted by the user.

[0775] Streaming content from people

In the content delivery system ex100, in addition to long-lasting content with high image quality distributed by a video distribution entity, unicast or multicast streaming of short content with low image quality from a person is also possible. The popularity of such content from people is likely to increase even more. The server may first perform editing processing on the content before encoding processing to refine content from people. This can be achieved, for example, using the following configuration.

[0776] In real time, when video or image content is captured or after the content is captured and accumulated, the server performs recognition processing based on the raw data or encoded data, such as capture error handling, scene retrieval processing, semantic meaning analysis, and /or object detection processing. Thereafter, based on the result of the recognition processing, the server (either directed or automatically) edits the content, examples of which include: correction such as correction of focus and/or motion blur; Removing scenes with low priority, such as scenes that have low brightness compared to other frames or are out of focus; adjusting the edges of objects; and regulation of color tones. The server encodes the edited data based on the result of the edit. It is known that overly long videos tend to have fewer views. Accordingly, in order to keep content within a specific duration that scales with the duration of the source video, the server can, in addition to the low priority scenes described above, automatically cut scenes with minor movement based on the result of image processing. Alternatively, the server can generate and encode a video digest based on the result of the analysis of the semantic meaning of the scene.

[0777] There may be cases in which content from people may include content that violates copyright, moral law, book rights, etc. Such a case may lead to an unfavorable situation for the creator, for example, when the content is shared beyond the scope intended by the creator. Accordingly, before encoding, the server, for example, can edit the images so as to, for example, blur the faces of people on the periphery of the screen or blur the inside of a house. Additionally, the server may be configured to recognize faces of people other than the registered person in images to be encoded, and when such faces appear in the image, may apply a mosaic filter to, for example, the person's face. Alternatively, as pre-processing or post-processing for encoding, the user may specify, for copyright reasons, an image area including a person or a background area to be processed. The server may process the specified area, for example, by replacing the area with another image or blurring the area. If the area includes a person, the person may be tracked in a moving frame, and the user's head region may be replaced with another image as the person moves.

[0778] Since there is a demand for real-time viewing of human-generated content, which often has a small data size, the decoder may first take the base layer as the highest priority and perform decoding and playback, although this may differ depending on the bandwidth. When the content is played back two or more times, for example, when the decoder receives the enhancement layer during decoding and playback of the base layer and loops playback, the decoder can play back the high image quality video including the enhancement layer. If a stream is encoded using such scalable encoding, the video may be low quality in the unselected state or at the beginning of the video, but it may offer an experience in which the quality of the images of the stream is gradually increased in an intelligent manner. This is not limited to just scalable coding; An identical experience can be offered by configuring one stream of the low quality stream played back the first time and a second stream encoded using the first stream as a reference.

[0779] Other Examples of Implementations and Applications

Encoding and decoding may be performed by an LSI ex500 (Large Integrated Circuit) (see FIG. 56), which is typically included in each terminal. The LSI ex500 can be configured with a single chip or multiple chips. Software for encoding and decoding moving frames may be integrated into some type of recording medium (such as a CD-ROM, floppy disk, or hard disk) that is readable, for example, by a computer ex111, and encoding and decoding may be performed using the software. In addition, when the ex114 smartphone is equipped with a camera, video data captured by the camera can be transmitted. In this case, video data can be encoded using the LSI ex500 included in the ex115 smartphone.

[0780] It should be noted that the LSI ex500 may be configured to download and activate an application. In such a case, the terminal first determines whether or not it is compatible with the scheme used to encode the content, or whether or not it is capable of performing a particular service. When the terminal is not compatible with the content encoding scheme, or when the terminal does not allow execution of a particular service, the terminal may first download the codec or application software, and then receive and play the content.

[0781] In addition to the example of a content provision system ex100 that uses the Internet ex101, at least a moving frame encoder (picture encoder) or a moving frame decoder (picture decoder) described in the above embodiments may be implemented in a digital broadcast system. Identical encoding processing and decoding processing can be used to transmit and receive broadcast radio waves overlaid with multiplexed audio and video data using, for example, a satellite, even though it accommodates multicast transmission, while unicast transmission is simpler with the ex100 system providing content.

[0782] Hardware configuration

Fig. 61 illustrates additional information regarding the smartphone ex115 shown in FIG. 56. Fig. 62 illustrates an ex115 smartphone configuration example. The ex115 smartphone includes an ex450 antenna for transmitting and receiving radio waves to/from the ex110 base station, an ex465 camera that allows video and still image capture, and an ex458 display that displays decoded data such as video captured through the ex465 camera and videos received via antenna ex450. The ex115 smartphone further includes an ex466 user interface such as a touch panel, an ex457 audio output module such as a speaker for outputting speech or other audio, an ex456 audio input module such as a microphone for audio input, an ex467 storage device capable of storing decoded data such as captured video or still images, recorded audio, received video or still images and mail messages, as well as decoded data, and the ex464 socket, which is an interface for the ex468 SIM card to authorize access to the network and various data. It should be noted that an external storage device can be used instead of the ex467 storage device.

[0783] Main controller ex460, which can comprehensively control the display ex458 and user interface ex466, power supply circuit ex461, user interface input controller ex462, video signal processor ex455, camera interface ex463, display controller ex459, modulator/demodulator ex452, multiplexer/demultiplexer ex453, ex454 audio processor, ex464 socket and ex467 memory are connected via ex470 bus.

[0784] When the user turns on the power button of the power supply circuit ex461, the smartphone ex115 is turned on to an operational state and each component is supplied with power from the battery pack.

[0785] The ex115 smartphone performs processing, for example, for calls and data transfers, based on control performed by the ex460 main controller, which includes a CPU, ROM and RAM. When calls are made, the audio signal recorded by the audio input module ex456 is converted to a digital audio signal by the audio signal processor ex454, which is applied with spread spectrum processing by the ex452 modulator/demodulator and applied with digital-to-analog conversion and frequency conversion processing by the ex451 transceiver. and the resulting signal is transmitted through the ex450 antenna. The received data is amplified, frequency-converted and A/D converted, despread spectrum processed by the ex452 modulator/demodulator, converted to an analog audio signal by the ex454 audio processor, and then output from the ex457 audio output module. In the data transmission mode, text, still image data or video data may be transmitted, for example, under the control of the main controller ex460 through the input controller ex462 from the user interface based on the operation of the main body user interface ex466. Similar transmission and reception processing is performed. In the data transfer mode, when sending video, still image or video and audio, the video signal processor ex455 compressively encodes, through the moving frame encoding method described in the above embodiments, the video signal stored in the storage device ex467 or the video signal input from the camera ex465 , and transmits the encoded video data to the ex453 multiplexer/demultiplexer. The ex454 audio processor encodes the audio signal recorded by the ex456 audio input module, while the ex465 camera captures video or still image, and transmits the encoded audio data to the ex453 multiplexer/demultiplexer. The ex453 multiplexer/demultiplexer multiplexes encoded video data and encoded audio data using a specific circuit, modulates and converts the data using the ex452 modulator/demodulator (modulator/demodulator circuit) and ex451 transceiver and transmits the result through the ex450 antenna. A specific schema can be predefined.

[0786] When a video attachment in an email message or chat, or a video containing a link from a web page, is received, for example, to decode multiplexed data received through antenna ex450, the ex453 multiplexer/demultiplexer demultiplexes the multiplexed data to separate multiplexes data onto a video bitstream and an audio bitstream, provides encoded video data to the ex455 video processor via the ex470 synchronous bus, and provides encoded audio data to the ex454 audio processor via the ex470 synchronous bus. The video signal processor ex455 decodes the video signal using a moving frame decoding method corresponding to the moving frame encoding method described in the above embodiments, and the video or still image included in the associated moving frame file is displayed on the display ex458 through the display controller ex459. The audio signal processor ex454 decodes the audio signal and outputs audio from the audio output module ex457. As live streaming becomes more popular, there may be cases in which audio playback may not be socially feasible, depending on the user's environment. Accordingly, as an initial value, a configuration in which only video data is reproduced, i.e., may be preferred. no audio signal is played; audio can only sync and play when input is received, such as when the user clicks on video data.

[0787] Although the smartphone ex115 is used in the above example, other implementations are possible: a transceiver terminal including both an encoder and a decoder; a transmission terminal including only an encoder; and a receiving terminal including only a decoder. In the description of a digital broadcast system, an example is given in which multiplexed data obtained by multiplexing video data with audio data is received or transmitted. However, the multiplexed data may be video data multiplexed with data other than audio data, such as text data associated with the video. Additionally, direct video data rather than multiplexed data may be received or transmitted.

[0788] Although the ex460 main controller, which includes a CPU, is described as controlling encoding or decoding processes, various terminals often include GPUs. Accordingly, it is acceptable to have a configuration in which a large area is processed at once by using performance characteristics of the GPU through a memory shared by the CPU and the GPU, or through a memory including an address that is controlled so as to enable a wide use via CPU and GPU. This reduces encoding time, maintains the real-time nature of the stream, and reduces latency. In particular, processing related to motion estimation, deblocking filtering, sampled adaptive offset (SAO), and transform/quantization can be efficiently performed by the GPU instead of the CPU, for example, in units of frames, simultaneously.

Industrial applicability

[0789] The present disclosure is applicable, for example, to television receivers, digital video recorders, automobile navigation systems, mobile phones, digital cameras, digital video cameras, teleconferencing systems, electronic mirrors, etc.

List of reference numbers

[0790] 100 - encoder

102 - partitioning module

104 - subtractor

106 - converter

108 - quantizer

110 - entropy encoder

112, 204 - inverse quantizer

114, 206 - inverse converter

116, 208 - adder

118, 210 - block storage device

120, 212 - contour filter

122, 214 - frame memory

124, 216 - internal forecasting module

126, 218 - external forecasting module

126a, a2, b2 - storage device

126b - module for extracting interpolated images

126c - gradient image extraction module

126d - optical flow extraction module

126e - correction value extraction module

126f - predictive image corrector

128, 220 - forecasting controller

200 - decoder

202 - entropy decoder

1201 - boundary detection module

1202, 1204, 1206 - switch

1203 - filter detection module

1205 - filtering module

1207 - module for determining filter characteristics

1208 - processing definition module

a1, b1 - processor

Claims (21)

Encoder containing Memory device; a processor coupled to the storage device and configured to generate a predictive image to encode a current block based on a first parameter, a second parameter, a third parameter, a fourth parameter and a fifth parameter that are extracted in a bidirectional optical flow (BDOF) extraction process, wherein: the first parameter is retrieved based on the following expression: ; the second parameter is retrieved based on the following expression: ; the third parameter is retrieved based on the following expression: ; the fourth parameter is retrieved based on the following expression: ; the fifth parameter is extracted based on the following expression, , where Ω indicates a set of relative pixel positions; [i, j] indicates a relative pixel position in the set Ω, specified by the horizontal position i and the vertical position j; I _x ⁰ indicates a horizontal gradient value at a first pixel position in the first gradient image, and I _x ¹ indicates a horizontal gradient value at a first pixel position in the second gradient image, wherein the first pixel position is determined based on the relative pixel position [i, j] of the first a gradient image and a second gradient image corresponding to the current block; I _y ⁰ indicates a vertical gradient value at a first pixel position in the first gradient image, and I _y ¹ indicates a vertical gradient value at a first pixel position in the second gradient image; I ⁰ indicates a pixel value at the first pixel position in the first interpolated image corresponding to the current block; I ¹ indicates a pixel value at a first pixel position in the second interpolated image corresponding to the current block; the abs function prints the absolute value of the argument; The sign function prints the sign of the argument, which is -1, 0, or 1.

Images