RU2817290C2 - Video encoding system and method - Google Patents

Abstract

FIELD: video and images encoding and decoding.

SUBSTANCE: invention relates to video and image encoding/decoding, particularly for performing a cross-component adaptive loop filtering (CCALF) process. Encoder includes a circuit and a memory device connected to the circuit. Circuit, in response to the first reconstructed sample of images located outside the virtual boundary, duplicates the reconstructed sample located inside and near the virtual boundary in order to form the first reconstructed image sample. Circuit generates the first coefficient value by applying the CCALF process to the first reconstructed sample of images of the luminance signal component. Circuit generates a second coefficient value by applying an adaptive loop filtering (ALF) process to a second recovered chroma component image sample. Circuit generates third coefficient value by summation of first coefficient value with second coefficient value and codes the third reconstructed sample of the chroma component images using the third coefficient value.

EFFECT: high efficiency of encoding/decoding.

10 cl, 164 dwg

Description

Field of technology to which the invention relates

[0001] This disclosure relates to video encoding and, in particular, to video encoding and decoding systems, components and methods for encoding and decoding video, for example, to perform a CCALF (Cross Component Adaptive Contour Filtering) process.

State of the art

[0002] Due to improvements in video encoding technology, from (H.261 and MPEG-1 to (H.264/AVC (Advanced Video Coding Standard), MPEG-LA, (H.265/HEVC (High Efficiency Video Coding Standard) and (H.266/VVC (Versatile Video Codec), there remains an ongoing need to provide improvements and optimizations to video encoding technology in order to handle the ever-increasing volume of digital video data in a variety of applications. This disclosure relates to additional improvements improvements and optimizations in video encoding, in particular in the CCALF (Cross Component Adaptive Contour Filtering) process.

The essence of the invention

[0003] According to one aspect, an encoder is provided that includes circuitry and a memory device coupled to the circuitry. The circuit, in response to the first reconstructed image sample located outside the virtual boundary, duplicates the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample. The circuit generates a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance component images. The circuit generates a second coefficient value by applying an ALF (adaptive loop filtering) process to a second reconstructed sample of chroma component images. The circuit generates a third coefficient value by adding the first coefficient value with a second coefficient value, and encodes a third reconstructed sample of chrominance component images using the third coefficient value.

[0004] According to a further aspect, the first reconstructed image sample is located adjacent to the second reconstructed image sample.

[0005] According to another aspect, the circuit, in operation: sets a first coefficient value equal to zero in response to a first coefficient value less than 64.

[0006] According to another aspect, an encoder is provided that includes: a block splitter that, in operation, splits a first image into a plurality of blocks; an intra prediction module that, when operated, predicts blocks included in the first image using reference blocks included in the first image; an inter-prediction module that, in operation, predicts blocks included in the first image using reference blocks included in a second image different from the first image; a contour filter which, in operation, filters the blocks included in the first image; a converter that, in operation, converts the prediction error between the original signal and the prediction signal generated by the intra prediction module or the inter prediction module to generate conversion coefficients; a quantizer that, in operation, quantizes the transform coefficients to generate quantized coefficients; and an entropy encoder that, in operation, variably encodes the quantized coefficients to generate a coded bitstream including the encoded quantized coefficients and control information. The loop filter does the following:

- in response to the first reconstructed image sample located outside the virtual boundary, duplicating the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample;

- generating a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images;

- generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images;

- generating a third coefficient value by summing the first coefficient value with the second coefficient value, and

- encoding the third reconstructed sample of chrominance component images using the third coefficient value.

[0007] According to a further aspect, a decoder is provided that includes circuitry and a memory device coupled to the circuitry. The circuit, in response to the first reconstructed image sample located outside the virtual boundary, duplicates the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample. The circuit generates a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance component images. The circuit generates a second coefficient value by applying an ALF (adaptive loop filtering) process to a second reconstructed sample of chroma component images. The circuit generates a third coefficient value by adding the first coefficient value with the second coefficient value, and decodes a third reconstructed sample of chrominance component images using the third coefficient value.

[0008] According to another aspect, decoding equipment is provided that includes: a decoder that, in operation, decodes a coded bit stream to output quantized coefficients; an inverse quantizer, which, when operated, inversely quantizes the quantized coefficients to output transform coefficients; an inverse converter that, in operation, inversely converts the conversion coefficients to output a prediction error; an intra prediction module that, when operated, predicts blocks included in the first image using reference blocks included in the first image; an inter-prediction module that, in operation, predicts blocks included in the first image using reference blocks included in a second image different from the first image; a contour filter which, in operation, filters the blocks included in the first image; and an output which, when operative, outputs a frame including the first image. The loop filter does the following:

- in response to the first reconstructed image sample located outside the virtual boundary, duplicating the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample;

- generating a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images;

- generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images;

- generating a third coefficient value by summing the first coefficient value with the second coefficient value, and

- decoding the third reconstructed sample of chrominance component images using the third coefficient value.

[0009] According to another aspect, an encoding method is provided that includes:

- in response to the first reconstructed image sample located outside the virtual boundary, duplicating the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample;

- generating a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images;

- generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images;

- generating a third coefficient value by summing the first coefficient value with the second coefficient value, and

- encoding the third reconstructed sample of chrominance component images using the third coefficient value.

[0010] According to a further aspect, a decoding method is provided that includes:

- in response to the first reconstructed image sample located outside the virtual boundary, duplicating the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample;

- generating a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images;

- generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images;

- generating a third coefficient value by summing the first coefficient value with the second coefficient value, and

- decoding the third reconstructed sample of chrominance component images using the third coefficient value.

[0011] In video encoding technology, it is desirable to propose new methods in order to improve encoding efficiency, improve image quality, and reduce circuit size. Certain implementations of embodiments of the present disclosure, including constituent elements of embodiments of the present disclosure, considered separately or in various combinations, may facilitate one or more of the following: increasing encoding efficiency, increasing image quality, reducing the utilization of processing resources associated with encoding/decoding, reducing the size of circuits, increasing the processing speed of encoding/decoding, etc.

[0012] In addition, some implementations of embodiments of the present disclosure, including constituent elements of embodiments of the present disclosure, considered separately or in various combinations, may facilitate, during encoding and decoding, the appropriate selection of one or more elements, such as a filter , block, size, motion vector, reference frame, reference block or operation. It should be noted that the present disclosure includes disclosure regarding configurations and methods that may provide advantages other than those described above. Examples of such configurations and methods include a configuration or method for increasing encoding efficiency while reducing the increase in processing resource usage.

[0013] Additional benefits and advantages of the disclosed embodiments will become apparent from the detailed description and drawings. Benefits and/or advantages may be separately obtained through various embodiments and features of the detailed description and drawings, not all of which must necessarily be provided in order to obtain one or more such benefits and/or advantages.

[0014] It should be noted that general or specific embodiments may be implemented as a system, method, integrated circuit, computer program, storage medium, or any selective combination of the foregoing.

Brief description of drawings

[0015] FIG. 1 is a circuit diagram illustrating one example of a functional configuration of a transmission system according to an embodiment.

Fig. 2 is a conceptual diagram to illustrate one example of a hierarchical data structure in a stream.

Fig. 3 is a conceptual diagram for illustrating one example of a slice configuration.

Fig. 4 is a conceptual diagram for illustrating one example of a tile configuration.

Fig. 5 is a conceptual diagram for illustrating one example of a coding structure in scalable coding.

Fig. 6 is a conceptual diagram for illustrating one example of a coding structure in scalable coding.

Fig. 7 is a block diagram illustrating a functional configuration of an encoder according to the embodiment.

Fig. 8 is a functional block diagram illustrating an installation example of the encoder.

Fig. 9 is a flowchart indicating one example of a complete decoding process performed by the encoder.

Fig. 10 is a conceptual diagram for illustrating one example of block partitioning.

Fig. 11 is a block diagram illustrating one example of a functional configuration of a partitioning module according to the embodiment.

Fig. 12 is a conceptual diagram for illustrating examples of partitioning patterns.

Fig. 13A is a conceptual diagram for illustrating one example of a split pattern syntax tree.

Fig. 13B is a conceptual diagram for illustrating another example of a split pattern syntax tree.

Fig. 14 is a diagram indicating exemplary transformation basis functions for various types of transformation.

Fig. 15 is a conceptual diagram for illustrating exemplary spatially varying transforms (SVT).

Fig. 16 is a flowchart illustrating one example of a process performed by the converter.

Fig. 17 is a flowchart illustrating another example of a process performed by the converter.

Fig. 18 is a block diagram illustrating one example of a functional configuration of a quantizer according to the embodiment.

Fig. 19 is a flowchart illustrating one example of a quantization process performed by a quantizer.

Fig. 20 is a block diagram illustrating one example of a functional configuration of an entropy encoder according to an embodiment.

Fig. 21 is a conceptual diagram for illustrating an exemplary flow of a context adaptive binary arithmetic coding (CABAC) process in an entropy encoder.

Fig. 22 is a block diagram illustrating one example of a functional configuration of a loop filter according to the embodiment.

Fig. 23A is a conceptual diagram illustrating one example of a filter shape used in an adaptive loop filter (ALF).

Fig. 23B is a conceptual diagram for illustrating another example of a filter shape used in the ALF.

Fig. 23C is a conceptual diagram for illustrating another example of a filter shape used in the ALF.

Fig. 23D is a conceptual diagram for illustrating an exemplary cross-component ALF (CCALF) process flow.

Fig. 23E is a conceptual diagram for illustrating an example of a filter shape used in CCALF.

Fig. 23F is a conceptual diagram for illustrating an exemplary JC-CCALF process flow.

Fig. 23G is a table illustrating exemplary possible weighting indices that may be used in JC-CCALF.

Fig. 24 is a block diagram indicating one example of a specific configuration of a loop filter that functions as a deblocking filter (DBF).

Fig. 25 is a conceptual diagram for illustrating an example of a deblocking filter having a symmetrical filtering characteristic with respect to a block boundary.

Fig. 26 is a conceptual diagram for illustrating a block boundary on which a filtering process to remove blocking is performed.

Fig. 27 is a conceptual diagram for illustrating examples of boundary intensity values (Bs).

Fig. 28 is a flowchart illustrating one example of a process performed by the encoder prediction module.

Fig. 29 is a flowchart illustrating another example of a process performed by the encoder prediction module.

Fig. 30 is a flowchart illustrating another example of a process performed by the encoder prediction module.

Fig. 31 is a conceptual diagram for illustrating sixty-seven intra prediction modes used in intra prediction in the embodiment.

Fig. 32 is a flowchart illustrating one example of a process performed by the intra prediction module.

Fig. 33 is a conceptual diagram for illustrating examples of reference frames.

Fig. 34 is a conceptual diagram for illustrating examples of reference frame lists.

Fig. 35 is a flowchart illustrating an exemplary basic flow of inter-prediction processing.

Fig. 36 is a flowchart illustrating one example of a motion vector extraction process.

Fig. 37 is a flowchart illustrating another example of a motion vector extraction process.

Fig. 38A is a conceptual diagram for illustrating exemplary mode characteristics for MV extraction.

Fig. 38B is a conceptual diagram for illustrating exemplary mode characteristics for MV extraction.

Fig. 39 is a flowchart illustrating an example of an inter-prediction process in a normal inter-mode.

Fig. 40 is a flowchart illustrating an example of an inter-prediction process in the normal combining mode.

Fig. 41 is a conceptual diagram for illustrating one example of a motion vector extraction process in a fusion mode.

Fig. 42 is a conceptual diagram for illustrating one example of an MV extraction process for a current frame by HMVP combining mode.

Fig. 43 is a flowchart illustrating one example of a movie frame rate upconversion (FRUC) process.

Fig. 44 is a conceptual diagram for illustrating one example of pattern matching (bilateral matching) between two blocks along a motion path.

Fig. 45 is a conceptual diagram for illustrating one example of pattern matching (pattern matching) between a pattern in the current frame and a block in the reference frame.

Fig. 46A is a conceptual diagram for illustrating one example of extracting the motion vector of each sub-block based on the motion vectors of a plurality of neighboring blocks.

Fig. 46B is a conceptual diagram for illustrating one example of extracting the motion vector of each sub-block in an affine mode in which three control points are used.

Fig. 47A is a conceptual diagram for illustrating exemplary MV extraction at control points in affine mode.

Fig. 47B is a conceptual diagram for illustrating an example MV extraction at control points in affine mode.

Fig. 47C is a conceptual diagram for illustrating an example MV extraction at control points in affine mode.

Fig. 48A is a conceptual diagram for illustrating an affine mode in which two control points are used.

Fig. 48B is a conceptual diagram for illustrating an affine mode that uses three control points.

Fig. 49A is a conceptual diagram for illustrating one example of a method for MV extraction at control points when the number of control points for a coded block and the number of control points for a current block are different from each other.

Fig. 49B is a conceptual diagram for illustrating another example of a method for MV extraction at control points when the number of control points for a coded block and the number of control points for a current block are different from each other.

Fig. 50 is a flowchart illustrating one example of a process in the affine combining mode.

Fig. 51 is a flowchart illustrating one example of an affine reciprocal mode process.

Fig. 52A is a conceptual diagram for illustrating the generation of two triangular projection images.

Fig. 52B is a conceptual diagram for illustrating examples of a first portion of a first segment that overlaps with a second segment, and first and second sets of samples that may be weighted as part of the correction process.

Fig. 52C is a conceptual diagram for illustrating a first part of a first segment, which constitutes a part of the first segment that overlaps with a part of an adjacent segment.

Fig. 53 is a flowchart illustrating one example of a triangular mode process.

Fig. 54 is a conceptual diagram for illustrating one example of an advanced temporal motion vector prediction (ATMVP) mode in which the MV is extracted in units of subblocks.

Fig. 55 is a flowchart illustrating the relationship between the combining mode and dynamic motion vector updating (DMVR).

Fig. 56 is a conceptual diagram to illustrate one example of a DMVR.

Fig. 57 is a conceptual diagram for illustrating another example of a DMVR for determining MV.

Fig. 58A is a conceptual diagram for illustrating one example of motion estimation in DMVR.

Fig. 58B is a flowchart illustrating one example of a motion estimation process in a DMVR.

Fig. 59 is a flowchart illustrating one example of a predictive image generation process.

Fig. 60 is a flowchart illustrating another example of a predictive image generation process.

Fig. 61 is a flowchart illustrating one example of a prediction image correction process by overlapping block motion compensation (OBMC).

Fig. 62 is a conceptual diagram for illustrating one example of a process for correcting prediction images by OBMC.

Fig. 63 is a conceptual diagram to illustrate the model under the condition of uniform linear motion.

Fig. 64 is a flowchart illustrating one example of an inter-prediction process according to BIO.

Fig. 65 is a functional block diagram illustrating one example of a functional configuration of an inter-prediction module that can perform inter-prediction according to BIO.

Fig. 66A is a conceptual diagram for illustrating one example process for a method for generating predictive images using a brightness correction process performed by an LIC.

Fig. 66B is a flowchart illustrating one example process for a predictive image generation method using LIC.

Fig. 67 is a block diagram illustrating a functional configuration of a decoder according to an embodiment.

Fig. 68 is a functional block diagram illustrating an installation example of a decoder.

Fig. 69 is a flowchart illustrating one example of a complete decoding process performed by the decoder.

Fig. 70 is a conceptual diagram for illustrating the relationship between the partition determination module and other constituent elements.

Fig. 71 is a block diagram illustrating one example of a functional configuration of an entropy decoder.

Fig. 72 is a conceptual diagram for illustrating an exemplary flow of the CABAC process in an entropy decoder.

Fig. 73 is a block diagram illustrating one example of a functional configuration of an inverse quantizer.

Fig. 74 is a flowchart illustrating one example of an inverse quantization process performed by an inverse quantizer.

Fig. 75 is a flowchart illustrating one example of a process performed by an inverter.

Fig. 76 is a flowchart illustrating another example of a process performed by an inverter.

Fig. 77 is a block diagram illustrating one example of a functional configuration of a loop filter.

Fig. 78 is a flowchart illustrating one example of a process performed by the decoder prediction module.

Fig. 79 is a flowchart illustrating another example of a process performed by the decoder prediction module.

Fig. 80A is a flowchart illustrating another example of a process performed by the decoder prediction module.

Fig. 80B is a flowchart illustrating another example of a process performed by the decoder prediction module.

Fig. 80C is a flowchart illustrating another example of a process performed by the decoder prediction module.

Fig. 81 is a diagram illustrating one example of a process performed by the decoder intra prediction unit.

Fig. 82 is a flowchart illustrating one example of an MV extraction process in a decoder.

Fig. 83 is a flowchart illustrating another example of an MV extraction process in a decoder.

Fig. 84 is a flowchart illustrating an example of the inter-prediction process by the normal inter-mode in the decoder.

Fig. 85 is a flowchart illustrating an example of an inter-prediction process by means of a normal combining mode in a decoder.

Fig. 86 is a flowchart illustrating an example of an inter-prediction process by FRUC mode in a decoder.

Fig. 87 is a flowchart illustrating an example of an inter-prediction process via an affine combining mode in a decoder.

Fig. 88 is a flowchart illustrating an example of a mutual prediction process by affine mutual mode in a decoder.

Fig. 89 is a flowchart illustrating an example of an inter-prediction process by triangular mode in a decoder.

Fig. 90 is a flowchart illustrating an example of a motion estimation process by DMVR in a decoder.

Fig. 91 is a flowchart illustrating one exemplary motion estimation process by a DMVR in a decoder.

Fig. 92 is a flowchart illustrating one example of a prediction image generation process in a decoder.

Fig. 93 is a flowchart illustrating another example of a prediction image generation process in a decoder.

Fig. 94 is a flowchart illustrating an example of a prediction image correction process by OBMC in a decoder.

Fig. 95 is a flowchart illustrating an example of a prediction image correction process by BIO in a decoder.

Fig. 96 is a flowchart illustrating an example of a prediction image correction process by an LIC in a decoder.

Fig. 97 is a flowchart for an exemplary processing flow for decoding an image applying a CCALF (Cross Component Adaptive Contour Filtering) process according to the first aspect.

Fig. 98 is a block diagram illustrating a functional configuration of an encoder and decoder according to an embodiment.

Fig. 99 is a block diagram illustrating a functional configuration of an encoder and decoder according to an embodiment.

Fig. 100 is a block diagram illustrating a functional configuration of an encoder and decoder according to an embodiment.

Fig. 101 is a block diagram illustrating a functional configuration of an encoder and decoder according to an embodiment.

Fig. 102 is a flowchart for an exemplary processing flow for decoding an image applying the CCALF process according to the second aspect.

Fig. 103 illustrates select locations of cut parameters that must be parsed, for example, from a VPS, APS, SPS, PPS, slice header, CTU, or TU bitstream.

Fig. 104 illustrates examples of clipping parameters.

Fig. 105 is a flowchart for an exemplary processing flow for decoding an image applying a CCALF process using a filter coefficient according to the third aspect.

Fig. 106 is a conceptual diagram of an example indicating locations of filter coefficients to be used in the CCALF process.

Fig. 107 is a conceptual diagram of an example indicating locations of filter coefficients to be used in the CCALF process.

Fig. 108 is a conceptual diagram of an example indicating locations of filter coefficients to be used in the CCALF process.

Fig. 109 is a conceptual diagram of an example indicating locations of filter coefficients to be used in the CCALF process.

Fig. 110 is a conceptual diagram of an example indicating locations of filter coefficients to be used in the CCALF process.

Fig. 111 is a conceptual diagram of an additional example indicating locations of filter coefficients to be used in the CCALF process.

Fig. 112 is a conceptual diagram of an additional example indicating locations of filter coefficients to be used in the CCALF process.

Fig. 113 is a block diagram illustrating the functional configuration of a CCALF process executed by an encoder and decoder according to an embodiment.

Fig. 114 is a flowchart for an exemplary processing flow for decoding an image applying a CCALF process using a filter selected from a plurality of filters according to the fourth aspect.

Fig. 115 illustrates an example filter selection processing flow.

Fig. 116 illustrates examples of filters.

Fig. 117 illustrates examples of filters.

Fig. 118 is a flowchart for an exemplary processing flow for decoding an image applying a CCALF process using a parameter according to the fifth aspect.

Fig. 119 illustrates examples of the number of coefficients that must be parsed from the bitstream.

Fig. 120 is a flowchart for an exemplary processing flow for decoding an image applying a CCALF process using a parameter according to the sixth aspect.

Fig. 121 is a conceptual diagram illustrating an example of generating a luma component CCALF value for a current chrominance signal sample by calculating a weighted average of neighboring samples.

Fig. 122 is a conceptual diagram illustrating an example of generating a luma component CCALF value for a current chroma sample by calculating a weighted average of neighboring samples.

Fig. 123 is a conceptual diagram illustrating an example of generating a luma component CCALF value for a current chroma sample by calculating a weighted average of neighboring samples.

Fig. 124 is a conceptual diagram illustrating an example of generating a luma component CCALF value for a current sample by calculating a weighted average of adjacent samples, wherein the locations of adjacent samples are determined adaptively with respect to the type of chrominance signals.

Fig. 125 is a conceptual diagram illustrating an example of generating a luma component CCALF value for a current sample by calculating a weighted average of neighboring samples, wherein the locations of neighboring samples are determined adaptively with respect to the type of chrominance signals.

Fig. 126 is a conceptual diagram illustrating an example of generating a CCALF value of a luminance signal component by applying a bit shift to a weighted calculation output value.

Fig. 127 is a conceptual diagram illustrating an example of generating a CCALF value of a luminance signal component by applying a bit shift to a weighted calculation output value.

Fig. 128 is a flowchart for an exemplary processing flow for decoding an image applying a CCALF process using a parameter according to the seventh aspect.

Fig. 129 illustrates sample locations of one or more parameters that are to be parsed from the bitstream.

Fig. 130 shows exemplary processes for retrieving one or more parameters.

Fig. 131 shows sample values of the second parameter.

Fig. 132 shows an example of parsing the second parameter using arithmetic encoding.

Fig. 133 is a conceptual diagram of a variation of this embodiment applied to rectangular segments and non-rectangular segments such as triangular segments.

Fig. 134 is a flowchart for an exemplary processing flow for decoding an image applying a CCALF process using a parameter according to eight aspects.

Fig. 135 is a flowchart for an exemplary processing flow for decoding an image applying a CCALF process using a parameter according to the eighth aspect.

Fig. 136 shows exemplary locations of chroma sample types 0-5.

Fig. 137 is a conceptual diagram illustrating selective symmetrical complement.

Fig. 138 is a conceptual diagram illustrating selective symmetrical complement.

Fig. 139 is a conceptual diagram illustrating selective symmetrical complement.

Fig. 140 is a conceptual diagram illustrating selective non-symmetrical padding.

Fig. 141 is a conceptual diagram illustrating selective non-symmetrical padding.

Fig. 142 is a conceptual diagram illustrating selective non-symmetrical padding.

Fig. 143 is a conceptual diagram illustrating selective non-symmetrical padding.

Fig. 144 is a conceptual diagram illustrating additional selective non-symmetrical padding.

Fig. 145 is a conceptual diagram illustrating additional selective non-symmetrical padding.

Fig. 146 is a conceptual diagram illustrating additional selective non-symmetrical padding.

Fig. 147 is a conceptual diagram illustrating additional selective non-symmetrical padding.

Fig. 148 is a conceptual diagram illustrating additional selective symmetrical addition.

Fig. 149 is a conceptual diagram illustrating additional selective symmetrical addition.

Fig. 150 is a conceptual diagram illustrating additional selective symmetrical addition.

Fig. 151 is a conceptual diagram illustrating additional selective non-symmetrical padding.

Fig. 152 is a conceptual diagram illustrating additional selective non-symmetrical padding.

Fig. 153 is a conceptual diagram illustrating additional selective non-symmetrical padding.

Fig. 154 is a conceptual diagram illustrating additional selective non-symmetrical padding.

Fig. 155 illustrates additional examples of padding with a horizontal and vertical virtual border.

Fig. 156 is a block diagram illustrating a functional configuration of an encoder and decoder according to an example in which symmetric padding is used at virtual boundary locations in virtual boundary locations for ALF, and symmetric or non-symmetric padding is used at virtual boundary locations for CCALF.

Fig. 157 is a block diagram illustrating the functional configuration of an encoder and decoder according to an example in which other symmetrical padding is used in virtual boundary locations for ALF, and one-sided padding is used in virtual boundary locations for CCALF.

Fig. 158 is a conceptual diagram illustrating an example of one-sided padding with a horizontal or vertical virtual border.

Fig. 159 is a conceptual diagram illustrating an example of one-way padding with a horizontal and vertical virtual border.

Fig. 160 is a diagram illustrating an exemplary general configuration of a content delivery system for implementing a content distribution service.

Fig. 161 is a conceptual diagram illustrating an example of a web page display screen.

Fig. 162 is a conceptual diagram illustrating an example of a web page display screen.

Fig. 163 is a block diagram illustrating one example of a smartphone.

Fig. 164 is a block diagram illustrating an example of a functional configuration of a smartphone.

Detailed Description of Embodiments

[0016] In the drawings, identical reference numbers identify like elements unless the context indicates otherwise. Dimensions and relative positions of elements in drawings are not necessarily drawn to scale.

[0017] Embodiments will be described hereinafter with reference to the drawings. It should be noted that the embodiments described below show a general or specific example. The numerical values, shapes, materials, components, arrangement and connection of components, steps, relationship and order of steps, etc., indicated in the following embodiments are merely examples and are therefore not intended to limit the scope of the claims.

[0018] Embodiments of the encoder and decoder are described below. The embodiments are examples of an encoder and decoder to which the processes and/or configurations presented in the description of aspects of the present disclosure are applicable. The processes and/or configurations may also be implemented in an encoder and decoder different from the encoder and decoder of the embodiments. For example, with respect to the processes and/or configurations applied to the embodiments, any of the following may be implemented:

[0019] (1) Any of the components of the encoder or decoder according to the embodiments presented in the description of aspects of this disclosure may be replaced or combined with another component presented anywhere in the description of aspects of this disclosure.

[0020] (2) In the encoder or decoder according to the embodiments, discretionary changes may be made to functions or processes performed by one or more components of the encoder or decoder, such as adding, replacing, deleting, etc. functions or processes. For example, any function or process may be substituted for or combined with another function or process presented anywhere in the description of aspects of this disclosure.

[0021] (3) In the methods implemented by the encoder or decoder according to the embodiments, discretionary changes such as adding, replacing and deleting one or more processes included in the method may be made. For example, any process in the method may be replaced by or combined with another process presented anywhere in the description of aspects of the present disclosure.

[0022] (4) One or more components included in an encoder or decoder according to embodiments may be combined with a component presented anywhere in the description of aspects of the present disclosure may be combined with a component including one or more functions presented anywhere in the description of aspects of the present disclosure, and may be combined with a component that implements one or more of the processes implemented by the component presented in the description of aspects of the present disclosure.

[0023] (5) A component that includes one or more encoder or decoder functions according to embodiments, or a component that implements one or more encoder or decoder processes according to embodiments, may be combined or replaced by a component presented anywhere in the description aspects of the present disclosure, a component that includes one or more of the functions presented anywhere in the description of aspects of the present disclosure, or a component that implements one or more processes presented anywhere in the description of aspects of the present disclosure.

[0024] (6) In methods implemented by an encoder or decoder according to embodiments, any of the processes included in the method may be replaced or combined with a process presented anywhere in the description of aspects of this disclosure, or with any corresponding or equivalent process.

[0025] (7) One or more processes included in the methods implemented by the encoder or decoder according to the embodiments may be combined with a process presented anywhere in the description of aspects of the present disclosure.

[0026] (8) The implementation of the processes and/or configurations presented in the description of aspects of the present disclosure is not limited to the encoder or decoder according to the embodiments. For example, the processes and/or configurations may be implemented in an apparatus used for a purpose other than the moving frame encoder or moving frame decoder disclosed in the embodiments.

[0027] Definitions of terms

The corresponding terms may be specified as below as examples.

[0028] An image is a unit of data configured with a set of pixels, is a frame, or includes blocks smaller than a pixel. Images include still image in addition to video.

[0029] A frame is an image processing unit configured with a set of pixels, and may also be referred to as a "frame" or "field". A frame, for example, may take the form of an array of luma samples in monochrome, or an array of luma samples and two corresponding arrays of chrominance samples in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0030] A block is a processing unit that is a collection of a specific number of pixels. Blocks can have any number of different shapes. For example, a block may have a rectangular shape of MxN (M columns by N rows) pixels, a square shape of MxM pixels, a triangular shape, a circular shape, etc. Examples of blocks include slices, tiles, bricks, CTUs, superblocks, basic partitioning units, VPDUs, hardware partitioning units, CUs, processing block units, prediction block units (PUs), orthogonal transform block units (TUs) , units and subunits. The block may take the form of an MxN array of samples or an MxN array of transform coefficients. For example, a block may be a square or rectangular region of pixels including one luma matrix and two chroma matrices.

[0031] A pixel or sample is the smallest point in an image. The pixels or samples include a pixel at an integer position as well as pixels at sub-pixel positions, such as those generated from the pixel at an integer position.

[0032] A pixel value or sample value is a pixel's own value. The pixel values or sample values may include one or more of a luma value, a chrominance value, an RGB gradation level, a depth value, binary values of zero or 1, etc.

[0033] The chrominance signal or chrominance represents color intensity, typically represented by the symbols Cb and Cr, which indicate that the sample array values or one sample value represent the values of one of two chrominance signals associated with the primary colors.

[0034] A luminance signal or luminance is the luminance of an image, typically represented by a symbol or subscript Y or L, which indicates that the sample array values or one sample value represent monochrome signal values associated with the primary colors.

[0035] The flag contains one or more bits that indicate the value of, for example, a parameter or an index. The flag may be a binary flag, which indicates the binary value of the flag, which may also indicate the non-binary value of the parameter.

[0036] The signal conveys information that is converted into symbol form by or encoded into a signal. Signals include discrete digital signals and continuous analog signals.

[0037] A stream or bit stream is a string of digital data of a digital data stream. The stream or bitstream may be a single stream or may be configured with multiple streams having multiple hierarchical layers. The stream or stream of bits may be transmitted in serial communication using a single transmission path, or may be transmitted in packet switched communication using multiple transmission paths.

[0038] Difference means various mathematical differences, for example, simple difference (x-y), absolute value of difference (|x-y|), squared difference (x^2-y^2), square root difference (√(x-y)), weighted difference (ax-by: a and b are constants), offset difference (x-y+a: a is offset), etc. In the case of a scalar quantity, a simple difference may be sufficient and a difference calculation may be enabled.

[0039] Sum means various mathematical sums, for example, the simple sum (x+y), the absolute value of the sum (|x+y|), the squared sum (x^2+y^2), the square root of the sum (√ (x+y)), weighted difference (ax+by: a and b are constants), offset sum (x+y+a: a is offset), etc. In the case of a scalar quantity, a simple sum may be sufficient and the calculation of sums may be involved.

[0040] A frame is a composition of a top field and a bottom field, with lines 0, 2, 4, ..., samples being initiated from the top field, and lines 1, 3, 5, ..., samples being initiated from the bottom field.

[0041] A slice is an integer number of coding tree units contained in one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) in an identical access unit.

[0042] A tile is a rectangular area of coding tree blocks in a particular column of tiles and a particular row of tiles in a frame. A tile can be a rectangular area of a movie frame that should be able to be decoded and encoded independently, although edge filtering can still be applied to the edges of the tiles.

[0043] A coding tree unit (CTU) may be a frame luminance sample coding tree block that has three sample arrays, or two corresponding chrominance sample coding tree blocks. Alternatively, the CTU may be a sample encoding tree block of one of a monochrome frame and a frame that is encoded using three separate color planes and syntax structures used to encode the samples. A superblock may be a 64x64 pixel square block that consists of either 1 or 2 mode information blocks, or is recursively segmented into four 32x32 blocks, which themselves can be further segmented.

[0044] System Configuration

First, the transmission system according to the embodiment is described. Fig. 1 is a circuit diagram illustrating one example configuration of transmission system 400 according to an embodiment.

[0045] The transmission system 400 is a system that transmits a stream generated by image encoding and decodes the transmitted stream. As illustrated, transmission system 400 includes an encoder 100, a network 300, and a decoder 200, as illustrated in FIG. 1.

[0046] An image is input to encoder 100. Encoder 100 generates a stream by encoding the input image and outputs the stream to network 300. The stream includes, for example, encoded image information and control information for decoding the encoded image. The image is compressed through encoding.

[0047] It should be noted that the image before encoding by the encoder 100 is also referred to as the “original image,” “original signal,” or “original sample.” The image may be a video or a still image. The image represents the general principle of sequence, frame and block and is therefore not limited to a spatial region having a specific size and a temporal region having a specific size unless otherwise specified. An image is an array of pixels or pixel values, and the signal representing the image or pixel values is also called "samples". The stream may be referred to as a "bitstream", "coded bitstream", "compressed bitstream" or "coded signal". In addition, the encoder 100 may be referred to as an "image encoder" or a "video encoder". The encoding method performed by the encoder 100 may be called "encoding method", "image encoding method" or "video encoding method".

[0048] Network 300 transmits the stream generated by encoder 100 to decoder 200. Network 200 may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination of networks. Network 300 is not limited to a bidirectional communication network and may be a unidirectional communication network that transmits digital terrestrial broadcast, satellite broadcast, and the like broadcast waves. Alternatively, the network 300 may be replaced by a recording medium such as a digital versatile disc (DVD) and a Blu-ray disc (BD), etc., on which the stream is recorded.

[0049] The decoder 200 generates, for example, a decoded image, which is an uncompressed image, by decoding a stream transmitted via the network 300. For example, the decoder decodes the stream according to a decoding method corresponding to the encoding method used by the encoder 100.

[0050] It should be noted that the decoder 200 may also be referred to as an "image decoder" or a "video decoder", and that the decoding method performed by the decoder 200 may also be referred to as a "decoding method", "image decoding method" or "video decoding method" .

[0051] Data structure

Fig. 2 is a conceptual diagram to illustrate one example of a hierarchical data structure in a stream. For convenience, FIG. 2 is described with reference to the transmission system 400 of FIG. 1. The stream includes, for example, a video sequence. As illustrated in (a) of FIG. 2, a video sequence includes one or more video parameter sets (VPS), one or more sequence parameter sets (SPS), one or more frame parameter sets (PPS), additional enhancement information (SEI), and a plurality of frames.

[0052] In a video having multiple layers, the VPS may include an encoding parameter that is common among some of the plurality of layers, and an encoding parameter associated with some of the multiple layers included in the video or with a particular layer.

[0053] SPS includes a parameter that is used for the sequence, i.e. an encoding parameter that is accessed by decoder 200 in order to decode the sequence. For example, the encoding parameter may indicate the width or height of the frame. It should be noted that multiple SPS may be present.

[0054] PPS includes a parameter that is used for the frame, i.e. an encoding parameter that is accessed by the decoder 200 in order to decode each of the frames in the sequence. For example, the encoding parameter may include a reference value for the quantization width that is used to decode the frame, and a flag indicating the use of weighted prediction. It should be noted that multiple PPS may be present. Each of the SPS and PPS may be referred to simply as a "parameter set".

[0055] As illustrated in (b) of FIG. 2, a frame may include a frame header and one or more slices. The frame header includes an encoding parameter that is accessed by the decoder 200 in order to decode one or more slices.

[0056] As illustrated in (c) of FIG. 2, a slice includes a slice header and one or more bricks. The slice header includes an encoding parameter that is accessed by the decoder 200 in order to decode one or more bricks.

[0057] As illustrated in (d) of FIG. 2, a brick includes one or more coding tree units (CTUs).

[0058] It should be noted that the frame may not include a slice and may include a group of tiles instead of a slice. In this case, the tile group includes at least one tile. In addition, the brick may include a cut.

[0059] A CTU is also called a "superblock" or "basic partitioning unit". As illustrated in (e) of FIG. 2, the CTU includes a CTU header and at least one coding unit (CU). As illustrated, the CTU includes four encoding units CU(10), CU(11), CU(12), and CU(13). The CTU header includes an encoding parameter that is accessed by the decoder 200 in order to decode at least one CU.

[0060] A CU may be broken down into multiple smaller CUs. As shown, CU(10) is not broken down into smaller coding units; CU(11) is broken down into four smaller coding units CU(110), CU(111), CU(112) and CU(113); CU(12) is not broken down into smaller coding units; and CU(13) is broken down into seven smaller encoding units CU(1310), CU(1311), CU(1312), CU(1313), CU(132), CU(133) and CU(134), as illustrated in ( f) according to FIG. 2, the CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting the CU, and the residual coefficient information is information indicating the prediction residual, which is described below. Although a CU is essentially identical to a prediction unit (PU) and a transformation unit (TU), it should be noted that, for example, a sub-block transform (SBT), which is described below, may include multiple TUs smaller than the CU. In addition, the CU may be processed for each virtual pipelined decoding unit (VPDU) included in the CU. A VPDU, for example, is a fixed unit that can be processed once when pipelined in hardware.

[0061] It should be noted that the stream may not include all of the hierarchical layers illustrated in FIG. 2. The order of the hierarchical layers can be changed, or any of the hierarchical layers can be replaced by another hierarchical layer. Here, a frame that represents a target for a process to be performed by a device such as an encoder 100 or a decoder 200 is called a “current frame.” The current frame means the current frame to be encoded when the process is an encoding process, and the current frame means the current frame to be decoded when the process is a decoding process. Likewise, for example, a CU or block CU that represents a target for a process to be performed by a device such as encoder 100 or decoder 200 is called a “current block.” The current block means the current block to be encoded when the process is an encoding process, and the current block means the current block to be decoded when the process is a decoding process.

[0062] Frame structure: slice/tile

A frame may be configured with one or more slice units or one or more tile units to facilitate parallel frame encoding/decoding.

[0063] Slices are basic coding units included in a frame. A frame may include, for example, one or more slices. In addition, a slice includes one or more coding tree units (CTUs).

[0064] FIG. 3 is a conceptual diagram for illustrating one example of a slice configuration. For example, in FIG. 3, the frame includes an 11x8 CTU and is divided into four slices (slices 1-4). Slice 1 includes sixteen CTUs, slice 2 includes twenty-one CTUs, slice 3 includes twenty-nine CTUs, and slice 4 includes twenty-two CTUs. Here, each CTU in the frame belongs to one of the slices. The shape of each slice is the shape obtained by splitting the frame horizontally. The boundary of each slice does not have to coincide with the end of the image and can coincide with any of the boundaries between CTUs in the image. The processing order of CTUs in a slice (encoding order or decoding order), for example, is the raster scan order. A slice includes a slice header and encoded data. Slice attributes can be written to the slice header. Features may include the CTU address of the top CTU in the slice, the slice type, etc.

[0065] A tile is a unit of rectangular area included in a frame. Frame tiles can be assigned a number called "TileId" in raster scan order.

[0066] FIG. 4 is a conceptual diagram for illustrating one example of a tile configuration. For example, in FIG. 4, the frame includes an 11x8 CTU and is divided into four rectangular region tiles (tiles 1-4). When tiles are used, the CTU processing order may be different from the processing order when tiles are not used. When tiles are not in use, the multiple CTUs in a frame are generally processed in raster scan order. When multiple tiles are used, at least one CTU in each of the plurality of tiles is processed in raster scan order. For example, as illustrated in FIG. 4, the order of processing CTUs included in tile 1 is from the left end of the first column of tile 1 to the right end of the first column of tile 1, and then continues from the left end of the second column of tile 1 to the right end of the second column of tile 1.

[0067] It should be noted that one tile may include one or more slices, and one slice may include one or more tiles.

[0068] It should be noted that a frame may be configured with one or more tile sets. A set of tiles may include one or more groups of tiles or one or more tiles. A frame can be configured with one of a tile set, a tile group, and a tile. For example, the order for scanning multiple tiles for each set of tiles in raster scan order is presumably the basic order for encoding the tiles. A set of one or more tiles that are contiguous in the basic encoding order within each tile set is assumed to constitute a group of tiles. Such a frame may be configured by a splitter 102 (see FIG. 7), which is described below.

[0069] Scalable Coding

Fig. 5 and 6 are conceptual diagrams illustrating examples of scalable flow structures and are described for convenience with reference to FIGS. 1.

[0070] As illustrated in FIG. 5, encoder 100 may generate a temporally/spatially scalable stream by dividing each of a plurality of frames into any of a plurality of layers and encoding the frame within the layer. For example, encoder 100 encodes a frame for each layer, thereby achieving scalability in which an enhancement layer is present above the base layer. This encoding of each frame is also called "scalable encoding". Thus, the decoder 200 allows switching of image quality for an image that is displayed by decoding the stream. In other words, the decoder 200 can determine which layer to decode based on internal factors such as the processing characteristics of the decoder 200 and external factors such as the bandwidth state of the communication links. As a result, decoder 200 allows content to be decoded while freely switching between low resolution and high resolution. For example, a stream user watches a video stream halfway using a smartphone on the way home and continues watching the video at home on a device such as an Internet-connected TV. It should be noted that each of the smartphone and device described above includes a decoder 200 having identical or different performance. In this case, when the device decodes layers up to the top layer in the stream, the user can watch the video in high quality at home. Thus, the encoder 100 does not have to generate multiple streams having different image qualities of identical content, and thereby the processing load can be reduced.

[0071] In addition, the enhancement layer may include meta-information based on statistical information regarding the image. The decoder 200 can generate video whose image quality is enhanced by performing super-resolution imaging on a frame in the base layer based on metadata. Super-resolution imaging may include, for example, improving the SN ratio at the same resolution, increasing the resolution, etc. The metadata may include, for example, information to identify a linear or nonlinear filtering coefficient used in a super-resolution process, or information identifying the value of a parameter in a filtering process, machine learning, or least squares method used in a super-resolution process and etc.

[0072] In an embodiment, a configuration may be provided in which a frame is divided, for example, into tiles, for example, according to the semantic meaning of an object in the frame. In this case, the decoder 200 can decode only a partial area in the frame by selecting a tile to be decoded. In addition, the attribute of the object (person, car, ball, etc.) and the position of the object in the frame (coordinates in identical images) can be stored as metadata. In this case, decoder 200 is capable of identifying the position of the desired object based on the metadata and determining a tile including the object. For example, as illustrated in FIG. 6, metadata may be stored using a data storage structure different from image data, such as an SEI (Additional Enhancement Information) message in HEVC. This metadata indicates, for example, the position, size or color of the underlying object.

[0073] Metadata may be stored in units of multiple frames, for example, as a stream, sequence, or random access unit. Thus, the decoder 200 is capable of obtaining, for example, the time at which a particular person appears in a video, and by fitting the time information to the frame unit information, is capable of identifying a frame in which an object (person) is present and determining the position of the object in the frame.

[0074] Encoder

Next, an encoder according to an embodiment will be described. Fig. 7 is a block diagram illustrating a functional configuration of the encoder 100 according to the embodiment. Encoder 100 is a video encoder that encodes video in units of blocks.

[0075] As illustrated in FIG. 7, the encoder 100 is equipment that encodes an image in units of blocks, and includes a partitioner 102, a subtractor 104, a converter 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse converter 114, an adder 116, and a memory 118. blocks, a loop filter 120, a movie frame memory 122, an internal prediction module 124, an inter-prediction module 126, a prediction controller 128, and a prediction parameter generator 130. As illustrated, the intra prediction module 124 and the inter prediction module 126 are part of the prediction controller.

[0076] Encoder 100 is implemented, for example, as a general processor and storage device. In this case, when the software program stored in the memory device is executed by the processor, the processor functions as a partitioner 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116 , a loop filter 120, an intra prediction module 124, an inter prediction module 126, and a prediction controller 128. Alternatively, encoder 100 may be implemented as one or more dedicated electronic circuits corresponding to partitioner 102, subtractor 104, transformer 106, quantizer 108, entropy encoder 110, inverse quantizer 112, inverse converter 114, adder 116, loop filter 120, module 124 internal prediction, inter-prediction module 126, and prediction controller 128.

[0077] Encoder installation example

Fig. 8 is a functional block diagram illustrating an installation example of the encoder 100. The encoder 100 includes a processor a1 and a memory a2. For example, the plurality of constituent elements of the encoder 100 illustrated in FIG. 7 are mounted in the processor a1 and memory a2 illustrated in FIG. 8.

[0078] Processor a1 is a circuit that performs information processing and is connected to storage device a2. For example, processor a1 is a specialized or general electronic circuit that encodes an image. Processor a1 may be a processor such as a CPU. In addition, processor a1 may be a collection of multiple electronic circuits. In addition, for example, processor a1 may serve as two or more constituent elements of the plurality of constituent elements of the encoder 100 illustrated in FIG. 7, etc.

[0079] Memory a2 is a dedicated or general memory for storing information that is used by processor a1 to encode an image. The storage device a2 may be an electronic circuit and may be coupled to the processor a1. In addition, the storage device a2 may be included in the processor a1. In addition, the storage device a2 may be a collection of a plurality of electronic circuits. In addition, the storage device a2 may be a magnetic disk, an optical disk or the like. or may be represented as a storage device, recording medium, or the like. In addition, the storage device a2 may be a non-volatile storage device or a volatile storage device.

[0080] For example, memory a2 may store an image to be encoded or a bit stream corresponding to an encoded image. In addition, the storage device a2 may store a program for instructing the processor a1 to encode an image.

[0081] In addition, for example, the storage device a2 can serve as two or more constituent elements for storing information from a plurality of constituent elements of the encoder 100 illustrated in FIG. 7, etc. For example, memory a2 may serve as block memory 118 and movie frame memory 122 illustrated in FIG. 7. More specifically, the storage device a2 can store a reconstructed block, a reconstructed frame, etc.

[0082] It should be noted that, in the encoder 100, all of the plurality of constituent elements indicated in FIG. 7, etc. may not be implemented, and all processes described herein may not be performed. Some of the constituent elements shown in FIG. 7, etc., may be included in another device, or part of the processes described herein may be performed by another device.

[0083] Hereinafter, this document describes the complete flow of processes performed by the encoder 100, and thereafter describes each of the constituent elements included in the encoder 100.

[0084] General flow of the encoding process

Fig. 9 is a flowchart indicating one example of a complete decoding process performed by the encoder 100, and for convenience is described with reference to FIG. 7.

[0085] First, the partitioning unit 102 of the encoder 100 partitions each of the frames included in the input image into a plurality of blocks having a fixed size (eg, 128×128 pixels) (step Sa_1). The partitioning module 102 then selects a partitioning pattern for a fixed-size block (also called “block shape”) (step Sa_2). In other words, the partitioning module 102 further partitions the fixed-size block into a plurality of blocks that form a selected partitioning pattern. Encoder 100 performs, for each of a plurality of blocks, steps Sa_3-Sa_9 for the block (which is the current block to be encoded).

[0086] The prediction controller 128 and the prediction execution module (which includes the intra prediction module 124 and the inter prediction module 126) generate a prediction image of the current block (step Sa 3). The prediction image may also be referred to as a "prediction signal", "prediction block" or "prediction samples".

[0087] Next, the subtractor 104 generates the difference between the current block and the prediction image as a prediction residual (step Sa_4). The forecast residual may also be called "forecast error".

[0088] Next, the converter 106 converts the prediction image, and the quantizer 108 quantizes the result to generate a plurality of quantized coefficients (step Sa_5). A set of quantized coefficients may sometimes be referred to as a "coefficient block".

[0089] Next, the entropy encoder 110 encodes (specifically, entropy encodes) a plurality of quantized coefficients and a prediction parameter associated with generating a prediction image to generate a stream (step Sa_6). The stream may sometimes be called a "coded bitstream" or a "compressed bitstream".

[0090] Next, the inverse quantizer 112 inversely quantizes the plurality of quantized coefficients, and the inverse transformer 114 inversely transforms the result to recover the prediction remainder (step Sa_7).

[0091] Next, the adder 116 adds the prediction image with the reconstructed prediction residual to reconstruct the current block (step Sa_8). Thus, a reconstructed image is formed. The reconstructed image may also be referred to as a "recovered block" or a "decoded image block".

[0092] When the reconstructed image is generated, the loop filter 120 filters the reconstructed image as necessary (step Sa_9).

[0093] The encoder 100 then determines whether or not encoding of the entire frame is complete (step Sa_10). When it is determined that encoding is not yet complete ("No" in step Sa_10), the processes from step Sa_2 are repeated for the next frame block.

[0094] Although the encoder 100 selects one partitioning pattern for a fixed-size block and encodes each block according to the partitioning pattern in the above example, it should be noted that each block may be encoded according to a corresponding one of a plurality of partitioning patterns. In this case, the encoder 100 may estimate costs for each of a plurality of partitioning patterns and, for example, may select a stream obtained by encoding according to the partitioning pattern that results in the lowest cost as the stream that is output.

[0095] As illustrated, the processes in steps Sa_1-Sa_10 are executed sequentially by the encoder 100. Alternatively, two or more processes may be executed in parallel, processes may be reordered, etc.

[0096] The encoding process used by the encoder 100 is a hybrid encoding using predictive encoding and transform encoding. In addition, predictive encoding is performed by an encoding loop configured with a subtractor 104, a transformer 106, a quantizer 108, an inverse quantizer 112, an inverse transformer 114, an adder 116, a loop filter 120, a block memory 118, a movie frame memory 122, and an intra prediction unit 124. , a mutual prediction module 126 and a prediction controller 128 . In other words, the prediction execution unit configured with the intra prediction unit 124 and the inter prediction unit 126 is part of a coding loop.

[0097] Partitioner

The partitioning unit 102 splits each frame included in the original image into a plurality of blocks, and outputs each block to the subtractor 104. For example, the partitioning unit 102 first partitions the frame into blocks of a fixed size (eg, 128×128 pixels). Other fixed block sizes may be used. A fixed size block is also called a "coding tree unit (CTU)". The partitioning module 102 then partitions each fixed-size block into variable-size blocks (eg, 64×64 pixels or less) based on recursive quadtree and/or binary block partitioning. In other words, the partitioning module 102 selects a partitioning pattern. A variable size block may also be referred to as a "coding unit (CU)", "prediction unit (PU)" or "transformation unit (TU)". It should be noted that, in various kinds of processing examples, there is no need to distinguish between CU, PU and TU; all or some of the blocks in a frame can be processed in CU, PU or TU units.

[0098] FIG. 10 is a conceptual diagram for illustrating one example of blocking according to an embodiment. In fig. 10, solid lines represent block boundaries for blocks divided by quadtree blocking, and dotted lines represent block boundaries for blocks divided by binary tree blocking.

[0099] Here, block 10 is a square block having 128×128 pixels (128×128 block). This 10 128x128 block is first split into four square 64x64 pixel blocks (quadtree blocking).

[0100] The top left 64x64 pixel block is further vertically split into two rectangular 32x64 pixel blocks, and the left 32x64 pixel block is further vertically split into two rectangular 16x64 pixel blocks (binary tree chunking ). As a result, the top left 64x64 pixel block is split into two 16x64 pixel blocks 11 and 12 and one 32x64 pixel block 13.

[0101] The top right 64x64 pixel block is horizontally split into two rectangular 64x32 pixel blocks 14 and 15 (binary tree blocking).

[0102] The lower left 64x64 pixel square block is first divided into four 32x32 pixel square blocks (quadtree blocking). The top left block and bottom right block of four 32x32 pixel square blocks are further broken down. The top left square 32x32 pixel block is vertically split into two rectangular 16x32 pixel blocks, and the right 16x32 pixel block is further split horizontally into two 16x16 pixel blocks (binary tree blocking). The bottom right 32x32 pixel block is horizontally split into two 32x16 pixel blocks (binary tree blocking). The top right 32x32 pixel square block is horizontally split into two 32x16 pixel rectangular blocks (binary tree blocking). As a result, the lower left square 64x64 pixel block is split into a rectangular 16x32 pixel block 16, two square 16x16 pixel blocks 17 and 18, two square 32x32 pixel blocks 19 and 20, and two rectangular blocks 32 ×16-pixel blocks 21 and 22.

[0103] The lower right 64x64 pixel block 23 is not split.

[0104] As described above, FIG. 10, block 10 is divided into thirteen blocks 11-23 of variable size based on recursive quadtree and binary tree blocking. This type of partitioning is also called "quadtree plus binary tree (QTBT) partitioning."

[0105] It should be noted that, in FIG. 10, one block is partitioned into four or two blocks (quadtree or binary tree partitioning), but partitioning is not limited to these examples. For example, one block can be split into three blocks (ternary block splitting). Partitioning that includes such ternary block partitioning is also called "multi-type tree (MBT) partitioning."

[0106] FIG. 11 is a block diagram illustrating one example of a functional configuration of the partitioning unit 102 according to one embodiment. As illustrated in FIG. 11, the partitioning unit 102 may include a block partitioning determining unit 102a. The block partition determining unit 102a may perform the following processes as examples.

[0107] For example, the block split determination unit 102a may obtain or retrieve block information from the block memory 118 and/or the movie frame memory 122 and determine a split pattern (eg, the above-described split pattern) based on the block information. The partitioning unit 102 partitions the original image according to the partitioning pattern and outputs at least one block obtained by partitioning to the subtractor 104.

[0108] In addition, for example, block partition determination module 102a outputs one or more parameters indicating a particular partitioning pattern (eg, the above-described partitioning pattern) to mapper 106, inverse mapper 114, intra prediction module 124, inter prediction module 126, and entropy encoder 110. Transformer 106 may transform the prediction remainder based on one or more parameters. The intra prediction module 124 and the inter prediction module 126 may generate a prediction image based on one or more parameters. In addition, entropy encoder 110 may entropy encode one or more parameters.

[0109] A parameter associated with a split pattern may be written to the stream, as set forth below as one example.

[0110] FIG. 12 is a conceptual diagram for illustrating examples of partitioning patterns. Examples of partitioning patterns include: quadruple partitioning (QT), which splits a block into two areas, both horizontally and vertically; splitting into three areas (HT or VT), in which the block is split in an identical direction in a ratio of 1:2:1; splitting into two areas (HB or VB), in which the block is split in an identical direction in a 1:1 ratio; and no partitioning (NS).

[0111] It should be noted that the partitioning pattern does not have block partitioning direction in the case of four-area partitioning and no partitioning, and that the partitioning pattern has partitioning direction information in the case of two-area partitioning or three-area partitioning.

[0112] FIG. 13A is a conceptual diagram for illustrating one example of a split pattern syntax tree.

[0113] FIG. 13B is a conceptual diagram for illustrating another example of a split pattern syntax tree.

[0114] FIG. 13A and 13B are conceptual diagrams for illustrating examples of a split pattern syntax tree. In the example of FIG. 13A, firstly, information indicating whether or not partitioning should be performed (S: partition flag) is present, and information indicating whether or not partitioning into four areas should be performed (QT: QT flag) is present after it. . Information indicating what of the three-area and two-area partitioning should be carried out (TT: TT-flag or BT: BT-flag) is present after it, and information indicating the direction of the partition (Ver: vertical flag, or Hor: horizontal flag) is present after this. It should be noted that each at least one block obtained by partitioning according to such a partitioning pattern may further be partitioned multiple times in a similar process. In other words, as one example, whether or not the partitioning is performed, whether or not the four-area partitioning is performed, which of the horizontal direction and the vertical direction is the direction in which the partitioning method should be performed, which of the three partitioning region and splitting into two regions must be performed, can be recursively determined, and the results of the determination can be encoded in the stream according to the encoding order disclosed by the syntax tree illustrated in FIG. 13A.

[0115] In addition, although information elements respectively indicating S, QT, TT and Ver are arranged in the listed order in the syntax tree illustrated in FIG. 13A, information elements respectively indicating S, QT, Ver and BT may be arranged in the listed order. In other words, in the example of FIG. 13B, firstly, information indicating whether or not to perform partitioning (S: partition flag) is present, and information indicating whether or not partitioning into four areas should be performed (QT: QT flag) is present after it. . Information indicating the direction of partitioning (Ver: vertical flag, or Hor: horizontal flag) is present after it, and information indicating what of two-area partitioning and three-area partitioning should be carried out (BT: BT-flag, or TT : TT flag) is present after this.

[0116] It should be noted that the partitioning patterns described above are examples, and partitioning patterns other than the described partitioning patterns may be used, or a portion of the described partitioning patterns may be used.

[0117] Subtractor

The subtractor 104 subtracts the prediction image (the prediction sample that is input from the prediction controller 128 specified below) from the original image, in units of blocks input from the partitioning unit 102 and partitioning by the partitioning unit 102. In other words, subtractor 104 calculates prediction residuals (also called "errors") of the current block. Subtractor 104 then outputs the calculated prediction residuals to converter 106.

[0118] The source image may be an image that is input to the encoder 100 as a signal representing the image of each frame included in the video (eg, a luma signal and two chrominance signals). The signal representing the image may also be called a "sample".

[0119] Converter

Transformer 106 converts the spatial domain prediction residuals into frequency domain transform coefficients and outputs the transform coefficients to quantizer 108. More specifically, transformer 106 applies, for example, a specified discrete cosine transform (DCT) or discrete sine transform (DST) to the spatial domain prediction residuals. areas. The specified DCT or DST can be preset.

[0120] It should be noted that the transformer 106 can adaptively select a transform type from a plurality of transform types and convert the prediction residuals into transform coefficients by using a transform basis function corresponding to the selected transform type. This kind of transformation is also called "explicit multiple base transformation (EMT)" or "adaptive multiple transformation (AMT)". The basis transformation function may also be called a "basis".

[0121] Conversion types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I and DST-VII. It should be noted that these conversion types can be represented as DCT2, DCT5, DCT8, DST1 and DST7. Fig. 14 is a diagram indicating exemplary transform basis functions for exemplary transform types. In fig. 14, N indicates the number of input pixels. For example, the selection of a transformation type from a plurality of transformation types may depend on the prediction type (one of intra-prediction and inter-prediction) and may depend on the intra-prediction mode.

[0122] Information indicating whether or not such EMT or AMT should be applied (called, for example, "EMT flag" or "AMT flag"), and information indicating the selected type of conversion are normally signaled at the CU level . It should be noted that signaling of this information does not necessarily have to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, or CTU level).

[0123] In addition, the converter 106 may re-convert the conversion coefficients (which represent the results of the conversion). This re-transformation is also called "adaptive secondary transform (AST)" or "non-separable secondary transform (NSST)". For example, transformer 106 performs re-conversion in units of a subblock (eg, a 4x4 pixel subblock) included in a block of transform coefficients corresponding to the intra prediction residual. Information indicating whether or not to apply NSST and information associated with a transformation matrix to be used in NSST are normally signaled at the CU level. It should be noted that signaling of this information does not necessarily need to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, or CTU level).

[0124] The converter 106 may use a separable transform and an inseparable transform. Separable transformation is a method in which the transformation is performed multiple times by separately performing the transformation for each of a certain number of directions according to the number of dimensions of the inputs. An inseparable transformation is a method of performing a collective transformation in which two or more dimensions in multidimensional inputs are collectively treated as one dimension.

[0125] In one example of a non-separable transformation, when the input is a 4x4 pixel block, the 4x4 pixel block is treated as a single array including sixteen elements, and the transformation applies a 16x16 transformation matrix to the array.

[0126] In another example of a non-separable transform, an input block of 4x4 pixels is treated as a single array including sixteen elements, and a transform (hypercubic Givens transform) can then be performed in which a Givens rotation is performed on the array multiple times.

[0127] When converting in converter 106, conversion types for basic conversion functions to be converted to the frequency domain according to domains in the CU can be switched. Examples include spatially varying transform (SVT).

[0128] FIG. 15 is a conceptual diagram for illustrating one example of an SVT.

[0129] In SVT, as illustrated in FIG. 5B, the CUs are divided into two equal regions horizontally or vertically, and only one of the regions is converted to the frequency domain. The basic transformation type can be specified for each area. For example, DST7 and DST8 are used. For example, from two areas obtained by dividing the CU vertically into two equal areas, DST7 and DCT8 can be used for the area at position 0. Alternatively, from two areas, DST7 is used for the area at position 1. Likewise, from two areas obtained by partitioning CU horizontally into two equal areas, DST7 and DCT8 are used for the area at position 0. Alternatively, of the two areas, DST7 is used for the area at position 1. Although only one of the two areas in the CU is converted and the other is not converted in the example illustrated in fig. 15, each of the two areas can be converted. In addition, the partitioning method may include not only partitioning into two areas, but also partitioning into four areas. In addition, the partitioning method can be more flexible. For example, information indicating the partitioning method may be encoded and signaled identical to the CU partitioning. It should be noted that SVT may also be referred to as "sub-block transform (SBT)".

[0130] AMT and EMT described above may be referred to as "MTS (Multiple Transform Selection)". When MTS is applied, a transformation type that is DST7, DCT8, etc. can be selected, and information indicating the selected transformation type can be encoded as index information for each CU. Another process called "IMTS (Implicit MTS)" is provided as a process for selecting the type of transformation to be used for the orthogonal transformation performed without encoding the index information. When IMTS is applied, for example, when the CU is rectangular in shape, the orthogonal transformation of the rectangular shape can be performed using DST7 for the short side and DST2 for the long side. In addition, for example, when the CU has a square shape, the orthogonal transformation of the rectangular shape can be performed using DCT2 when the MTS is effective in the sequence, and using DST7 when the MTS is ineffective in the sequence. DCT2 and DST7 are just examples. Other transformation types may be used, and it is also possible to change the combination of transformation types to use to another combination of transformation types. IMTS can be used only for intra prediction blocks or can be used for intra prediction blocks and inter prediction block.

[0131] Three processes of MTS, SBT and IMTS are described above as selection processes for selectively switching transform types for use in orthogonal transform. However, all three selection processes may be used, or only a subset of the selection processes may be selectively used. Whether or not one or more selection processes are used can be identified, for example, based on flag information and the like. in the header, for example, in SPS. For example, when all three selection processes are available for use, one of the three selection processes is selected for each CU, and the orthogonal transformation of the CU is performed. It should be noted that the selection processes for selectively switching transformation types may be selection processes different from the above three selection processes, or each of the three selection processes may be replaced by a different process. Typically, at least one of the following four transfer functions [1]-[4] is satisfied. Function [1] is a function for performing an orthogonal transformation of the entire CU and encoding information indicating the type of transformation used in the transformation. Function [2] is a function for performing an orthogonal transformation of the entire CU and determining the transformation type based on a certain rule without encoding information indicating the transformation type. Function [3] is a function for performing an orthogonal transformation of a partial CU region and encoding information indicating the type of transformation used in the transformation. Function [4] is a function for performing an orthogonal transformation of a partial CU region and determining the transformation type based on a certain rule without encoding information indicating the transformation type used in the transformation. Certain rules can be predefined.

[0132] It should be noted that whether or not MTS, IMTS and/or SBT is applied can be determined for each processing unit. For example, whether or not MTS, IMTS, and/or SBT is applied may be determined on a per-sequence, frame, brick, slice, CTU, or CU basis.

[0133] It should be noted that a tool for selectively switching transformation types in the present disclosure may be described as a method for selectively selecting a basis for use in a transformation process, in a selection process, or in a process for selecting a basis. In addition, a tool for selectively switching transformation types can be described as a mode for adaptively selecting transformation types.

[0134] FIG. 16 is a flowchart illustrating one example of a process performed by converter 106 and is described for convenience with reference to FIG. 7.

[0135] For example, the converter 106 determines whether or not to perform the orthogonal transform (step St_1). Here, upon determining whether to perform the orthogonal transform (Yes in step St_1), the transformer 106 selects a transform type to use for the orthogonal transform from a plurality of transform types (step St_2). Next, transformer 106 performs an orthogonal transform by applying the selected transform type to the prediction residual of the current block (step St_3). The converter 106 then outputs information indicating the selected transform type to the entropy encoder 110 so as to enable the entropy encoder 110 to encode the information (step St_4). On the other hand, when determining whether orthogonal transformation is not performed (“No” in step St_1), the converter 106 outputs information indicating that orthogonal transformation is not performed so as to allow the entropy encoder 110 to encode information (step St_5). It should be noted that whether or not the orthogonal transformation should be performed in step St_1 may be determined, for example, based on the size of the transformation block, the prediction mode applied to the CU, etc. Alternatively, the orthogonal transform may be performed using a specified transform type without encoding information indicating the transform type to use in the orthogonal transform. The specified conversion type can be preset.

[0136] FIG. 17 is a flowchart illustrating one example of a process performed by converter 106 and is described for convenience with reference to FIG. 7. It should be noted that the example illustrated in FIG. 17 is an example of an orthogonal transform in the case where the transform types to be used in the orthogonal transform are selectively switched, as in the case of the example illustrated in FIG. 16.

[0137] As one example, the first group of transformation types may include DCT2, DST7, and DCT8. As another example, the second group of transformation types may include DCT2. The transformation types included in the first transformation type group and the transformation types included in the second transformation type group may partially overlap each other or may be completely different from each other.

[0138] The converter 106 determines whether the transform size is less than or equal to the determined value (step Su_1). Here, when determining that the transform size is less than or equal to a certain value ("Yes" in step Su_1), the transformer 106 performs an orthogonal transform of the prediction residual of the current block using a transform type included in the first transform type group (step Su_2). Next, the transformer 106 outputs information indicating a transform type to be used from at least one transform type included in the first transform type group to the entropy encoder 110, so as to enable the entropy encoder 110 to encode information (step Su_3) . On the other hand, when determining that the transform size is not less than or equal to a predetermined value ("No" in step Su_1), the transformer 106 performs an orthogonal transform of the prediction residual of the current block using the second group of transform types (step Su_4). The determined value may be a threshold value and may be a predetermined value.

[0139] In step Su_3, information indicating a transform type to be used in the orthogonal transform may be information indicating a combination of a transform type to be applied vertically in the current block and a transform type to be applied horizontally in the current block. The first group of types may include only one transformation type, and information indicating the transformation type to be used in the orthogonal transformation may not be encoded. The second transformation type group may include a plurality of transformation types, and information indicating a transformation type to be used for orthogonal transformation from one or more transformation types included in the second transformation type group may be encoded.

[0140] Alternatively, the transform type may be indicated based on the transform size without encoding information indicating the transform type. It should be noted that such determinations are not limited to determining whether the transform size is less than or equal to a certain value, and other processes are also possible for determining the transform type to use in the orthogonal transform based on the transform size.

[0141] Quantizer

Quantizer 108 quantizes transform coefficients output from transformer 106. More specifically, quantizer 108 scans, in a specific scanning order, transform coefficients of the current block and quantizes the scanned transform coefficients based on quantization parameters (QPs) corresponding to the transform coefficients. Quantizer 108 then outputs quantized transform coefficients (hereinafter also referred to as “quantized coefficients”) of the current block to entropy encoder 110 and inverse quantizer 112. A specific scan order may be predetermined.

[0142] The determined scanning order is the order for quantizing/inverse quantizing the transform coefficients. For example, a particular scan order may be specified as ascending frequency order (low to high frequency) or descending frequency order (high to low frequency).

[0143] The quantization parameter (QP) is a parameter that specifies the quantization step (quantization width). For example, when the value of the quantization parameter increases, the quantization step size also increases. In other words, when the value of the quantization parameter increases, the error in the quantized coefficients (quantization error) increases.

[0144] In addition, the quantization matrix can be used for quantization. For example, several kinds of quantization matrices may be used, respectively, to perform frequency conversion for sizes such as 4x4 and 8x8, prediction modes such as intra prediction and inter prediction, and pixel components such as luminance and pixel components. color signal. It should be noted that quantization means digitizing values sampled at certain intervals, respectively, into certain levels. In this art, quantization may be referred to using other expressions such as rounding and scaling, and may use rounding and scaling. Certain intervals and certain levels can be predetermined.

[0145] Methods using quantization matrices may include a method using a quantization matrix that is set directly on the side of the encoder 100, and a method using a quantization matrix that is set as a default value (default matrix). At the encoder 100 side, a quantization matrix suitable for image features may be specified by directly specifying the quantization matrix. However, this case may have the disadvantage of increasing the encoding volume for encoding the quantization matrix. It should be noted that the quantization matrix to be used to quantize the current block may be generated based on the default quantization matrix or the coded quantization matrix, instead of directly using the default quantization matrix or the coded quantization matrix.

[0146] A method is provided for quantizing a high frequency coefficient and a low frequency coefficient without using a quantization matrix. It should be noted that this method can be considered equivalent to a method using a quantization matrix (flat matrix), the coefficients of which have identical meanings.

[0147] The quantization matrix may be encoded, for example, at the sequence level, frame level, slice level, brick level, or CTU level. The quantization matrix may be specified using, for example, a sequence parameter set (SPS) or a frame parameter set (PPS). SPS includes a parameter that is used for a sequence, and PPS includes a parameter that is used for a frame. Each of the SPS and PPS may be referred to simply as a "parameter set".

[0148] When using the quantization matrix, quantizer 108 scales, for each transform factor, eg, the quantization width, which may be calculated based on the quantization parameter, etc., using the value of the quantization matrix. The quantization process performed without using a quantization matrix may be a process for quantization transform coefficients based on a quantization width calculated based on a quantization parameter, etc. It should be noted that, in a quantization process performed without using quantization matrices, the quantization width may be multiplied by a certain value that is common to all transform coefficients in the block. A specific value can be predefined.

[0149] FIG. 18 is a block diagram illustrating one example of a functional configuration of a quantizer according to the embodiment. For example, the quantizer 108 includes a difference quantization parameter generator 108a, a predicted quantization parameter generator 108b, a quantization parameter generator 108c, a quantization parameter storage device 108d, and a quantization execution unit 108e.

[0150] FIG. 19 is a flowchart illustrating one example of a quantization process performed by the quantizer 108, and is described for convenience with reference to FIG. 7 and 18.

[0151] As one example, quantizer 108 may perform quantization on a per-CU basis based on the flowchart illustrated in FIG. 19. More specifically, the quantization parameter generator 108c determines whether or not to perform quantization (step Sv_1). Here, when determining whether to perform quantization (“Yes” in step Sv_1), the quantization parameter generator 108c generates a quantization parameter for the current block (step Sv_2) and stores the quantization parameter in the quantization parameter storage device 108d (step Sv_3).

[0152] Next, the quantization execution unit 108e quantizes the transform coefficients of the current block using the quantization parameter generated in step Sv_2 (step Sv_4). The predicted quantization parameter generator 108b then obtains the quantization parameter for a processing unit other than the current block from the quantization parameter storage device 108d (step Sv_5). The predicted quantization parameter generator 108b generates a predicted quantization parameter of the current block based on the received quantization parameter (step Sv_6). The difference quantization parameter generator 108a calculates the difference between the current block quantization parameter generated by the quantization parameter generator 108c and the predicted quantization parameter of the current block generated by the predicted quantization parameter generator 108b (step Sv_7). The difference quantization parameter can be generated by calculating the difference. The difference quantization parameter generator 108a outputs the difference quantization parameter to the entropy encoder 110 so as to enable the entropy encoder 110 to encode the difference quantization parameter (step Sv_8).

[0153] It should be noted that the difference quantization parameter may be encoded, for example, at the sequence level, frame level, slice level, brick level, or CTU level. In addition, the initial value of the quantization parameter may be encoded at the sequence level, frame level, slice level, brick level, or CTU level. During initialization, the quantization parameter can be generated using the initial value of the quantization parameter and the difference quantization parameter.

[0154] It should be noted that the quantizer 108 may include a plurality of quantizers and may employ dependent quantization, in which the transform coefficients are quantized using a quantization method selected from a plurality of quantization methods.

[0155] Entropy encoder

Fig. 20 is a block diagram illustrating one example of a functional configuration of the entropy encoder 110 according to the embodiment, and is described for convenience with reference to FIG. 7. The entropy encoder 110 generates a stream by entropy encoding the quantized coefficients input from the quantizer 108 and the prediction parameter input from the prediction parameter generator 130. For example, context adaptive binary arithmetic coding (CABAC) is used as entropy coding. More specifically, the entropy encoder 110, as illustrated, includes a binarization unit 110a, a context controller 110b, and a binary arithmetic encoder 110c. The binarization unit 110a performs binarization in which multi-level signals such as quantized coefficients and a prediction parameter are converted into binary signals. Examples of binarization methods include truncated Rice code binarization, exponential Golomb codes, and fixed-length code binarization. The context controller 110b extracts a context value according to a feature or surrounding state of a syntactic element that represents the probability of occurrence of a binary signal. Examples of methods for retrieving context value include traversal, syntactic element access, top and left adjacent block access, hierarchical information access, etc. Binary arithmetic encoder 110c arithmetically encodes the binary signal using the extracted context.

[0156] FIG. 21 is a conceptual diagram for illustrating an exemplary flow of the CABAC process in entropy encoder 110. First, initialization is performed in CABAC in entropy encoder 110. In initialization, initialization in binary arithmetic encoder 110c and setting an initial context value are performed. For example, the binarization unit 110a and the binary arithmetic encoder 110c may perform binarization and arithmetic encoding of a plurality of quantization coefficients in the CTU sequentially. The context controller 110b may update the context value each time arithmetic encoding is performed. The context controller 110b may then store the context value as post-processing. The stored context value can be used, for example, to initialize the context value for the next CTU.

[0157] Inverse Quantizer

The inverse quantizer 112 inversely quantizes the quantized coefficients that are input from the quantizer 108. More specifically, the inverse quantizer 112 inversely quantizes, in a specific scanning order, the quantized coefficients of the current block. The inverse quantizer 112 then outputs the inversely quantized transform coefficients of the current block to the inverse quantizer 114. A specific scan order may be predetermined.

[0158] Inverter

The inverter 114 recovers the prediction residuals by inversely transforming the transform coefficients that are input from the inverse quantizer 112. More specifically, the inverter 114 recovers the prediction residuals of the current block by performing an inverse transform corresponding to the transform applied to the transform coefficients by the inverter 106. The inverter 114 then outputs the reconstructed prediction residuals to the adder 116.

[0159] It should be noted that since information is lost in quantization, the reconstructed prediction errors are not the same as the prediction errors calculated by the subtractor 104. In other words, the reconstructed prediction errors normally include quantization errors.

[0160] Totalizer

The adder 116 reconstructs the current block by adding the prediction residuals that are input from the inverter 114 and the prediction images that are input from the prediction controller 128 . Consequently, a reconstructed image is formed. Adder 116 then outputs the reconstructed image to block memory 118 and loop filter 120. The reconstructed block may also be referred to as a “local decoded block.”

[0161] Block memory

Block memory 118 is a storage device for storing blocks in the current frame, for example, for use in intra prediction. More specifically, the block memory 118 stores the reconstructed images output from the adder 116.

[0162] Motion Picture Memory

The movie frame memory 122, for example, is a storage device for storing reference frames for use in inter-prediction, and is also called a “motion picture buffer”. More specifically, the movie frame memory 122 stores reconstructed images filtered by the edge filter 120.

[0163] Loop filter

The edge filter 120 applies the edge filter to the reconstructed image output by the adder 116 and outputs the filtered reconstructed image to the movie frame memory 122. A loop filter is a filter used in a coding loop (in-loop filter). Examples of loop filters include, for example, an adaptive loop filter (ALF), a deblocking filter (DB or DBF), a sampled adaptive offset filter (SAO), etc.

[0164] FIG. 22 is a block diagram illustrating one example of a functional configuration of the loop filter 120 according to the embodiment. For example, as illustrated in FIG. 22, the loop filter 120 includes a deblocking filtering execution unit 120a, an SAO execution unit 120b, and an ALF execution unit 120c. The deblocking filtering execution unit 120a performs a deblocking filtering process for the restored image. The SAO execution unit 120b performs the SAO process on the reconstructed image after subjecting it to the deblocking filtering process. The ALF execution unit 120c performs the ALF process on the reconstructed image after being subjected to the SAO process. The ALF and deblocking filter are described in detail below. The SAO process is a process for improving image quality by reducing ringing (a phenomenon in which pixel values are distorted as waves around an edge) and correcting deviation in pixel value. Examples of SAO processes include the edge shift process and the band shift process. It should be noted that loop filter 120, in some embodiments, may not include all of the constituent elements disclosed in FIG. 22, and may include certain constituent elements and may include additional elements. In addition, the loop filter 120 may be configured to perform the above processes in a processing order different from the processing order disclosed in FIG. 22 may not run all processes, etc.

[0165] Loop filter > adaptive loop filter

In ALF, a least squares error filter is applied to remove compression artifacts. For example, one filter, selected from a plurality of filters based on the direction and activity of local gradients, is applied to each 2x2-pixel subblock in the current block.

[0166] More specifically, first, each sub-block (eg, each 2x2-pixel sub-block) is classified into one of a plurality of classes (eg, fifteen or twenty-five classes). The subblock classification may be based, for example, on the directionality and activity of the gradients. For example, a classification index C (eg, C=5D+A) is extracted based on the directionality of the D gradients (eg, 0-2 or 0-4) and the activity of the A gradients (eg, 0-4). Then, based on the classification index C, each sub-block is classified into one of a variety of classes.

[0167] For example, the directionality D of the gradients is calculated by comparing the gradients of multiple directions (eg, horizontal, vertical, and two diagonal directions). In addition, for example, the activity A of the gradients is calculated by summing the gradients of multiple directions and quantizing the result of the summation.

[0168] The filter to be used for each sub-block may be determined from a plurality of filters based on the result of such classification.

[0169] The filter shape to be used in the ALF, for example, is a circular symmetrical filter shape. Fig. 23A-23C are conceptual diagrams to illustrate examples of filter shapes used in the ALF. Fig. 23A illustrates a 5x5 diamond shape filter, FIG. 23B illustrates a 7x7 diamond shape filter, and FIG. 23C illustrates a 9x9 diamond shaped filter. Information indicating the shape of the filter is normally carried in frame-level signaling. It should be noted that the signaling of this information indicating the shape of the filter need not be performed at the frame level and may be performed at another level (eg, sequence level, slice level, tile level, CTU level, or CU level).

[0170] Enabling or disabling ALF can be determined, for example, at the frame level or at the CU level. For example, a decision regarding whether or not to apply ALF to a luma signal may be made at the CU level, and a decision regarding whether or not to apply ALF to a chrominance signal may be made at the frame level. Information indicating whether ALF is enabled or disabled is normally carried in signaling at the frame level or at the CU level. It should be noted that the signaling of information indicating the enabling or disabling of ALF need not be performed at the frame level or the CU level and may be performed at another level (eg, sequence level, slice level, tile level, or CTU level).

[0171] In addition, as described above, one filter is selected from the plurality of filters, and the subblock ALF process is executed. A set of coefficients for the coefficients to be used for each of a plurality of filters (eg, up to the fifteenth or twenty-fifth filter) is normally signaled at the frame level. It should be noted that the signaling of the set of coefficients need not be performed at the frame level and may be performed at another level (eg, sequence level, slice level, tile level, CTU level, CU level, or subblock level).

[0172] Loop Filter > Cross Component Adaptive Loop Filter

Fig. 23D is a conceptual diagram for illustrating an exemplary cross-component ALF (CCALF) flowchart. Fig. 23E is a conceptual diagram for illustrating an example of a filter shape used in a CCALF, for example, the CCALF of FIG. 23D. The exemplary CCALF of FIG. 23D and 23E operate by applying a linear, diamond-shaped filter to the luma signal channel for each chrominance signal component. Filter coefficients, for example, can be passed to APS, scaled by a factor of 2^10, and rounded for fixed-point representation. For example, in FIG. 23D, Y samples (first component) are used for CCALF for Cb and CCALF for Cr (for components other than the first component).

[0173] The application of filters may be controlled for a variable block size and signaled via a context-coded flag received for each block of samples. The block size along with the CCALF enable flag may be received at the cut level for each chroma component. CCALF can support various block sizes, for example (in chroma samples) 16x16 pixels, 32x32 pixels, 64x64 pixels, 128x128 pixels.

[0174] Loop Filter > Combined Cross-Component Adaptive Chroma Loop Filter

One example of combined CCALF chrominance signals is illustrated in FIG. 23F and 23G. Fig. 23F is a conceptual diagram for illustrating an exemplary process flow of combined CCALF chrominance signals. Fig. 23G is a table illustrating approximate possible weight index options. As illustrated, one CCALF filter is used to generate one CCALF filtered output as the chrominance detail signal for one color component, while a weighted version of the identical chrominance detail signal is applied to the other color component. This reduces the complexity of the existing CCALF by about half. The weighting factor value may be encoded as a sign flag and a weighting index. The weight index (denoted as weight_index) may be encoded in 3 bits and indicates the absolute value of the JC-CCALF weighting factor JcCcWeight, which is a non-zero absolute value. The absolute value of JcCcWeight, for example, can be defined as follows:

If weight_index is less than or equal to 4, JcCcWeight is weight_index>>2;

Otherwise, JcCcWeight is 4/(weight_index-4).

[0175] The on/off control at the block level of ALF filtering for Cb and Cr may be separate. This is identical to CCALF, and two separate sets of block-level enable/disable control flags can be encoded. Unlike CCALF, in this document, the sizes of the on/off control blocks Cb, Cr are identical, and therefore only one block size variable can be encoded.

[0176] Loop Filter > Deblocking Filter

In the deblocking filtering process, the edge filter 120 performs a filtering process on a block boundary in the reconstructed image so as to reduce distortion that occurs at the block boundary.

[0177] FIG. 24 is a block diagram illustrating one example of a specific configuration of the deblocking filtering performing unit 120a of the loop filter 120 (see FIGS. 7 and 22) that functions as a deblocking filter.

[0178] The deblocking filtering module 120a includes: a boundary detection module 1201; filter determination module 1203; filtering execution module 1205; processing determination module 1208; filter characteristics determination module 1207; and switches 1202, 1204 and 1206.

[0179] Boundary determination module 1201 determines whether or not a pixel to be filtered for deblocking (ie, the current pixel) is present near a block near the block boundary. The boundary determination module 1201 then outputs the determination result to the switch 1202 and the processing determination module 1208 .

[0180] In the case where the edge detection module 1201 determines that the current pixel is present near the block boundary, the switch 1202 outputs an unfiltered image to the switch 1204. In the opposite case, in which the edge detection module 1201 determines that the current pixel is not present near the boundary block, switch 1202 outputs an unfiltered image to switch 1206. It should be noted that the unfiltered image is an image configured with the current pixel and at least one surrounding pixel located near the current pixel.

[0181] The filter determination unit 1203 determines whether or not to perform filtering to deblock the current pixel based on the pixel value of at least one surrounding pixel located near the current pixel. The filter determination unit 1203 then outputs the determination result to the switch 1204 and the processing determination unit 1208.

[0182] In case it is determined that the filter determination unit 1203 performs filtering to remove blockiness of the current pixel, the switch 1204 outputs the unfiltered image obtained through the switch 1202 to the filtering execution unit 1205. In the opposite case, in which it is determined that the filter determination module 1203 does not perform filtering to deblock the current pixel, the switch 1204 outputs the unfiltered image obtained through the switch 1202 to the switch 1206.

[0183] Upon receiving an unfiltered image via switches 1202 and 1204, filtering execution unit 1205 performs, for the current pixel, deblocking filtering with the filter characteristic determined by filter characteristic determining unit 1207. The filtering module 1205 then outputs the filtered pixel to the switch 1206.

[0184] Under the control of the processing determination unit 1208, the switch 1206 selectively outputs one of a pixel that is not filtered for deblocking and a pixel that is filtered for deblocking by the filtering execution unit 1205.

[0185] The processing determination module 1208 controls the switch 1206 based on the results of the determinations made by the boundary determination module 1201 and the filtering determination module 1203 . In other words, the processing determination module 1208 instructs the switch 1206 to output a pixel that is filtered to deblock when the edge determination module 1201 determines that the current pixel is present near a block boundary, and when the filter determination module 1203 determines that filtering to deblock the current pixel is necessary. . In addition, unlike the above-described case, the processing determination unit 1208 instructs the switch 1206 to output a pixel that is not filtered for deblocking. The filtered image is output from the switch 1206 by repeating the pixel output in this manner. It should be noted that the configuration illustrated in FIG. 24 is one example of a configuration in the blocking deblocking filtering module 120a. The deblocking filtering unit 120a may have various configurations.

[0186] FIG. 25 is a conceptual diagram for illustrating an example of a deblocking filter having a symmetrical filtering characteristic with respect to a block boundary.

[0187] In the deblocking filtering process, one of two deblocking filters having different characteristics, i.e. from a strong filter and a weak filter, can be selected using pixel values and quantization parameters. In the case of a strong filter, when pixels p0-p2 and pixels q0-q2 are present at the block boundary, as illustrated in FIG. 25, the pixel values of the corresponding pixel q0-q2 are changed to the pixel values q'0-q'2 by performing calculations according to the following expressions, for example.

[0188] q'0=(p1+2xp0+2xq0+2xq1+q2+4)/8

q'1=(p0+q0+q1+q2+2)/4

q'2=(p0+q0+q1+3xq2+2xq3+4)/8

[0189] It should be noted that, in the above expressions, p0-p2 and q0-q2 represent the pixel values of the corresponding pixels p0-p2 and pixels q0-q2. In addition, q3 is the pixel value of the neighboring pixel q3 located on the opposite side of pixel q2 relative to the block boundary. In addition, on the right side of each of the expressions, coefficients that are multiplied by the corresponding pixel values of the pixels to be used for filtering to remove blocking are filtering coefficients.

[0190] Additionally, when filtering to remove blocking, clipping may be performed such that the calculated pixel values do not change by more than a threshold value. For example, in the clipping process, pixel values calculated according to the above expressions may be clipped to a value obtained according to "calculated pixel value ± 2 x threshold value" using a threshold value determined based on the quantization parameter. In this way, excessive smoothing can be prevented.

[0191] FIG. 26 is a conceptual diagram for illustrating a block boundary on which a filtering process to remove blocking is performed. Fig. 27 is a conceptual diagram for illustrating examples of boundary intensity values (Bs).

[0192] A block boundary on which the deblocking filtering process is performed is, for example, a boundary between CUs, PUs or TUs having 8x8 pixel blocks, as illustrated in FIG. 26. The deblocking filtering process can be performed, for example, in units of four rows or four columns. First, the boundary intensity values (Bs) are determined as indicated in FIG. 27 for the P block and Q block illustrated in FIG. 26.

[0193] According to the Bs values in FIG. 27, it can be determined whether or not filtering processes for deblocking the boundaries of blocks belonging to the same image should be performed using different intensities. The deblocking filtering process for the chroma signal is performed when the Bs value is 2. The deblocking filtering process for the luma signal is performed when the Bs value is 1 or more and a certain condition is satisfied. A certain condition can be predefined. It should be noted that the conditions for determining the Bs values are not limited to those indicated in FIG. 27, and the Bs value may be determined based on another parameter.

[0194] Prediction Module (Intra Prediction Module, Inter Prediction Module, Prediction Controller)

Fig. 28 is a flowchart illustrating one example of a process performed by the encoder prediction module 100. It should be noted that the prediction module includes all or part of the following constituent elements: an intra prediction module 124; mutual prediction module 126; and a prediction controller 128. The prediction execution module includes, for example, an intra prediction module 124 and an inter prediction module 126.

[0195] The prediction module generates a prediction image of the current block (step Sb_1). This prediction image may also be referred to as a "prediction signal" or a "prediction block". It should be noted that the prediction signal is, for example, an internal prediction image (image prediction signal) or a mutual prediction image (mutual prediction signal). The prediction module generates a prediction image of the current block using a reconstructed image that has already been obtained through another block, through generating a prediction image, generating a prediction residual, generating quantized coefficients, restoring the prediction residual, and adding up the prediction image.

[0196] The reconstructed image, for example, may be an image in a reference frame or an image of a coded block (ie, another block described above) in the current frame, which is a frame including the current block. The coded block in the current frame, for example, is a block adjacent to the current block.

[0197] FIG. 29 is a flowchart illustrating another example of a process performed by the encoder prediction module 100.

[0198] The prediction module generates a prediction image using the first method (step Sc_1a), generates a prediction image using the second method (step Sc_1b), and generates a prediction image using the third method (step Sc_1c). The first method, the second method and the third method may be mutually different methods for generating a predictive image. Each of the first to third methods may be an inter-prediction method, an intra-prediction method, or another prediction method. The above-described reconstructed image can be used in these prediction methods.

[0199] Next, the prediction processor evaluates the prediction images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). For example, the prediction module calculates the costs C for the prediction images generated in steps Sc_1a, Sc_1b and Sc_1, and evaluates the prediction images by comparing the costs C of the prediction images. It should be noted that the cost C can be calculated, for example, according to the expression of the R-D optimization model, for example, C=D+λ x R. In this expression, D indicates the compression artifacts of the forecast image and is represented, for example, as the sum of the absolute differences between the pixel value the current block and the pixel value of the forecast image. In addition, R indicates the bit rate of the stream. In addition, λ indicates, for example, a multiplier according to the Lagrange multiplier method.

[0200] The prediction module then selects one of the prediction images generated in steps Sc_1a, Sc_1b and Sc_1c (step Sc_3). In other words, the prediction module selects a method or mode to obtain the final prediction image. For example, the prediction module selects the prediction image having the lowest cost C based on the costs C calculated for the prediction images. Alternatively, the evaluation in step Sc_2 and the selection of the prediction image in step Sc_3 may be performed based on a parameter that is used in the encoding process. Encoder 100 may convert information to identify the selected prediction image, method, or mode into a stream. The information may, for example, be a flag or the like. Thus, the decoder 200 is capable of generating a prediction image according to a method or mode selected by the encoder 100 based on the information. It should be noted that, in the example illustrated in FIG. 29, the prediction module selects any of the prediction images after the prediction images are generated using appropriate methods. However, the prediction module can select a method or mode based on a parameter to be used in the above-described encoding process before generating prediction images, and can generate a prediction image according to the selected method or mode.

[0201] For example, the first method and the second method may be intra-prediction and inter-prediction, respectively, and the prediction module may select the final prediction image for the current block from the prediction images generated according to the prediction methods.

[0202] FIG. 30 is a flowchart illustrating another example of a process performed by the encoder prediction module 100.

[0203] First, the prediction module generates a prediction image using intra prediction (step Sd_1a) and generates a prediction image using inter prediction (step Sd_1b). It should be noted that the prediction image generated by intra prediction is also called "intra prediction image", and the prediction image generated by inter prediction is also called "mutual prediction image".

[0204] Next, the prediction module evaluates each of the internal prediction image and the mutual prediction image (step Sd_2). The C costs described above can be used in the estimation. The prediction module can then select the prediction image for which the least cost C is calculated from the internal prediction image and the mutual prediction image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode is selected to generate a prediction image for the current block.

[0205] The prediction processor then selects the prediction image for which the least cost C is calculated from the internal prediction image and the mutual prediction image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode is selected to generate a prediction image for the current block.

[0206] Internal prediction module

The intra prediction module 124 generates a prediction signal (ie, an intra prediction signal) by performing intra prediction (also called “intra prediction”) of the current block by accessing the block or blocks in the current frame stored in the block memory 118. More specifically, the intra prediction module 124 generates an internal prediction image by performing intra prediction by referring to pixel values (eg, luma and/or chroma values) of the block or blocks adjacent to the current block, and then outputs the internal prediction image to the controller 128 forecasting.

[0207] For example, the intra prediction unit 124 performs intra prediction by using one mode of a plurality of intra prediction modes that are set. The intra prediction modes typically include one or more non-directional prediction modes and a plurality of directed prediction modes. Preset modes can be preset.

[0208] The one or more non-directional prediction modes include, for example, a planar prediction mode and a DC prediction mode defined in the H.265/High Efficiency Video Coding (HEVC) standard.

[0209] The plurality of directional prediction modes includes, for example, the thirty-three directional prediction modes defined in the H.265/HEVC standard. It should be noted that the plurality of directional prediction modes may further include thirty-two directional prediction modes in addition to the thirty-three directional prediction modes (for a total of sixty-five directional prediction modes). Fig. 31 is a conceptual diagram for illustrating a total of sixty-seven intra-prediction modes that can be used in intra-prediction (two non-directional prediction modes and sixty-five directional prediction modes). The solid arrows represent the thirty-three directions defined in the H.265/HEVC standard, and the dotted arrows represent the additional thirty-two directions (the two non-directional prediction modes are not illustrated in FIG. 31).

[0210] In various kinds of processing examples, a luma signal block may be referenced in internal prediction of a chrominance signal block. In other words, the chrominance signal component of the current block can be predicted based on the luminance signal component of the current block. This internal forecasting is also called “Cross Component Linear Model (CCLM) forecasting.” An intra-prediction mode for a chrominance block in which such a luma block is referenced (also called, for example, a “CCLM mode”) may be added as one of the intra-prediction modes for the chrominance blocks.

[0211] The internal prediction module 124 may adjust the internally predicted pixel values based on horizontal/vertical reference pixel gradients. The internal prediction followed by this kind of correction is also called "position-dependent combination with internal prediction (PDPC)". Information indicating whether or not to use PDPC (called, for example, a "PDPC flag") is normally signaled at the CU level. It should be noted that signaling of this information does not necessarily need to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, or CTU level).

[0212] FIG. 32 is a flowchart illustrating one example of a process performed by the intra prediction module 124.

[0213] The intra prediction unit 124 selects one intra prediction mode from a plurality of intra prediction modes (step Sw_1). The intra prediction unit 124 then generates a prediction image according to the selected intra prediction mode (step Sw_2). Next, the internal prediction module 124 determines the most probable modes (MPM) (step Sw_3). MPMs include, for example, six intraprediction modes. For example, two modes of the six intra prediction modes may be a planar mode and a DC prediction mode, and the other four modes may be directional prediction modes. The intra prediction unit 124 determines whether or not the intra prediction mode selected in step Sw_1 is turned on in the MPM (step Sw_4).

[0214] Here, when determining that the intra prediction mode selected in step Sw_1 is included in the MPM (“Yes” in step Sw_4), the intra prediction unit 124 sets the MPM flag to 1 (step Sw_5) and generates information indicating selected internal prediction mode, from MPM (step Sw_6). It should be noted that the MPM flag set to 1 and information indicating the intra prediction mode can be encoded as prediction parameters by the entropy encoder 110.

[0215] When determining that the selected intra prediction mode is not included in the MPM ("No" in step Sw_4), the intra prediction module 124 sets the MPM flag to 0 (step Sw_7). Alternatively, the intra prediction module 124 does not set the MPM flag at all. The intra prediction unit 124 then generates information indicating the selected intra prediction mode from at least one intra prediction mode that is not included in the MPM (step Sw_8). It should be noted that the MPM flag set to 0 and the information indicating the intra prediction mode can be encoded as prediction parameters by the entropy encoder 110. The information indicating the intra prediction mode indicates, for example, any of 0-60.

[0216] Internal Prediction Module

The inter prediction unit 126 generates a prediction image (cross prediction image) by performing inter prediction (also called “inter prediction”) of the current block by referring to a block or blocks in a reference frame that is different from the current frame and stored in the movie frame memory 122. Inter-prediction is performed in units of the current block or the current sub-block (eg, a 4x4 block) within the current block. A subblock is included in a block and represents a unit smaller than the block. The subblock size can be in the shape of a slice, brick, frame, etc.

[0217] For example, inter-prediction module 126 performs motion estimation in a reference frame for the current block or current sub-block and finds the reference block or reference sub-block that has the best match with the current block or current sub-block. The inter-prediction module 126 then receives motion information (eg, a motion vector) that compensates for the motion or change from the reference block or reference subblock to the current block or subblock. The mutual prediction unit 126 generates a mutual prediction image of the current block or sub-block by performing motion compensation (or motion prediction) based on the motion information. The mutual prediction module 126 outputs the generated mutual prediction image to the prediction controller 128 .

[0218] Motion information used in motion compensation may be signaled as mutual predictive signals in various forms. For example, the motion vector may be signaled. In other words, the difference between the motion vector and the motion vector predictor may be signaled.

[0219] List of reference frames

Fig. 33 is a conceptual diagram for illustrating examples of reference frames. Fig. 34 is a conceptual diagram for illustrating examples of reference frame lists. The reference frame list is a list indicating at least one reference frame stored in the movie frame memory 122. It should be noted that in FIG. 33, each of the rectangles indicates a frame, each of the arrows indicates the reference relationship of the frames, the horizontal axis indicates time, I, P and B in the rectangles indicate the internal prediction frame, uni-prediction frame and bi-prediction frame, respectively, and the numbers in the rectangles indicate the decoding order. As illustrated in FIG. 33, the decoding order of frames is the order of I0, P1, B2, B3 and B4, and the display order of frames is the order of I0, B3, B2, B4 and P1. As illustrated in FIG. 34, the reference frame list is a list representing candidate reference frames. For example, one frame (or slice) may include at least one list of reference frames. For example, one reference frame list is used when the current frame is a uni-prediction frame, and two reference frame lists are used when the current frame is a bi-prediction frame. In the examples of FIGS. 33 and 34, frame B3, which is the current frame currPic, has two reference frame lists, which are an L0 list and an L1 list. When the current frame currPic is a B3 frame, the possible reference frames for the current frame currPic are I0, P1 and B2, and the reference frame lists (which are an L0 list and an L1 list) indicate these frames. The inter-prediction module 126 or prediction controller 128 indicates which frame in each reference frame list should actually be accessed in the form of a reference frame index refidxLx. In fig. 34, reference frames P1 and B2 are indicated by reference frame indices refIdxL0 and refIdxL1.

[0220] Such a list of reference frames may be generated for each unit, such as a sequence, frame, slice, brick, CTU, or CU. In addition, of the reference frames indicated in the reference frame lists, a reference frame index indicating a reference frame accessed in inter-prediction may be signaled at the sequence level, frame level, slice level, brick level, CTU level, or C.U. In addition, a common reference frame list can be used in a variety of inter-prediction modes.

[0221] Basic Interprediction Flow

Fig. 35 is a flowchart illustrating an exemplary basic processing flow of the inter-prediction process.

[0222] First, the inter-prediction module 126 generates a prediction signal (steps Se_1 to Se_3). Next, the subtractor 104 generates the difference between the current block and the prediction image as a prediction residual (step Se_4).

[0223] Here, when generating a prediction image, the inter-prediction unit 126 generates a prediction image by determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and motion compensation (step Se_3). In addition, when determining the MV, the inter-prediction unit 126 determines the MV by selecting a motion vector candidate (MV candidate) (step Se_1) and extracting the MV (step Se_2). MV candidate selection is performed, for example, by internal prediction module 126 generating a list of MV candidate candidates and selecting at least one MV candidate candidate from the list of MV candidate candidates. It should be noted that MVs extracted previously may be added to the list of possible MV variants. Alternatively, when extracting the MV, the inter-prediction module 126 may further select at least one candidate MV from the at least one candidate MV and determine the selected at least one candidate MV as the MV for the current block. Alternatively, inter-prediction module 126 may determine the MV for the current block by performing an estimate in a reference frame region indicated by each of the selected at least one MV candidate. It should be noted that the estimation in the reference frame region may be referred to as "motion estimation".

[0224] In addition, although steps Se_1 to Se_3 are performed by the inter-prediction unit 126 in the above example, the process which is, for example, step Se_1, step Se_2, etc. may be performed by another component included in the encoder 100.

[0225] It should be noted that a list of MV candidates may be generated for each process in an inter-prediction mode, or a common list of MV candidates may be used in multiple inter-prediction modes. The processes in steps Se_3 and Se_4 correspond to steps Sa_3 and Sa_4 illustrated in FIG. 9, respectively. The process in step Se_3 corresponds to the process in step Sd_1b in FIG. thirty.

[0226] Motion Vector Extraction Flow

Fig. 36 is a flowchart illustrating one example of a motion vector extraction process.

[0227] The inter-prediction module 126 may extract the MV of the current block in a mode for encoding motion information (eg, MV). In this case, for example, motion information may be encoded as a prediction parameter and may be signaled. In other words, encoded motion information is included in the stream.

[0228] Alternatively, inter-prediction module 126 may extract MVs in a mode in which motion information is not encoded. In this case, motion information is not included in the stream.

[0229] Here, MV extraction modes may include normal mutual mode, merging mode, FRUC mode, affine mode, etc., which are described below. Modes in which motion information is encoded include normal reciprocal mode, normal combining mode, affine mode (specifically, affine reciprocal mode and affine combining mode), etc. It should be noted that the motion information may include not only MV, but also motion vector predictor selection information, which is described below. Modes in which motion information is not encoded include FRUC mode, etc. The inter-prediction unit 126 selects a mode for extracting the MV of the current block from a plurality of modes, and extracts the MV of the current block using the selected mode.

[0230] FIG. 37 is a flowchart illustrating another example of extracting motion vectors.

[0231] The inter prediction unit 126 may extract the MV of the current block in a mode in which the MV difference is encoded. In this case, for example, the MV difference may be encoded as a prediction parameter and may be signaled. In other words, the encoded MV difference is included in the stream. The MV difference is the difference between the MV of the current block and the MV predictor. It should be noted that the MV predictor is a motion vector predictor.

[0232] Alternatively, inter-prediction module 126 may extract MVs in a mode in which the MV difference is not encoded. In this case, the encoded MV difference is not included in the stream.

[0233] Here, as described above, MV extraction modes include normal mutual mode, normal pooling mode, FRUC mode, affine mode, etc., which are described below. Modes in which the MV difference is encoded include normal reciprocal mode, affine mode (in particular, affine reciprocal mode), etc. Modes in which the MV difference is not encoded include FRUC mode, normal concatenation mode, affine mode (specifically, affine concatenation mode), etc. The inter-prediction unit 126 selects a mode for extracting the MV of the current block from a plurality of modes, and extracts the MV of the current block using the selected mode.

[0234] Motion Vector Extraction Modes

Fig. 38A and 38B are conceptual diagrams for illustrating an exemplary categorization of modes for MV extraction. For example, as illustrated in FIG. 38A, the MV extraction modes are roughly categorized into three modes according to whether or not motion information should be encoded and whether or not MV differences should be encoded. The three modes are reciprocal mode, fusion mode, and frame rate upconversion (FRUC) mode. The reciprocal mode is a mode in which motion estimation is performed and in which motion information and MV difference are encoded. For example, as illustrated in FIG. 38B, the reciprocal mode includes an affine reciprocal mode and a normal reciprocal mode. The merging mode is a mode in which motion estimation is not performed, and in which the MV is selected from the encoded surrounding block, and the MV for the current block is extracted using the MV. The combining mode is a mode in which essentially motion information is encoded and the MV difference is not encoded. For example, as illustrated in FIG. 38B, the combining modes include normal combining mode (also called “normal combining mode” or “regular combining mode”), motion vector difference (MMVD) combining mode, combined inter combining and intra prediction (CIIP) mode, triangle mode, ATMVP mode and affine merging mode. Here, the MV difference is encoded exclusively in the MMVD mode from the modes included in the combining modes. It should be noted that affine union mode and affine reciprocal mode are modes included in affine modes. The affine mode is a mode for retrieving, as the MV of the current block, the MV of each of a plurality of sub-blocks included in the current block, subject to an affine transformation. The FRUC mode is a mode that serves to extract the MV of the current block by performing estimation between encoded regions, and in which neither motion information nor any MV difference is encoded. It should be noted that the corresponding modes are described in more detail below.

[0235] It should be noted that the categorization of modes illustrated in FIG. 38A and 38B are examples, and the categorization is not limited to this. For example, when the MV difference is encoded in the CIIP mode, the CIIP mode is categorized into mutual modes.

[0236] MV extract > normal mutual mode

The normal mutual mode is a mutual prediction mode for extracting the MV of the current block based on a block similar to the image of the current block from the reference frame area indicated by the MV candidate. In this normal reciprocal mode, the MV difference is encoded.

[0237] FIG. 39 is a flowchart illustrating an example of an inter-prediction process in a normal inter-mode.

[0238] First, the inter-prediction module 126 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sg_1). In other words, the inter-prediction module 126 generates a list of possible MV options.

[0239] Next, the inter-prediction module 126 extracts N (an integer of 2 or more) MV candidates from the plurality of MV candidates obtained in step Sg_1 as motion vector predictor candidates (also called “MV candidates”). predictors") according to a certain order of priorities (stage Sg_2). It should be noted that the priority order can be determined in advance for each of the N possible MV options.

[0240] Next, the inter-prediction unit 126 selects one motion vector predictor candidate from the motion vector predictor candidates N as the motion vector predictor (also called “MV predictor”) of the current block (step Sg_3). At this time, the inter-prediction unit 126 encodes, in the stream, motion vector predictor selection information to identify the selected motion vector predictor. In other words, the inter-prediction module 126 outputs the MV predictor selection information as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130.

[0241] Next, the inter-prediction unit 126 extracts the MV of the current block by referring to the encoded reference frame (step Sg_4). At this time, the inter-prediction unit 126 further encodes, in the stream, a difference value between the extracted MV and the motion vector predictor as an MV difference. In other words, the inter-prediction module 126 outputs the MV difference as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 . It should be noted that an encoded reference frame is a frame including a plurality of blocks that are reconstructed after encoding.

[0242] Finally, the inter prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Sg_5). Processes in stages Sg_1-Sg_5 are executed for each block. For example, when the processes in steps Sg_1 to Sg_5 are executed for all blocks in the slice, the slice mutual prediction using the normal mutual mode ends. For example, when the processes in steps Sg_1 to Sg_5 are executed for all blocks in a frame, frame mutual prediction using the normal mutual mode ends. It should be noted that not all blocks are included in the slice, processes may be subjected to steps Sg_1-Sg_5, and slice mutual prediction using normal reciprocal mode may end when a portion of the blocks are subject to processes. This also applies to processes in stages Sg_1-Sg_5. Frame reciprocal prediction using normal reciprocal mode may end when processes are running on a portion of the blocks in the frame.

[0243] It should be noted that the prediction image is a mutual prediction signal as described above. In addition, information indicating the inter prediction mode (normal inter mode in the above example) used to generate a prediction image, for example, is encoded as a prediction parameter in the encoded signal.

[0244] It should be noted that the list of possible MV options can also be used as a list for use in another mode. In addition, processes associated with a list of possible MV options can be applied to processes associated with a list for use in another mode. Processes associated with a list of MV candidates include, for example, retrieving or selecting a candidate MV from a list of candidate MVs, reordering the candidate MVs, or removing a candidate MV.

[0245] MV extract > normal merge mode

The normal pooling mode is a cross-prediction mode for selecting a candidate MV from a list of candidate MVs as the MV of the current block, thereby extracting the MV. It should be noted that the normal combining mode is a type of combining mode and may simply be referred to as "combining mode". In this embodiment, the normal combining mode and the combining mode are different, and the combining mode is used in a broader sense.

[0246] FIG. 40 is a flowchart illustrating an example of inter-prediction in the normal combining mode.

[0247] First, the inter-prediction module 126 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sh_1). In other words, the inter-prediction module 126 generates a list of possible MV options.

[0248] Next, the inter-prediction module 126 selects one candidate MV from the plurality of candidate MVs obtained in step Sh_1, thereby extracting the MV of the current block (step Sh_2). At this time, the inter-prediction module 126 encodes, in the stream, MV selection information to identify the selected MV candidate. In other words, the inter-prediction module 126 outputs the MV selection information as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 .

[0249] Finally, the inter prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Sh_3). The processes in stages Sh_1-Sh_3 are performed, for example, for each block. For example, when the processes in steps Sh_1-Sh_3 are executed for all blocks in the slice, the cross-prediction of the slice using the normal pooling mode ends. In addition, when the processes in steps Sh_1 to Sh_3 are executed for all blocks in a frame, frame mutual prediction using the normal combining mode ends. It should be noted that not all blocks included in the slice may be subject to processes in steps Sh_1-Sh_3, and cross-prediction of the slice using the normal pooling mode may end when a portion of the blocks are subject to processes. This also applies to processes in stages Sh_1-Sh_3. Mutual frame prediction using normal merging mode may end when processes are running on a portion of the blocks in the frame.

[0250] In addition, information indicating the inter-prediction mode (normal combining mode in the above example) used to generate the prediction image and included in the encoded signal, for example, is encoded as a prediction parameter in the stream.

[0251] FIG. 41 is a conceptual diagram for illustrating one example of a process for extracting motion vectors of the current frame by the normal combining mode.

[0252] First, the inter-prediction module 126 generates a list of possible MV options in which the possible MV options are registered. Examples of MV candidates include: spatially adjacent MV candidates, which are MVs of a plurality of coded blocks located spatially surrounding the current block; temporarily adjacent MV candidates, which are the MVs of surrounding blocks onto which the position of the current block in the encoded reference frame is projected; combined candidate MVs, which are MVs formed by combining the MV value of a spatially adjacent MV predictor and the MV value of a temporally adjacent MV predictor; and a null candidate MV, which is an MV having a value of zero.

[0253] Next, the inter-prediction unit 126 selects one MV candidate from the plurality of MV candidates registered in the MV candidate list, and determines the MV candidate as the MV of the current block.

[0254] In addition, entropy encoder 110 records and encodes, on stream, merge_idx, which is a signal indicating which MV candidate is selected.

[0255] It should be noted that the MV candidate candidates registered in the MV candidate list described in FIG. 41 are examples. The number of possible MV options may be different from the number of possible MV options in the diagram, the list of possible MV options may be configured so that some kinds of possible MV options in the diagram may not be included, or so that one or more possible MVs -options other than the types of possible MV options in the diagram are included.

[0256] The final MV may be determined by performing dynamic motion vector updating (DMVR), which is described below, using the MV of the current block extracted through the normal merging mode. It should be noted that in the normal combining mode, motion information is encoded, but the MV difference is not encoded. In MMVD mode, one possible MV variant is selected from a list of possible MV variants, as in the case of normal combining mode, the MV difference is encoded. As illustrated in FIG. 38B, MMVD can be categorized into combining modes along with a normal combining mode. It should be noted that the MV difference in MMVD mode does not always have to be identical to the MV difference for use in reciprocal mode. For example, retrieving MV differences in MMVD mode may be a process that requires less processing than the amount of processing required for retrieving MV differences in reciprocal mode.

[0257] In addition, a combined inter-combining and intra-prediction (CIIP) mode may be performed. The mode serves to overlap the prediction image generated by inter-prediction and the prediction image generated by intra-prediction to generate a prediction image for the current block.

[0258] It should be noted that the list of MV candidate options may be referred to as a “candidate list.” In addition, merge_idx represents MV selection information.

[0259] MV extraction > HMVP mode

Fig. 42 is a conceptual diagram for illustrating one example of an MV extraction process for a current frame using the HMVP combining mode.

[0260] In normal combining mode, the MV for, for example, a CU that represents the current block is determined by selecting one candidate MV from an MV list generated by referring to a coded block (eg, the CU). Here, another possible MV option may be registered in the list of possible MV options. The mode in which such other possible MV variant is registered is called "HMVP mode".

[0261] In HMVP mode, MV candidates are managed using a first-in, first-out (FIFO) server for HMVP, separate from the list of MV candidates for normal aggregation mode.

[0262] In the FIFO buffer, motion information such as MV of blocks processed previously is stored "newest first". When managing the FIFO buffer, each time one block is processed, the MV for the newest block (which is the CU processed immediately before) is stored in the FIFO buffer, and the MV of the oldest CU (i.e. the CU processed earlier) total) is removed from the FIFO buffer. In the example illustrated in FIG. 42, HMVP1 is the MV for the newest block, and HMVP5 is the MV for the oldest MV.

[0263] The inter-prediction module 126 then, for example, checks whether each MV managed in the FIFO buffer is or is not an MV different from all possible MVs that are already registered in the list of possible MVs for the normal mode. association starting with HMVP1. When determining that the MV is different from all possible MV cases, the inter-prediction module 126 may add the MV managed in the FIFO buffer to the list of possible MV cases for the normal merge mode as a possible MV case. At this time, one or more MV candidates in the FIFO buffer may be registered (added to the list of MV candidates).

[0264] By using the HMVP mode in this manner, it is possible to add not only the MV of a block that is spatially or temporally adjacent to the current block, but also the MV for a block that was processed previously. As a result, the variety of possible MV options for the normal combining mode is expanded, which increases the possibility that coding efficiency can be improved.

[0265] It should be noted that the MV may be motion information. In other words, the information stored in the MV candidate list and the FIFO buffer may include not only the MV values, but also information about reference frames, reference directions, frame numbers, etc. In addition, the block may, for example, be a CU.

[0266] It should be noted that the MV candidate list and FIFO buffer illustrated in FIG. 42 are examples. The MV candidate list and FIFO buffer may be different in size from the MV candidate list and FIFO buffer in FIG. 42 or may be configured to register possible MV options in an order different from that in FIG. 42. In addition, the process described herein may be shared between encoder 100 and decoder 200.

[0267] It should be noted that the HMVP mode can be used for modes other than the normal combining mode. For example, it is also possible that motion information, such as MVs of blocks processed in affine mode previously, can be stored “newest first” and can be used as candidate MVs that can contribute to better performance. The mode obtained by applying the HMVP mode to the affine mode may be called a "history-based affine mode".

[0268] MV extraction > FRUC mode

Motion information can be extracted at the decoder side without being signaled from the encoder side. For example, motion information may be extracted by performing motion estimation on the decoder 200 side. In an embodiment, on the decoder side, motion estimation is performed without using pixel values in the current block. Modes for performing motion estimation on the decoder 200 side without using the pixel value in the current block at all include Frame Rate Upconversion (FRUC) mode, Pattern Matching Based Motion Vector Extraction (PMMVD) mode, etc.

[0269] One example of a FRUC process in flowchart form is illustrated in FIG. 43. First, a list that specifies, as possible MV options, MVs for encoded blocks, each of which is adjacent to the current block spatially or temporally by referring to the MV (the list may be a list of possible MV options and can also be used as a list of possible MV options for the normal combining mode) (step Si_1).

[0270] Next, the best possible MV option is selected from the plurality of possible MV options registered in the list of possible MV options (step Si_2). For example, the evaluation values of the corresponding MV candidates included in the MV candidate list are calculated, and one MV candidate is selected based on the evaluation values. Based on the selected motion vector candidates, the motion vector for the current block is then extracted (step Si_4). More specifically, for example, the selected motion vector candidate (the best possible MV candidate) is retrieved directly as the motion vector for the current block. In addition, for example, a motion vector for the current block may be retrieved using pattern matching in the surrounding region of a position in a reference frame, wherein the position in the reference frame corresponds to a selected motion vector candidate. In other words, scoring using template matching and scoring values can be performed on the surrounding region of the best possible MV, and when there is an MV that results in a better scoring value, the best possible MV can be updated to the MV that results in the best estimate value, and the updated MV can be determined as the final MV for the current block. In some embodiments, updating the motion vector that results in a better estimate value may not be performed.

[0271] Finally, the inter prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the encoded reference frame (step Si_5). Processes in stages Si_1-Si_5 are performed, for example, for each block. For example, when the processes in steps Si_1-Si_5 are executed for all blocks in the slice, the cross-prediction of the slice using FRUC mode ends. For example, when the processes in steps Si_1-Si_5 are executed for all blocks in a frame, frame inter-prediction using FRUC mode ends. It should be noted that not all blocks included in the slice may be subject to processes in steps Si_1-Si_5, and cross-prediction of the slice using FRUC mode may end when a portion of the blocks are subject to processes. When the processes in steps Si_1-Si_5 are executed for a portion of the blocks included in the frame in a similar manner, mutual frame prediction using the FRUC mode may end.

[0272] A similar process can be performed in subblock units.

[0273] The evaluation values may be calculated according to various kinds of methods. For example, a comparison is made between a reconstructed image in a region in a reference frame corresponding to a motion vector and a reconstructed image in a specific region (the region, for example, may be a region in another reference frame or a region in an adjacent block of the current frame, as defined below). The specific area can be pre-defined.

[0274] The difference between the pixel values of the two reconstructed images can be used for the motion vector estimation value. It should be noted that the score value may be calculated using information other than the difference value.

[0275] An example of pattern matching is described in detail next. First, one MV candidate included in the list of MV candidate candidates (eg, a merge list) is selected as the starting evaluation point through template matching. For example, as a pattern match, a first pattern match or a second pattern match may be used. The first template matching and the second template matching may be referred to as "bilateral matching" and "template matching", respectively.

[0276] MV extraction > FRUC > bilateral matching

During the first pattern matching, pattern matching is performed between two blocks that are located along the current block's motion path and are included in two different reference frames. Accordingly, in the first template matching, a region in another reference frame along the motion path of the current block is used as a determined region for calculating the evaluation value of the above-described candidate. The specific area can be pre-defined.

[0277] FIG. 44 is a conceptual diagram for illustrating one example of first template matching (bilateral matching) between two blocks in two reference frames along a motion path. As illustrated in FIG. 44, in the first template matching, two motion vectors (MV ₀ , MV ₁ ) are extracted by evaluating the pair that has the best match from among the pairs in two blocks included in two different reference frames (Ref ₀ , Ref ₁ ) and located along the trajectory of the current block (Cur block). More specifically, the difference between the reconstructed image at a specified location in the first coded reference frame (Ref ₀ ) indicated by the MV option, and the reconstructed image at the specified location in the second coded reference frame (Ref ₁ ) indicated by the symmetric MV, obtained by scaling the MV candidate in the display time interval is retrieved for the current block, and a score value is calculated using the resulting difference value. One can choose, as the final MV, a possible MV option that results in the best evaluation value from the set of possible MV options and which is likely to lead to good results.

[0278] Assuming a continuous motion path, the motion vectors (MV ₀ , MV ₁ ) indicating the two reference blocks are proportional to the time distances (TD ₀ , TD ₁ ) between the current frame (Cur Pic) and the two reference frames (Ref ₀ ,Ref ₁ ). For example, when the current frame is temporarily between two reference frames, and the temporal distances from the current frame to the corresponding two reference frames are equal to each other, mirror-symmetric bidirectional motion vectors are extracted in the first template matching.

[0279] MV extraction > FRUC > reference matching

In the second pattern matching (reference matching), pattern matching is performed between a block in the reference frame and a reference in the current frame (the reference is the block adjacent to the current block in the current frame (the adjacent block, for example, represents the top and/or left adjacent block(s)). Therefore, in the second pattern matching, a block adjacent to the current block in the current frame is used as a predetermined region for the above-described calculation of the MV candidate evaluation value.

[0280] FIG. 45 is a conceptual diagram for illustrating one example of pattern matching (pattern matching) between a pattern in the current frame and a block in the reference frame. As illustrated in FIG. 45, in the second template matching, the motion vector of the current block (Cur block) is extracted by estimating, in the reference frame (Ref ₀ ), the block that has the best match to the block adjacent to the current block in the current frame (Cur Pic). More specifically, the difference between a reconstructed image in a coded region that borders both left and above or either left or above, and a reconstructed image that is in a corresponding region in a coded reference frame (Ref0) and is indicated by an MV candidate, is extracted and the score value is calculated using the resulting difference value. The MV candidate that results in the best score value from the plurality of possible MV candidates may be selected as the best possible MV candidate.

[0281] This information indicating whether or not to use FRUC mode (called, for example, a “FRUC flag”) may be signaled at the CU level. In addition, when the FRUC mode is applied (eg, when the FRUC flag is true), information indicating the applicable pattern matching method (eg, first pattern matching or second pattern matching) may be signaled at the CU level. It should be noted that signaling of this information does not necessarily have to be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, CTU level, or subblock level).

[0282] MV extraction > affine mode

Affine mode is a mode for generating MVs using an affine transformation. For example, the MV may be extracted in units of sub-blocks based on the motion vectors of multiple neighboring blocks. This mode is also called "motion compensated affine prediction mode".

[0283] FIG. 46A is a conceptual diagram for illustrating one example of MV extraction in units of sub-blocks based on motion vectors of multiple neighboring blocks. In fig. 46A, the current block includes, for example, sixteen 4x4 subblocks. Here, the motion vector V ₀ at the upper left corner control point in the current block is extracted based on the motion vector of the adjacent block, and similarly, the motion vector V ₁ at the upper right corner control point in the current block is extracted based on the motion vector of the adjacent sub-block. The two motion vectors v ₀ and v ₁ may be projected according to the expression (1A) below, and the motion vectors (v _x , v _y ) for the corresponding sub-blocks in the current block can be extracted.

[0284] Mathematical expression 1

[0285] Here, x and y indicate the horizontal position and vertical position of the subblock, respectively, and w indicates a certain weighting factor. The specific weighting factor may be predetermined.

[0286] This information indicating the affine mode (eg, called an "affine flag") may be signaled at the CU level. It should be noted that the signaling of information indicating the affine mode need not be performed at the CU level and may be performed at another level (eg, sequence level, frame level, slice level, tile level, CTU level, or subblock level).

[0287] In addition, the affine mode may include multiple modes for different methods for extracting motion vectors at the top left and top right corner control points. For example, the affine mode includes two modes, which are the affine reciprocal mode (also called the "affine normal reciprocal mode") and the affine union mode.

[0288] MV extraction > affine mode

Fig. 46B is a conceptual diagram for illustrating one example of MV extraction in subblock units in affine mode, in which three control points are used. In fig. 46B, the current block includes, for example, sixteen 4×4 blocks. Here, the motion vector v ₀ at the top left corner control point in the current block is extracted based on the motion vector of the adjacent block. Here, the motion vector V ₁ at the upper right corner control point in the current block is extracted based on the motion vector of the adjacent block, and similarly, the motion vector V ₂ at the lower left corner control point for the current block is extracted based on the motion vector of the adjacent block. The three motion vectors v ₀ , v ₁ and v ₂ can be projected according to expression (1B) below, and the motion vectors (v _x , v _y ) for the corresponding sub-blocks in the current block can be extracted.

[0289] Mathematical expression 2

[0290] Here, x and y indicate the horizontal position and vertical position of the sub-block, respectively, and w and h may be weights that may constitute predetermined weights. In an embodiment, w may indicate the width of the current block, and h may indicate the height of the current block.

[0291] Affine modes that use different numbers of control points (eg, two and three control points) can be switched and signaled at the CU level. It should be noted that information indicating the number of control points in the affine mode used at the CU level may be signaled at another level (eg, sequence level, frame level, slice level, tile level, CTU level, or subblock level).

[0292] In addition, such an affine mode that uses three control points may include various methods for extracting motion vectors at the top left, top right, and bottom left corner control points. For example, affine modes that use three control points may include two modes that are an affine mutual mode and an affine union mode, as is the case for affine modes that use two control points.

[0293] It should be noted that, in affine modes, the size of each sub-block included in the current block may not be limited to 4x4 pixels and may be a different size. For example, the size of each sub-block may be 8x8 pixels.

[0294] MV Extract > Affine Mode > Control Point

Fig. 47A, fig. 47B and FIG. 47C are conceptual diagrams for illustrating examples of MV extraction at control points in affine mode.

[0295] As illustrated in FIG. 47A, in the affine mode, for example, motion vector predictors at the corresponding control points of the current block are calculated based on a plurality of motion vectors corresponding to blocks coded according to the affine mode from coded block A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) that are adjacent to the current block. More specifically, the encoded block A (left), block B (upper), block C (upper right), block D (lower left) and block E (upper left) are checked in the listed order, and the first effective block encoded according to the affine mode , is identified. Motion vector predictors at control points of the current block are calculated based on the set of motion vectors corresponding to the identified block.

[0296] For example, as illustrated in FIG. 47B, when block A, which is adjacent to the left of the current block, is encoded according to an affine mode that uses two control points, motion vectors v ₃ and v ₄ , projected at the upper left corner position and upper right corner position of the encoded block including block A itself is extracted. The motion vector v ₀ at the top left corner control point of the current block and the motion vector v ₁ at the top right corner control point of the current block are then calculated from the extracted motion vectors v ₃ and v ₄ .

[0297] For example, as illustrated in FIG. 47C, when block A, which is adjacent to the left of the current block, is encoded according to an affine mode that uses three control points, motion vectors v ₃ , v ₄ and v ₅ projected at the top left corner position, top right corner position and bottom the left corner position of the coded block including block A is retrieved. The motion vector v ₀ at the top left corner control point of the current block, the motion vector v ₁ at the top right corner control point of the current block, and the motion vector v ₂ at the bottom left corner control point of the current block are then calculated from the extracted vectors v ₃ , v ₄ , and v ₅ movements.

[0298] The MV extraction methods illustrated in FIG. 47A-47C may be used in MV extraction at each control point for the current block in step Sk_1 illustrated in FIG. 50, or may be used to extract an MV predictor at each control point for the current block in step Sj_1 illustrated in FIG. 51 described below.

[0299] FIG. 48A and 48B are conceptual diagrams for illustrating examples of MV extraction at control points in affine mode.

[0300] FIG. 48A is a conceptual diagram for illustrating an example affine mode in which two control points are used.

[0301] In affine mode, as illustrated in FIG. 48A, the MV selected from the MVs in the coded block A, block B and block C that adjoin the current block are used as the motion vector v ₀ at the upper left corner control point for the current block. Likewise, an MV selected from the MVs of the coded block D and block E that adjoin the current block is used as the motion vector v ₁ at the upper right corner control point for the current block.

[0302] FIG. 48B is a conceptual diagram for illustrating an example affine mode that uses three control points.

[0303] In affine mode, as illustrated in FIG. 48B, the MV selected from the MVs in the coded block A, block B and block C that adjoin the current block are used as the motion vector v ₀ at the upper left corner control point for the current block. Likewise, an MV selected from the MVs of the coded block D and block E that adjoin the current block is used as the motion vector v ₁ at the upper right corner control point for the current block. In addition, an MV selected from the MVs of a coded block F and a block G that adjoin the current block is used as a motion vector v ₂ at the lower left corner control point for the current block.

[0304] It should be noted that the MV extraction methods illustrated in FIGS. 48A and 48B may be used in MV extraction at each control point for the current block in step Sk_1 illustrated in FIG. 50 described below, or may be used to extract the MV predictor at each control point for the current block in step Sj_1 illustrated in FIG. 51 described below.

[0305] Here, when affine modes that use different numbers of control points (eg, two and three control points) can be switched and signaled at the CU level, the number of control points for the coded block and the number of control points for the current block may differ from each other.

[0306] FIG. 49A and 49B are conceptual diagrams for illustrating examples of a method for MV extraction at control points when the number of control points for a coded block and the number of control points for a current block are different from each other.

[0307] For example, as illustrated in FIG. 49A, the current block has three control points at the upper left corner, the upper right corner and the lower left corner, and block A, which is adjacent to the left of the current block, is encoded according to an affine mode in which two control points are used. In this case, the motion vectors v ₃ and v ₄ projected at the upper left corner position and the upper right corner position in the coded block including block A are extracted. The motion vector v ₀ at the top left corner control point and the motion vector v ₁ at the top right corner control point for the current block are then calculated from the extracted motion vectors v ₃ and v ₄ . In addition, the motion vector v ₂ at the lower left corner control point is calculated from the extracted motion vectors v ₀ and v ₁ .

[0308] For example, as illustrated in FIG. 49B, the current block has two control points in the upper left corner and in the upper right corner, and block A, which is adjacent to the left of the current block, is encoded according to an affine mode in which three control points are used. In this case, motion vectors v ₃ , v ₄ and v ₅ projected at the upper left corner position in the coded block including block A, at the upper right corner position in the coded block and at the lower left corner position in the coded block are extracted . The motion vector v ₀ at the top left corner control point for the current block and the motion vector v ₁ at the top right corner control point for the current block are then calculated from the extracted motion vectors v ₃ , v ₄ and v ₅ .

[0309] It should be noted that the MV extraction methods illustrated in FIGS. 49A and 49B may be used in MV extraction at each control point for the current block in step Sk_1 illustrated in FIG. 50 described below, or may be used to extract the MV predictor at each control point for the current block in step Sj_1 illustrated in FIG. 51 described below.

[0310] MV extract > affine mode > affine join mode

Fig. 50 is a flowchart illustrating one example of a process in the affine combining mode.

[0311] In the affine combining mode, as illustrated first, the inter-prediction module 126 extracts MVs at the corresponding control points for the current block (step Sk_1). The control points are the top left corner point of the current block and the top right corner point of the current block, as illustrated in FIG. 46A, or the top left corner point of the current block, the top right corner point of the current block, and the bottom left corner point of the current block, as illustrated in FIG. 46B. The inter-prediction module 126 may encode MV selection information to identify two or three extracted MVs in a stream.

[0312] For example, when the MV extraction methods illustrated in FIG. 47A-47C are used as illustrated in FIGS. 47A, the inter-prediction module 126 examines the coded block A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) in the listed order and identifies the first effective block coded according to the affine regime.

[0313] Inter prediction module 126 extracts the MV at the control point using the identified first effective block encoded according to the identified affine mode. For example, when block A is identified and block A has two control points, as illustrated in FIG. 47B, the mutual prediction unit 126 calculates a motion vector v ₀ at the upper left corner control point of the current block and a motion vector v ₁ at the upper right corner control point of the current block from the motion vectors v ₃ and v ₄ at the upper left corner of the coded block including block A, and in the upper right corner of the coded block. For example, the mutual prediction module 126 calculates the motion vector v ₀ at the top left corner control point of the current block and the motion vector v ₁ at the top right corner control point of the current block by projecting the motion vectors v ₃ and v ₄ into the top left corner and the top right corner coded block to the current block.

[0314] Alternatively, when block A is identified and block A has three control points, as illustrated in FIG. 47C, the mutual prediction unit 126 calculates the motion vector v ₀ at the top left corner control point of the current block, the motion vector v ₁ at the top right corner control point of the current block, and the motion vector v ₂ at the bottom left corner control point of the current block from the vectors v ₃ . v ₄ and v ₅ movements in the upper left corner of the coded block including block A, in the upper right corner of the coded block and in the lower left corner of the coded block. For example, the mutual prediction module 126 calculates the motion vector v ₀ at the top left corner control point of the current block, the motion vector v ₁ at the top right corner control point of the current block, and the motion vector v ₂ at the bottom left corner control point of the current block by projecting the vectors v ₃ , v ₄ and v ₅ moves in the top left corner, top right corner and bottom left corner of the coded block onto the current block.

[0315] It should be noted that, as illustrated in FIG. 49A described above, MVs at three control points can be calculated when block A is identified and block A has two control points, and that, as illustrated in FIG. 49B described above, MVs at two control points can be calculated when block A is identified and block A has three control points.

[0316] Next, the inter-prediction module 126 performs motion compensation of each of the plurality of sub-blocks included in the current block. In other words, the inter-prediction unit 126 calculates an MV for each of a plurality of sub-blocks as an affine MV, for example, using two motion vectors v ₀ and v ₁ and the above expression (1A) or three motion vectors v ₀ , v ₁ and v ₂ and the above expression (1B) (step Sk_2). The inter-prediction module 126 then performs motion compensation of the sub-blocks using these affine MVs and coded reference frames (step Sk_3). When the processes in steps Sk_2 and Sk_3 are executed for each of all sub-blocks included in the current block, the process for generating a prediction image using the affine fusion mode for the current block ends. In other words, motion compensation of the current block is performed in order to generate a predictive image of the current block.

[0317] It should be noted that the above-described list of possible MV options can be generated at step Sk_1. The MV candidate list, for example, may be a list including MV candidate candidates extracted using a plurality of MV extraction methods for each control point. The plurality of MV extraction methods, for example, may be any combination of the MV extraction methods illustrated in FIG. 47A-47C, the MV extraction methods illustrated in FIGS. 48A and 48B, the MV extraction methods illustrated in FIGS. 49A and 49B, and other MV extraction methods.

[0318] It should be noted that the MV candidate lists may include MV candidate candidates in a mode in which prediction is performed in subblock units other than the affine mode.

[0319] It should be noted that, for example, a list of possible MV options including possible MV options in the affine pooling mode in which two control points are used and in the affine combining mode in which three control points are used may be generated as a list of possible MV options. Alternatively, a list of possible MV options including possible MV options in an affine join mode that uses two control points, and a list of possible MV options including possible MV options in an affine join mode that uses three control points can be formed separately. Alternatively, a list of possible MV options including possible MV options in one of an affine join mode that uses two control points and an affine join mode that uses three control points can be generated. The possible MV variant(s), for example, could be MV for coded block A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) or MV for an efficient block of blocks.

[0320] It should be noted that an index indicating one of the MVs in the list of possible MV options may be transmitted as MV selection information.

[0321] MV Extract > Affine Mode > Affine Reciprocal Mode

Fig. 51 is a flowchart illustrating one example of an affine reciprocal mode process.

[0322] In the affine mutual mode, as illustrated, first, the mutual prediction module 126 extracts the MV predictors (v ₀ , v ₁ ) or (v ₀ , v ₁ , v ₂ ) of the corresponding two or three control points of the current block (step Sj_1 ). The control points, for example, may be the top left corner point for the current block, the top right corner point of the current block, and the top right corner point for the current block, as illustrated in FIG. 46A or FIG. 46B.

[0323] For example, when the MV extraction methods illustrated in FIG. 48A and 48B are used, the inter-prediction module 126 retrieves the MV predictors (v ₀ , v ₁ ) or (v ₀ , v ₁ , v ₂ ) at the corresponding two or three control points for the current block by selecting the MVs of any of the encoded blocks blocks near the corresponding control points for the current block illustrated in FIG. 48A or FIG. 48B. At this time, the inter-prediction module 126 encodes, in-stream, MV predictor selection information to identify the selected two or three MV predictors.

[0324] For example, the inter-prediction module 126 may determine, using a cost estimate or the like, a block from which an MV is selected as a control point MV predictor from among the coded blocks adjacent to the current block, and may record , in the bit stream, a flag indicating which MV predictor is selected. In other words, the inter-prediction module 126 outputs, as a prediction parameter, MV predictor selection information such as a flag to the entropy encoder 110 via the prediction parameter generator 130.

[0325] Next, the mutual prediction unit 126 performs motion estimation (step Sj_3 and Sj_4) while updating the MV vector predictor selected or extracted in step Sj_1 (step Sj_2). In other words, the inter-prediction unit 126 calculates, as an affine MV, the MV of each of the sub-blocks that corresponds to the updated MV predictor using expression (1A) or expression (1B) described above (step Sj_3). The inter-prediction module 126 then performs motion compensation of the sub-blocks using these affine MVs and coded reference frames (step Sj_4). The processes in step Sj_3 and Sj_4 are executed for all blocks in the current block when the MV predictor is updated in step Sj_2. As a result, for example, the mutual prediction unit 126 determines the MV predictor that results in the lowest cost as the MV at the control point in the motion estimation loop (step Sj_5). At this time, the inter-prediction unit 126 further encodes, in the stream, the difference value between the determined MV and the MV predictor as an MV difference. In other words, the inter-prediction module 126 outputs the MV difference as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 .

[0326] Finally, the inter prediction unit 126 generates a prediction image for the current block by performing motion compensation of the current block using the determined MV and the encoded reference frame (step Sj_6).

[0327] It should be noted that the above-described list of possible MV options may be generated in step Sj_1. The MV candidate list, for example, may be a list including MV candidate candidates extracted using a plurality of MV extraction methods for each control point. The plurality of MV extraction methods, for example, may be any combination of the MV extraction methods illustrated in FIG. 47A-47C, the MV extraction methods illustrated in FIGS. 48A and 48B, the MV extraction methods illustrated in FIGS. 49A and 49B, and other MV extraction methods.

[0328] It should be noted that the MV candidate lists may include MV candidate candidates in a mode in which prediction is performed in subblock units other than the affine mode.

[0329] It should be noted that, for example, a list of possible MV options including possible MV options in the affine reciprocal mode in which two control points are used and in the affine reciprocal mode in which three control points are used may be generated as a list of possible MV options. Alternatively, a list of possible MV options including possible MV options in an affine reciprocal mode that uses two control points, and a list of possible MV options including possible MV options in an affine reciprocal mode that uses three control points can be formed separately. Alternatively, a list of possible MV options including possible MV options in one of an affine reciprocal mode in which two control points are used and an affine reciprocal mode in which three control points are used may be generated. The possible MV variant(s), for example, could be MV for coded block A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) or MV for an efficient block of blocks.

[0330] It should be noted that an index indicating one of the possible MV options in the list of possible MV options may be transmitted as MV predictor selection information.

[0331] MV extract > triangle mode

The inter-prediction module 126 generates one rectangular prediction image for the current rectangular block in the above example. However, the inter-prediction module 126 can generate a plurality of prediction images having a shape other than a rectangle for the current rectangular block, and can combine the plurality of prediction images to generate a final rectangular prediction image. A shape other than a rectangle may, for example, be a triangle.

[0332] FIG. 52A is a conceptual diagram for illustrating the formation of two triangular projection images.

[0333] The inter-prediction unit 126 generates a triangular prediction image by performing motion compensation of the first segment having a triangular shape in the current block by using the first MV of the first segment to generate the triangular prediction image. Likewise, the inter-prediction unit 126 generates a triangular prediction image by performing motion compensation of the second segment having a triangular shape in the current block by using the second MV of the second segment to generate the triangular prediction image. The mutual prediction unit 126 then generates a prediction image having a rectangular shape identical to the rectangular shape of the current block by combining these prediction images.

[0334] It should be noted that the first prediction image having a rectangular shape corresponding to the current block may be generated as a prediction image for the first segment using the first MV. In addition, a second prediction image having a rectangular shape corresponding to the current block may be generated as a prediction image for the second segment using the second MV. The prediction image for the current block may be generated by performing weighting summation of the first prediction image and the second prediction image. It should be noted that the portion that is subject to weighting summation may be a partial region at the boundary between the first segment and the second segment.

[0335] FIG. 52B is a conceptual diagram for illustrating examples of a first portion of a first segment that overlaps with a second segment, and first and second sets of samples that may be weighted as part of the correction process. The first part, for example, may be one quarter of the width or height of the first segment. In another example, the first portion may have a width corresponding to N samples adjacent to the edge of the first segment, where N is an integer greater than zero, for example, N may be the integer 2. As illustrated, the left example of FIG. 52B shows a rectangular segment having a rectangular portion with a width that is one quarter of the width of the first segment, with a first set of samples including samples outside the first portion and samples within the first portion, and a second set of samples including samples within the first portion. parts. The central example of FIG. 52B shows a rectangular segment having a rectangular portion with a height that is one-quarter the height of the first segment, with a first set of samples including samples outside the first portion and samples within the first portion, and a second set of samples including samples within the first portion. parts. The right example in Fig. 52B shows a triangular segment having a polygonal portion with a height that corresponds to two samples, with a first set of samples including samples outside the first portion and samples inside the first portion, and a second set of samples including samples inside the first portion.

[0336] The first portion may constitute a portion of the first segment that overlaps with an adjacent segment. Fig. 52C is a conceptual diagram for illustrating a first part of a first segment, which constitutes a part of the first segment that overlaps with a part of an adjacent segment. For ease of illustration, a rectangular segment is shown having an overlapping portion with a spatially adjacent rectangular segment. Segments having other shapes, for example, triangular segments, can be used, and overlapping portions can overlap with a spatially adjacent or temporally adjacent segment.

[0337] In addition, although an example is given in which a prediction image is generated for each of two segments using inter prediction, a prediction image may be generated for at least one segment using intra prediction.

[0338] FIG. 53 is a flowchart illustrating one example of a triangular mode process.

[0339] In the triangular mode, first, the inter-prediction unit 126 splits the current block into a first segment and a second segment (step Sx_1). At this time, the inter-prediction unit 126 may encode, in the stream, segment information, which is segmentation-related information, as a prediction parameter. In other words, the inter-prediction module 126 may output the segment information as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 .

[0340] First, inter-prediction module 126 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sx_2). In other words, the inter-prediction module 126 generates a list of possible MV options.

[0341] The inter-prediction module 126 then selects a candidate MV for the first segment and a candidate MV for the second segment as the first MV and the second MV, respectively, from the plurality of candidate MVs obtained in step Sx_1 (step Sx_3). At this time, the inter-prediction module 126 encodes, in the stream, MV selection information to identify the selected MV candidate. In other words, the inter-prediction module 126 outputs the MV selection information as a prediction parameter to the entropy encoder 110 via the prediction parameter generator 130 .

[0342] Next, the inter prediction unit 126 generates a first prediction image by performing motion compensation using the selected first MV and the encoded reference frame (step Sx_4). Likewise, the inter-prediction module 126 generates a second prediction image by performing motion compensation using the selected second MV and the encoded reference frame (step Sx_5).

[0343] Finally, the inter-prediction unit 126 generates a prediction image for the current block by performing weighting summation of the first prediction image and the second prediction image (step Sx_6).

[0344] It should be noted that although the first segment and the second segment are triangles in the example illustrated in FIG. 52A, the first segment and the second segment may be trapezoidal or other shapes different from each other. In addition, although the current block includes two segments in the examples illustrated in FIGS. 52A and 52C, the current block may include three or more segments.

[0345] In addition, the first segment and the second segment may overlap each other. In other words, the first segment and the second segment may include an identical pixel region. In this case, the prediction image for the current block may be generated using the prediction image in the first segment and the prediction image in the second segment.

[0346] In addition, although an example is illustrated in which a prediction image is generated for each of two segments using inter prediction, a prediction image may be generated for at least one segment using intra prediction.

[0347] It should be noted that the list of possible MV options for selecting the first MV and the list of possible MV options for selecting the second MV may be different from each other, or the list of possible MV options for selecting the first MV may also be used as the list of possible MVs -options for choosing the second MV.

[0348] It should be noted that the segment information may include an index indicating a partitioning direction in which at least the current block is partitioned into a plurality of segments. The MV selection information may include an index indicating the selected first MV and an index indicating the selected second MV. A single index can specify many pieces of information. For example, one index jointly indicating part or all of the segment information and part or all of the MV selection information may be encoded.

[0349] MV extraction > ATMVP mode

Fig. 54 is a conceptual diagram for illustrating one example of an advanced temporal motion vector prediction (ATMVP) mode in which the MV is extracted in units of subblocks.

[0350] The ATMVP mode is a mode categorized as an aggregation mode. For example, in ATMVP mode, a candidate MV for each subblock is registered in the list of candidate MVs for use in normal aggregation mode.

[0351] More specifically, in ATMVP mode, first, as illustrated in FIG. 54, a temporary reference MV block associated with the current block is identified in a coded reference frame indicated by the MV (MV0) of a neighboring block located at the lower left position relative to the current block. Then, in each sub-block in the current block, an MV used to encode the region corresponding to the sub-block in the temporary reference MV block is identified. The MV identified in this manner is included in the list of MV candidates as a candidate MV for the subblock in the current block. When a candidate MV for each sub-block is selected from the list of candidate MVs, the sub-block undergoes motion compensation in which the candidate MV is used as the MV for the sub-block. Thus, a forecast image for each subblock is formed.

[0352] Although the block located at the lower left position relative to the current block is used as the surrounding MV reference block in the example illustrated in FIG. 54, it should be noted that a different block may be used. In addition, the subblock size may be 4x4 pixels, 8x8 pixels, or other size. The sub-block size can be switched per unit, such as slice, brick, frame, etc.

[0353] MV extract > DMVR

Fig. 55 is a flowchart illustrating the relationship between the combining mode and decoder motion vector detail (DMVR).

[0354] The inter-prediction unit 126 extracts the motion vector for the current block according to the combining mode (step Sl_1). Next, the mutual prediction unit 126 determines whether or not to perform motion vector estimation, i.e. motion assessment (stage Sl_2). Here, when determining whether to not perform motion estimation (“No” in step Sl_2), the inter-prediction unit 126 determines the motion vector extracted in step Sl_1 as the final motion vector for the current block (step Sl_4). In other words, in this case, the motion vector of the current block is determined according to the merging mode.

[0355] When determining whether to perform motion estimation in step Sl_1 (Yes in step Sl_2), the inter-prediction unit 126 extracts the final motion vector for the current block by estimating the surrounding area of the reference frame indicated by the motion vector extracted in step Sl_1 (step Sl_3). In other words, in this case, the motion vector of the current block is determined according to the DMVR.

[0356] FIG. 56 is a conceptual diagram for illustrating one example of a DMVR process for determining MV.

[0357] First, in merge mode, for example, candidate MVs (L0 and L1) are selected for the current block. The reference pixel is identified from the first reference frame (L0), which is a coded frame in the L0 list according to the MV candidate (L0). Likewise, a reference pixel is identified from a second reference frame (L1), which is a coded frame in the L1 list according to the MV candidate (L1). The reference is formed by calculating the average of these reference pixels.

[0358] Next, each of the surrounding regions of candidate MVs of the first reference frame (L0) and the second reference frame (L1) is estimated using the reference, and the MV that results in the lowest cost is determined as the final MV. It should be noted that costs can be calculated, for example, using the difference value between each of the pixel values in the reference and the corresponding one of the pixel values in the evaluation area, the values of the MV candidate values, etc.

[0359] The completely identical processes described here do not always have to be performed. Another process for allowing the final MV to be extracted by evaluating candidate MVs in surrounding areas can be used.

[0360] FIG. 57 is a conceptual diagram for illustrating another example of a DMVR for determining MV. Unlike the DMVR example illustrated in FIG. 56, in the example illustrated in FIG. 57, costs are calculated without forming a standard.

[0361] First, the inter-prediction module 126 estimates the surrounding area of the reference block included in each of the reference frames in the L0 list and the L1 list based on the initial MV, which is a candidate MV obtained from each candidate list. MV options. For example, as illustrated in FIG. 57, the initial MV corresponding to the reference block in the L0 list is InitMV_L0, and the initial MV corresponding to the reference block in the L1 list is InitMV_L1. When estimating motion, the inter-prediction module 126 first sets a search position for the reference frame in the L0 list. Based on the position indicated by the vector difference indicating the search position to be specified, namely, the initial MV (ie, InitMV_L0), the vector difference for the search position is MVd_L0. The inter-prediction module 126 then determines the estimated position in the reference frame in the L1 list. This search position is indicated by a vector difference for the search position from the position indicated by the initial MV (ie, InitMV_L1). More specifically, the inter-prediction unit 126 determines the vector difference as MVd_L1 by mirroring MVd_L0. In other words, the inter-prediction unit 126 determines a position that is symmetrical with respect to the position indicated by the initial MV as a search position in each reference frame in the L0 list and L1 list. The inter-prediction module 126 calculates, for each search position, the total sum of absolute differences (SAD) between the pixel values at the search positions in the blocks as costs and learns the search position that results in the lowest cost.

[0362] FIG. 58A is a conceptual diagram for illustrating one example of motion estimation in DMVR, and FIG. 58B is a flowchart illustrating one example of a motion estimation process.

[0363] First, in step 1, the inter-prediction module 126 calculates the costs between the search position (also called the “starting point”) indicated by the initial MV and the eight surrounding search positions. The mutual prediction module 126 then determines whether or not the costs at each of the search positions other than the starting point are the smallest. Here, upon determining that the cost at a search position other than the starting point is the smallest, the inter-prediction unit 126 changes the target to the search position at which the smallest cost is obtained and executes the process in step 2. When the cost at the starting point is the smallest , the mutual prediction module 126 skips the process in step 2 and executes the process in step 3.

[0364] In step 2, the mutual prediction module 126 performs a search similar to the process in step 1 regarding, as a new starting point, the search position after the target is changed according to the result of the process in step 1. The mutual prediction module 126 then determines whether or not the costs at each of the search positions other than the starting point are the smallest. Here, when determining that the cost at a search position other than the starting point is the smallest, the inter-prediction unit 126 executes the process in step 4. When the cost at the starting point is the smallest, the inter-prediction unit 126 carries out the process in step 3.

[0365] In step 4, the inter-prediction unit 126 considers the search position at the starting point as the final search position, and determines the difference between the position indicated by the starting MV and the final search position as a vector difference.

[0366] In step 3, the inter-prediction module 126 determines the pixel position with sub-pixel precision at which the lowest cost is obtained, based on the costs at four points located at the top, bottom, left and right positions relative to the starting point in step 1 or step 2, and considers the pixel position as the final search position. The pixel position is determined with sub-pixel precision by performing a summation weighting each of the four top, bottom, left and right vectors ((0, 1), (0,-1), (-1, 0) and (1, 0)), using, as a weighting factor, the costs corresponding to one of the four search positions. The inter-prediction unit 126 then determines the difference between the position indicated by the initial MV and the final search position as a vector difference.

[0367] Motion compensation > BIO/OBMC/LIC

Motion compensation includes a mode for generating a predictive image and correcting a predictive image. The mode, for example, is Bidirectional Optical Flow (BIO), Overlapping Block Motion Compensation (OBMC), Local Illumination Compensation (LIC), which are described below, etc.

[0368] FIG. 59 is a flowchart illustrating one example of a predictive image generation process.

[0369] The mutual prediction unit 126 generates a prediction image (step Sm_1) and corrects the prediction image according to any of the modes described above (step Sm_2).

[0370] FIG. 60 is a flowchart illustrating another example of a predictive image generation process.

[0371] The mutual prediction unit 126 determines the motion vector of the current block (step Sn_1). Next, the mutual prediction unit 126 generates a prediction image using the motion vector (step Sn_2) and determines whether or not to perform the correction process (step Sn_3). Here, upon determining whether to carry out the correction process (“Yes” in step Sn_3), the inter-prediction unit 126 generates the final prediction image by correcting the prediction image (step Sn_4). It should be noted that in the LIC described below, the brightness and chrominance can be adjusted in step Sn_4. When determining whether to not perform the correction process (“No” in step Sn_3), the inter-prediction unit 126 outputs the prediction image as the final prediction image without correcting the prediction image (step Sn_5).

[0372] Motion compensation > OBMC

It should be noted that the mutual prediction image can be generated using motion information for an adjacent block in addition to motion information for the current block obtained by motion estimation. More specifically, a mutual prediction image can be generated for each sub-block in the current block by performing weighting summation of the prediction image based on the motion information obtained by motion estimation (in the reference frame) and the prediction image based on the motion information of the adjacent block (in the current frame) . This mutual prediction (motion compensation) is also called "overlapping block motion compensation (OBMC)" or "OBMC mode".

[0373] In OBMC mode, information indicating the subblock size for the OBMC (referred to, for example, as "OBMC block size") may be signaled at the sequence level. In addition, information indicating whether or not to use the OBMC mode (called, for example, an “OBMC flag”) is signaled at the CU level. It should be noted that the signaling of this information need not necessarily be performed at the sequence level and the CU level and may be performed at another level (eg, frame level, slice level, brick level, CTU level, or subblock level).

[0374] The OBMC mode is described in more detail below. Fig. 61 and 62 are a flowchart and a conceptual diagram for illustrating a brief view of the prediction image correction process performed by OBMC.

[0375] First, as illustrated in FIG. 62, a prediction image (Pred) by normal motion compensation is obtained using the MV assigned to the current block. In fig. 62, the arrow "MV" points to the reference frame and indicates what the current block of the current frame is referenced to obtain the prediction image.

[0376] Next, the prediction image (Pred_L) is obtained by applying the motion vector (MV_L) that is already extracted for the coded block adjacent to the left of the current block to the current block (reusing the motion vector for the current block). The motion vector (MV_L) is indicated by an arrow "MV_L" indicating a reference frame from the current block. The first correction of the prediction image is performed by overlapping two prediction images Pred and Pred_L. This provides the advantage of blending the boundary between adjacent blocks.

[0377] Similarly, a prediction image (Pred_U) is obtained by applying the MV (MV_U) that is already extracted for the coded block adjacent above the current block to the current block (reusing the MV for the current block). MV (MV_U) is indicated by an arrow "MV_U" indicating a reference frame from the current block. The second correction to the prediction image is performed by overlapping the prediction image Pred_U with the prediction images (eg, Pred and Pred_L) for which the first correction is performed. This provides the advantage of blending the boundary between adjacent blocks. The prediction image obtained by the second correction is a prediction image in which the boundary between adjacent blocks is blended (smoothed) and therefore represents the final prediction image of the current block.

[0378] Although the above example is a two-path correction method using a left and an upper adjacent block, it should be noted that the correction method may be a three or more path correction method using also a right adjacent block and/or a lower adjacent block.

[0379] It should be noted that the area in which such overlap is performed may be only a portion of the area near the block boundary instead of the pixel area of the entire block.

[0380] Note that the above describes a process for correcting prediction images according to OBMC to obtain one prediction image Pred from one reference frame by overlapping the additional prediction image Pred_L and Pred_U. However, when the prediction image is corrected based on a plurality of reference pictures, a similar process may be applied to each of the plurality of reference frames. In such a case, after corrected prediction images are obtained from the corresponding reference frames by performing OBMC image correction based on a plurality of reference frames, the obtained corrected prediction images are further overlapped to obtain a final prediction image.

[0381] It should be noted that, in OBMC, the current block unit may be a PU or a sub-block unit obtained by further partitioning the PU.

[0382] One example of a method for determining whether or not to apply OBMC is a method for using obmc_flag, which is a signal indicating whether or not to apply OBMC. As one specific example, encoder 100 may determine whether or not the current block belongs to a region having complex motion. The encoder sets obmc_flag to a value of "1" when the block belongs to a region having complex motion and applies OBMC when encoding, and sets obmc_flag to a value of "0" when the block does not belong to a region having complex motion and encodes the block without applying OBMC. Decoder 200 switches between applying and not applying OBMC by decoding the obmc_flag written to the stream.

[0383] Motion compensation > BIO

The following describes the MV extraction method. First, a mode for model-based MV extraction under the condition of uniform straight-line motion is described. This mode is also called "Bidirectional Optical Flow (BIO) mode". In addition, this bidirectional optical flow can be recorded as BDOF instead of BIO.

[0384] FIG. 63 is a conceptual diagram to illustrate the model under the condition of uniform linear motion. In fig. 63, (v _x , v _y ) indicates the velocity vector, and τ ₀ and τ ₁ indicate the time distances between the current frame (Cur Pic) and the two reference frames (Ref ₀ , Ref ₁ ). (MVx ₀ , MVy ₀ ) indicate MVs corresponding to the reference frame Ref ₀ , and (MVx ₁ , MVy ₁ ) indicate MVs corresponding to the reference frame Ref ₀ .

[0385] Here, according to the assumption of relatively uniform rectilinear motion demonstrated by the velocity vector, (v _x , v _y ), (MVx ₀ , MVy ₀ ) and (MVx ₁ , MVy ₁ ) are represented as (v _x τ ₀ , v _y τ ₀ ) and (-v _x τ ₁ ,- v _y τ ₁ ), respectively, and the following optical flow equation (2) is specified:

[0386] Mathematical expression 3

[0387] Here, I(k) indicates the value of the motion-compensated luminance signal of reference frame k (k=0, 1) after motion compensation. This optical flow equation indicates that the sum of (i) the time derivative of the luminance signal value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is zero. The motion vector of each block, obtained, for example, from a list of possible MV variants, can be adjusted in units of pixels, based on a combination of the optical flow equation and Hermitian interpolation.

[0388] It should be noted that the motion vector may be extracted at the decoder 200 side using a method other than model-based motion vector extraction under the condition of uniform straight-line motion. For example, a motion vector may be extracted in sub-block units based on the motion vectors of multiple neighboring blocks.

[0389] FIG. 64 is a flowchart illustrating one example of an inter-prediction process according to BIO. Fig. 65 is a functional block diagram illustrating one example of a functional configuration of the inter-prediction unit 126, which can perform inter-prediction according to the BIO.

[0390] As illustrated in FIG. 65, the mutual prediction unit 126 includes, for example, a storage unit 126a, an interpolated image extracting unit 126b, a gradient image extracting unit 126c, an optical flow extracting unit 126d, a correction value extracting unit 126e, and a prediction image corrector 126f. It should be noted that the memory device 126a may be a movie frame memory 122.

[0391] The inter-prediction unit 126 extracts two motion vectors (M ₀ , M ₁ ) using two reference frames (Ref ₀ , Ref ₁ ) different from the frame (Cur Pic) including the current block. The mutual prediction unit 126 then extracts the prediction image for the current block using two motion vectors (M ₀ , M ₁ ) (step Sy_1). It should be noted that the motion vector M0 is a motion vector (MV _x0 , MV _y0 ) corresponding to the reference frame Ref ₀ , and the motion vector M ₁ is a motion vector (MV _x1 , MV _y1 ) corresponding to the reference frame Ref ₁ .

[0392] Next, the interpolated image retrieval unit 126b retrieves the interpolated image I ⁰ for the current block using the motion vector M ₀ and the reference frame L ₀ by accessing the storage device 126a. In addition, the interpolated image retrieving unit 126b retrieves the interpolated image I ¹ for the current block using the motion vector M ₁ and the reference frame L ₁ by accessing the storage device 126a (step Sy_2). Here, the interpolated image I ⁰ is an image included in the reference frame Ref ₀ and which is to be extracted for the current block, and the interpolated image I ¹ is an image included in the reference frame Ref ₁ and which is to be extracted for the current block. Each of the interpolated image I ⁰ and the interpolated image I ¹ may have a size identical to the size of the current block. Alternatively, each of the interpolated image I ⁰ and the interpolated image I ¹ may be an image larger than the current block. In addition, the interpolated image I ⁰ and the interpolated image I ¹ may include a prediction image obtained by using motion vectors (M ₀ , M ₁ ) and reference frames (L ₀ , L ₁ ) and applying a motion compensation filter.

[0393] In addition, the gradient image extracting unit 126c extracts the gradient images (Ix ⁰ , Ix ¹ , Iy ⁰ , Iy ¹ ) of the current block from the interpolated image I ⁰ and the interpolated image I ¹ (step Sy_3). It should be noted that the gradient images in the horizontal direction are (Ix ⁰ , Ix ¹ ), and the gradient images in the vertical direction are (Iy ⁰ , Iy ¹ ). The gradient image extraction module 126c may extract each gradient image, for example, by applying a gradient filter to the interpolated images. A gradient image may indicate the amount of spatial variation of a pixel value along a horizontal direction, along a vertical direction, or along both directions.

[0394] Next, optical flow extractor 126d extracts, for each sub-block of the current block, optical flow (vx, vy), which is a velocity vector, using interpolated images (I ⁰ , I ¹ ) and gradient images (Ix ⁰ , Ix ¹ , Iy ⁰ , Iy ¹ ) (stage Sy_4). The optical flow specifies coefficients for correcting the amount of spatial pixel movement and may be referred to as a “local motion estimation value,” “adjusted motion vector,” or “adjusted weight vector.” As one example, the subunit may be a 4x4 pixel sub-CU. It should be noted that optical flow extraction may be performed for each pixel unit and the like. instead of executing for each subblock.

[0395] Next, the inter-prediction module 126 corrects the prediction image for the current block using the optical flow (vx, vy). For example, the correction value extracting unit 126e extracts a correction value for a pixel value included in the current block using optical flow (vx, vy) (step Sy_5). The prediction image corrector 126f can then correct the prediction image for the current block using the correction value (step Sy_6). It should be noted that the correction value may be retrieved in units of a pixel, or may be retrieved in units of a plurality of pixels or in units of subblocks.

[0396] It should be noted that the BIO process flow is not limited to the process disclosed in FIG. 64. For example, only a portion of the processes disclosed in FIG. 64 may be executed, or another process may be added or used as a replacement, or the processes may be executed in a different processing order, etc.

[0397] Motion compensation > LIC

Next, one example of a mode for generating a forecast image using a local illuminance compensation (LIC) process is described.

[0398] FIG. 66A is a conceptual diagram for illustrating one example process for a method for generating predictive images using a brightness correction process performed by an LIC. Fig. 66B is a flowchart illustrating one example process for a predictive image generation method using LIC.

[0399] First, the inter-prediction unit 126 extracts the MV from the encoded reference frame and obtains a reference image corresponding to the current block (step Sz_1).

[0400] Next, the inter-prediction unit 126 extracts, for the current block, information indicating how the luminance signal value is changed between the current block and the reference frame (step Sz_2). This extraction is performed based on the luminance pixel values of the encoded left adjacent reference region (surrounding reference region) and the encoded top adjacent reference region (surrounding reference region) in the current frame and the pixel values of the luminance signal at corresponding positions in the reference frame indicated by the extracted MVs. The inter-prediction unit 126 calculates the luminance correction parameter using information indicating how the luminance signal value is changed (step Sz_3).

[0401] The inter-prediction unit 126 generates a prediction image for the current block by performing a luminance correction process in which a luminance correction parameter is applied to the reference image in the reference frame indicated by the MV (step Sz_4). In other words, the prediction image, which is a reference image in the reference frame indicated by the MV, is subject to correction based on the luminance correction parameter. With this correction, brightness may be adjusted, or chroma may be adjusted, or both may be adjusted. In other words, the chrominance correction parameter can be calculated using information indicating how the chrominance is changed, and the chrominance correction process can be performed.

[0402] It should be noted that the shape of the surrounding support region illustrated in FIG. 66A is one example; another form may be used.

[0403] In addition, although a process in which a prediction image is generated from a single reference frame is described here, cases in which a prediction image is generated from a plurality of reference frames may be similarly described. The prediction image can be generated after performing the brightness correction process of the reference images obtained from the reference frames, identical to that described above.

[0404] One example of a method for determining whether or not to apply LIC is a method for using lic_flag, which is a signal indicating whether or not to apply LIC. As one specific example, encoder 100 determines whether or not the current block belongs to an area having a brightness change. The encoder 100 sets lic_flag to a value of "1" when the block belongs to a region having a luminance change and applies LIC when encoding, and sets lic_flag to a value of "0" when the block does not belong to a region having a luminance change and performs encoding without applying LIC. The decoder 200 can decode the lic_flag recorded in the stream and decode the current block by switching between applying and not applying the LIC according to the value of the flag.

[0405] One example of another method for determining whether or not to apply an LIC process is a method for determining according to whether or not an LIC process is applied to a surrounding block. As one specific example, when the current block is processed in the merge mode, the inter-prediction unit 126 determines whether or not the encoded surrounding block selected in MV extraction in the merge mode is encoded using LIC. The inter-prediction unit 126 performs encoding by switching between applying and not applying LIC according to the result. It should be noted that also in this example, identical processes are applied in the processes on the decoder 200 side.

[0406] The luma correction (LIC) process is described with reference to FIG. 66A and 66B and is further described below.

[0407] First, the inter-prediction unit 126 extracts the MV to obtain a reference picture corresponding to the current block to be encoded from a reference frame that is a coded frame.

[0408] Next, the inter-prediction module 126 extracts information indicating how the luminance signal value of the reference frame is changed to the luminance signal value of the current frame, using the pixel luminance signal values of encoded surrounding reference regions that are adjacent to the left of or above the current block, and luminance signal values at corresponding positions in the reference frames indicated by the MV, and calculates a luminance correction parameter. For example, it is assumed that the pixel value of the luminance signal of a given pixel in the surrounding reference region in the current frame is p0, and that the pixel value of the luminance signal of the pixel corresponding to a given pixel in the surrounding reference region in the reference frame is p1. The inter-prediction unit 126 calculates coefficients A and B to optimize A*p1+B=p0 as a luminance correction parameter for a plurality of pixels in the surrounding reference region.

[0409] Next, the inter-prediction unit 126 performs a luminance correction process using the luminance correction parameter for the reference image in the reference frame indicated by the MV to generate a prediction image for the current block. For example, it is assumed that the pixel value of the luminance signal in the reference image is p2, and that the luminance-corrected pixel value of the luminance signal of the prediction image is p3. The mutual prediction unit 126 generates a prediction image after undergoing the brightness correction process by calculating A*p2+B=p3 for each of the pixels in the reference image.

[0410] For example, an area having a certain number of pixels extracted from each of the top adjacent pixel and the left adjacent pixel can be used as a surrounding reference area. In addition, the surrounding reference area is not limited to an area that is adjacent to the current block, and may be an area that is not adjacent to the current block. In the example illustrated in FIG. 66A, the surrounding reference area in the reference frame may be an area indicated by another MV in the current frame from the surrounding reference area in the current frame. For example, the other MV may be an MV in the surrounding reference region in the current frame.

[0411] Although the operations performed by the encoder 100 are described here, it should be noted that the decoder 200 performs similar operations.

[0412] It should be noted that LIC can be applied not only to the luma signal, but also to the chrominance signal. At this time, a correction parameter can be extracted separately for each of Y, Cb and Cr, or a common correction parameter can be used for any of Y, Cb and Cr.

[0413] In addition, the LIC process may be applied in subblock units. For example, the correction parameter may be extracted using a surrounding reference area in the current sub-block and a surrounding reference area in the reference sub-block in the reference frame indicated by the MV of the current sub-block.

[0414] Forecast controller

The prediction controller 128 selects one of the internal prediction signal (the image or signal output from the intra prediction unit 124) and the mutual prediction signal (the image or signal output from the inter prediction unit 126) and outputs the selected prediction image to the subtractor 104 and the adder 116 as forecast signal.

[0415] Prediction parameter generator

The prediction parameter generator 130 may output information related to intra prediction, inter prediction, prediction image selection in prediction controller 128, etc. as a prediction parameter to the entropy encoder 110. The entropy encoder 110 may generate a stream based on the prediction parameter that is input from the prediction parameter generator 130 and the quantized coefficients that are input from the quantizer 108. The prediction parameter may be used in the decoder 200. The decoder 200 may receive and decode the stream and perform processes identical to the prediction processes performed by the intra prediction module 124, the inter prediction module 126, and the prediction controller 128. The prediction parameter may include, for example, (i) a selectable prediction signal (e.g., MV, prediction type, or prediction mode used by intra prediction module 124 or inter prediction module 126), or (ii) an optional index, flag, or value, which is based on the prediction process performed in each of the intra prediction module 124, the inter prediction module 126 and the prediction controller 128, or which indicates the prediction process.

[0416] Decoder

The following describes a decoder 200 capable of decoding a stream output from the encoder 100 described above. Fig. 67 is a block diagram illustrating a functional configuration of the decoder 200 according to this embodiment. The decoder 200 is equipment that decodes a stream that represents an encoded image in units of blocks.

[0417] As illustrated in FIG. 67, the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse converter 206, an adder 208, a block memory 210, a loop filter 212, a movie frame memory 214, an intra prediction module 216, an inter prediction module 218, a prediction controller 220, a prediction parameter generator 222 and a partition determination module 224. It should be noted that the intra prediction module 216 and the inter prediction module 218 are configured as part of the prediction execution module.

[0418] Decoder installation example

Fig. 68 is a functional block diagram illustrating an installation example of the decoder 200. The decoder 200 includes a processor b1 and a memory b2. For example, the plurality of constituent elements of the decoder 200 illustrated in FIG. 67 are mounted in the processor b1 and memory b2 illustrated in FIG. 68.

[0419] Processor b1 is a circuit that performs information processing and is connected to memory b2. For example, processor b1 is a specialized or general electronic circuit that decodes the stream. Processor b1 may be a processor such as a CPU. In addition, processor b1 may be a collection of multiple electronic circuits. In addition, for example, the processor b1 may serve as two or more constituent elements other than the information storing constituent element of the plurality of constituent elements of the decoder 200 illustrated in FIG. 67, etc.

[0420] Memory b2 is a dedicated or general memory for storing information that is used by processor b1 to decode the stream. Memory b2 may be an electronic circuit and may be coupled to processor b1. In addition, memory b2 may be included in processor b1. In addition, the storage device b2 may be a collection of a plurality of electronic circuits. In addition, the storage device b2 may be a magnetic disk, an optical disk or the like. or may be represented as a storage device, recording medium, or the like. In addition, the storage device b2 may be a non-volatile storage device or a volatile storage device.

[0421] For example, storage device b2 may store an image or stream. In addition, memory b2 may store a program for instructing processor b1 to decode the stream.

[0422] In addition, for example, the storage device b2 may serve as two or more constituent elements for storing information from a plurality of constituent elements of the decoder 200 illustrated in FIG. 67, etc. More specifically, the memory b2 may serve as the block memory 210 and the movie frame memory 214 illustrated in FIG. 67. More specifically, the storage device b2 can store a reconstructed image (specifically, a reconstructed block, a reconstructed frame, etc.).

[0423] It should be noted that, in the decoder 200, not all of the plurality of constituent elements illustrated in FIG. 67, etc. may be implemented, and not all processes described in this document may be performed. Some of the constituent elements shown in Fig. 67, etc., may be included in another device, or part of the processes described herein may be performed by another device.

[0424] Hereinafter, a complete flow of processes performed by the decoder 200 is described, and thereafter each of the constituent elements included in the decoder 200 is described. It should be noted that some of the constituent elements included in the decoder 200 perform processes identical to the processes performed by the encoder portion 100, and therefore the identical processes are not described in detail again. For example, the inverse quantizer 204, inverse converter 206, adder 208, block memory 210, movie frame memory 214, intra prediction module 216, inter prediction module 218, prediction controller 220, and loop filter 212 included in the decoder 200 perform processes similar to processes performed by the inverse quantizer 112, inverse converter 114, adder 116, block memory 118, movie frame memory 122, intra prediction module 124, inter prediction module 126, prediction controller 128, and loop filter 120 included in the decoder 200, respectively.

[0425] General flow of decoding process

Fig. 69 is a flowchart illustrating one example of a complete decoding process performed by decoder 200.

[0426] First, the partition determination unit 224 in the decoder 200 determines the partition pattern of each of the plurality of fixed size blocks (128×128 pixels) included in the frame based on a parameter that is input from the entropy decoder 202 (step Sp_1). This split pattern is a split pattern selected by the encoder 100. The decoder 200 then performs the processes of step Sp_2-Sp_6 for each of the plurality of split pattern blocks.

[0427] The entropy decoder 202 decodes (specifically, entropy decodes) the encoded quantized coefficients and the prediction parameter of the current block (step Sp_2).

[0428] Next, the inverse quantizer 204 inversely quantizes the plurality of quantized coefficients, and the inverse transformer 206 inversely transforms the result to recover the prediction residuals (ie, the difference block) (step Sp_3).

[0429] Next, the prediction execution unit including all or part of the intra prediction unit 216, the inter prediction unit 218, and the prediction controller 220 generates a prediction signal of the current block (step Sp_4).

[0430] Next, the adder 208 adds the predicted image with the prediction residual to generate a reconstructed image (also called a “decoded block of images”) of the current block (step Sp_5).

[0431] When the reconstructed image is generated, the contour filter 212 performs filtering on the reconstructed image (step Sp_6).

[0432] The decoder 200 then determines whether or not decoding of the entire frame is complete (step Sp_7). Upon determining that decoding is not yet complete ("No" in step Sp_7), the decoder 200 repeats the processes starting from step Sp_1.

[0433] It should be noted that the processes of these steps Sp_1-Sp_7 may be executed sequentially by the decoder 200, or two or more processes may be executed in parallel. The processing order of two or more processes can be modified.

[0434] Partition detection module

Fig. 70 is a conceptual diagram for illustrating the relationship between the partition determination module 224 and other constituent elements in the embodiment. The partition determination unit 224 may perform the following processes as examples.

[0435] For example, the partition determination module 224 collects block information from the block memory 210 or the movie frame memory 214 and further obtains a parameter from the entropy decoder 202. The partition determination module 224 can then determine a fixed-size block partition pattern based on the information blocks and parameter. The partition determination module 224 may then output information indicating the determined partition pattern to the inverter 206, the intra prediction module 216, and the inter prediction module 218. The inverse converter 206 may perform inverse conversion of the transform coefficients based on the partitioning pattern indicated by the information from the partition determination unit 224 . The intra prediction unit 216 and the inter prediction unit 218 may generate a prediction image based on the split pattern indicated by the information from the split determination unit 224.

[0436] Entropy decoder

Fig. 71 is a block diagram illustrating one example of a functional configuration of the entropy decoder 202.

[0437] Entropy decoder 202 generates quantized coefficients, a prediction parameter, and a parameter associated with the partitioning pattern by entropy decoding the stream. For example, CABAC is used in entropy decoding. More specifically, entropy decode 202 includes, for example, a binary arithmetic decoder 202a, a context controller 202b, and a debinarizer 202c. Binary arithmetic decoder 202a arithmetically decodes the stream using the context value retrieved by context controller 202b into a binary signal. The context controller 202b extracts the context value according to the attribute or surrounding state of the syntactic element, which represents the probability of occurrence of the binary signal, identical to that performed by the context controller 110b of the encoder 100. The debinarization unit 202c performs debinarization to convert the binary signal output from the binary arithmetic decoder 202a into a multi-level signal indicating quantized coefficients, as described above. This binary conversion may be performed according to the binary conversion method described above.

[0438] Consequently, the entropy decoder 202 outputs the quantized coefficients of each block to the inverse quantizer 204. The entropy decoder 202 may output the prediction parameter included in the stream (see FIG. 1) to the intra prediction module 216, the inter prediction module 218, and the controller. 220 forecasting. The intra prediction module 216, the inter prediction module 218, and the prediction controller 220 are capable of performing prediction processes identical to the prediction processes performed by the intra prediction module 124, the inter prediction module 126, and the prediction controller 128 on the encoder 100 side.

[0439] FIG. 72 is a conceptual diagram for illustrating the flow of an example CABAC process in entropy decoder 202.

[0440] First, initialization is performed in CABAC in the entropy decoder 202. In initialization, initialization in the binary arithmetic decoder 202a and setting the initial value of the context is performed. The binary arithmetic decoder 202a and the debinarization unit 202c then perform arithmetic decoding and debinarization of, for example, the encoded CTU data. At this time, the context controller 202b updates the context value every time arithmetic decoding is performed. The context controller 202b then stores the context value as post-processing. The stored context value is used, for example, to initialize the context value for the next CTU.

[0441] Inverse Quantizer

The inverse quantizer 204 inversely quantizes the quantized coefficients of the current block, which are inputs from the entropy decoder 202. More specifically, the inverse quantizer 204 inversely quantizes the quantized coefficients of the current block, based on the quantization parameters corresponding to the quantized coefficients. The inverse quantizer 204 then outputs the inversely quantized transform coefficients (which are transform coefficients) of the current block to the inverse transformer 206.

[0442] FIG. 73 is a block diagram illustrating one example of the functional configuration of the inverse quantizer 204.

[0443] The inverse quantizer 204 includes, for example, a quantization parameter generator 204a, a predicted quantization parameter generator 204b, a quantization parameter storage device 204d, and an inverse quantization execution unit 204e.

[0444] FIG. 74 is a flowchart illustrating one example of an inverse quantization process performed by the inverse quantizer 204.

[0445] The inverse quantizer 204 may perform the inverse quantization process as one example for each CU based on the process flow illustrated in FIG. 74. More specifically, the quantization parameter generator 204a determines whether or not to perform inverse quantization (step Sv_11). Here, upon determining whether to perform inverse quantization (“Yes” in step Sv_11), the quantization parameter generator 204a obtains the difference quantization parameter for the current block from the entropy decoder 202 (step Sv_12).

[0446] Next, the predicted quantization parameter generator 204b then obtains the quantization parameter for a processing unit other than the current block from the quantization parameter storage device 204d (step Sv_13). The predicted quantization parameter generator 204b generates a predicted quantization parameter of the current block based on the received quantization parameter (step Sv_14).

[0447] The quantization parameter generator 204a then generates a quantization parameter for the current block based on the difference quantization parameter for the current block obtained from the entropy decoder 202 and the predicted quantization parameter for the current block generated by the predicted quantization parameter generator 204b (step Sv_15). For example, a difference quantization parameter for the current block obtained from the entropy decoder 202 and a predicted quantization parameter for the current block generated by the predicted quantization parameter generator 204b may be summed together to generate a quantization parameter for the current block. In addition, the quantization parameter generator 204a stores the quantization parameter for the current block in the quantization parameter storage device 204d (step Sv_16).

[0448] Next, the inverse quantization execution unit 204e inversely quantizes the quantized coefficients of the current block into transform coefficients using the quantization parameter generated in step Sv_15 (step Sv_17).

[0449] It should be noted that the difference quantization parameter may be decoded at the bit sequence level, frame level, slice level, brick level, or CTU level. In addition, the initial value of the quantization parameter may be decoded at the sequence level, frame level, slice level, brick level, or CTU level. At this time, the quantization parameter can be generated using the initial value of the quantization parameter and the difference quantization parameter.

[0450] It should be noted that the inverse quantizer 204 may include a plurality of inverse quantizers and may inversely quantize the quantized coefficients using an inverse quantization method selected from a plurality of inverse quantization methods.

[0451] Inverter

The inverse transformer 206 recovers the prediction residuals by inversely transforming the transform coefficients that are the inputs from the inverse quantizer 204.

[0452] For example, when information parsed from the stream indicates that EMT or AMT should be applied (eg, when the AMT flag is true), the inverter 206 inverts the transform coefficients of the current block based on the information indicating the parsed type transformations.

[0453] Additionally, for example, when information parsed from the stream indicates that NSST should be applied, inverter 206 applies a secondary inverse transform to the transform coefficients.

[0454] FIG. 75 is a flowchart illustrating one example of a process performed by the inverter 206.

[0455] For example, the inverter 206 determines whether or not information indicating that the orthogonal transform is not performed is present in the stream (step St_11). Here, upon determining that this information is not present ("No" in step St_11) (for example: absence of indicators regarding whether the orthogonal transformation is performed or not; presence of an indicator that the orthogonal transformation should be performed); the inverse converter 206 receives information indicating the type of conversion decoded by the entropy decoder 202 (step St_12). Then, based on the information, the inverse transformer 206 determines the transform type used for the orthogonal transform in the encoder 100 (step St_13). The inverter 206 then performs the inverse orthogonal transform using a certain transform type (step St_14). As illustrated in FIG. 75, upon determining that information indicating that the orthogonal transformation is not performed is present ("Yes" in step St_11) (eg, a clear indicator that the orthogonal transformation is not performed; no indicator that the orthogonal transformation is performed), the orthogonal transformation is not performed.

[0456] FIG. 76 is a flowchart illustrating one example of a process performed by the inverter 206.

[0457] For example, the inverter 206 determines whether the transform size is less than, equal to, or not the determined value (step Su_11). A specific value can be predefined. Here, upon determining that the transform size is less than or equal to a certain value ("Yes" in step Su_11), the inverter 206 obtains, from the entropy decoder 202, information indicating what type of transform is used by the encoder 100 at least from one transformation type included in the first group of transformation types (step Su_12). It should be noted that this information is decoded by the entropy decoder 202 and output to the inverter 206.

[0458] Based on the information, the inverter 206 determines the transform type used for the orthogonal transform in the encoder 100 (step Su_13). The inverter 206 then performs an inverse orthogonal transform on the transform coefficients of the current block using a certain transform type (step Su_14). Upon determining that the transform size is not less than or equal to a certain value ("No" in step Su_11), the inverse converter 206 inverts the transform coefficients of the current block using a second group of transform types (step Su_15).

[0459] It should be noted that the inverse orthogonal transform by the inverter 206 can be performed according to the process flow illustrated in FIG. 75 or fig. 76 for each TU, as one example. In addition, the inverse orthogonal transform can be performed by using a specified transform type without decoding information indicating the transform type used for the orthogonal transform. The specified transformation type may be a predefined transformation type or a default transformation type. In addition, the conversion type may specifically be DST7, DCT8, or the like. For an inverse orthogonal transform, the inverse transform basis function corresponding to the type of transform is used.

[0460] Totalizer

The adder 208 reconstructs the current block by summing the prediction residual, which is the input from the inverter 206, and the prediction image, which is the input from the prediction controller 220. In other words, a reconstructed image of the current block is formed. Adder 208 then outputs the reconstructed image of the current block to block memory 210 and loop filter 212.

[0461] Block memory

The block memory 210 is a storage device for storing a block that is included in the current frame, and may be referred to in intra prediction. More specifically, the block memory 210 stores the reconstructed image output from the adder 208.

[0462] Loop filter

The contour filter 212 applies the contour filter to the reconstructed image generated by the adder 208, and outputs the filtered reconstructed image to the movie frame memory 214 and provides an output to the decoder 200, for example, and outputs to a display device, etc.

[0463] When information indicating ALF on or off parsed from the stream indicates that ALF is on, one filter from a plurality of filters may be selected, for example, based on the direction and activity of local gradients, and the selected filter is applied to the reconstructed image.

[0464] FIG. 77 is a block diagram illustrating one example of a functional configuration of loop filter 212. It should be noted that loop filter 212 has a configuration similar to that of loop filter 120 of encoder 100.

[0465] For example, as illustrated in FIG. 77, the loop filter 212 includes a deblocking filtering execution unit 212a, an SAO execution unit 212b, and an ALF execution unit 212c. The deblocking filtering execution unit 212a performs a deblocking filtering process for the restored image. The SAO execution unit 212b performs the SAO process on the reconstructed image after subjecting it to the deblocking filtering process. The ALF execution unit 212c performs the ALF process on the reconstructed image after being subjected to the SAO process. It should be noted that loop filter 212 need not always include all of the constituent elements disclosed in FIG. 77, and may include only part of the constituent elements. In addition, the loop filter 212 may be configured to perform the above processes in a processing order different from the processing order disclosed in FIG. 77 may not perform all of the processes illustrated in FIG. 77, etc.

[0466] Motion Picture Memory

The movie frame memory 214, for example, is a storage device for storing reference frames for use in inter-prediction, and may also be referred to as a “motion picture buffer.” More specifically, the movie frame memory 214 stores the reconstructed image filtered by the edge filter 212.

[0467] Prediction Module (Intra Prediction Module, Inter Prediction Module, Prediction Controller)

Fig. 78 is a flowchart illustrating one example of a process performed by the decoder prediction module 200. It should be noted that the prediction execution module may include all or part of the following constituent elements: intra prediction module 216; mutual prediction module 218; and a prediction controller 220. The prediction execution module includes, for example, an intra prediction module 216 and an inter prediction module 218.

[0468] The prediction module generates a prediction image of the current block (step Sq_1). This prediction image is also called a "prediction signal" or a "prediction block". It should be noted that the prediction signal is, for example, an internal prediction signal or a mutual prediction signal. More specifically, the prediction module generates a prediction image of the current block using a reconstructed image that has already been obtained for another block through generating the prediction image, reconstructing the prediction residual, and summing the prediction image. The prediction module of the decoder 200 generates a prediction image identical to the prediction image generated by the prediction module of the encoder 100. In other words, the prediction images are generated according to a method common among prediction modules or mutually corresponding methods.

[0469] The reconstructed image, for example, may be an image in a reference frame or an image of a decoded block (ie, another block described above) in the current frame, which is a frame including the current block. The decoded block in the current frame, for example, is a block adjacent to the current block.

[0470] FIG. 79 is a flowchart illustrating another example of a process performed by the decoder prediction module 200.

[0471] The prediction module determines a method or mode for generating a prediction image (step Sr_1). For example, the method or mode may be determined, for example, based on a prediction parameter, etc.

[0472] When determining the first method as the mode for generating a prediction image, the prediction module generates a prediction image according to the first method (step Sr_2a). When determining the second method as the mode for generating the prediction image, the prediction module generates the prediction image according to the second method (step Sr_2b). When determining the third method as the mode for generating the prediction image, the prediction module generates the prediction image according to the third method (step Sr_2c).

[0473] The first method, the second method and the third method may be mutually different methods for generating a predictive image. Each of the first to third methods may be an inter-prediction method, an intra-prediction method, or another prediction method. The above-described reconstructed image can be used in these prediction methods.

[0474] FIG. 80A-80C (collectively, FIG. 80) are a flowchart illustrating another example of a process performed by the decoder prediction module 200.

[0475] The prediction module may perform the prediction process according to the flowchart illustrated in FIG. 80, as one example. It should be noted that the intra-block copying illustrated in FIG. 80 is one mode that belongs to inter-prediction, and in which a block included in the current frame is called a “reference image” or a “reference block”. In other words, a frame different from the current frame is not accessed during intra-block copying. In addition, the PCM mode illustrated in FIG. 80 is one mode that belongs to intra prediction, and in which conversion and quantization are not performed.

[0476] Internal prediction module

The intra prediction unit 216 performs intra prediction by accessing a block in the current frame stored in the block memory 210 based on the intra prediction mode parsed from the stream to generate a prediction image of the current block (ie, the intra prediction block). More specifically, the intra prediction module 216 performs intra prediction by accessing pixel values (eg, luma and/or chroma values) of a block or blocks adjacent to the current block to generate an internal prediction image, and then outputs the internal prediction image to the controller. 220 forecasting.

[0477] It should be noted that when an intra prediction mode is selected in which the luminance signal block is accessed in intra prediction of a chroma block, the intra prediction unit 216 can predict the chrominance signal component of the current block based on the luminance signal component of the current block.

[0478] In addition, when information parsed from the stream indicates that PDPC should be applied, intra prediction module 216 adjusts internally predicted pixel values based on horizontal/vertical reference pixel gradients.

[0479] FIG. 81 is a diagram illustrating one example of a process performed by the intra prediction unit 216 of the decoder 200.

[0480] The internal prediction module 216 first determines whether or not the MPM should be used. As illustrated in FIG. 81, the internal prediction unit 216 determines whether or not an MPM flag indicating 1 is present in the stream (step Sw_11). Here, upon determining that the MPM flag indicating 1 is present (“Yes” in step Sw_11), the intra prediction unit 216 obtains, from the entropy decoder 202, information indicating the intra prediction mode selected in the encoder 100 from the MPM. It should be noted that this information is decoded by the entropy decoder 202 and output to the intra prediction module 216. Next, the intra prediction unit 216 determines the MPM (step Sw_13). MPMs include, for example, six intraprediction modes. The intra prediction unit 216 then determines an intra prediction mode that is included in a plurality of intra prediction modes included in the MPM and is indicated by the information obtained in step Sw_12 (step Sw_14).

[0481] Upon determining that the MPM flag indicating 1 is not present (“No” in step Sw_11), the intra prediction unit 216 obtains information indicating the intra prediction mode selected in the encoder 100 (step Sw_15). In other words, the intra prediction module 216 receives, from the entropy decoder 202, information indicating the intra prediction mode selected in the encoder 100 from among at least one intra prediction mode that is not included in the MPM. It should be noted that this information is decoded by the entropy decoder 202 and output to the intra prediction module 216. The intra prediction unit 216 then determines an intra prediction mode that is not included in the plurality of intra prediction modes included in the MPM and is indicated by the information obtained in step Sw_15 (step Sw_17).

[0482] The intra prediction unit 216 generates a prediction image according to the intra prediction mode determined in step Sw_14 or in step Sw_17 (step Sw_18).

[0483] Mutual Prediction Module

The inter-prediction unit 218 predicts the current block by referring to a reference frame stored in the movie frame memory 214 . Prediction is performed in units of the current block or the current subblock within the current block. It should be noted that a subblock is included in a block and represents a unit smaller than the block. The subblock size can be 4x4 pixels, 8x8 pixels, or other size. The sub-block size can be switched per unit, such as slice, brick, frame, etc.

[0484] For example, the mutual prediction module 218 generates a mutual prediction signal of the current block or sub-block by performing motion compensation by using motion information (e.g., a motion vector) parsed from the encoded bitstream (e.g., a prediction parameter output from the entropy decoder 202) , and outputs the mutual prediction signal to the prediction controller 220.

[0485] When information parsed from the stream indicates that the OBMC mode should be applied, the inter-prediction module 218 generates a inter-prediction image using the motion information of the adjacent block in addition to the motion information of the current block obtained through motion estimation.

[0486] In addition, when information parsed from the stream indicates that the FRUC mode should be applied, the inter-prediction module 218 extracts the motion information by performing motion estimation in accordance with a template matching method (for example, bilateral matching or matching with a reference) parsed from the stream. The mutual prediction module 218 then performs motion compensation (prediction) using the extracted motion information.

[0487] In addition, when the BIO mode is to be applied, the inter-prediction module 218 extracts the MV based on the model under the condition of uniform straight-line motion. In addition, when information parsed from the stream indicates that the affine mode should be applied, the inter-prediction module 218 extracts the MV for each sub-block based on the MVs of multiple neighboring blocks.

[0488] Flow for MV Extraction

Fig. 82 is a flowchart illustrating one example of an MV extraction process in the decoder 200.

[0489] The inter-prediction module 218 determines whether or not motion information (eg, MV) should be decoded, for example. For example, inter-prediction module 218 may make a determination according to a prediction mode included in the stream, or may make a determination based on other information included in the stream. Here, when determining whether to decode the motion information, the inter-prediction unit 218 extracts the MV for the current block in a mode in which the motion information is decoded. When determining whether to not decode the motion information, the inter-prediction module 218 extracts the MV in a mode in which the motion information is not decoded.

[0490] Here, MV extraction modes may include normal mutual mode, normal pooling mode, FRUC mode, affine mode, etc., which are described below. Modes in which motion information is decoded include normal reciprocal mode, normal combining mode, affine mode (specifically, affine reciprocal mode and affine combining mode), etc. It should be noted that the motion information may include not only MVs, but also MV predictor selection information, which is described below. Modes in which motion information is not decoded include FRUC mode, etc. The inter-prediction unit 218 selects a mode for extracting the MV for the current block from a plurality of modes, and extracts the MV for the current block using the selected mode.

[0491] FIG. 83 is a flowchart illustrating one example of an MV extraction process in the decoder 200.

[0492] For example, the inter-prediction module 218 may determine whether or not to decode the MV difference, i.e., for example, may make a determination according to a prediction mode included in the stream, or may make a determination based on other information included in flow. Here, when determining whether to decode the MV difference, the inter-prediction unit 218 may extract the MV for the current block in a mode in which the MV difference is decoded. In this case, for example, the MV difference included in the stream is decoded as a prediction parameter.

[0493] When determining whether to decode the MV difference at all, the inter-prediction module 218 extracts the MV in a mode in which the MV difference is not decoded. In this case, the encoded MV difference is not included in the stream.

[0494] Here, as described above, MV extraction modes include normal mutual mode, normal pooling mode, FRUC mode, affine mode, etc., which are described below. Modes in which the MV difference is encoded include normal reciprocal mode and affine mode (specifically, affine reciprocal mode), etc. Modes in which the MV difference is not encoded include FRUC mode, normal concatenation mode, affine mode (specifically, affine concatenation mode), etc. The inter-prediction unit 218 selects a mode for extracting the MV for the current block from a plurality of modes, and extracts the MV for the current block using the selected mode.

[0495] MV extract > normal mutual mode

For example, when the information parsed from the stream indicates that the normal reciprocal mode should be applied, the reciprocal prediction module 218 extracts the MV based on the information parsed from the stream and performs motion compensation (prediction) using the MV.

[0496] FIG. 84 is a flowchart illustrating an example of the inter-prediction process by the normal inter-mode in the decoder 200.

[0497] The inter-prediction module 218 of the decoder 200 performs motion compensation for each block. First, the inter-prediction module 218 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Sg_11). In other words, the inter-prediction module 218 generates a list of possible MV options.

[0498] Next, the inter-prediction module 218 extracts N (an integer of 2 or more) MV candidates from the plurality of MV candidates obtained in step Sg_11 as motion vector predictor candidates (also called “MV candidates”). predictors") according to certain ranks in order of priority (stage Sg_12). It should be noted that the priority order ranks may be predetermined for the corresponding N candidate MV predictors and may be predetermined.

[0499] Next, the inter-prediction module 218 decodes the MV predictor selection information from the input stream and selects one MV predictor candidate from the N candidate MV predictor candidates as the MV predictor for the current block using the decoded MV predictor selection information ( stage Sg_13).

[0500] Next, the inter-prediction unit 218 decodes the MV difference from the input stream and extracts the MV for the current block by summing the difference value, which is the decoded MV difference, and the selected MV predictor (step Sg_14).

[0501] Finally, the inter prediction unit 218 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the decoded reference frame (step Sg_15). Processes in stages Sg_11-Sg_15 are executed for each block. For example, when the processes in steps Sg_11 to Sg_15 are executed for each of all blocks in the slice, the slice reciprocal prediction using the normal reciprocal mode ends. For example, when the processes in steps Sg_11 to Sg_15 are executed for each of all blocks in a frame, frame inter-prediction using the normal inter-mode ends. It should be noted that not all blocks included in the slice may be subject to processes in steps Sg_11 to Sg_15, and slice mutual prediction using the normal mutual mode may end when a portion of the blocks are subject to processes. This also applies to frames in stages Sg_11-Sg_15. Frame reciprocal prediction using normal reciprocal mode may end when processes are running on a portion of the blocks in the frame.

[0502] MV extract > normal merge mode

For example, when information parsed from the stream indicates that the normal combining mode should be applied, the inter-prediction module 218 extracts the MV and performs motion compensation (prediction) using the MV.

[0503] FIG. 85 is a flowchart illustrating an example of an inter-prediction process via the normal combining mode in the decoder 200.

[0504] The inter-prediction module 218 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Sh_11). In other words, the inter-prediction module 218 generates a list of possible MV options.

[0505] Next, the inter-prediction module 218 selects one candidate MV from the plurality of candidate MVs obtained in step Sh_11 by extracting the MV of the current block (step Sh_12). More specifically, the inter-prediction unit 218 receives the MV selection information included as a prediction parameter in the stream, and selects the candidate MV identified by the MV selection information as the MV for the current block.

[0506] Finally, the inter prediction unit 218 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the decoded reference frame (step Sh_13). The processes in stages Sh_11-Sh_13 are performed, for example, for each block. For example, when the processes in steps Sh_11 to Sh_13 are executed for each of all blocks in the slice, cross-prediction of the slice using the normal pooling mode ends. In addition, when the processes in steps Sh_11 to Sh_13 are executed for each of all blocks in a frame, frame inter-prediction using the normal combining mode ends. It should be noted that not all blocks included in the slice are processed in steps Sh_11 to Sh_13, and cross-prediction of the slice using the normal pooling mode may end when a portion of the blocks are processed. This also applies to frames in stages Sh_11-Sh_13. Mutual frame prediction using normal merging mode may end when processes are running on a portion of the blocks in the frame.

[0507] MV extraction > FRUC mode

For example, when information parsed from the stream indicates that the FRUC mode should be applied, the inter-prediction module 218 extracts the MV in the FRUC mode and performs motion compensation (prediction) using the MV. In this case, motion information is extracted at the decoder 200 side without being signaled from the encoder 100 side. For example, the decoder 200 may extract motion information by performing motion estimation. In this case, decoder 200 performs motion estimation without using the pixel values in the current block at all.

[0508] FIG. 86 is a flowchart illustrating an example of an inter-prediction process by FRUC mode in the decoder 200.

[0509] First, the inter-prediction module 218 generates a list indicating the MVs of decoded blocks spatially or temporally adjacent to the current block by referring to the MVs as MV candidates (the list is a list of MV candidates and, for example, can also be used as a candidate MV candidate list for the normal combining mode (step Si_11). Then, the best possible MV candidate is selected from the plurality of candidate MV candidates registered in the candidate MV candidate list (step Si_12). For example, module 218. mutual prediction calculates the score value of each possible MV option included in the list of possible MV options, and selects one of the possible MV options as the best possible MV option based on the score values Based on the selected best possible MV options, module 218. The mutual prediction then extracts the MV for the current block (step Si_14). More specifically, for example, the selected best possible MVs are directly extracted as the MV for the current block. In addition, for example, the MV for the current block may be retrieved using pattern matching in the surrounding region of a position that is included in the reference frame and corresponds to the selected best possible MV candidate. In other words, evaluation using template matching on the reference frame and evaluation values can be performed on the surrounding region of the best possible MV candidate, and when there is an MV that results in a better evaluation value, the best possible MV candidate can be updated to the MV that results in a better score value, and the updated MV can be determined as the final MV for the current block. In an embodiment, the update to the MV that results in a better score value may not be performed.

[0510] Finally, the inter prediction unit 218 generates a prediction image for the current block by performing motion compensation of the current block using the extracted MV and the decoded reference frame (step Si_15). The processes in stages Si_11-Si_15 are performed, for example, for each block. For example, when the processes in steps Si_11-Si_15 are executed for each of all blocks in a slice, the cross-prediction of the slice using the FRUC mode ends. For example, when the processes in steps Si_11 to Si_15 are executed for each of all blocks in a frame, frame inter-prediction using the FRUC mode ends. Each subblock can be processed similarly to the case of each block.

[0511] MV extraction > FRUC mode

For example, when information parsed from the stream indicates that the affine combining mode should be applied, the inter-prediction module 218 extracts the MV in the affine combining mode and performs motion compensation (prediction) using the MV.

[0512] FIG. 87 is a flowchart illustrating an example of an inter-prediction process via the affine combining mode in the decoder 200.

[0513] In the affine merge mode, first, the inter-prediction module 218 extracts the MVs at the corresponding control points for the current block (step Sk_11). The control points are the top left corner point of the current block and the top right corner point of the current block, as illustrated in FIG. 46A, or the top left corner point of the current block, the top right corner point of the current block, and the bottom left corner point of the current block, as illustrated in FIG. 46B.

[0514] For example, when the MV extraction methods illustrated in FIG. 47A-47C are used as illustrated in FIGS. 47A, inter-prediction module 218 examines the decoded block A (left), block B (upper), block C (upper right), block D (lower left), and block E (upper left) in that order and identifies the first effective block decoded according to the affine regime. The inter-prediction module 218 extracts the MV at the control point using the identified first effective block decoded according to the affine mode. For example, when block A is identified and block A has two control points, as illustrated in FIG. 47B, the inter-prediction module 218 calculates the motion vector v ₀ at the top left corner control point of the current block and the motion vector v ₁ at the top right corner control point of the current block from the motion vectors v ₃ and v ₄ at the top left corner and the top right corner of the decoded block including block A. Thus, the MV at each control point is extracted.

[0515] It should be noted that, as illustrated in FIG. 49A, MVs at three control points can be calculated when block A is identified and block A has two control points, and that, as illustrated in FIG. 49B, MVs at two control points can be calculated when block A is identified and when block A has three control points.

[0516] In addition, when MV selection information is included as a prediction parameter in a stream, inter-prediction module 218 can extract MVs at each control point for the current block using the MV selection information.

[0517] Next, the inter-prediction module 218 performs motion compensation of each of the plurality of sub-blocks included in the current block. In other words, the inter-prediction module 218 calculates the MV for each of the plurality of sub-blocks as an affine MV using either the two motion vectors v ₀ and v ₁ and the above expression (1A) or the three motion vectors v ₀ , v ₁ and v ₂ and the above expression (1B) (step Sk_12). The inter-prediction module 218 then performs motion compensation of the sub-blocks using these affine MVs and the decoded reference frames (step Sk_13). When the processes in steps Sk_12 and Sk_13 are executed for each of the sub-blocks included in the current block, mutual prediction using the affine join mode for the current block is completed. In other words, motion compensation of the current block is performed in order to generate a predictive image of the current block.

[0518] It should be noted that the above-described list of possible MV options can be generated at step Sk_11. The MV candidate list, for example, may be a list including MV candidate candidates extracted using a plurality of MV extraction methods for each control point. The plurality of MV extraction methods, for example, may be any combination of the MV extraction methods illustrated in FIG. 47A-47C, the MV extraction methods illustrated in FIGS. 48A and 48B, the MV extraction methods illustrated in FIGS. 49A and 49B, and other MV extraction methods.

[0519] It should be noted that the list of possible MV options may include possible MV options in a mode in which prediction is performed in subblock units other than the affine mode.

[0520] It should be noted that, for example, a list of possible MV options including possible MV options in the affine pooling mode in which two control points are used and in the affine combining mode in which three control points are used can be generated as a list of possible MV options. Alternatively, a list of possible MV options including possible MV options in an affine join mode that uses two control points, and a list of possible MV options including possible MV options in an affine join mode that uses three control points can be formed separately. Alternatively, a list of possible MV options including possible MV options in one of an affine join mode that uses two control points and an affine join mode that uses three control points can be generated.

[0521] MV Extraction > Affine Mutual Mode

For example, when information parsed from the stream indicates that the affine reciprocal mode should be applied, the reciprocal prediction module 218 extracts the MV in the affine reciprocal mode and performs motion compensation (prediction) using the MV.

[0522] FIG. 88 is a flowchart illustrating an example of a mutual prediction process by affine mutual mode in the decoder 200.

[0523] In the affine mutual mode, first, the mutual prediction module 218 extracts the MV predictors (v ₀ , v ₁ ) or (v ₀ , v ₁ , v ₂ ) of the corresponding two or three control points of the current block (step Sj_11). The control points are the top left corner point of the current block, the top right corner point of the current block, and the bottom left corner point of the current block, as illustrated in FIG. 46A or FIG. 46B.

[0524] The inter-prediction unit 218 receives the MV predictor selection information included as a prediction parameter in the stream, and extracts an MV predictor at each control point for the current block using the MV identified by the MV predictor selection information. For example, when the MV extraction methods illustrated in FIG. 48A and 48B are used, the inter-prediction module 218 retrieves the ( _v0 , _v1 ) or ( _v0 , _v1 , _v2 ) motion vector predictors at the control points for the current block by selecting the MV block identified by the MV predictor selection information. , from the decoded blocks around the corresponding control points for the current block illustrated in FIG. 48A or in FIG. 48B.

[0525] Next, the inter-prediction unit 218 receives each MV difference included as a prediction parameter in the stream, and sums the MV predictor at each control point for the current block and the MV difference corresponding to the MV predictor (step Sj_12). Thus, the MV at each control point for the current block is retrieved.

[0526] Next, the inter-prediction module 218 performs motion compensation of each of the plurality of sub-blocks included in the current block. In other words, the inter-prediction module 218 calculates the MV for each of the plurality of sub-blocks as an affine MV using either the two motion vectors v ₀ and v ₁ and the above expression (1A) or the three motion vectors v ₀ , v ₁ and v ₂ and the above expressions (1B) (step Sj_13). The inter-prediction module 218 then performs motion compensation of the sub-blocks using these affine MVs and the decoded reference frames (step Sj_14). When the processes in steps Sj_13 and Sj_14 are executed for each of the sub-blocks included in the current block, the inter-prediction using the affine join mode for the current block is completed. In other words, motion compensation of the current block is performed in order to generate a predictive image of the current block.

[0527] It should be noted that the above-described list of possible MV options can be generated in step Sj_11, similar to step Sk_11.

[0528] MV extract > triangle mode

For example, when information parsed from the stream indicates that the triangle mode should be applied, the inter-prediction module 218 extracts the MV in the triangle mode and performs motion compensation (prediction) using the MV.

[0529] FIG. 89 is a flowchart illustrating an example of an inter-prediction process by triangular mode in the decoder 200.

[0530] In the triangle mode, first, the inter-prediction unit 218 splits the current block into a first segment and a second segment (step Sx_11). For example, the inter-prediction module 218 may obtain, from the stream, segment information, which is partition-related information, as a prediction parameter. The inter-prediction module 218 can then split the current block into a first segment and a second segment according to the segment information.

[0531] Next, inter-prediction module 218 obtains a plurality of candidate MVs for the current block based on information such as the MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Sx_12). In other words, the inter-prediction module 218 generates a list of possible MV options.

[0532] The inter-prediction module 218 then selects a candidate MV for the first segment and a candidate MV for the second segment as the first MV and the second MV, respectively, from the plurality of candidate MVs obtained in step Sx_11 (step Sx_13). At this time, the inter-prediction module 218 may obtain, from the stream, MV selection information to identify a selected MV candidate as a prediction parameter. The inter-prediction unit 218 can then select the first MV and the second MV according to the MV selection information.

[0533] Next, the inter prediction unit 218 generates a first prediction image by performing motion compensation using the selected first MV and the decoded reference frame (step Sx_14). Likewise, the inter-prediction module 218 generates a second prediction image by performing motion compensation using the selected second MV and the decoded reference frame (step Sx_15).

[0534] Finally, the inter prediction unit 218 generates a prediction image for the current block by performing a weighting summation of the first prediction image and the second prediction image (step Sx_16).

[0535] MV evaluation > DMVR

For example, the information parsed from the stream indicates that the DMVR should be applied, the inter-prediction module 218 performs motion estimation using the DMVR.

[0536] FIG. 90 is a flowchart illustrating an example of a motion estimation process by DMVR in the decoder 200.

[0537] The inter-prediction unit 218 extracts the MV for the current block according to the combining mode (step Sl_11). Next, the inter-prediction unit 218 extracts the final MV for the current block by searching in the area surrounding the reference frame indicated by the MV extracted at Sl_11 (step Sl_12). In other words, in this case, the MV of the current block is determined according to the DMVR.

[0538] FIG. 91 is a flowchart illustrating an example of a motion estimation process by DMVR in the decoder 200 and is identical to FIG. 58B.

[0539] First, in step 1 illustrated in FIG. 58A, the inter-prediction module 218 calculates the costs between the search position (also called the “starting point”) indicated by the initial MV and the eight surrounding search positions. The mutual prediction module 218 then determines whether or not the costs at each of the search positions other than the starting point are the smallest. Here, upon determining that the cost at one of the search positions other than the starting point is the smallest, the inter-prediction unit 218 changes the target to the search position at which the smallest cost is obtained and carries out the process in step 2 illustrated in FIG. 58. When the cost at the starting point is the smallest, the inter-prediction module 218 skips the process in step 2 illustrated in FIG. 58A, and carries out the process in step 3.

[0540] In step 2, illustrated in FIG. 58A, the inter-prediction unit 218 performs a search similar to the process in step 1 regarding the search position after the target is changed as a new starting point according to the result of the process in step 1. The inter-prediction unit 218 then determines whether or not the costs are in each of the search positions other than the starting point, the smallest. Here, when determining that the cost at one of the search positions other than the starting point is the smallest, the inter-prediction unit 218 executes the process in step 4. When the cost at the starting point is the smallest, the inter-prediction unit 218 executes the process in step 3.

[0541] In step 4, the inter-prediction unit 218 considers the search position at the start point as the search end position, and determines the difference between the position indicated by the start MV and the search end position as a vector difference.

[0542] In step 3, illustrated in FIG. 58A, the inter-prediction module 218 determines the sub-pixel precision pixel position at which the lowest cost is obtained based on the costs at four points located at the top, bottom, left and right positions relative to the starting point in step 1 or step 2, and considers the pixel position position as the final search position.

[0543] The pixel position with sub-pixel precision is determined by performing a summation weighting each of the four top, bottom, left and right vectors ((0, 1), (0,-1), (-1, 0) and (1, 0 )), using, as a weighting factor, the costs corresponding to one of the four search positions. The inter-prediction unit 218 then determines the difference between the position indicated by the initial MV and the final search position as a vector difference.

[0544] Motion compensation > BIO/OBMC/LIC

For example, when information parsed from the stream indicates that prediction image correction should be performed when generating the prediction image, the inter-prediction unit 218 corrects the prediction image based on the mode for correction. The mode, for example, is one of BIO, OBMC and LIC described above.

[0545] FIG. 92 is a flowchart illustrating one example of a prediction image generation process in the decoder 200.

[0546] The mutual prediction unit 218 generates a prediction image (step Sm_11) and corrects the prediction image according to any of the modes described above (step Sm_12).

[0547] FIG. 93 is a flowchart illustrating another example of a predictive image generation process in the decoder 200.

[0548] The inter-prediction unit 218 extracts the MV for the current block (step Sn_11). Next, the mutual prediction unit 218 generates a prediction image using the MV (step Sn_12) and determines whether or not to perform the correction process (step Sn_13). For example, the inter-prediction unit 218 receives the prediction parameter included in the flow and determines whether or not to perform the correction process based on the prediction parameter. This prediction parameter, for example, is a flag indicating whether or not one or more of the above-described modes should be applied. Here, upon determining whether to perform the correction process (Yes in step Sn_13), the inter-prediction unit 218 generates the final prediction image by correcting the prediction image (step Sn_14). It should be noted that, in LIC, brightness and chrominance can be adjusted in step Sn_14. When determining whether to not perform the correction process (“No” in step Sn_13), the inter-prediction unit 218 outputs the final prediction image without correcting the prediction image (step Sn_15).

[0549] Motion compensation > OBMC

For example, when information parsed from the stream indicates that OBMC should be performed when generating a prediction image, inter-prediction module 218 adjusts the prediction image according to the OBMC.

[0550] FIG. 94 is a flowchart illustrating an example of a prediction image correction process by OBMC in the decoder 200. Note that the flowchart in FIG. 94 indicates the process of correcting a prediction image using a current frame and a reference frame illustrated in FIG. 62.

[0551] First, as illustrated in FIG. 62, the mutual prediction unit 218 obtains a prediction image (Pred) by normal motion compensation using the MV assigned to the current block.

[0552] Next, the prediction image (Pred_L) is obtained by applying the motion vector (MV_L) that is already extracted for the coded block adjacent to the left of the current block to the current block (reusing the motion vector for the current block). The mutual prediction module 218 then performs a first correction of the prediction image by overlapping the two prediction images Pred and Pred_L. This provides the advantage of blending the boundary between adjacent blocks.

[0553] Similarly, the inter prediction unit 218 obtains the prediction image (Pred_U) by applying the MV (MV_U) that is already extracted for the decoded block adjacent above the current block to the current block (reusing the motion vector for the current block). The inter-prediction module 218 then performs a second correction to the prediction image by overlapping the prediction image Pred_U with the prediction images (eg, Pred and Pred_L) for which the first correction has been performed. This provides the advantage of blending the boundary between adjacent blocks. The prediction image obtained by the second correction is a prediction image in which the boundary between adjacent blocks is blended (smoothed) and therefore represents the final prediction image of the current block.

[0554] Motion compensation > BIO

For example, when information parsed from the stream indicates that a BIO should be executed when generating a prediction image, the inter-prediction module 218 adjusts the prediction image according to the BIO.

[0555] FIG. 95 is a flowchart illustrating an example of a prediction image correction process by BIO in the decoder 200.

[0556] As illustrated in FIG. 63, the inter-prediction unit 218 extracts two motion vectors ( _M0 , _M1 ) using two reference frames (Ref0, Ref1) different from the frame (Cur Pic) including the current block. The mutual prediction unit 218 then extracts the prediction image for the current block using two motion vectors (M ₀ , M ₁ ) (step Sy_11). It should be noted that the motion vector M0 is a motion vector (MV _x0 , MVy ₀ ) corresponding to the reference frame Ref ₀ , and the motion vector _M1 is a motion vector (MV _x1 , MV _y1 ) corresponding to the reference frame Ref ₀ .

[0557] Next, inter-prediction module 218 extracts the interpolated image I ⁰ for the current block using the motion vector M ₀ and the reference frame L ₀ . In addition, the inter-prediction module 218 extracts the interpolated image I ¹ for the current block using the motion vector M ₁ and the reference frame L ₁ (step Sy_12). Here, the interpolated image I ⁰ is an image included in the reference frame Ref ₀ and which is to be extracted for the current block, and the interpolated image I ¹ is an image included in the reference frame Ref ₁ and which is to be extracted for the current block. Each of the interpolated image I ⁰ and the interpolated image I ¹ may have a size identical to the size of the current block. Alternatively, each of the interpolated image I ⁰ and the interpolated image I ¹ may be an image larger than the current block. In addition, the interpolated image I ⁰ and the interpolated image I ¹ may include a prediction image obtained by using motion vectors (M ₀ , M ₁ ) and reference frames (L ₀ , L ₁ ) and applying a motion compensation filter.

[0558] In addition, the inter-prediction module 218 extracts the gradient images (Ix ⁰ , Ix ¹ , Iy ⁰ , Iy ¹ ) of the current block from the interpolated image I ⁰ and the interpolated image I ¹ (step Sy_13). It should be noted that the gradient images in the horizontal direction are (Ix ⁰ , Ix ¹ ), and the gradient images in the vertical direction are (Iy ⁰ , Iy ¹ ). Mutual prediction module 218 may extract gradient images, for example, by applying a gradient filter to the interpolated images. The gradient images may be gradient images each indicating an amount of spatial variation in a pixel value along a horizontal direction or an amount of spatial variation in a pixel value along a vertical direction.

[0559] Next, inter-prediction module 218 extracts, for each sub-block of the current block, optical flow (vx, vy), which is a velocity vector, using interpolated images (I ⁰ , I ¹ ) and gradient images (Ix ⁰ , Ix ¹ , Iy ⁰ , Iy ¹ ) (stage Sy_14). As one example, the subunit may be a 4x4 pixel sub-CU.

[0560] Next, the inter prediction module 218 corrects the prediction image for the current block using the optical flow (vx, vy). For example, the inter-prediction module 218 extracts a correction value for a pixel value included in the current block using optical flow (vx, vy) (step Sy_15). The mutual prediction unit 218 can then correct the prediction image for the current block using the correction value (step Sy_16). It should be noted that the correction value may be retrieved in units of a pixel, or may be retrieved in units of multiple pixels or in units of subblocks, etc.

[0561] It should be noted that the BIO process flow is not limited to the process disclosed in FIG. 95. Only part of the processes disclosed in FIG. 95 may be executed, or another process may be added or used as a replacement, or the processes may be executed in a different processing order.

[0562] Motion compensation > LIC

For example, when information parsed from the stream indicates that LIC should be performed when generating a prediction image, the inter-prediction module 218 adjusts the prediction image according to the LIC.

[0563] FIG. 96 is a flowchart illustrating an example of a prediction image correction process by an LIC in the decoder 200.

[0564] First, the inter-prediction unit 218 obtains a reference picture corresponding to the current block from the decoded reference frame using the MV (step Sz_11).

[0565] Next, the inter-prediction unit 218 extracts, for the current block, information indicating how the luminance value is changed between the current frame and the reference frame (step Sz_12). This extraction may be performed based on the luminance signal pixel values for the decoded left adjacent reference region (surrounding reference region) and the decoded upper adjacent reference region (surrounding reference region) and the luminance signal pixel values at corresponding positions in the reference frame indicated by the extracted MVs. The inter-prediction unit 218 calculates the luminance correction parameter using information indicating how the luminance signal value is changed (step Sz_13).

[0566] The inter-prediction unit 218 generates a prediction image for the current block by performing a luminance correction process in which a luminance correction parameter is applied to the reference image in the reference frame indicated by the MV (step Sz_14). In other words, the prediction image, which is a reference image in the reference frame indicated by the MV, is subject to correction based on the luminance correction parameter. With this correction, the brightness can be adjusted, or the color can be adjusted.

[0567] Forecast controller

The prediction controller 220 selects an internal prediction image or a cross prediction image and outputs the selected image to the adder 208. In general, the configurations, functions and processes of the prediction controller 220, the intra prediction module 216 and the cross prediction module 218 on the decoder 200 side may correspond to the configurations, functions and processes of the prediction controller 128, the internal prediction module 124, and the inter-prediction module 126 on the side of the encoder 100.

[0568] First aspect

Fig. 97 is a flowchart for an example of a processing flow 1000 for decoding an image using a CCALF (Cross Component Adaptive Contour Filtering) process according to the first aspect. The process flow 1000 may be performed, for example, by the decoder 200 of FIG. 67, etc.

[0569] In step S1001, a filtering process is applied to the reconstructed first component image samples. The first component, for example, may be a luminance signal component. The luminance signal component may be represented as a Y component. The reconstructed luminance signal image samples may represent the output signals of the ALF process. The ALF output signals may be reconstructed luminance samples generated through the SAO process. In some embodiments, this filtering process performed in step S1001 may be represented as a CCALF process. The numbers of reconstructed luminance samples may be identical to the number of filter coefficients to be used in the CCALF process. In other embodiments, the clipping process may be performed on the filtered reconstructed luminance signal samples.

[0570] In step S1002, the reconstructed sample of the second component images is modified. The second component may be a chrominance signal component. The chrominance signal component may be represented as a Cb and/or Cr component. The reconstructed chrominance signal image samples may represent the output signals of the ALF process. The ALF output signals can be reconstructed from chroma samples generated through the SAO process. The modified reconstructed image sample may be the sum of the reconstructed chroma samples and the filtered reconstructed luma samples, which is the output of step S1001. In other words, the modification process can be performed by adding the filtered value of the reconstructed luma signal samples generated by the CCALF process of step S1001 with the filtered value of the reconstructed chrominance signal samples generated by the ALF process. In some embodiments, the clipping process may be performed on the reconstructed chroma samples. The first component and the second component may belong to an identical block or may belong to different blocks.

[0571] In step S1003, the value of the modified reconstructed chroma component image sample is clipped. By performing the pruning process, the value of the samples can be guaranteed to be within a certain range. Additionally, pruning can promote better convergence in a least squares optimization process, etc., to minimize the difference between the residual (the difference between the original sample value and the reconstructed sample value) and the filtered value of the chroma samples to determine filter coefficients.

[0572] In step S1004, the image is decoded using the clipped reconstructed sample of chroma component images. In some embodiments, step S1003 does not need to be performed. In this case, the image is decoded using a modified reconstructed chrominance sample that is not clipped.

[0573] FIG. 98 is a block diagram illustrating a functional configuration of an encoder and decoder according to an embodiment. In this embodiment, the clipping process is applied to the modified reconstructed sample of chrominance component images, similar to step S1003 in FIG. 97. For example, the modified reconstructed image sample may be clipped such that it is in the range [0, 1023] for a 10-bit output. When the filtered reconstructed luma component image samples generated by the CCALF process are clipped, it may be optional to clip the modified reconstructed chroma component image sample in some embodiments.

[0574] FIG. 99 is a block diagram illustrating a functional configuration of an encoder and decoder according to an embodiment. In this embodiment, the clipping process is applied to the modified reconstructed sample of chrominance component images, similar to step S1003 in FIG. 97. The clipping process is not applied to filtered reconstructed luminance samples generated by the CCALF process. The filtered value of the reconstructed chrominance samples generated by the ALF process must not be clipped, as shown by "no clipping" in FIG. 99. In other words, the reconstructed image sample to be modified is generated using the filtered value (chrominance ALF signal) and the difference value (Cb/Cr CCALF), and no clipping is applied to the output of the generated sample value.

[0575] FIG. 100 is a block diagram illustrating a functional configuration of an encoder and decoder according to an embodiment. In this embodiment, the clipping process is applied to the filtered reconstructed luminance signal samples generated by the CCALF process ("output sample clipping") and to the modified reconstructed chrominance component image samples ("post-stack clipping"). The filtered value of the reconstructed chrominance samples generated by the ALF process is not clipped (“no clipping”). As an example, the cut range applied to the filtered reconstructed sample of luminance component images may be [-2^15, 2^15-1] or [-2^7, 2^7-1].

[0576] FIG. 101 shows another example in which the clipping process is applied to filtered reconstructed luma samples generated by the CCALF process ("output clipping"), to modified reconstructed chrominance component image samples ("post-stack clipping"), and to filtered reconstructed samples. chrominance signals generated through the ALF process ("clipping"). In other words, the output values from the CCALF process and the ALF process for chrominance signals are separately clipped and clipped again after they are added. In this embodiment, the modified reconstructed chrominance component image sample does not need to be clipped. As an example, the final output from the ALF process for chrominance signals may be clipped to a 10-bit value. As an example, the cut range applied to the filtered reconstructed luminance component image samples may be [-2^15, 2^15-1] or [-2^7, 2^7-1]. This range may be fixed or may be adaptively determined. In either case, the range may be signaled in header information, for example in an SPS (Sequence Parameter Set) or APS (Adaptation Parameter Set). In the case where non-linear ALF is used, clipping parameters can be set for "post-stack clipping" in FIG. 101.

[0577] The reconstructed samples of luma component images to be filtered by the CCALF process may be adjacent samples that are adjacent to the currently reconstructed sample of chrominance component images. Thus, a modified current reconstructed image sample may be formed by summing the filtered value of neighboring luma component image samples adjacent to the current image sample with the filtered value of the current chrominance component image sample. The filtered value of the luminance component image samples may be represented as a difference value.

[0578] The processes disclosed in this aspect can reduce the hardware size of the internal storage required to store filtered sample image values.

[0579] Second aspect

Fig. 102 is a flowchart for an example of a processing flow 2000 for decoding an image applying a CCALF process using specified information according to the second aspect. The processing flow 2000 may be performed, for example, by the decoder 200 of FIG. 67, etc.

[0580] In step S2001, the clipping parameter is parsed from the bit stream. The cut parameter may be parsed from the VPS, APS, SPS, PPS, slice header, at the CTU or TU level, as described in FIG. 103. Fig. 103 is a conceptual diagram indicating the location(s) of the clipping parameters. The parameter described in FIG. 103 may be replaced by another type of cut parameter, flag, or index. Two or more cut parameters may be parsed from two or more sets of parameters in the bitstream.

[0581] In step S2002, the difference is clipped using the clipping parameter. The difference is generated based on the reconstructed image samples of the first component (for example, the difference value (Cb/Cr CCALF) in Fig. 98-101). As an example, the first component is the luminance signal component, and the difference is the filtered reconstructed luminance signal samples generated by the CCALF process. In this case, the clipping process is applied to the filtered reconstructed luminance samples using the parsed clipping parameter.

[0582] The cutoff parameter limits the value so that it is within the required range. If the required range is [-3, 3], according to the example, the value 5 is clipped to 3 using the clip(-3, 3, 5) operation. In this example, the value -3 represents the lower range, and the value 3 represents the upper range.

[0583] The cut parameter may specify an index to extract the lower range and the upper range, as shown in FIG. 104(i). In this example, ccalf_luma_clip_idx[] represents the index, -range_array[] represents the lower range, and range_array[] represents the upper range. In this example, range_array[] is a specific range array, which may be different from the range array used for ALF. A specific array of ranges can be predefined.

[0584] The cutoff parameter may indicate a lower range and an upper range, as shown in FIG. 104(ii). In this example, -ccalf_luma_clip_low_range[] represents the lower range, and ccalf_luma_clip_up_range[] represents the upper range.

[0585] The cutoff parameter may indicate the overall range for both the lower range and the upper range, as shown in FIG. 104(iii). In this example, -ccalf_luma_clip_range represents the lower range, and ccalf_luma_clip_range represents the upper range.

[0586] The difference is generated by multiplying, dividing, summing, or subtracting at least two reconstructed first component image samples. The two reconstructed image samples, for example, may come from the current and adjacent image samples or two adjacent image samples. The locations of the current and neighboring image samples can be predetermined.

[0587] In step S2003, the reconstructed image sample of the second component different from the first component is modified using the clipped value. The clipped value may be a clipped value of the reconstructed sample of luminance signal component images. The second component may be a chrominance signal component. The modification may include an operation to multiply, divide, sum, or subtract the clipped value relative to the reconstructed sample of the second component images.

[0588] In step S2004, the image is decoded using the modified reconstructed image sample.

[0589] In the present disclosure, one or more clipping parameters for cross-component adaptive loop filtering are signaled in a bit stream. In this signaling, cross-component adaptive loop filtering syntax and adaptive loop filter syntax can be combined to simplify the syntax. In addition, with this signaling, the design of cross-component adaptive loop filtering can be more flexible to improve coding efficiency.

[0590] Clipping parameters may be specified or preset for encoders and decoders without being signaled. The clipping parameters can also be extracted using luminance signal information without being signaled. For example, clipping parameters corresponding to a large clipping range can be extracted if a strong gradient or edge is detected in the reconstructed luminance signal image, and clipping parameters corresponding to a short clipping range can be extracted if a weak gradient or edge is detected in the reconstructed luminance signal image.

[0591] Third aspect

Fig. 105 is a flowchart for an example of a processing flow 3000 for decoding an image applying a CCALF process using a filter coefficient according to the third aspect. The processing flow 3000 may be performed, for example, by the decoder 200 of FIG. 67, etc. The filtering factor is used in the filtering step of the CCALF process to generate a filtered reconstructed sample of luminance component images.

[0592] In step S3001, it is determined whether or not the filter coefficient is located in the predetermined symmetric filter region. Optionally, an additional step of judging whether or not the shape of the filter coefficients is symmetrical may be performed. Information indicating whether the sample of filter coefficients is symmetric or not may be encoded into a bitstream. If the shape is symmetrical, the locations of the coefficients that are within the symmetrical region may be determined or predetermined.

[0593] In step S3002, if the filter coefficient is within the specified symmetric region (Yes in step S3001), the filter coefficient is copied to the symmetric position, and a set of filter coefficients is generated.

[0594] In step S3003, filter coefficients are used to filter the reconstructed first component image samples. The first component may be a luminance signal component.

[0595] In step S3004, the filter output is used to modify the reconstructed sample of images of a second component different from the first component. The second component may be a chrominance signal component.

[0596] In step S3005, the image is decoded using the modified reconstructed image sample.

[0597] If the filter coefficients are not symmetrical ("No" in step S3001), all filter coefficients can be encoded from the bitstream, and a set of filter coefficients can be generated without copying.

[0598] This aspect may reduce the amount of information that must be encoded into the bitstream. Thus, only one of the filter coefficients, which are symmetric, may need to be encoded in the bit stream.

[0599] FIG. 106, 107, 108, 109 and 110 are conceptual diagrams of examples indicating locations of filter coefficients to be used in the CCALF process. In these examples, some coefficients included in the set of coefficients are signaled, provided that symmetry exists.

[0600] In particular, FIG. 106(a), 106(b), 106(c) and 106(d) indicate examples in which a portion of a set of CCALF coefficients (labeled by diagonal lines and grid patterns) are located in a given symmetrical region. In these examples, the symmetrical regions have a linearly symmetrical shape. Only some marked coefficients (marked by diagonal lines or grid patterns) and white coefficients may be encoded into the bitstream, and other coefficients may be generated by using the encoded coefficients. As other examples, only labeled coefficients can be generated and used in the filtering process. Other white coefficients (not marked by any pattern) should not be used in the filtering process.

[0601] FIG. 106(e), 106(f), 106(g), and 106(h) indicate examples in which the shape of the symmetrical region is horizontal, vertical, diagonal along with direction, point symmetrical, or point symmetrical with direction.

[0602] In these drawings, only a portion of the coefficients that are marked by diagonal lines or grid patterns may need to be encoded. The locations of the coefficients to be encoded may be determined or predetermined. For example, the coefficients may be encoded in a particular scanning order, and the coefficient that appears first may be encoded first. A coefficient in a symmetrical region whose coefficient is not encoded may be copied from a coefficient that is located at its symmetrical position. In some embodiments, it may be optional to process the coefficients based on symmetry. For example, when it is determined that the i-th coefficient is identical to the j-th coefficient in scan order, the process may simply copy the j-th value to the i-th value. The location may be determined based on other parameters.

[0603] FIG. 107(a), 107(b), 107(c) and 107(d) indicate examples in which a portion of a set of CCALF coefficients (labeled by diagonal lines and grid patterns) are located in a given symmetrical region. In these examples, the number of symmetric coefficients may differ. The number of symmetric coefficients may be determined, predetermined, or signaled at the frame level, slice level, or block level.

[0604] FIG. 107(e), 107(f), 107(g) and 107(h) indicate examples in which a portion of a set of CCALF coefficients (labeled by diagonal lines and grid patterns) are located in a given symmetrical region. In these examples, the symmetric coefficients on one symmetric side may be different from the corresponding coefficients on the other symmetric side, i.e. some coefficients (eg a set of coefficients) on one side are symmetrical with different coefficient values (eg a different set of coefficients) on the other side. As an example, only a portion of the "mesh pattern" coefficients on one symmetrical side may be encoded into a bit stream and copied to form the "diagonal line pattern" coefficients on the other symmetrical side.

[0605] FIG. 108(a), 108(b), 108(c), and 108(d) indicate examples of filter shapes in which the type of chrominance signals serves as the specific format. The particular format may, for example, be YUV 420 type 0. The marked coefficients (diagonal line pattern coefficients or grid pattern coefficients) are symmetrical with respect to the position of the chrominance signal type. The filter shape can be designed to be symmetrical with respect to the chrominance position of other YUV formats. For example, these filter shapes of FIG. 108(a)-108(d) may be used as a default, and other filter forms may be defined for use in the filtering process when the parameter encoded in the bitstream indicates other formats.

[0606] FIG. 108(e), 108(f), 108(g), and 108(h) indicate examples of filter shapes in which the type of chrominance signals serves as the specific format. The particular format may be a YUV chrominance signal format other than YUV 420 type 0. Different filter shapes may be used for other formats.

[0607] FIG. 109(a), 109(b), 109(c), and 109(d) indicate other examples of filter shapes. In fig. 109(a), the number of symmetric coefficients may be zero, and all coefficients are signaled independently. The number of symmetric coefficients need not necessarily be encoded into the bit stream. In fig. 109(b), the number of symmetrical coefficients may be one half of all coefficients.

[0608] FIG. 110(a), 110(b), 110(c) and 110(d) indicate other examples of filter shapes and signals to be encoded, with the scanning order indicated by arrows. In fig. 110(a), the raster scan order is applied to the filter coefficients regardless of the symmetry type. In fig. 110(b), the raster scan order is applied to the filter coefficients, regardless of the symmetry type, and only the white coefficients and the grid pattern coefficients are signaled in the bitstream in raster scan order. The decoder can use a LUT (lookup table) to duplicate the copied grid coefficients to form the diagonal line pattern coefficients. In fig. 110(c), the copied grid coefficients located in the symmetric region are scanned and signaled, and then the white coefficients located in the asymmetric region are scanned and signaled. In fig. 110(d), the coefficients located in the asymmetric region are scanned and signaled, and then the grid pattern coefficients are scanned and signaled.

[0609] FIG. 111 and 112 are conceptual diagrams of additional examples indicating locations of filter coefficients to be used in the CCALF process. In these examples, the symmetric positions, locations, or numbers of coefficients in the set of filter coefficients may be adaptive with respect to the type of chrominance signals.

[0610] FIG. 113 is a block diagram illustrating the functional configuration of a CCALF process executed by an encoder and decoder according to an embodiment. After filtering the luma frame using the generated filter coefficients, the output samples are applied to the chrominance frame. A filter with generated filter coefficients is applied to the output SAO frame of luminance signals. The filtered samples (CC ALF Cb and CC ALF Cr) are then added to the output chrominance ALF frame.

[0611] Fourth aspect

Fig. 114 is a flowchart for an example of a processing flow 4000 for decoding an image applying a CCALF process using a filter selected from a plurality of filters according to the fourth aspect. The processing sequence 4000 may be performed, for example, by the decoder 200 of FIG. 67, etc. This embodiment discloses methods for modifying reconstructed samples of a component using information from another component.

[0612] In step S4001, the parameter is determined. The parameter may be parsed from the VPS, APS, SPS, PPS, slice header, or at the CTU or TU level, as described in FIG. 103. The parameter is parsed from the bitstream to indicate a filter. For example, a parameter may specify an index to select a filter from a specific set of filters. The parameter can be parsed from the bitstream to indicate the chroma downsampling format as 4:4:4, 4:2:0, 4:2:2, or 4:1:1. The parameter can be parsed from the bitstream to indicate the color space as YCbCr or RGB. The parameter can be parsed from the bitstream to indicate the frame resolution as 4K, FHD, CIF, QCIF. The parameter can specify the color component as Y, Cb or Cr. The parameter can also be retrieved using luminance signal information without signaling. According to an example, a parameter corresponding to a filter with a small number of taps can be extracted if a strong gradient or edge is detected in the reconstructed luminance signal image, and a parameter corresponding to a filter with a large number of taps can be extracted if a weak gradient or edge is detected in the reconstructed signal image brightness As another example, a parameter corresponding to a high-tap filter may be extracted if a strong gradient or edge is detected in the reconstructed luminance signal image, and a parameter corresponding to a low-tap filter may be extracted if a weak gradient or edge is detected in the reconstructed image. brightness signal image.

[0613] In step S4002, a filter is selected from a plurality of filters based on the parameter. A variety of filters may have different shapes or sizes. Multiple filters can have an identical shape and have different coefficient values. The parameter can specify coefficient values that should be used to generate a set of filter coefficients.

[0614] FIG. 115 shows an example of a filter selection processing flow.

[0615] In step S4011, it is determined whether or not the parameter indicates a specific format. The format can be predefined. The specified format may specify the chrominance subsampling format as 4:4:4, 4:2:0, 4:2:2, or 4:1:1. A particular format may specify the color component as Y, Cb or Cr. A particular format may specify a color space as YCbCr or RGB. A specific format can indicate frame resolution as 4K, FHD, CIF, QCIF.

[0616] In step S4012, if it is determined that the parameter indicates a specific format ("Yes" in step S4011), the first filter of the plurality of filters is selected.

[0617] In step S4013, if it is determined that the parameter does not indicate a specific format ("No" in step S4011), a filter different from the first filter is selected from the plurality of filters. The shape, size or filter coefficient values may differ between S4012 and S4013.

[0618] FIG. 116 and fig. 117 illustrate some examples of filters. In fig. 116 showing filters (1a)-(1i), the total number of rows having the maximum number of coefficients is even (eg, 2, 4, or 6). In fig. 117 showing filters (2a)-(2i), the total number of rows having the maximum number of coefficients is odd (eg, 1, 3 or 5).

[0619] For example, the filter of FIG. 116 may be selected if the parameter indicates that a 4:2:0 chroma downsampling format is applied while the filter of FIG. 117 can be selected if the parameter indicates that a 4:2:2, 4:4:4 or 4:1:1 chroma subsampling format is being used. Selection of filters from Fig. 116 and fig. 117 can be reversed.

[0620] For example, the filter of FIG. 116 may be selected if the parameter indicates that Y is used to modify Cb or Cr, while the filter of FIG. 117 may be selected if the parameter indicates that Cb is used to modify Cr, or Cr is used to modify Cb. Selection of filters from Fig. 116 and fig. 117 can be reversed.

[0621] For example, the filter of FIG. 116 may be selected if the parameter indicates that the YCbCr color space is applied while the filter of FIG. 117 can be selected if the parameter indicates that the RGB color space is being used. Selection of filters from Fig. 116 and fig. 117 can be reversed.

[0622] For example, the first filter of FIG. 116 may be selected if the parameter indicates that the resolution of the images is high (eg, 4K or 8K) while a filter other than the first filter of FIG. 116 may be selected if the parameter indicates that the resolution of the images is small (eg, QCIF or CIF). The size of the two selected filters may differ. For example, filter (1a) may be selected for QCIF image resolution, filter (1c) may be selected for FHD image resolution, and filter (1e) may be selected for 8K image resolution.

[0623] For example, the first filter of FIG. 117 may be selected if the parameter indicates that the resolution of the images is large (eg, 4K or 8K) while a filter other than the first filter of FIG. 117 may be selected if the parameter indicates that the resolution of the images is small (eg, QCIF or CIF). The size of the two selected filters may differ. For example, filter (2a) may be selected for QCIF image resolution, filter (2c) may be selected for FHD image resolution, and filter (2e) may be selected for 8K image resolution.

[0624] In step S4003, the reconstructed first component image samples are filtered using the selected filter. The first component may be a luminance signal component. The filtering process comprises at least a multiplication, division, summation or subtraction operation on at least two reconstructed first component image samples. For example, the two reconstructed image samples may come from the current and adjacent image samples, or may come from two adjacent image samples. The locations of the current and neighboring image samples can be predetermined.

[0625] In step S4004, the reconstructed sample of images of the second component different from the first component is modified using the filter output. The second component may be a chrominance signal component. The modification involves an operation to multiply, divide, sum, or subtract the filtering output with the reconstructed image sample.

[0626] In step S4005, the image is decoded using the modified reconstructed image sample.

[0627] The present disclosure relates to adaptive selection of one filter from a plurality of filters for cross-component filtering. Different filters may have different shapes or sizes. Adaptive filter selection provides greater flexibility in cross-component filtering for improved coding efficiency.

[0628] More than one set of filters may be signaled. Different filter sets can come in different shapes and sizes. Which filter should be used can be parsed or determined afterwards (eg from filter_id).

[0629] Fifth aspect

Fig. 118 is a flowchart for an example of a processing flow 5000 for decoding an image applying a CCALF process using a parameter according to the fifth aspect. The processing flow 5000 may be performed, for example, by the decoder 200 of FIG. 67, etc.

[0630] In step S5001, the first parameter is parsed from the bitstream. The first parameter may be parsed from the VPS, APS, SPS, PPS, slice header, or at the CTU or TU level (FIG. 103, with "parameter" corresponding to the "first parameter"). The first parameter can specify the chroma subsampling format as 4:4:4, 4:2:0, 4:2:2 or 4:1:1. The first parameter can specify the color space as YCbCr or RGB. The first parameter can indicate the frame resolution in 4K, FHD, CIF, QCIF quality. The first parameter can specify the color component as Y, Cb or Cr.

[0631] In step S5002, it is determined whether or not the first parameter is equal to the determined value. A specific value can be predefined.

[0632] In step S5003, if it is determined that the first parameter is equal to a certain value ("Yes" in step S5002), the first number of coefficients is parsed from the bitstream. The first number of coefficients may be parsed from the VPS, APS, SPS, PPS, slice header, or at the CTU or TU level (FIG. 103, with "parameter" corresponding to the "first number of coefficients").

[0633] In step S5004, if it is determined that the first parameter is not equal to a certain value ("No" in step S5003), a second number of coefficients not equal to the first number of coefficients is parsed from the bitstream. The second number of coefficients may be parsed from the VPS, APS, SPS, PPS, slice header, or at the CTU or TU level (FIG. 103, with “parameter” corresponding to the “second number of coefficients”). The first number and the second number from step S5002 and step S5003 may be different.

[0634] For example, as shown in FIG. 119(i), the number of coefficients when the first parameter indicates that the 4:2:0 chrominance subsampling format is applied is different from the number of coefficients when the first parameter indicates that the 4:2:2, 4 chrominance subsampling format is applied: 4:4 or 4:1:1 applies.

[0635] For example, as shown in FIG. 119(ii), the number of coefficients when the first parameter indicates that the YCbCr color space is applied is different from the number of coefficients when the first parameter indicates that the RGB color space is applied.

[0636] For example, as shown in FIG. 119(iii), the number of coefficients when the first parameter indicates that Y is used to modify Cb or Cr is different from the number of coefficients when the first parameter indicates that Cb is used to modify Cr or Cr is used in order to modify Cb.

[0637] For example, as shown in FIG. 119(iv), the number of coefficients when the first parameter indicates that the resolution of the images is large (for example, 4K, or 8K) is different from the number of coefficients when the first parameter indicates that the resolution of the images is small (for example, QCIF or CIF ).

[0638] In FIG. 119, information such as chroma subsampling format and image resolution can be obtained if the SPS_id is encoded in APS.

[0639] In step S5005, the reconstructed first component image samples are filtered using the parsed coefficients. The filtering process comprises at least a multiplication, division, summation or subtraction operation on at least two reconstructed first component image samples. The two reconstructed image samples, for example, may come from the current and adjacent image samples, or may come from two adjacent image samples. The locations of the current and neighboring image samples can be predetermined.

[0640] In step S5006, the reconstructed sample of images of a component different from the first component is modified using the filter output. The modification involves an operation to multiply, divide, sum, or subtract the filtering output with the reconstructed image sample.

[0641] In step S5007, the image is decoded using the modified reconstructed image sample.

[0642] The present disclosure relates to adaptively extracting the number of filter coefficients for cross-component filtering. Adaptive extraction of the number of filter coefficients provides greater flexibility for cross-component filtering to improve coding efficiency.

[0643] More than one set of coefficients may be signaled. Different sets of coefficients may have different numbers of coefficients. Different sets of coefficients may have the same number of coefficients. The number of coefficients from these coefficient sets can be fixed. Which set of coefficients to use is parsed or determined afterwards (eg from coeff_set_id or filter_id).

[0644] Sixth aspect

Fig. 120 is a flowchart for an example of a processing flow 6000 for decoding an image applying a CCALF process using a parameter according to the sixth aspect. The processing flow 6000 may be performed, for example, by the decoder 200 of FIG. 67, etc.

[0645] At S6001, the process selects at least a set of reconstructed samples from the first component.

[0646] At S6002, the process extracts a value based on the selected set of reconstructed samples.

[0647] At S6003, the process filters the reconstructed samples based on the extracted value.

[0648] At S6004, the process modifies the reconstructed image sample of the second component using the filtering output

[0649] At S6005, the process decodes the image using the filtered reconstructed image sample.

[0650] FIG. 121, 122 and 123 are conceptual diagrams illustrating examples of generating a luma component CCALF value (see step S6002) for a current chrominance signal sample by calculating a weighted average of neighboring samples. In other words, in this example, the CCALF value of the luma samples for the chroma sample is generated by calculating the weighted sum of the luma samples located in the adjacent chrominance sample region. The luma samples include a sample adjacent to the chrominance sample.

[0651] In FIG. 121, the location indicated by the diamond shape is the location of the current chroma sample. For example, the value corresponding to the location (curr) for CCALF can be extracted by calculating the average of neighboring luminance signal samples that are labeled with grid patterns. White luminance samples should not be used for the averaging process. In other words, the value for CCALF may be retrieved by accessing the sample value of a luma sample adjacent to the current chrominance sample. Two such luminance signal samples are provided in the example of FIG. 121.

[0652] FIG. 122 describes example equations for calculating the CCALF value. The CCALF value can be extracted by using filter coefficient values and sample luminance signal values. The filter coefficient value is multiplied by a subconstruct of two values of the adjacent luminance signal sample. The luminance sample used in each subconstruct calculation can be located adjacent to the current chrominance sample. The form of the equations may be identical to the form used in the ALF filtering process. In some embodiments, if the filter coefficient value is less than 64, the coefficient value may be set to zero.

[0653] As described in FIG. 123, different numbers of luminance samples may be averaged, and the number of averaged adjacent luminance samples may be preset or signaled at the frame, slice, or block level. The positions of averaged neighboring luminance samples may be preset or signaled at the frame, slice, or block level. Weights of averaged neighboring luminance samples may be preset or signaled at the frame, slice, or block level.

[0654] FIG. 124 and 125 are conceptual diagrams illustrating examples of generating a luma component CCALF value for a current sample by calculating a weighted average of neighboring samples, wherein the locations of neighboring samples are determined adaptively with respect to the type of chrominance signals. In other words, the locations of the luma samples to be used in the weighted calculation are determined based on the location of the current chrominance sample.

[0655] Samples labeled with different patterns may represent different weights. The number of average samples may be adaptive (can match) with respect to the type of chrominance signals. The weighting coefficients of the average samples can be adaptive (can match) with respect to the type of chrominance signals.

[0656] FIG. 126 and 127 are conceptual diagrams illustrating examples of generating a CCALF value of a luminance signal component by applying a bit shift to a weighted calculation output value. In other words, a downscaling shift process is applied to the filtered value of the luma samples, identical to the ALF process. In some embodiments, if the coefficient value is less than 64, the coefficient value may be set to zero.

[0657] The number of bits to shift for CCALF is represented as x. The value x may be defined to be identical to the value in the ALF process. In some examples, the value of x may be fixed to 10. In some examples, the value of x may be fixed to 7.

[0658] Seventh aspect

Fig. 128 is a flowchart for an example of a processing flow 7000 for decoding an image applying a CCALF process using a parameter according to the seventh aspect. The processing flow 7000 may be performed, for example, by the decoder 200 of FIG. 67, etc. The following describes methods for determining the reconstructed samples that should be filtered using one or more parameters.

[0659] In step S7001, one or more parameters are parsed from the bit stream. One or more parameters may be encoded in at least one of the APS, SPS, PPS, slice header, or CTU level, as shown in FIG. 129. Fig. 130 shows exemplary processes for retrieving one or more parameters.

[0660] One or more parameters may be in the SPS. The slice first finds the PPS according to the PPS_id, which is encoded in the slice. PPS then finds the SPS according to the SPS_id, which is encoded in PPS. Through this connection, the slice can retrieve one or more parameters in the SPS, as shown in FIG. 130(a).

[0661] One or more parameters may be in a frame-level parameter set, such as a PPS. The slice first finds the PPS according to the PPS_id, which is encoded in the slice. Through this connection, the slice can retrieve one or more parameters in the PPS, as shown in FIG. 130(b).

[0662] One or more parameters may be in the APS. The slice first finds the APS according to the APS_id, which is encoded in the slice. Through this connection, the slice can retrieve one or more parameters in the APS, as shown in FIG. 130(c).

[0663] One or more parameters may be in the slice (Fig. 130(d)). A slice may receive one or more parameters from its internal header or data.

[0664] The one or more parameters may include a first parameter that selects the size of the samples to be modified. One or more parameters may indicate whether or not the CCALF process is activated. The one or more parameters may include parameters indicating whether or not the CCALF process is enabled and parameters indicating values of filter coefficients to be used.

[0665] The samples may be grouped into a square shape having a specific size, for example, 4x4, 8x8, 16x16, 16x16, 32x32, 64x64, or 128x128 samples.

[0666] The first parameter may be parsed before the slice header or slice data is parsed. For example, the first parameter may be parsed from APS, SPS, or PPS.

[0667] The first parameter may be parsed from the slice header.

[0668] The first parameter can be parsed from the coding tree units (CTU) data.

[0669] The first parameter may depend on the type of chroma subsampling or CTU size or both. If the chrominance subsampling type is 4:4:4, the block size selected by the first parameter may be 4x4, 8x8, 16x16, 32x32, 64x64, or 128x128. If the chrominance subsampling type is 4:2:2 or 4:2:0, the block size selected by the first parameter may be 4x4, 8x8, 16x16, 32x32, or 64x64.

[0670] The first parameter may depend on the CTU size, and the selected block size cannot exceed its CTU size.

[0671] The one or more parameters may include a second parameter that indicates whether or not the block of samples should be filtered.

[0672] The second parameter may be a flag with a value of 1 or 0, with 1 indicating that the reconstructed samples should be modified and 0 indicating that the reconstructed samples should not be modified.

[0673] The second parameter may be parsed before the first coding tree unit (CTU) data is parsed. For example, the second parameter may be parsed from APS, SPS, PPS, slice header, or slice data.

[0674] The second parameter can be parsed from the coding tree units (CTU) data. Fig. 131 shows sample values of the second parameter. A plurality of second parameters may indicate whether or not a plurality of blocks in a coding tree unit (CTU) having a particular size should be modified.

[0675] The second parameters may indicate whether or not modification of the reconstructed samples in the frame or sequence is disabled. If the second parameters indicate that modification of the reconstructed samples is deactivated, step S7002 of FIG. 128 should lead to step S7005 directly, which corresponds to the "No" branch in FIG. 128.

[0676] The one or more parameters may include a parameter that can be parsed using non-arithmetic encoding, for example, fixed length encoding, exponential Golomb encoding, or VLC.

[0677] The one or more parameters may include a parameter that can be parsed using an arithmetic encoding, for example, CAVLC or CABAC.

[0678] According to an example, as shown in FIG. 132, the second parameter may be parsed using arithmetic encoding before parsing the first encoding tree unit data in the slice and then the byte alignment or bit alignment data. In this example, arithmetic encoding initialization to parse parameters after the second parameter in an identical slice can be used.

[0679] In step S7002, it is determined whether or not a filter should be used based on the parsed parameters.

[0680] If a filter is to be used, in step S7003, at least the reconstructed sample from the first component is filtered. The first component may represent luminance samples.

[0681] In step S7004, the reconstructed samples are modified using at least one filtered reconstructed sample from a component different from the first component. The component different from the first component may be chroma samples.

[0682] In step S7005, the block of image samples is decoded using the modified reconstructed samples.

[0683] The present disclosure illustrates the characteristics of one or more parameters at multiple levels for filtering, including generation methods, functions, and encoding methods. Using these control parameters, filtering design can be optimized to reduce the number of encoding bits, improve high-frequency components using modified samples, and reduce redundancy between different channels, thereby improving image quality.

[0684] FIG. 133 is a conceptual diagram of a variation of this embodiment.

[0685] One or more parameters may depend on a segment of a coding tree unit. When the segment has a size different from the size indicated by the first parameter, the second parameter indicating whether or not the segment is filtered is not encoded and filtering of the segment is deactivated. In this example, the encoded bits of the second parameters are decremented.

[0686] The shape or samples described in the seventh aspect may be replaced by a rectangular or non-rectangular shaped segment. Examples of the non-rectangular-shaped segment may be at least one of a triangular-shaped segment, an L-shaped segment, a pentagonal-shaped segment, a hexagonal-shaped segment, and a polygonal-shaped segment, as shown in FIG. 133.

[0687] Eighth aspect

Fig. 134 is a flowchart for an example of a processing flow 8000 for decoding an image applying a CCALF process using a parameter according to the eighth aspect. The processing sequence 8000 may be performed, for example, by the decoder 200 of FIG. 67, etc.

[0688] In step S8001, it is determined whether or not the first sample of the first component is outside the virtual boundary.

[0689] In step S8002, if it is determined that the first sample of the first component is outside the virtual boundary, the second sample of the first component is copied to the first sample, with the second sample located inside the virtual boundary.

[0690] In step S8003, the reconstructed sample of the first component, which includes the first and second samples, is filtered.

[0691] In step S8004, the reconstructed sample of a component different from the first component is modified using the filter output.

[0692] In step S8005, the modified reconstructed sample is used to decode the image.

[0693] The padding method on S8002 may be identical regardless of the chrominance signal sampling format. For example, symmetrical complement can be used. The padding method, for example, may vary depending on the sampling format of the chrominance signals between symmetrical padding and unbalanced padding.

[0694] The first component may be luma samples, and the other component whose sample values are modified may be chrominance samples. The chrominance signal may be Cb, Cr, or both.

[0695] FIG. 135 is a flowchart for an example of a processing flow 8100 for decoding an image applying a CCALF process using a parameter according to the eighth aspect. The processing flow 8100 may be performed, for example, by the decoder 200 of FIG. 67, etc.

[0696] In S8101, it is determined whether the type of chrominance samples is a first type or not. Fig. 136 shows exemplary locations of chroma sample types 0-5.

[0697] At S8102, if it is determined that the type of the chrominance signal samples is of the first type, the first sample and the second sample are used when filtering the reconstructed samples of the first component, wherein the first sample is duplicated from the second reconstructed sample.

[0698] At S8103, if it is determined that the type of the chrominance signal samples is not of the first type, the second sample and the third sample are used when filtering the reconstructed samples of the first component, and the first sample is eliminated in which the third sample is different from the first or second sample.

[0699] At S8104, the reconstructed sample from a component different from the first component is modified using the filter output. For example, the first component indicates the luminance, and the filtering output is added to the reconstructed sample from the chrominance component. In another example, the first component indicates the chrominance component Cr, and the filtering output is added to the reconstructed sample from the chrominance component Cb. In another example, the filter output and the reconstructed sample may be added, subtracted, multiplied, divided, or subjected to any combination of mathematical processes to produce a modified reconstructed sample.

[0700] At S8105, the modified reconstructed sample is used to decode the image. For example, the modified reconstructed sample is stored in a reference frame buffer.

[0701] FIG. 137, 138 and 139 are conceptual diagrams of examples of symmetrical complement.

[0702] For example, in S8002 of FIG. 134 if the chroma samples type is 0 or 1 and the virtual boundary is between C0 and C2 as shown in FIG. 137(a), the reconstructed sample value of the second sample (C15) is duplicated into the first sample (C17). Likewise, the reconstructed sample value of the second sample (C2) is duplicated into the first sample (C0).

[0703] As another example, at S8002, if the chroma samples type is 0 or 1, and the virtual boundary is between C15 and C17, as shown in FIG. 137(b), the reconstructed sample value of the second sample (C2) is duplicated into the first sample (C0). Likewise, the reconstructed sample value of the second sample (C15) is duplicated into the first sample (C17).

[0704] As another example, at S8002, if the type of chroma samples is 0 or 1, and the virtual boundary is between C2 and C6, as shown in FIG. 137(c), the reconstructed sample values of the second samples (C10, C11 and C12) are duplicated into the first samples (C14, C15, C16 and C17). Likewise, the reconstructed sample values of the second samples (C5, C6 and C7) are duplicated into the first samples (C0, C1, C2 and C3).

[0705] As another example, at S8002, if the type of chrominance samples is 0 or 1, and the virtual boundary is between C11 and C15, as shown in FIG. 137(d), the reconstructed sample values of the second samples (C5, C6 and C7) are duplicated into the first samples (C0, C1, C2 and C3). Likewise, the reconstructed sample values of the second samples (C10, C11 and C12) are duplicated into the first samples (C14, C15, C16 and C17).

[0706] FIG. 138 and 139 show examples of samples that are duplicated in FIGS. 137(a) and FIG. 137(c), respectively.

[0707] After duplication, the duplicate samples are used when filtering the reconstructed samples of the first component.

[0708] FIG. 140, 141, 142, and 143 are conceptual diagrams of examples of unsymmetrical complement.

[0709] For example, in S8002, if the chroma samples type is 2 or 3, and the virtual boundary is between C0 and C2, as shown in FIG. 140(a), the reconstructed sample values of the second samples (C10, C11 and C12) are duplicated into the first samples (C14, C15, C16 or C17). Likewise, the reconstructed sample value of the second sample (C2) is duplicated into the first sample (C0).

[0710] In another example S8002, if the chroma samples type is 2 or 3, and the virtual boundary is between C15 and C17, as shown in FIG. 140(b), the recovered sample value of the second sample (C15) is duplicated into the first sample (C17).

[0711] In another example S8002, if the chroma samples type is 2 or 3, and the virtual boundary is between C2 and C6, as shown in FIG. 140(c), the reconstructed sample values of the second samples (C4, C5, C6, C7 and C8) are duplicated into the first samples (C9, C10, C11, C12, C13, C14, C15, C16 and C17). Likewise, the reconstructed sample values of the second samples (C5, C6 and C7) are duplicated into the first samples (C0, C1, C2 and C3).

[0712] In another example S8002, if the chroma samples type is 2 or 3, and the virtual boundary is between C11 and C15, as shown in FIG. 140(d), the reconstructed sample values of the second samples (C5, C6 and C7) are duplicated into the first samples (C0, C1, C2 and C3).

[0713] FIG. 141, 142 and 143 show examples of samples that are duplicated in FIGS. 140(a), fig. 140(c) and FIG. 140(d), accordingly.

[0714] After duplication, the duplicate samples are used when filtering the reconstructed samples of the first component.

[0715] FIG. 144, 145, 146, and 147 are conceptual diagrams of additional examples of asymmetrical complementation.

[0716] In example S8002, if the chroma samples type is 4 or 5, and the virtual boundary is between C0 and C2, as shown in FIG. 144(a), the reconstructed sample value of the second sample (C2) is duplicated into the first sample (C0).

[0717] In another example S8002, if the chroma samples type is 4 or 5, and the virtual boundary is between C15 and C17, as shown in FIG. 144(b), the reconstructed sample values of the second samples (C5, C6 and C7) are duplicated into the first samples (C0, C1, C2 and C3). Likewise, the reconstructed sample value of the second sample (C15) is duplicated into the first sample (C17).

[0718] In another example S8002, if the chroma samples type is 4 or 5, and the virtual boundary is between C2 and C6, as shown in FIG. 144(c), the reconstructed sample value of the second sample (C15) is duplicated into the first sample (C17). Likewise, the reconstructed sample values of the second samples (C5, C6 and C7) are duplicated into the first samples (C0, C1, C2 and C3).

[0719] In another example S8002, if the chroma samples type is 4 or 5, and the virtual boundary is between C11 and C15, as shown in FIG. 144(d), the reconstructed sample values of the second samples (C9, C10, C11, C12 and C13) are duplicated into the first samples (C0, C1, C2, C3, C4, C5, C6, C7 and C8).

[0720] FIG. 145, 146 and 147 show examples of samples that are duplicated in FIGS. 144(b), FIG. 144(c) and FIG. 144(d), accordingly.

[0721] After duplication, the duplicate samples are used when filtering the reconstructed samples of the first component.

[0722] FIG. 148, 149, and 150 are conceptual diagrams of additional examples of symmetrical complementation.

[0723] In example S8002, if the type of chrominance samples is 0, 2 or 4, and the virtual boundary is between C4 and C5, as shown in FIG. 148(a), the reconstructed sample values of the second samples (C7 and C12) are duplicated into the first samples (C8 and C13). Likewise, the reconstructed sample values of the second samples (C5 and C10) are duplicated into the first samples (C4 and C9).

[0724] In another example S8002, if the type of chroma samples is 0, 2 or 4, and the virtual boundary is between C7 and C8, as shown in FIG. 148(b), the reconstructed sample values of the second samples (C5 and C10) are duplicated into the first samples (C4 and C9). Likewise, the reconstructed sample values of the second samples (C7 and C12) are duplicated into the first samples (C8 and C13).

[0725] In another example S8002, if the type of chroma samples is 0, 2 or 4, and the virtual boundary is between C5 and C6, as shown in FIG. 148(c), the reconstructed sample values of the second samples (C2, C6, C11 and C15) are duplicated into the first samples (C1, C4, C5, C9, C10 and C14). Likewise, the reconstructed sample values of the second samples (C2, C6, C11 and C15) are duplicated into the first samples (C3, C7, C8, C12, C13 and C16).

[0726] In another example S8002, if the type of chroma samples is 0, 2 or 4, and the virtual boundary is between C6 and C7, as shown in FIG. 148(d), the reconstructed sample values of the second samples (C2, C6, C11 and C15) are duplicated into the first samples (C1, C4, C5, C9, C10 and C14). Likewise, the reconstructed sample values of the second samples (C2, C6, C11 and C15) are duplicated into the first samples (C3, C7, C8, C12, C13 and C16).

[0727] FIG. 149 and fig. 150 show examples of samples that are duplicated in FIG. 148(a) and FIG. 148(c), accordingly.

[0728] After duplication, the duplicate samples are used when filtering the reconstructed samples of the first component.

[0729] FIG. 151, 152, 153, 154, and 155 are conceptual diagrams of additional examples of asymmetrical complementation.

[0730] In example S8002, if the chroma samples type is 1, 3 or 5, and the virtual boundary is between C4 and C5, as shown in FIG. 151(a), the reconstructed sample values of the second samples (C5 and C10) are duplicated into the first samples (C4 and C9).

[0731] In another example S8002, if the type of chroma samples is 1, 3 or 5, and the virtual boundary is between C7 and C8, as shown in FIG. 151(b), the reconstructed sample values of the second samples (C2, C6, C11 and C15) are duplicated into the first samples (C1, C4, C5, C9, C10 and C14). Likewise, the reconstructed sample values of the second samples (C7 and C12) are duplicated into the first samples (C8 and C13).

[0732] In another example S8002, if the type of chroma samples is 1, 3 or 5, and the virtual boundary is between C5 and C6, as shown in FIG. 151(c), the reconstructed sample values of the second samples (C7 and C12) are duplicated into the first samples (C8 and C13). Likewise, the reconstructed sample values of the second samples (C2, C6, C11 and C15) are duplicated into the first samples (C1, C4, C5, C9, C10 and C14).

[0733] FIG. 152, 153 and 154 show examples of samples that are duplicated in FIGS. 151(a), fig. 151(b) and FIG. 151(c), respectively.

[0734] FIG. 155 shows additional examples of padding with a horizontal and vertical virtual border.

[0735] After duplication, the duplicate samples are used when filtering the reconstructed samples of the first component.

[0736] The present disclosure illustrates the addition or duplication of samples used in a filter based on the type of chrominance samples and the virtual boundary location in the filter. This method of supplementing or duplicating samples improves the quality of frames.

[0737] Varieties

The type of chrominance signal samples may be replaced by other information that indicates the relationship between the first component and another component different from the first component.

[0738] The type of chroma samples may be replaced by a flag that selects symmetrical or unsymmetrical padding within a virtual boundary, where 0 selects symmetrical padding and 1 selects single-ended padding, or 1 selects symmetrical padding and 0 selects single-ended padding.

[0739] The flag may be signaled from the bitstream or may be retrieved.

[0740] The default value of the flag may be symmetrical padding within the virtual boundary.

[0741] The default value of the flag may be unsymmetrical padding within the virtual boundary.

[0742] For example, a flag may be retrieved based on the on/off state of another filter. For example, if the ALF filter is enabled, the flag may select symmetrical complement. If the ALF filter is disabled, the flag can select unsymmetrical padding.

[0743] As another example, if the ALF filter is enabled, the flag may select single-ended complement. If the ALF filter is turned off, the flag can select symmetrical complement.

[0744] Other filters that may be used include LMCS, SAO, DBF and other post filters.

[0745] In some embodiments, the flag may be set based on a profile.

[0746] The virtual boundary may be replaced by a frame, slice, brick, tile, or subframe boundary.

[0747] FIG. 156 is a block diagram illustrating the functional configuration of an encoder and decoder according to an example in which symmetrical padding is used at virtual boundary locations for ALF, and symmetrical or non-symmetrical padding is used at virtual boundary locations for CCALF based on the type of chroma samples and the virtual boundary location.

[0748] FIG. 157 is a block diagram illustrating the functional configuration of an encoder and decoder according to an example in which other symmetrical padding is used in virtual boundary locations for ALF, and one-sided padding is used in virtual boundary locations for CCALF.

[0749] FIG. 158 is a conceptual diagram illustrating an example of one-sided padding with a horizontal or vertical virtual border.

[0750] FIG. 159 is a conceptual diagram illustrating an example of one-way padding with a horizontal and vertical virtual border.

[0751] The input to CCALF (reconstructed samples of the first component used for filtering) is not limited to the SAO output. The input may be provided from the output of a luma chrominance scaling (LMCS) transform, a bilateral/Hadamard filter or a deblocking filter, or any combination of post-filters.

[0752] CCALF can be enabled/disabled in each block. A block must not necessarily overlap more than one CTU. Since ALF can be enabled/disabled in a CTU, CCALF can be enabled for a CTU that has ALF enabled.

[0753] One or more aspects disclosed herein may be performed in combination with at least a portion of the other aspects in the present disclosure. In addition, one or more aspects disclosed herein may be performed by combining, with other aspects, portions of the processes indicated in any of the flowcharts of the aspects, portions of the configuration of any of the devices, portions of syntaxes, etc. . The aspects described with reference to the encoder component may be performed similarly by the corresponding decoder component.

[0754] Implementations and Applications

As described in each of the above-described embodiments, each functional or operating unit can typically be implemented, for example, as an MPU (microprocessor) and a memory device. In addition, processes carried out by each of the functional blocks may be implemented as a program execution unit such as a processor that reads and executes software (program) recorded on a recording medium such as ROM. Software can be distributed. The software may be recorded on a variety of recording media, such as a semiconductor storage device. It should be noted that each functional block can also be implemented as hardware (specialized circuitry). Various combinations of hardware and software may be used.

[0755] In addition, the processing described in each of the embodiments may be implemented through integrated processing using a single piece of equipment (system) and, alternatively, may be implemented through decentralized processing using multiple pieces of equipment. Moreover, the processor that executes the above-described program may be a single processor or a plurality of processors. In other words, integrated processing may be performed, and alternatively, decentralized processing may be performed.

[0756] Embodiments of the present disclosure are not limited to the above exemplary embodiments; Various modifications may be made to the exemplary embodiments, the effects of which are also included within the scope of the embodiments of the present disclosure.

[0757] The following describes application examples of the moving frame encoding method (image encoding method) and the moving frame decoding method (image decoding method) described in each of the above embodiments, as well as various systems that implement the example applications. Such a system may be characterized as including an image encoder that uses an image encoding method, an image decoder that uses an image decoding method, and an image codec that includes both an image encoder and an image decoder. Other configurations of such a system can be modified for each individual case.

[0758] Examples of use

Fig. 160 illustrates a general configuration of a content delivery system ex100 suitable for implementing a content distribution service. The area in which the communication service is provided is divided into cells of required sizes, and base stations ex106, ex107, ex108, ex109 and ex110, which are fixed wireless stations in the illustrated example, are located in the corresponding cells.

[0759] In the content delivery system ex100, devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 through an Internet service provider ex102 or a communications network ex104 and base stations ex106- ex110. The content delivery system ex100 can combine and connect any combination of the above devices. In various implementations, devices may directly or indirectly communicate with each other through the telephone network or short-range communications rather than through the ex106-ex110 base stations. Additionally, the streaming server ex103 can connect with devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114 and a smartphone ex115, for example, via the Internet ex101. The ex103 streaming server can also connect, for example, to a terminal on a public hotspot on an ex117 aircraft via an ex116 satellite.

[0760] It should be noted that instead of base stations ex106-ex110, wireless access points or public access points may be used. The streaming server ex103 can connect to the communications network ex104 directly rather than via the Internet ex101 or Internet service provider ex102, and can connect to the aircraft ex117 directly rather than via a satellite ex116.

[0761] The camera ex113 may be a device capable of capturing still images and video, such as a digital camera. The ex115 smartphone can be a smartphone, a cellular phone or a personal mobile phone system (PHS) phone that can operate according to mobile phone system standards for 2G, 3G, 3.9G and 4G systems, as well as 5G systems next generation.

[0762] Home appliance ex114, for example, is a refrigerator or device included in a home fuel cell cogeneration system.

[0763] In the content delivery system ex100, a terminal including an image and/or video capture function allows, for example, live streaming through a connection to a streaming server ex103, for example, through a base station ex106. When live streaming, a terminal (eg, a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or an airplane terminal ex117) may perform the encoding processing described in the above embodiments for still image content or video content captured by the user through the terminal, can multiplex video data obtained through encoding and audio data obtained through audio encoding corresponding to the video, and can transmit the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to one aspect of the present disclosure.

[0764] Streaming server ex103 streams transmitted content data to clients that request the stream. Examples of the client include a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and airplane terminals ex117 that are capable of decoding the above-described encoded data. Devices that receive streaming data can decode and play back the received data. In other words, the devices may function as an image decoder in accordance with one aspect of the present disclosure.

[0765] Decentralized processing

The ex103 streaming server can be implemented as a plurality of servers or computers that share tasks such as processing, recording, and streaming data. For example, the ex103 streaming server may be implemented as a content delivery network (CDN) that streams content over a network connecting multiple edge servers located around the world. In a CDN, an edge server physically near the client can be dynamically assigned to the client. Content is cached and streamed to the edge server to reduce loading times. In the event of, for example, a certain type of error or connection change, such as due to a spike in traffic, data can be streamed reliably at high speeds because affected parts of the network can be avoided, for example by dividing processing among multiple edge servers or switching streaming loads transfer to another edge server and continue streaming.

[0766] Decentralization is not limited to simply sharing processing for streaming; Encoding of the captured data can be shared between and performed via terminals, on the server side, or both. In one example, in typical encoding, processing is performed in two loops. The first loop serves to detect how complex the image is on a frame-by-frame or scene-by-scene basis, or to detect encoding load. The second circuit provides processing that preserves image quality and improves encoding efficiency. For example, it is possible to reduce the processing load of the terminals and improve the quality and efficiency of content encoding by causing the terminals to perform a first encoding loop and causing the server side that receives the content to perform a second encoding loop. In such a case, when a decoding request is received, the encoded data resulting from the first loop performed by one terminal can be received and reproduced at the other terminal in approximately real time. This allows for smooth, real-time streaming.

[0767] In another example, camera ex113, etc. extracts the feature count (number of features or characteristics) from the image, compresses the data associated with the feature count as metadata, and transmits the compressed metadata to the server. For example, the server determines the significance of an object based on the number of features and changes the quantization precision accordingly to perform compression appropriate for the semantic meaning (or content significance) of the image. These feature counts are particularly effective in improving the accuracy and efficiency of predicting motion vectors during a second pass of compression performed by the server. In addition, coding that has a relatively low processing load, such as variable length coding (VLC), can be processed by the terminal, and coding that has a relatively high processing load, such as context adaptive binary arithmetic coding (CABAC), can be processed via the server.

[0768] In yet another example, cases arise in which multiple videos of an approximately identical scene are captured through multiple terminals, for example, in a stadium, a shopping mall, or a factory. In such a case, for example, encoding can be decentralized by dividing processing tasks among multiple terminals that capture video and, if necessary, other terminals that do not capture video, and a server on a per-unit basis. The units, for example, can be groups of frames (GOPs), frames, or tiles resulting from dividing a frame. This allows you to reduce loading times and achieve streaming that is closer to real time.

[0769] Since the videos have approximately the same scene, control and/or instructions can be performed by the server such that videos captured through the terminals can be cross-referenced. In addition, the server may receive encoded data from the terminals, change the reference relationship between data elements, or adjust or replace the frames themselves, and then perform encoding. This allows you to generate a stream with increased quality and efficiency for individual data elements.

[0770] In addition, the server may stream video data after performing transcoding to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (for example, VP9), can convert (H.264 to (H.265, etc.)

[0771] Thus, encoding may be performed by a terminal or one or more servers. Accordingly, although the device that performs the encoding is referred to as a “server” or “terminal” in the following description, some or all of the processes performed by the server may be performed by the terminal, and similarly, some or all of the processes performed by the terminal may be performed by the server . This also applies to decoding processes.

[0772] 3D Multi-View Mode

There is an increase in the use of images or videos that are combined from images or videos of different scenes captured in parallel, or an identical scene captured from different angles through multiple terminals, such as the ex113 camera and/or the ex115 smartphone. Video captured by the terminals may be combined, for example, based on a separately obtained relative positional relationship between terminals or regions in the video having matching feature points.

[0773] In addition to encoding two-dimensional moving frames, the server may encode the still image based on scene analysis of the moving frame, for example, automatically or at a time specified by the user, and transmit the encoded still image to the receiving terminal. In addition, when the server can obtain the relative positional relationship between video capture terminals, in addition to two-dimensional moving frames, the server can generate three-dimensional scene geometry based on video of an identical scene captured from different angles. The server may separately encode 3D data generated from, for example, a point cloud, and, based on the result of recognizing or tracking a person or object using the 3D data, may select or reconstruct and generate videos to be transmitted to the receiving terminal from the videos captured through multiple terminals.

[0774] This allows the user to enjoy the scene by freely selecting videos corresponding to the video capture terminals, and allows the user to enjoy the content obtained by extracting video at a selected viewpoint from 3D data recovered from a plurality of images or video, video. Additionally, similar to video, audio can be recorded from relatively different angles, and the server can multiplex the audio from a particular angle or space with the corresponding video and transmit the multiplexed video and audio.

[0775] In recent years, content that is a composite of the real world and the virtual world, such as virtual reality (VR) and augmented reality (AR) style content, has also become popular. In the case of VR images, the server may create images from the left and right eye viewpoints and perform encoding that allows a link between the two viewpoint images, such as multi-view encoding (MVC), and alternatively, may encode the images as separate streams without a link . When images are decoded as separate streams, the streams can be synchronized during playback in such a way as to recreate the virtual 3D space according to the user's point of view.

[0776] In the case of AR images, the server may overlay virtual object information existing in virtual space with camera information representing real world space, for example, based on three-dimensional position or movement from the user's perspective. The decoder can receive or store virtual object information and 3D data, generate 2D images based on movement from the user's point of view, and then generate overlay data by transparently connecting the images. Alternatively, the decoder may transmit, to the server, motion from the user's point of view in addition to a request for virtual object information. The server may generate overlay data based on three-dimensional data stored in the server according to the received motion, and encode and stream the generated overlay data to the decoder. It should be noted that the overlay data typically includes, in addition to RGB values, an α value indicating transparency, and the server sets the α value for sections other than the object formed from the 3D data, for example, to 0, and may perform the encoding while these sections are transparent. Alternatively, the server may set the background to a specific RGB value, such as a chroma key, and generate data in which areas other than the object are set as the background. A specific RGB value can be predefined.

[0777] Decoding of similarly streamed data can be performed by the client (ie, terminals), on the server side, or shared between them. In one example, one terminal may transmit a receive request to a server, the requested content may be received and decoded by the other terminal, and the decoded signal may be transmitted to a device having a display. It is possible to reproduce data with high image quality by decentralizing processing and appropriately selecting content regardless of the processing characteristics of the communication terminal itself. In yet another example, while a television set, for example, receives image data that is large in size, an area of a frame such as a tile obtained by dividing the frame can be decoded and displayed on the personal terminal or terminals of the viewer or viewers of the television set. This allows viewers to share a view with large frames, as well as each viewer to check their assigned area or explore an area in more detail with a close-up view.

[0778] In situations in which multiple wireless connections are possible over short, medium and long distances, indoors or outdoors, it may be possible to transparently receive content using a streaming system standard such as MPEG DASH. The user can switch between real-time data by freely selecting a decoder or display equipment, including a user terminal, indoor or outdoor displays, etc. In addition, for example, using information regarding the user's position, decoding can be performed when switching which terminal processes decoding and which terminal processes content display. This allows information to be converted and displayed while the user is on the move on a route to a destination, on the wall of a nearby building into which a device capable of displaying the content is embedded, or on a piece of ground. In addition, it is also possible to switch the bit rate of the received data based on the reachability of the encoded data over the network, for example, when the encoded data is cached on a server readily accessible from the receiving terminal, or when the encoded data is copied to an edge server in a content delivery service.

[0779] (Web page optimization)

Fig. 161, for example, illustrates an example of a web page display screen on computer ex111. Fig. 162, for example, illustrates an example of a web page display screen on a smartphone ex115. As illustrated in FIG. 161 and fig. 162, a web page may include a plurality of image links that are links to image content, and the appearance of the web page may differ depending on the device used to view the web page. When multiple image links are viewable on the screen until the user explicitly selects the image link, either until the image link is approximated in the center of the screen, or the entire image link fits into the screen, display hardware (decoder) may display, as image links, still images included in the content, or I-frames, may display video, such as an animated GIF using multiple still images or I-frames, for example, or may only receive base layer and decode and display the video.

[0780] When an image reference is selected by the user, the display equipment performs decoding, for example, by providing the highest priority to the base layer. It should be noted that if there is information in the HTML code of the web page indicating that the content is scalable, the display equipment can decode down to the enhancement layer. Additionally, to facilitate real-time playback, before a selection is made, or when bandwidth is severely limited, the display equipment may reduce the delay between the time at which the leading frame is decoded and the time at which the decoded frame is displayed (i.e., e. the delay from the start of decoding the content until the content is displayed) by decoding and displaying only direct reference frames (I-frame, P-frame, forward reference B-frame). Still further, the display equipment may deliberately ignore the reference relationship between frames and approximately decode all B and P frames as direct reference frames and then perform normal decoding as the number of frames received in time increases.

[0781] Autonomous driving

When transmitting and receiving still image or video data such as two- or three-dimensional map information for autonomous or assisted driving, the receiving terminal may receive, in addition to image data belonging to one or more layers, information such as regarding weather or roadworks as metadata, and associate the metadata with image data when decoding. It should be noted that the metadata may be assigned on a per-layer basis and, alternatively, may simply be multiplexed with the image data.

[0782] In such a case, since the vehicle, unmanned vehicle, aircraft, etc. containing the receiving terminal is mobile, the receiving terminal can transparently receive and perform decoding when switching between base stations among the base stations ex106 to ex110 by transmitting information , indicating the position of the receiving terminal. In addition, according to the selection made by the user, the user's situation and/or the connection bandwidth, the receiving terminal can dynamically select the extent to which metadata is received or the extent to which, for example, map information is updated.

[0783] In the content delivery system ex100, the client can receive, decode and reproduce, in real time, encoded information transmitted by the user.

[0784] Streaming content from people

In the content delivery system ex100, in addition to long-lasting content with high image quality distributed by a video distribution entity, unicast or multicast streaming of short content with low image quality from a person is also possible. The popularity of such content from people is likely to increase even more. The server may first perform editing processing on the content before encoding processing to refine content from people. This can be achieved, for example, using the following configuration.

[0785] In real time, when video or image content is captured or after the content is captured and accumulated, the server performs recognition processing based on raw data or encoded data, such as capture error handling, scene retrieval processing, semantic meaning analysis, and /or object detection processing. Thereafter, based on the result of the recognition processing, the server (eg, by direction or automatically) edits the content, examples of which include: correction, such as correction of focus and/or motion blur; Removing scenes with low priority, such as scenes that have low brightness compared to other frames or are out of focus; adjusting the edges of objects; and regulation of color tones. The server encodes the edited data based on the result of the edit. It is known that overly long videos tend to have fewer views. Accordingly, in order to keep content within a specific duration that scales with the duration of the source video, the server can, in addition to the low priority scenes described above, automatically cut scenes with minor movement based on the result of image processing. Alternatively, the server can generate and encode a video digest based on the result of the analysis of the semantic meaning of the scene.

[0786] There may be cases in which content from people may include content that violates copyright, moral law, book rights, etc. Such a case may lead to an unfavorable situation for the creator, for example, when the content is shared beyond the scope intended by the creator. Accordingly, before encoding, the server, for example, can edit the images so as to, for example, blur the faces of people on the periphery of the screen or blur the inside of a house. Additionally, the server may be configured to recognize faces of people other than the registered person in images to be encoded, and when such faces appear in the image, may apply a mosaic filter to, for example, the person's face. Alternatively, as pre-processing or post-processing for encoding, the user may specify, for copyright reasons, an image area including a person or a background area to be processed. The server may process the specified area, for example, by replacing the area with another image or blurring the area. If the area includes a person, the person may be tracked in a moving frame, and the user's head region may be replaced with another image as the person moves.

[0787] Since there is a demand for real-time viewing of human generated content, which is often of small data size, the decoder may first take the base layer as the highest priority and perform decoding and playback, although this may differ depending on the bandwidth. When the content is played back two or more times, for example, when the decoder receives the enhancement layer during decoding and playback of the base layer and loops playback, the decoder can play back the high image quality video including the enhancement layer. If a stream is encoded using such scalable encoding, the video may be low quality in the unselected state or at the beginning of the video, but it may offer an experience in which the quality of the images of the stream is gradually increased in an intelligent manner. This is not limited to just scalable coding; An identical experience can be offered by configuring one stream of the low quality stream played back the first time and a second stream encoded using the first stream as a reference.

[0788] Other Examples of Implementations and Applications

Encoding and decoding may be performed by an LSI ex500 (Large Integrated Circuit) (see FIG. 160), which is typically included in each terminal. The LSI ex500 can be configured with a single chip or multiple chips. Software for encoding and decoding moving frames may be integrated into some type of recording medium (such as a CD-ROM, floppy disk, or hard disk) that is readable, for example, by a computer ex111, and encoding and decoding may be performed using the software. In addition, when the ex114 smartphone is equipped with a camera, video data captured by the camera can be transmitted. In this case, video data can be encoded using the LSI ex500 included in the ex115 smartphone.

[0789] It should be noted that the LSI ex500 may be configured to download and activate an application. In such a case, the terminal first determines whether or not it is compatible with the scheme used to encode the content, or whether or not it is capable of performing a particular service. When the terminal is not compatible with the content encoding scheme, or when the terminal does not allow execution of a particular service, the terminal may first download the codec or application software, and then receive and play the content.

[0790] In addition to the example of a content provision system ex100 that uses the Internet ex101, at least a moving frame encoder (picture encoder) or a moving frame decoder (picture decoder) described in the above embodiments may be implemented in a digital broadcast system. Identical encoding processing and decoding processing can be used to transmit and receive broadcast radio waves overlaid with multiplexed audio and video data using, for example, a satellite, even though it accommodates multicast transmission, while unicast transmission is simpler with the ex100 system providing content.

[0791] Hardware configuration

Fig. 163 illustrates additional details regarding the exemplary smartphone ex115 shown in FIG. 160. Fig. 164 illustrates an example of the functional configuration of the ex115 smartphone. The ex115 smartphone includes an ex450 antenna for transmitting and receiving radio waves to/from the ex110 base station, an ex465 camera that allows video and still image capture, and an ex458 display that displays decoded data such as video captured through the ex465 camera and videos received via antenna ex450. The ex115 smartphone further includes ex466 user interface such as touchpad; an audio output module ex457, such as a speaker for outputting speech or other audio; ex456 audio input module, such as a microphone for audio input; a storage device ex467 capable of storing decoded data such as captured videos or still images, recorded audio, received videos or still images and mail messages, as well as decoded data; and the ex464 socket, which is an interface for the ex468 SIM card to authorize access to the network and various data. It should be noted that an external storage device may be used instead of or in addition to the ex467 storage device.

[0792] Main controller ex460, which can comprehensively control the display ex458 and user interface ex466, power supply circuit ex461, user interface input controller ex462, video signal processor ex455, camera interface ex463, display controller ex459, modulator/demodulator ex452, multiplexer/demultiplexer ex453, ex454 audio processor, ex464 socket and ex467 memory are connected via ex470 bus.

[0793] When the user turns on the power button of the power supply circuit ex461, the smartphone ex115 is turned on to an operational state and each component is supplied with power, for example, from a battery pack.

[0794] The ex115 smartphone performs processing, for example, for calls and data transfers, based on control performed by the ex460 main controller, which includes a CPU, ROM and RAM. When calls are made, the audio signal recorded by the audio input module ex456 is converted to a digital audio signal by the audio signal processor ex454, which is applied with spread spectrum processing by the ex452 modulator/demodulator and applied with digital-to-analog conversion and frequency conversion processing by the ex451 transceiver. and the resulting signal is transmitted through the ex450 antenna. The received data is amplified, frequency-converted and A/D converted, despread spectrum processed by the ex452 modulator/demodulator, converted to an analog audio signal by the ex454 audio processor, and then output from the ex457 audio output module.

[0795] In the data transmission mode, text, still image data or video data may be transmitted, for example, under the control of the main controller ex460 through the input controller ex462 from the user interface based on the operation of the main body user interface ex466. Similar transmission and reception processing is performed. In the data transfer mode, when sending video, still image or video and audio, the video signal processor ex455 compressively encodes, through the moving frame encoding method described in the above embodiments, the video signal stored in the storage device ex467 or the video signal input from the camera ex465 , and transmits the encoded video data to the ex453 multiplexer/demultiplexer. The ex454 audio processor encodes the audio signal recorded by the ex456 audio input module, while the ex465 camera captures video or still image, and transmits the encoded audio data to the ex453 multiplexer/demultiplexer. The ex453 multiplexer/demultiplexer multiplexes encoded video data and encoded audio data using a specific circuit, modulates and converts the data using the ex452 modulator/demodulator (modulator/demodulator circuit) and ex451 transceiver and transmits the result through the ex450 antenna. A specific schema can be predefined.

[0796] When a video attachment in an email message or chat, or a video containing a link from a web page, is received, for example, to decode multiplexed data received through antenna ex450, the ex453 multiplexer/demultiplexer demultiplexes the multiplexed data to separate multiplexes the data onto a video bitstream and an audio bitstream, provides encoded video data to the ex455 video processor via the ex470 synchronous bus, and provides encoded audio data to the ex454 audio processor via the ex470 synchronous bus. The video signal processor ex455 decodes the video signal using a moving frame decoding method corresponding to the moving frame encoding method described in the above embodiments, and the video or still image included in the associated moving frame file is displayed on the display ex458 through the display controller ex459. The audio signal processor ex454 decodes the audio signal and outputs audio from the audio output module ex457. As live streaming becomes more popular, there may be cases in which audio playback may not be socially feasible, depending on the user's environment. Accordingly, as an initial value, a configuration in which only video data is reproduced, for example, no audio signal is reproduced, may be preferred; audio can only sync and play when input is received, such as when the user clicks on video data.

[0797] Although the smartphone ex115 is used in the above example, other implementations are possible: a transceiver terminal including both an encoder and a decoder; a transmission terminal including only an encoder; and a receiving terminal including only a decoder. In the description of a digital broadcast system, an example is given in which multiplexed data obtained by multiplexing video data with audio data is received or transmitted. However, the multiplexed data may be video data multiplexed with data other than audio data, such as text data associated with the video. Additionally, direct video data rather than multiplexed data may be received or transmitted.

[0798] Although the ex460 main controller, which includes a CPU, is described as controlling encoding or decoding processes, various terminals often include graphics processing units (GPUs). Accordingly, a configuration in which a large area is processed at once by using the performance characteristics of the GPU through a memory shared by the CPU and the GPU, or through a memory including an address that is controlled so as to enable a wide use via CPU and GPU, or via separate storage devices. This reduces encoding time, maintains the real-time nature of the stream, and reduces latency. In particular, processing related to motion estimation, deblocking filtering, sampled adaptive offset (SAO), and transform/quantization can be efficiently performed by the GPU instead of the CPU, for example, in units of frames, simultaneously.

Claims (72)

1. Encoder containing: - a processor that performs information processing; And - a storage device connected to the processor; - while the processor, during operation: - in response to the first reconstructed image sample located outside the virtual boundary, duplicates the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample; - generates a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images; - sets the first coefficient value to zero in response to the first coefficient value less than 64; - generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images; - generates the third coefficient value by summing the first coefficient value with the second coefficient value; And - outputs a third reconstructed sample of chrominance component images using the third coefficient value. 2. Encoder according to claim 1, in which: - the first restored sample of images is located next to the second restored sample of images. 3. Encoder containing: - a block splitting module, which, during operation, splits the first image into many blocks; - an internal prediction module that, in operation, predicts blocks included in the first image using reference blocks included in the first image; - an inter-prediction module which, in operation, predicts blocks included in the first image using reference blocks included in a second image different from the first image; - a contour filter, which, when in operation, filters the blocks included in the first image; - a converter that, in operation, converts the prediction error between the original signal and the prediction signal generated by the intra prediction module or the inter prediction module to generate conversion coefficients; - a quantizer, which, when in operation, quantizes the transformation coefficients to generate quantized coefficients; And - an entropy encoder which, in operation, variably encodes the quantized coefficients to generate a coded bitstream including the encoded quantized coefficients and control information, - in this case, the contour filter performs: - in response to the first reconstructed image sample located outside the virtual boundary, duplicating the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample; - generating a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images; - setting the first coefficient value to zero in response to the first coefficient value less than 64; - generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images; - formation of the third coefficient value by summing the first coefficient value with the second coefficient value; And - outputting a third reconstructed sample of chrominance signal component images using the third coefficient value. 4. Encoder according to claim 3, in which: - the first restored sample of images is located next to the second restored sample of images. 5. Decoder containing: - a processor that performs information processing; And - a storage device connected to the processor; - while the processor, during operation: - in response to the first reconstructed image sample located outside the virtual boundary, duplicates the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample; - generates a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images; - sets the first coefficient value to zero in response to the first coefficient value less than 64; - generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images; - generates the third coefficient value by summing the first coefficient value with the second coefficient value; And - outputs a third reconstructed sample of chrominance component images using the third coefficient value. 6. Decoder according to claim 5, in which: - the first restored sample of images is located next to the second restored sample of images. 7. Decoding equipment containing: - a decoder which, in operation, decodes the encoded bit stream to output quantized coefficients; - an inverse quantizer, which, when in operation, inversely quantizes the quantized coefficients to output the conversion coefficients; - an inverse converter, which, when in operation, inversely converts the conversion coefficients to output the prediction error; - an internal prediction module that, in operation, predicts blocks included in the first image using reference blocks included in the first image; - an inter-prediction module which, in operation, predicts blocks included in the first image using reference blocks included in a second image different from the first image; - a contour filter, which, when in operation, filters the blocks included in the first image; And - an output which, when operated, outputs a frame including the first image, - in this case, the contour filter performs: - in response to the first reconstructed image sample located outside the virtual boundary, duplicating the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample; - generating a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images; - setting the first coefficient value to zero in response to the first coefficient value less than 64; - generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images; - formation of the third coefficient value by summing the first coefficient value with the second coefficient value; And - outputting a third reconstructed sample of chrominance signal component images using the third coefficient value. 8. Decoding equipment according to claim 7, in which: - the first restored sample of images is located next to the second restored sample of images. 9. An encoding method containing the steps of: - in response to the first reconstructed image sample located outside the virtual boundary, duplicating the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample; - generating a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images; - setting the first coefficient value equal to zero in response to the first coefficient value less than 64; - generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chroma component images; - the third value of the coefficient is formed by summing the first value of the coefficient with the second value of the coefficient; And - outputting a third reconstructed sample of chrominance signal component images using the third coefficient value. 10. A decoding method comprising the steps of: - in response to the first reconstructed image sample located outside the virtual boundary, duplicating the reconstructed sample located within and adjacent to the virtual boundary so as to generate the first reconstructed image sample; - generating a first coefficient value by applying a CCALF (Cross Component Adaptive Contour Filtering) process to the first reconstructed sample of luminance signal component images; - setting the first coefficient value equal to zero in response to the first coefficient value less than 64; - generating a second coefficient value by applying an ALF (adaptive loop filtering) process to the second reconstructed sample of chrominance component images; - the third value of the coefficient is formed by summing the first value of the coefficient with the second value of the coefficient; And - outputting a third reconstructed sample of chrominance signal component images using the third coefficient value.