RU2808103C2 - Method for decoding videos and images using block segmentation - Google Patents

Abstract

FIELD: video and image decoding.

SUBSTANCE: result is achieved by the decoder segmenting the encoded signal into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, after which the decoder decodes the plurality of blocks, the plurality of modes of block segmentation includes a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one of the blocks, obtained after the first block is segmented in segmentation, when the total number of segments of the first block segmentation mode is three, the second block is the centre block from among the blocks obtained after the first block is segmented, and the segmentation direction for the second segmentation mode is identical to the segmentation direction for the first block segmentation mode, wherein the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three, and wherein the parameter for identifying the second block segmentation mode includes a first flag that indicates whether the block is to be segmented horizontally or vertically, and does not include a second flag that indicates the total number of segments into which the block is to be segmented.

EFFECT: increasing the efficiency of video compression.

1 cl, 56 dwg

Description

Field of technology to which the invention relates

[0001] The present disclosure relates to methods and equipment for encoding and decoding video and images using block segmentation.

State of the art

[0002] In traditional image and video encoding methods, the image is typically segmented into blocks, and the encoding and decoding processes are performed at the block level. With recent developments in video standards, encoding and decoding processes can be performed at different block sizes other than the typical 16x16 or 8x8. In an image, a size range from 4x4 to 256x256 can be used for image encoding and decoding processes.

List of bibliographic references

Non-patent documents

[0003] NPL 1. H.265 (ISO/IEC 23008-2 HEVC (High Efficiency Video Coding))

The essence of the invention

Technical problem

[0004] To represent a size range from 4x4 to 256x256, block segmentation information such as block segmentation modes (eg, quadtree, binary tree, and ternary tree) and segmentation flags (eg, split flag) are defined and is transmitted in block signaling. The amount of transmitted overhead information increases when the segmentation depth increases. In this case, the increased amount of overhead information reduces the efficiency of video compression.

[0005] In view of this, an encoder according to an aspect of the present disclosure provides an encoder or the like capable of increasing compression efficiency when encoding block segmentation information.

The solution of the problem

[0006] An encoder according to an aspect of the present disclosure is an encoder that encodes a frame (picture), and includes: a processor; and a storage device, wherein the processor includes: a block segmentation determining module that segments a frame into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein the frame is read from the storage device; and an encoding module that encodes a plurality of blocks, the set of block segmentation modes including a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting the first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block which is one of the blocks obtained after the first block is segmented, and when the total number of segments of the first block segmentation mode is three, the second block is the central block from among the blocks obtained after the first block segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first block segmentation mode, the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three.

[0007] A decoder according to an aspect of the present disclosure is a decoder that decodes an encoded signal, and includes: a processor; and a storage device, the processor including: a block segmentation determining module that segments the encoded signal into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type , wherein the encoded signal is read from the storage device; and a decoding module that decodes a plurality of blocks, the set of block segmentation modes including a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting the first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block which is one of the blocks obtained after the first block is segmented, and when the total number of segments of the first block segmentation mode is three, the second block is the central block from among the blocks obtained after the first block segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first block segmentation mode, the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three.

[0008] It should be noted that the general and specific aspects described above may be implemented using a system, method, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM, or any arbitrary combination of systems, methods, integrated circuits, computer programs or machine-readable recording media.

Advantages of the invention

[0009] The present disclosure provides improved compression efficiency when encoding segmentation information into blocks.

Brief description of drawings

[0010] FIG. 1 is a block diagram illustrating a functional configuration of an encoder according to Embodiment 1.

Fig. 2 illustrates one example of blocking according to Embodiment 1.

Fig. 3 is a diagram indicating the basis transformation functions for each type of transformation.

Fig. 4A illustrates one example of a filter shape used in ALF.

Fig. 4B illustrates another example of a filter shape used in ALF.

Fig. 4C illustrates another example of a filter shape used in ALF.

Fig. 5A illustrates 67 intra prediction modes used in intra prediction.

Fig. 5B is a flowchart for illustrating a brief view of the prediction image correction process performed through OBMC processing.

Fig. 5C is a conceptual diagram for illustrating a brief view of a prediction image correction process performed through OBMC processing.

Fig. 5D illustrates one example of FRUC.

Fig. 6 serves to illustrate pattern matching (bilateral matching) between two blocks along a motion path.

Fig. 7 serves to illustrate pattern matching (pattern matching) between a pattern in the current frame and a block in the reference frame.

Fig. 8 serves to illustrate the model under the condition of uniform rectilinear motion.

Fig. 9A serves to illustrate the extraction of the motion vector of each sub-block based on the motion vectors of neighboring blocks.

Fig. 9B serves to illustrate a brief representation of a process for extracting a motion vector through the combining mode.

Fig. 9C is a conceptual diagram for illustrating a brief representation of DMVR processing.

Fig. 9D serves to illustrate a brief presentation of a method for generating prediction images using a brightness correction process performed through LIC processing.

Fig. 10 is a block diagram illustrating a functional configuration of a decoding apparatus according to Embodiment 1.

Fig. 11 is a flowchart of a video encoding process according to Embodiment 2.

Fig. 12 is a flowchart of a method for a video decoding process according to Embodiment 2.

Fig. 13 is a flowchart for a video encoding process according to Embodiment 3.

Fig. 14 is a flowchart of a method for a video decoding process according to Embodiment 3.

Fig. 15 is a block diagram illustrating the structure of a video/image encoder according to Embodiment 2 or 3.

Fig. 16 is a block diagram illustrating the structure of a video/image decoder according to Embodiment 2 or 3.

Fig. 17 illustrates examples of possible locations of a first parameter in a compressed video bit stream according to Embodiment 2 or 3.

Fig. 18 illustrates examples of possible locations of a second parameter in a compressed video bit stream according to embodiment 2 or 3.

Fig. 19 illustrates an example of a second parameter following the first parameter according to embodiment 2 or 3.

Fig. 20 illustrates an example in which the second segmentation mode is not selected to segment a block of 2N pixels into N pixels as illustrated in step (2c) in Embodiment 2.

Fig. 21 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into 2N pixels as illustrated in step (2c) in Embodiment 2.

Fig. 22 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into N pixels as illustrated in step (2c) in Embodiment 2.

Fig. 23 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into N pixels as illustrated in step (2c) in Embodiment 2.

Fig. 24 illustrates an example of segmenting a block of 2N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2.

Fig. 25 illustrates an example of segmenting a block of N pixels into 2N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2.

Fig. 26 illustrates an example of segmenting a block of N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2.

Fig. 27 illustrates an example of segmenting a block of N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2.

Fig. 28 illustrates examples of segmentation modes for segmenting a block of N pixels into N pixels in Embodiment 2; (a)-(h) show different segmentation modes.

Fig. 29 illustrates examples of segmentation types and segmentation directions for segmenting a block of N pixels by N pixels in Embodiment 3; (1), (2), (3) and (4) represent different segmentation types, (1a), (2a), (3a) and (4a) represent different segmentation modes from related segmentation types in the vertical segmentation direction, and (1b), (2b), (3b) and (4b) represent different segmentation modes from related segmentation types in the horizontal segmentation direction.

Fig. 30 illustrates the advantage of encoding the segmentation type before the segmentation direction compared to encoding the segmentation direction before the segmentation type according to Embodiment 3.

Fig. 31A illustrates an example of dividing a block into sub-blocks using a set of segmentation modes with fewer bins when encoding the segmentation modes.

Fig. 31B illustrates an example of dividing a block into sub-blocks using a set of segmentation modes with fewer bins when encoding the segmentation modes.

Fig. 32A illustrates an example of dividing a block into subblocks using a segmentation mode set that appears first in a predetermined order of a plurality of segmentation mode sets.

Fig. 32B illustrates an example of dividing a block into subblocks using a segmentation mode set that appears first in a predetermined order of a plurality of segmentation mode sets.

Fig. 32C illustrates an example of dividing a block into subblocks using a segmentation mode set that appears first in a predetermined order of a plurality of segmentation mode sets.

Fig. 33 illustrates a general configuration of a content delivery system for implementing a content distribution service.

Fig. 34 illustrates one example of a coding structure in scalable coding.

Fig. 35 illustrates one example of a coding structure in scalable coding.

Fig. 36 illustrates an example of a web page display screen.

Fig. 37 illustrates an example of a web page display screen.

Fig. 38 illustrates one example of a smartphone.

Fig. 39 is a block diagram illustrating an example configuration of a smartphone.

Fig. 40 is a diagram illustrating an example of a limitation on a segmentation mode for dividing a rectangular block into three sub-blocks.

Fig. 41 is a diagram illustrating an example of a limitation on a segmentation mode for dividing a block into two sub-blocks.

Fig. 42 is a diagram illustrating an example of a limitation on a segmentation mode for dividing a square block into three sub-blocks.

Fig. 43 is a diagram illustrating an example of a limitation on a segmentation mode for dividing a rectangular block into two sub-blocks.

Fig. 44 is a diagram illustrating an example of a segmentation direction-based restriction of a segmentation mode for dividing a non-rectangular block into two sub-blocks.

Fig. 45 is a diagram illustrating an example of a valid segmentation direction of a segmentation mode for dividing a non-rectangular block into two sub-blocks.

Detailed Description of Embodiments

[0011] Embodiments will be described in detail hereinafter with reference to the drawings.

[0012] It should be noted that each of the embodiments described below shows a general or specific example. The numerical values, shapes, materials, components, arrangement and connection of components, steps, order of steps, etc., indicated in the following embodiments are merely examples and are therefore not intended to limit the scope of the claims. In addition, among the components in the following embodiments, components not set forth in any of the independent claims defining the broadest concepts of the invention are described as optional components.

[0013] Embodiment 1

First, a brief introduction to Embodiment 1 is provided as one example of an encoder and decoder to which the processes and/or configurations presented in the following description of aspects of the present disclosure are applicable. It should be noted that Embodiment 1 is simply one example of an encoder and decoder to which the processes and/or configurations presented in the description of aspects of the present disclosure are applicable. The processes and/or configurations presented in the description of aspects of the present disclosure may also be implemented in an encoder and decoder different from the encoder and decoder according to Embodiment 1.

[0014] When the processes and/or configurations presented in the description of aspects of the present disclosure are applied to embodiment 1, for example, any of the following may be performed:

[0015] (1) with respect to the encoder or decoder according to Embodiment 1, from among the components included in the encoder or decoder according to Embodiment 1, replacing the component corresponding to the component presented in the description of aspects of the present disclosure with the component presented in the description of aspects of this disclosure disclosures;

(2) with respect to the encoder or decoder according to Embodiment 1, implementing discretionary changes of functions or implemented processes performed by one or more components included in the encoder or decoder according to Embodiment 1, such as adding, replacing or deleting, etc. such functions or implemented processes, then replacing the component corresponding to the component presented in the description of aspects of this disclosure with the component presented in the description of aspects of this disclosure;

(3) with respect to the method implemented by the encoder or decoder according to Embodiment 1, implementing discretionary changes such as adding processes and/or replacing, deleting one or more processes included in the method, and then replacing processes corresponding to the process presented in the description aspects of this disclosure, to the process presented in the description of aspects of this disclosure;

(4) combining one or more components included in the encoder or decoder according to Embodiment 1 with a component presented in the description of aspects of the present disclosure, a component including one or more functions included in the component presented in the description of aspects of the present disclosure, or a component that implements one or more of the processes implemented by the component presented in the description of aspects of this disclosure;

(5) combining a component including one or more functions included in one or more components included in an encoder or decoder according to Embodiment 1, or a component that implements one or more processes implemented by one or more components included in an encoder or decoder according to Embodiment 1, with a component presented in the description of aspects of this disclosure, a component including one or more functions included in a component presented in the description of aspects of this disclosure, or a component that implements one or more processes implemented by means of the component presented in the description of aspects of this disclosure;

(6) with respect to the method implemented by the encoder or decoder according to Embodiment 1, from among the processes included in the method, replacing the process corresponding to the process presented in the description of aspects of the present disclosure with the process presented in the description of aspects of the present disclosure; And

(7) combining one or more processes included in the method implemented by the encoder or decoder according to Embodiment 1 with the process presented in the description of aspects of the present disclosure.

[0016] It should be noted that the implementation of the processes and/or configurations presented in the description of aspects of the present disclosure is not limited to the above examples. For example, the processes and/or configurations presented in the description of aspects of the present disclosure may be implemented in an apparatus used for a purpose different from the moving frame/frame encoder or the moving frame/frame decoder disclosed in Embodiment 1 of the implementation. In addition, the processes and/or configurations presented in the description of aspects of the present disclosure may be independently implemented. In addition, the processes and/or configurations described in various aspects may be combined.

[0017] Encoder Brief Introduction

First, the encoder according to Embodiment 1 is briefly introduced. Fig. 1 is a block diagram illustrating a functional configuration of the encoder 100 according to Embodiment 1. The encoder 100 is a moving frame/frame encoder that encodes a moving frame/frame block by block.

[0018] As illustrated in FIG. 1, the encoder 100 is a device that encodes a frame block by block, and includes a partitioner 102, a subtractor 104, a converter 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse converter 114, an adder 116, a block memory 118, a loop filter 120, a frame memory 122, an intra prediction module 124, an inter prediction module 126, and a prediction controller 128.

[0019] Encoder 100 is implemented, for example, as a general processor and storage device. In this case, when the software program stored in the memory device is executed by the processor, the processor functions as a partitioner 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116 , a loop filter 120, an intra prediction module 124, an inter prediction module 126, and a prediction controller 128. Alternatively, encoder 100 may be implemented as one or more dedicated electronic circuits corresponding to partitioner 102, subtractor 104, converter 106, quantizer 108, entropy encoder 110, inverse quantizer 112, inverse converter 114, adder 116, loop filter 120, module 124 internal prediction module 126 inter prediction and controller 128 prediction.

[0020] Each component included in the encoder 100 is further described herein.

[0021] Partitioner

The partitioning unit 102 splits each frame included in the input moving frame into blocks and outputs each block to the subtractor 104. For example, the partitioning unit 102 first partitions the frame into blocks of a fixed size (eg, 128*128). A fixed size block is also called a "coding tree unit (CTU)". Partitioner 102 then partitions each fixed-size block into variable-size blocks (eg, 64*64 or smaller) based on recursive quadtree and/or binary tree blocking. The variable size block is also called a "coding unit (CU)", "prediction unit (PU)" or "transformation unit (TU)". It should be noted that in this embodiment, there is no need to distinguish between CU, PU and TU; all or some of the blocks in a frame can be processed per CU, PU or TU.

[0022] FIG. 2 illustrates one example of blocking according to Embodiment 1. In fig. 2, solid lines represent block boundaries for blocks divided by quadtree blocking, and dotted lines represent block boundaries for blocks divided by binary tree blocking.

[0023] Here, block 10 is a 128*128 square pixel block (128*128 block). This 10,128*128 block is first split into four 64*64 square blocks (quadtree blocking).

[0024] The top left 64*64 block is further vertically split into two 32*64 rectangular blocks, and the left 32*64 block is further vertically split into two 16*64 rectangular blocks (binary tree chunking). As a result, the upper left block 64*64 is divided into two blocks 11 and 12 16*64 and one block 13 32*64.

[0025] The top right 64*64 block is horizontally split into two 64*32 rectangular blocks 14 and 15 (binary tree blocking).

[0026] The lower left 64*64 block is first divided into four 32*32 square blocks (quadtree blocking). The top left block and bottom right block of four 32*32 blocks are further broken down. The top left 32*32 block is vertically split into two 16*32 rectangular blocks, and the right 16*32 block is further split horizontally into two 16*16 blocks (binary tree blocking). The bottom right 32*32 block is horizontally split into two 32*16 blocks (binary tree blocking). As a result, the lower left block 64*64 is divided into a block 16 16*32, two blocks 17 and 18 16*16, two blocks 19 and 20 32*32 and two blocks 21 and 22 32*16.

[0027] The bottom right block 23 64*64 is not broken.

[0028] As described above, FIG. 2, block 10 is divided into 13 blocks 11-23 of variable size based on recursive quadtree and binary tree blocking. This type of partitioning is also called "quadtree plus binary tree (QTBT) partitioning."

[0029] It should be noted that in FIG. 2, one block is split into four or two blocks (quadtree or binary tree chunking), but splitting is not limited to this example. For example, one block can be split into three blocks (ternary block splitting). A partition involving such a ternary partition is also called a "multi-type tree (MBT) partition."

[0030] Subtractor

The subtractor 104 subtracts the predictive signal (predictive sample) from the original signal (original sample) per block divided by the partitioning unit 102. In other words, the subtractor 104 calculates the prediction errors (also called “residuals”) of the block to be encoded (hereinafter referred to as the “current block.”) The subtractor 104 then outputs the calculated prediction errors to the mapper 106.

[0031] The original signal is a signal input to the encoder 100 and is a signal representing an image for each frame included in the moving frame (eg, a luma signal and two chrominance signals). Hereinafter in this document, a signal representing an image is also called a “sample”.

[0032] Converter

Converter 106 converts prediction errors in the spatial domain into transform coefficients in the frequency domain and outputs the transform coefficients to quantizer 108. More specifically, converter 106 applies, for example, a predefined discrete cosine transform (DCT) or discrete sine transform (DST) to the prediction errors in spatial area.

[0033] It should be noted that the transformer 106 can adaptively select a transform type from a plurality of transform types and convert prediction errors into transform coefficients by using a transform basis function corresponding to the selected transform type. This kind of transformation is also called "explicit multiple base transformation (EMT)" or "adaptive multiple transformation (AMT)".

[0034] Conversion types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I and DST-VII. Fig. 3 is a diagram indicating the basis transformation functions for each type of transformation. In fig. 3, N indicates the number of input pixels. For example, the selection of a transformation type from a plurality of transformation types may depend on the prediction type (intra-prediction and inter-prediction) and may depend on the intra-prediction mode.

[0035] Information indicating whether such EMT or AMT should be applied (called, for example, an "AMT flag"), and information indicating the type of mapping selected are signaled at the CU level. It should be noted that signaling of this information need not be performed at the CU level and may be performed at another level (eg, sequence level, frame level, run level, tile level, or CTU level).

[0036] In addition, the converter 106 may apply a secondary transform to the transform coefficients (to the result of the transform). This secondary transform is also called "adaptive secondary transform (AST)" or "non-separable secondary transform (NSST)". For example, transformer 106 applies a secondary transform to each sub-block (eg, each 4*4 sub-block) included in the block of transform coefficients corresponding to the intra prediction errors. Information indicating whether NSST should be applied and information associated with the transformation matrix used in NSST is signaled at the CU level. It should be noted that signaling of this information need not be performed at the CU level and may be performed at another level (eg, sequence level, frame level, run level, tile level, or CTU level).

[0037] Here, separable transformation is a method in which the transformation is performed multiple times by separately performing the transformation for each direction according to the input number of dimensions. An inseparable transformation is a method of performing a collective transformation in which two or more dimensions in a multidimensional input are collectively treated as a single dimension.

[0038] In one example of a non-separable transformation, when the input is a 4*4 block, the 4*4 block is considered a simple array including 16 components, and the transformation applies a 16*16 transformation matrix to the array.

[0039] Additionally, similar to the above, after the 4*4 input block is treated as a simple array including 16 components, a transformation that performs multiple Givens rotations on the array (i.e., a hypercubic Givens transform) also represents one example of an inseparable transformation.

[0040] Quantizer

Quantizer 108 quantizes transform coefficients output from transformer 106. More specifically, quantizer 108 scans, in a predetermined scanning order, transform coefficients of the current block and quantizes the scanned transform coefficients based on quantization parameters (QPs) corresponding to the transform coefficients. Quantizer 108 then outputs the quantized transform coefficients (hereinafter referred to as “quantized coefficients”) of the current block to entropy encoder 110 and inverse quantizer 112.

[0041] The predetermined order is the order for quantization/inverse quantization of the transform coefficients. For example, the predetermined scanning order is set to ascending frequency order (low to high frequency) or descending frequency order (high to low frequency).

[0042] The quantization parameter is a parameter that specifies the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. In other words, if the value of the quantization parameter increases, the quantization error increases.

[0043] Entropy encoder

Entropy encoder 110 generates a coded signal (coded bitstream) by variable length encoding of quantized coefficients that are inputs from quantizer 108. More specifically, entropy encoder 110, for example, binarizes the quantized coefficients and arithmetically encodes the binary signal.

[0044] Inverse Quantizer

Inverse quantizer 112 inversely quantizes the quantized coefficients that are inputs from quantizer 108. More specifically, inverse quantizer 112 inversely quantizes, in a predetermined scan order, the quantized coefficients of the current block. Inverse quantizer 112 then outputs the inversely quantized transform coefficients of the current block to inverse quantizer 114.

[0045] Inverter

The inverse transformer 114 recovers the prediction errors by inversely transforming the transform coefficients that are the inputs from the inverse quantizer 112. More specifically, the inverse transformer 114 recovers the prediction errors of the current block by applying an inverse transform corresponding to the transform applied by the transformer 106 to the transform coefficients. Inverter 114 then outputs the recovered prediction errors to adder 116.

[0046] It should be noted that since information is lost in quantization, the reconstructed prediction errors are not the same as the prediction errors calculated by the subtractor 104. In other words, the reconstructed prediction errors include quantization errors.

[0047] Adder

The adder 116 reconstructs the current block by summing the prediction errors, which are the inputs from the inverter 114, and the prediction samples, which are the inputs from the prediction controller 128. Adder 116 then outputs the reconstructed block to block memory 118 and loop filter 120. The reconstructed block is also referred to as a "local decoded block."

[0048] Block storage device

The block memory 118 is a storage device for storing blocks in a frame to be encoded (hereinafter referred to as the “current frame”) for reference in intra prediction. More specifically, block storage 118 stores reconstructed blocks output from adder 116.

[0049] Loop filter

Loop filter 120 applies a loop filter to the blocks reconstructed by the adder 116 and outputs the filtered reconstructed blocks to frame memory 122. A loop filter is a filter used in a coding loop (in-loop filter) and includes, for example, a deblocking filter (DF), a sampled adaptive offset (SAO), and an adaptive loop filter (ALF).

[0050] In ALF, a least squares error filter is applied to remove compression artifacts. For example, one filter from a plurality of filters is selected for each 2*2 sub-block in the current block based on the direction and activity of local gradients and applied.

[0051] More specifically, first, each sub-block (eg, each 2*2 sub-block) is classified into one of a plurality of classes (eg, 15 or 25 classes). The subblock classification is based on the directionality and activity of the gradients. For example, a classification index C is extracted based on the directionality D of gradients (eg, 0-2 or 0-4) and the activity of A gradients (eg, 0-4) (eg, C=5D+A). Then, based on the classification index C, each sub-block is classified into one of a plurality of classes (eg, 15 or 25 classes).

[0052] For example, the directionality D of the gradients is calculated by comparing the gradients of multiple directions (eg, horizontal, vertical, and two diagonal directions). In addition, for example, the activity A of the gradients is calculated by summing the gradients of multiple directions and quantizing the sum.

[0053] The filter to be used for each sub-block is determined from the plurality of filters based on the result of such classification.

[0054] The filter shape to be used in the ALF, for example, is a circular symmetrical filter shape. Fig. 4A-4C illustrate examples of filter shapes used in ALF. Fig. 4A illustrates a 5*5 diamond shape filter, FIG. 4B illustrates a 7*7 diamond shape filter, and FIG. 4C illustrates a 9*9 diamond shape filter. Information indicating the shape of the filter is carried in frame-level signaling. It should be noted that signaling information indicating the shape of the filter need not be performed at the frame level and may be performed at another level (eg, sequence level, run level, tile level, CTU level, or CU level).

[0055] Activation or deactivation of ALF is determined at the frame level or CU level. For example, for a luma signal, the decision as to whether to apply ALF or not is made at the CU level, and for a chroma signal, the decision as to whether to apply ALF or not is made at the frame level. Information indicating whether ALF is activated or deactivated is signaled at the frame level or the CU level. It should be noted that signaling information indicating whether the ALF is activated or deactivated need not be performed at the frame level or CU level and may be performed at another level (eg, sequence level, run level, tile level, or CTU level).

[0056] A set of coefficients for a plurality of selectable filters (eg, 15 or 25 filters) is signaled at the frame level. It should be noted that signaling of the set of coefficients need not be performed at the frame level and may be performed at another level (eg, sequence level, run level, tile level, CTU level, CU level, or subblock level).

[0057] Frame memory

The frame memory 122 is a storage device for storing reference frames used in inter-prediction and is also called a “frame buffer”. More specifically, frame memory 122 stores reconstructed blocks filtered by loop filter 120.

[0058] Internal prediction module

The intra prediction unit 124 generates a prediction signal (intra prediction signal) by intra prediction of the current block with reference to the block or blocks in the current frame and stores in the block memory 118 (also called “intra prediction”). More specifically, the intra prediction module 124 generates an internal prediction signal by intra prediction with reference to samples (eg, luma and/or chroma values) of the block or blocks adjacent to the current block, and then outputs the internal prediction signal to the prediction controller 128.

[0059] For example, intra prediction module 124 performs intra prediction by using one mode from a plurality of preset intra prediction modes. The intra prediction modes include one or more non-directional prediction modes and a plurality of directed prediction modes.

[0060] One or more omnidirectional prediction modes include, for example, a planar prediction mode and a DC prediction mode defined in the H.265/High Efficiency Video Coding (HEVC) standard (see NPL 1).

[0061] The plurality of directional prediction modes includes, for example, the 33 directional prediction modes defined in the H.265/HEVC standard. It should be noted that the plurality of directional prediction modes may further include 32 directional prediction modes in addition to 33 directional prediction modes (for a total of 65 directional prediction modes). Fig. 5A illustrates 67 intra prediction modes used in intra prediction (two non-directional prediction modes and 65 directional prediction modes). The solid arrows represent the 33 directions defined in the H.265/HEVC standard, and the dotted arrows represent the additional 32 directions.

[0062] It should be noted that a luma block may be referenced in block intra-chrominance prediction. In other words, the chrominance signal component of the current block can be predicted based on the luminance signal component of the current block. This internal forecasting is also called “Cross Component Linear Model (CCLM) forecasting.” Such a block intra-chrominance prediction mode that references a luma block (called, for example, a “CCLM mode”) may be added as one of the block intra-chrominance prediction modes.

[0063] The intra prediction module 124 may adjust the pixel values after the intra prediction based on horizontal/vertical reference pixel gradients. The internal prediction followed by this kind of correction is also called "position-dependent combination with internal prediction (PDPC)". Information indicating whether or not to apply PDPC (called, for example, a “PDPC flag”) is, for example, signaled at the CU level. It should be noted that signaling of this information need not be performed at the CU level and may be performed at another level (eg, sequence level, frame level, run level, tile level, or CTU level).

[0064] Mutual prediction module

The inter-prediction unit 126 generates a prediction signal (mutual prediction signal) by inter-predicting the current block with reference to a block or blocks in a reference frame that is different from the current frame, and stores in the frame memory 122 (also called “inter-frame prediction”). Inter-prediction is performed on a per-current block basis or on a per-sub-block basis (eg, per 4*4 block) within the current block. For example, inter-prediction module 126 performs motion estimation in the reference frame for the current block or sub-block. The mutual prediction unit 126 then generates a mutual prediction signal of the current block or sub-block by motion compensation by using motion information (eg, motion vector) obtained from the motion estimate. The mutual prediction module 126 then outputs the generated mutual prediction signal to the prediction controller 128 .

[0065] Motion information used in motion compensation is transmitted in overhead signals. A motion vector predictor may be used to signal a motion vector. In other words, the difference between the motion vector and the motion vector predictor may be signaled.

[0066] It should be noted that the mutual prediction signal may be generated using motion information for an adjacent block in addition to motion information for the current block obtained from the motion estimate. More specifically, the mutual prediction signal may be generated per subblock in the current block by calculating a weighted sum of the prediction signal based on the motion information obtained from the motion estimate and the prediction signal based on the motion information for the adjacent block. This mutual prediction (motion compensation) is also called "overlapping block motion compensation (OBMC)".

[0067] In such an OBMC mode, information indicating the sub-block size for the OBMC (called, for example, “OBMC block size”) is signaled at the sequence level. In addition, information indicating whether or not to apply the OBMC mode (called, for example, an “OBMC flag”) is signaled at the CU level. It should be noted that the signaling of this information need not be performed at the sequence level and the CU level and may be performed at another level (eg, frame level, run level, tile level, CTU level, or subblock level).

[0068] The OBMC mode is described in more detail later in this document. Fig. 5B is a flowchart of the method, and FIG. 5C is a conceptual diagram for illustrating a brief view of a prediction image correction process performed through OBMC processing.

[0069] First, a prediction image (Pred) is obtained through typical motion compensation using a motion vector (MV) assigned to the current block.

[0070] Next, the prediction image (Pred_L) is obtained by applying the motion vector (MV_L) of the encoded adjacent left block to the current block, and the first pass of correction of the prediction image is performed by superimposing the prediction image and Pred_L.

[0071] Similarly, a prediction image (Pred_U) is obtained by applying a motion vector (MV_U) of a coded adjacent upper block to the current block, and a second pass of correction of the prediction image is performed by superimposing the prediction image resulting from the first pass and Pred_U. The result of the second pass is the final prediction image.

[0072] It should be noted that the above example is for a two-pass correction method using adjacent left and top blocks, but the method could be a three-pass or higher correction method that also uses the adjacent right and/or bottom block.

[0073] It should be noted that the area subject to overlap may be the entire pixel area of a block and, alternatively, may be a partial block boundary area.

[0074] It should be noted that here the prediction image correction process is described as being based on a single reference frame, but the same applies when the prediction image is corrected based on multiple reference frames. In such a case, after the corrected prediction images resulting from performing correction based on each of the reference frames are obtained, the obtained corrected prediction images are further superimposed to obtain a final prediction image.

[0075] It should be noted that the unit of the current block may be a prediction block and, alternatively, may be a subblock obtained by further dividing the prediction block.

[0076] One example of a method for determining whether to implement OBMC processing is to use obmc_flag, which is a signal that indicates whether to implement OBMC processing. As one specific example, the encoder determines whether the current block belongs to a region including a complex movement. The encoder sets obmc_flag to a value of "1" when the block belongs to a region including a complex motion, and implements OBMC processing when encoding, and sets obmc_flag to a value of "0" when the block does not belong to a region including a complex motion, and encodes without implementing OBMC processing. The decoder switches between implementing OBMC processing or not by decoding the obmc_flag written to the stream and performing decoding according to the flag value.

[0077] It should be noted that motion information can be extracted at the decoder side without being signaled. For example, the aggregation mode specified in the H.265/HEVC standard can be used. In addition, for example, motion information can be extracted by performing motion estimation on the decoder side. In this case, motion estimation is performed without using the pixel values of the current block.

[0078] Hereinafter, a mode for performing motion estimation on the decoder side is described. The mode for performing motion estimation on the decoder side is also called "pattern matching based motion vector extraction (PMMVD) mode" or "frame rate upconversion (FRUC) mode".

[0079] One example of FRUC processing is illustrated in FIG. 5D. First, a candidate list (the candidate list may be a union list) for candidates including a motion vector predictor is generated with reference to the motion vectors of coded blocks that are spatially or temporally adjacent to the current block. Then, the best possible MV is selected from the set of possible MVs registered in the candidate list. For example, the score values for the candidate options included in the candidate list are calculated and one candidate is selected based on the calculated score values.

[0080] Next, the motion vector for the current block is extracted from the motion vector of the selected candidate. More specifically, for example, the motion vector for the current block is calculated as the motion vector of the selected candidate (the best candidate MV), as is. Alternatively, the motion vector for the current block may be retrieved by template matching performed near a position in the reference frame corresponding to the motion vector of the selected candidate. In other words, when the neighborhood of the best possible MV is searched through an identical method, and the MV having the best evaluation value is found, the best possible MV can be updated to the MV having the better evaluation value, and the MV having the best evaluation value can be used as the ending MV for the current block. It should be noted that a configuration in which this processing is not implemented is also acceptable.

[0081] Identical processes may be performed in cases in which processing is performed in units of subblocks.

[0082] It should be noted that the evaluation value is calculated by calculating the difference in the reconstructed image by template matching performed between the region in the reference frame corresponding to the motion vector and the predetermined region. It should be noted that the score value may be calculated by using some other information in addition to the difference.

[0083] The pattern match used is either a first pattern match or a second pattern match. The first pattern matching and the second pattern matching are also referred to as "bilateral matching" and "pattern matching", respectively.

[0084] In the first pattern matching, pattern matching is performed between two blocks along the path of the current block in two different reference frames. Therefore, in the first template matching, a region in another reference frame corresponding to the motion path of the current block is used as a predetermined region for the above-described calculation of the candidate evaluation value.

[0085] FIG. 6 serves to illustrate one example of pattern matching (bilateral matching) between two blocks along a motion path. As illustrated in FIG. 6, in the first template matching, two motion vectors (MV0, MV1) are extracted by finding the best match between two blocks along the motion path of the current block (Cur block) in two different reference frames (Ref0, Ref1). More specifically, the difference between (i) the reconstructed image at a specified position in the first coded reference frame (Ref0) indicated by the MV option, and (ii) the reconstructed frame at the specified position in the second coded reference frame (Ref1) indicated by the symmetric MV , scaled with the candidate MV display time interval, may be retrieved, and a score value for the current block may be calculated by using the extracted difference. The MV candidate having the best score value from the plurality of MV candidate candidates may be selected as the final MV.

[0086] According to the continuous motion path assumption, the motion vectors (MV0, MV1) pointing to the two reference blocks must be proportional to the time distances (TD0, TD1) between the current frame (Cur Pic) and the two reference frames (Ref0, Ref1) . For example, when the current frame is located in time between two reference frames, and the temporal distance from the current frame to the two reference frames is identical, the first template matching extracts a bidirectional motion vector based on mirroring.

[0087] In the second pattern matching, pattern matching is performed between a reference in the current frame (blocks adjacent to the current block in the current frame (eg, top and/or left adjacent blocks)) and a block in the reference frame. Therefore, in the second pattern matching, a block adjacent to the current block in the current frame is used as a predetermined region for the above-described calculation of the candidate evaluation value.

[0088] FIG. 7 serves to illustrate one example of pattern matching (pattern matching) between a pattern in the current frame and a block in a reference frame. As illustrated in FIG. 7, in the second template matching, the motion vector of the current block is retrieved by performing a search on the reference frame (Ref0) to find the block that has the best match with neighboring blocks relative to the current block (Cur block) in the current frame (Cur Pic). More specifically, the difference between (i) a reconstructed image of a coded region that is both or one of an adjacent left and an adjacent upper region, and (ii) a reconstructed frame at an identical position in a coded reference frame (Ref0) indicated by the MV option, may be retrieved, and a score value for the current block may be calculated by using the extracted difference. The MV candidate having the best score value from the plurality of MV candidate candidates may be selected as the best possible MV candidate.

[0089] Information indicating whether or not to apply the FRUC mode (referred to as "eg, FRUC flag") is signaled at the CU level. In addition, when the FRUC mode is applied (eg, when the FRUC flag is set to true), information indicating the pattern matching method (first pattern matching or second pattern matching) is signaled at the CU level. It should be noted that signaling of this information need not be performed at the CU level and may be performed at another level (eg, sequence level, frame level, run level, tile level, CTU level, or subblock level).

[0090] Hereinafter, a mode for extracting a motion vector based on a model under the condition of uniform straight-line motion is described. This mode is also called "Bidirectional Optical Flow (BIO) mode".

[0091] FIG. 8 serves to illustrate the model under the condition of uniform rectilinear motion. In fig. 8, (v _x , v _y ) denotes the velocity vector, and τ ₀ and τ ₁ denote the time distances between the current frame (Cur Pic) and two reference frames (Ref0, Ref1). (MVx ₀ , MVy ₀ ) denotes a motion vector corresponding to the reference frame Ref ₀ , and (MVx ₁ , MVy ₁ ) denotes a motion vector corresponding to the reference frame Ref ₁ .

[0092] Here, according to the assumption of relatively uniform rectilinear motion demonstrated by the velocity vector (v _x , v _y ), (MVx ₀ , MVy ₀ ) and (MVx ₁ , MVy ₁ ), are represented as (v _x τ ₀ , v _y τ ₀ ) and (-v _x τ ₁ ,- v _y τ ₁ ), respectively, and the following optical flow equation is specified:

[0093] Mathematical expression 1

[0094] Here, I ^(k) denotes the value of the luminance signal from the reference frame k (k=0, 1) after motion compensation. This optical flow equation indicates that the sum of (i) the time derivative of the luminance signal value, (ii) the product of the horizontal velocity and the horizontal component of the reference frame's spatial gradient, and (iii) the product of the vertical velocity and the vertical component of the reference frame's spatial gradient is zero. The motion vector of each block, obtained from, for example, a union list, is adjusted pixel-by-pixel based on the union of the optical flow equation and Hermitian interpolation.

[0095] It should be noted that the motion vector can be extracted at the decoder side using a method other than model-based motion vector extraction under the condition of uniform straight-line motion. For example, a motion vector may be extracted for each sub-block based on the motion vectors of neighboring blocks.

[0096] Hereinafter, a mode in which a motion vector is extracted for each sub-block based on the motion vectors of neighboring blocks is described. This mode is also called "motion compensated affine prediction mode".

[0097] FIG. 9A serves to illustrate the extraction of the motion vector of each sub-block based on the motion vectors of neighboring blocks. In fig. 9A, the current block includes 16 4*4 sub-blocks. Here, the motion vector v ₀ of the left upper corner control point in the current block is extracted based on the motion vectors of neighboring sub-blocks, and the motion vector v ₁ of the right upper corner control point in the current block is extracted based on the motion vectors of neighboring blocks. After that, using two motion vectors v ₀ and v ₁ , the motion vector (vx, vy) of each sub-block in the current block is extracted using the following equation 2:

[0098] Mathematical expression 2

[0099] Here, x and y are the horizontal and vertical positions of the sub-block, respectively, and w is a predetermined weight coefficient.

[0100] Such an affine motion-compensated prediction mode may include a number of modes of different methods for extracting the motion vectors of the top left and top right corner control points. Information indicating such an affine motion-compensated prediction mode (called, for example, an "affine flag") is signaled at the CU level. It should be noted that the signaling of information indicating the affine prediction mode with motion compensation need not be performed at the CU level and may be performed at another level (e.g., sequence level, frame level, slice level, tile level, CTU level or subunit level).

[0101] Prediction controller

The prediction controller 128 selects either the internal prediction signal or the mutual prediction signal and outputs the selected prediction signal to the subtractor 104 and the adder 116.

[0102] Hereinafter, an example of extracting the motion vector through the combining mode in the current frame is given. Fig. 9B serves to illustrate a brief representation of a process for extracting a motion vector through the combining mode.

[0103] First, a list of MV predictors in which possible variants of the MV predictors are registered is generated. Examples of possible MV predictor options include: spatially adjacent MV predictors, which are MVs of encoded blocks positioned in the spatial neighborhood of the current block; a temporally adjacent MV predictor, which is an MV of a block in a coded reference frame that is adjacent to a block at a location identical to the location of the current block; a combined MV predictor, which is an MV formed by combining the MV values of a spatially adjacent MV predictor and a temporally adjacent MV predictor; and a null MV predictor, which is an MV whose value is zero.

[0104] Next, the MV of the current block is determined by selecting one MV predictor from a plurality of MV predictors registered in the MV predictor list.

[0105] Additionally, in a variable length encoder, merge_idx, which is a signal indicating which MV predictor is selected, is written and encoded into the stream.

[0106] It should be noted that the MV predictors registered in the MV predictor list illustrated in FIG. 9B constitute one example. The number of MV predictors registered in the MV predictor list may be different from the number illustrated in FIG. 9B, MV predictors registered in the MV predictor list may omit one or more types of MV predictors shown in the example of FIG. 9B, and the MV predictors registered in the MV predictor list may include one or more types of MV predictors additional and different from the types shown in the example of FIG. 9B.

[0107] It should be noted that the final MV can be determined by performing DMVR processing (described below) by using the MV of the current block extracted through the combining mode.

[0108] Hereinafter, an example of determining MV by using DMVR processing is given.

[0109] FIG. 9C is a conceptual diagram for illustrating a brief representation of DMVR processing.

[0110] First, the best-fitting MVP set for the current block is considered the MV candidate, the reference pixels are obtained from the first reference frame, which is a frame processed in the L0 direction according to the MV candidate, and the second reference frame, which is a frame processed in the L1 direction according to the MV candidate, and a template is formed by calculating the average of the reference pixels.

[0111] Next, using the reference, the surrounding regions are searched for candidate MVs of the first and second reference frames, and the least costly MV is determined as the final MV. It should be noted that the cost value is calculated using, for example, the difference between each pixel value in the reference and each pixel value in the regions in which the search is performed, as well as the MV value.

[0112] It should be noted that the summary of the processes described herein are fundamentally identical in the encoder and in the decoder.

[0113] It should be noted that processing other than processing exactly as described above may be used as long as the processing allows retrieval of the final MV by performing a search in the vicinity of the candidate MV.

[0114] Hereinafter, an example of a mode that generates a predictive image by using LIC processing is given.

[0115] FIG. 9D serves to illustrate a brief presentation of a method for generating prediction images using a brightness correction process performed through LIC processing.

[0116] First, the MV is extracted to obtain, from the encoded reference frame, a reference image corresponding to the current block.

[0117] Next, information indicating how the luminance value is changed between the reference frame and the current frame is extracted, and a luminance correction parameter is calculated by using the pixel luminance values for the coded left adjacent reference region and the coded top adjacent reference region and the pixel luminance value in identical location in the reference frame indicated by the MV.

[0118] The prediction image for the current block is generated by performing a luminance correction process by using a luminance correction parameter for the reference image in the reference frame indicated by the MV.

[0119] It should be noted that the shape of the surrounding support region illustrated in FIG. 9D is just one example; the surrounding support area may have a different shape.

[0120] In addition, although the prediction image is generated from a single reference frame in this example, in cases in which the prediction image is also generated from multiple reference frames, the prediction image is generated after performing the brightness correction process, through an identical method for the reference images obtained from reference frames.

[0121] One example of a method for determining whether LIC processing should be implemented is to use lic_flag, which is a signal that indicates whether LIC processing should be implemented. As one specific example, the encoder determines whether the current block belongs to the dimming region. The encoder sets lic_flag to the value "1" when the block belongs to the dimming region and implements LIC processing when encoding, and sets lic_flag to the value "0" when the block does not belong to the dimming region and encodes without implementing LIC processing. The decoder switches between implementing LIC processing or not by decoding the lic_flag written to the stream and performing decoding according to the value of the flag.

[0122] One example of another method for determining whether to implement LIC processing is to determine whether LIC processing is determined to be implemented for a surrounding block. In one specific example, when the merge mode is used for the current block, it can be determined whether LIC processing is applied when encoding a surrounding coded block selected after MV extraction in the merge mode processing, and whether or not LIC processing should be implemented. can switch based on the result of the determination. It should be noted that in this example, this also applies to the processing performed on the decoder side.

[0123] Decoder Brief Introduction

Next, a decoder capable of decoding a coded signal (coded bit stream) output from the encoder 100 will be described. FIG. 10 is a block diagram illustrating a functional configuration of the decoder 200 according to Embodiment 1. The decoder 200 is a moving frame/frame decoder that decodes a moving frame/frame block by block.

[0124] As illustrated in FIG. 10, decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse converter 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra prediction module 216, an inter prediction module 218, and a prediction controller 220.

[0125] The decoder 200 is implemented, for example, as a general processor and storage device. In this case, when the software program stored in the memory device is executed by the processor, the processor functions as an entropy decoder 202, an inverse quantizer 204, an inverse converter 206, an adder 208, a loop filter 212, an intra prediction unit 216, an inter-prediction unit 218 prediction and prediction controller 220. Alternatively, decoder 200 may be implemented as one or more dedicated electronic circuits corresponding to entropy decoder 202, inverse quantizer 204, inverse converter 206, adder 208, loop filter 212, intra prediction module 216, inter prediction module 218, and prediction controller 220.

[0126] Each component included in the decoder 200 is further described herein.

[0127] Entropy decoder

Entropy decoder 202 entropy decodes the encoded bit stream. More specifically, for example, entropy decoder 202 arithmetically decodes the encoded bit stream into a binary signal. Entropy decoder 202 then debinarizes the binary signal. Consequently, the entropy decoder 202 outputs the quantized coefficients of each block to the inverse quantizer 204.

[0128] Inverse Quantizer

The inverse quantizer 204 inversely quantizes the quantized coefficients of the block to be decoded (hereinafter referred to as the “current block”), which are inputs from the entropy decoder 202. More specifically, the inverse quantizer 204 inversely quantizes the quantized coefficients of the current block based on the quantization parameters corresponding to the quantized coefficients. The inverse quantizer 204 then outputs the inversely quantized coefficients (i.e., transform coefficients) of the current block to the inverse quantizer 206.

[0129] Inverter

The inverse transformer 206 recovers the prediction errors by inversely transforming the transform coefficients that are the inputs from the inverse quantizer 204.

[0130] For example, when information parsed from the coded bitstream indicates the use of EMT or AMT (eg, when the AMT flag is set to true), the inverter 206 inverts the transform coefficients of the current block based on the information indicating the detected by parse conversion type.

[0131] In addition, for example, when information parsed from the encoded bitstream indicates the application of NSST, the inverter 206 applies a secondary inverse transform to the transform coefficients.

[0132] Totalizer

The adder 208 reconstructs the current block by summing the prediction errors, which are the inputs from the inverter 206, and the prediction samples, which are the inputs from the prediction controller 220. The adder 208 then outputs the reconstructed block to a block memory 210 and a loop filter 212.

[0133] Block memory

The block memory 210 is a storage device for storing blocks in a frame to be decoded (hereinafter referred to as a “current frame”) for reference in intra prediction. More specifically, block storage 210 stores reconstructed blocks output from adder 208.

[0134] Loop filter

A loop filter 212 applies a loop filter to the blocks reconstructed by the adder 208 and outputs the filtered reconstructed blocks to a frame memory 214 and, for example, a display device.

[0135] When information indicating activation or deactivation of the ALF parsed from the encoded bitstream indicates "activated", one filter from the plurality of filters is selected based on the direction and activity of the local gradients, and the selected filter is applied to the reconstructed block.

[0136] Frame memory

The frame memory 214 is a storage device for storing reference frames used in inter-prediction and is also called a “frame buffer”. More specifically, frame memory 214 stores reconstructed blocks filtered by loop filter 212.

[0137] Internal prediction module

The intra prediction unit 216 generates a prediction signal (intra prediction signal) by intra prediction with reference to the block or blocks in the current frame and stores in the block memory 210. More specifically, the intra prediction module 216 generates an internal prediction signal by intra prediction with reference to samples (eg, luma and/or chroma values) of the block or blocks adjacent to the current block, and then outputs the internal prediction signal to the prediction controller 220.

[0138] It should be noted that when an intra prediction mode in which a chrominance signal block is internally predicted from a luma signal block is selected, the intra prediction unit 216 can predict the chrominance signal component of the current block based on the luma signal component of the current block.

[0139] In addition, when information indicating the application of PDPC is obtained by parsing from the encoded bitstream, the intra prediction unit 216 corrects the pixel values after the intra prediction based on the horizontal/vertical reference pixel gradients.

[0140] Mutual Prediction Module

The inter-prediction module 218 predicts the current block with reference to a reference frame stored in the frame memory 214. Inter-prediction is performed on a per-current block basis or on a per-sub-block basis (eg, per 4*4 block) within the current block. For example, the mutual prediction module 218 generates a mutual prediction signal of the current block or sub-block by motion compensation by using motion information (eg, motion vector) obtained by parsing from the encoded bit stream, and outputs the mutual prediction signal to the prediction controller 220.

[0141] It should be noted that when information obtained by parsing from the encoded bitstream indicates application of the OBMC mode, the inter-prediction unit 218 generates a mutual prediction signal using the motion information for the adjacent block in addition to the motion information for the current block. obtained from the motion estimate.

[0142] In addition, when information obtained by parsing from the encoded bit stream indicates the use of the FRUC mode, the inter-prediction module 218 extracts the motion information by performing motion estimation in accordance with a template matching method (bilateral matching or pattern matching) , obtained by parsing from the encoded bitstream. The mutual prediction module 218 then performs motion compensation using the extracted motion information.

[0143] In addition, when the BIO mode is to be applied, the inter-prediction module 218 extracts a motion vector based on the model under the condition of uniform straight-line motion. In addition, when information obtained by parsing from the encoded bitstream indicates that the motion-compensated affine prediction mode should be applied, the inter-prediction module 218 extracts the motion vector of each sub-block based on the motion vectors of neighboring blocks.

[0144] Prediction Controller

The prediction controller 220 selects either the internal prediction signal or the mutual prediction signal and outputs the selected prediction signal to the adder 208.

[0145] Embodiment 2

In the following, the encoding process and the decoding process according to Embodiment 2 will be described in detail with reference to FIGS. 11 and fig. 12, and the encoder and decoder according to Embodiment 2 are described in detail hereinafter with reference to FIG. 15 and fig. 16.

[0146] Encoding process

Fig. 11 illustrates a video encoding process according to Embodiment 2.

[0147] First, in step S1001, a first parameter for identifying, from a plurality of segmentation modes, a segmentation mode for segmenting the first block into subblocks is written into the bitstream. Using the segmentation mode should result in the block being segmented into subblocks. Using different segmentation modes can result in a block being segmented into subblocks with different shapes, either with different heights or different widths.

[0148] FIG. 28 illustrates examples of segmentation modes for segmenting a block of N pixels into N pixels in Embodiment 2. In fig. 28(a)-(h) show different segmentation modes. As illustrated in FIG. 28, using segmentation mode (a) segments a block of N pixels by N pixels (the value of N can be any value in the range of 8 to 128, which are integer multiples of 4, for example, a block of 16 pixels by 16 pixels) into two N/2 subblocks pixels by N pixels (for example, subblocks of 8 pixels by 16 pixels). Using segmentation mode (b) segments the N pixels by N pixels block into an N/4 pixels by N pixels subblock and a 3N/4 pixels by N pixels subblock (eg, into a 4 pixels by 16 pixels subblock and a 12 pixels by 16 pixels subblock). Using segmentation mode (c) segments the N pixels by N pixels block into a 3N/4 pixels by N pixels subblock and an N/4 pixels by N pixels subblock (eg, into a 12 pixels by 16 pixels subblock and a 4 pixels by 16 pixels subblock). Using segmentation mode (d) segments a block of N pixels by N pixels into a subblock of N/4 pixels by N pixels, a subblock of N/2 pixels by N pixels, and a subblock of N/4 pixels by N pixels (for example, into a subblock of 4 pixels by 16 pixels , an 8 pixel by 16 pixel subblock and a 4 pixel by 16 pixel subblock). Using segmentation mode (e) segments a block of N pixels by N pixels into two subblocks of N pixels by N/2 pixels (eg, into 16 pixels by 8 pixels subblocks). Using segmentation mode (f) segments the N pixels by N pixels block into an N pixels by N/4 pixels subblock and an N pixels by 3N/4 pixels subblock (eg, into a 16 pixels by 4 pixels subblock and a 16 pixels by 12 pixels subblock). Using segmentation mode (g) segments a block of N pixels by N pixels into a subblock of N pixels by 3N/4 pixels and a subblock of N pixels by N/4 pixels (eg, a subblock of 16 pixels by 12 pixels and a subblock of 16 pixels by 4 pixels). Using segmentation mode (h) segments a block of N pixels by N pixels into a subblock of N pixels by N/4 pixels, a subblock of N pixels by N/2 pixels, and a subblock of N pixels by N/4 pixels (for example, into a subblock of 16 pixels by 4 pixels , a 16 pixel by 8 pixel subblock and a 16 pixel by 4 pixel subblock).

[0149] Next, in step S1002, it is determined whether or not the first parameter identifies the first segmentation mode.

[0150] Next, in step S1003, based on at least determining whether the first parameter identifies the first segmentation mode, it is determined whether the second segmentation mode should not or should be selected as an option for segmenting the second block.

[0151] Two different sets of segmentation modes can split a block into subblocks of identical shapes and sizes. For example, as illustrated in FIG. 31A, the subblocks of (1b) and (2c) have identical shapes and sizes. The set of segmentation modes may include at least two segmentation modes. For example, a set of segmentation modes may include vertical ternary tree partitioning followed by vertical binary tree partitioning for a central subblock and no partitioning for other subblocks, as illustrated in (1a) and (1b) of FIG. 31A. Another set of segmentation modes, for example, may include a vertical binary tree split followed by a vertical binary tree split for both of the subblocks, as illustrated in (2a), (2b) and (2c) of FIG. 31A. Both sets of segmentation modes should result in subblocks of identical shapes and sizes.

[0152] When choosing from two sets of segmentation modes that result in a block being divided into subblocks of identical shapes and sizes, and wherein each of the sets of segmentation modes when encoded into the bitstream has a different number of bins or a different number of bits, the set of segmentation modes that has fewer bins, or fewer bits, are selected from two sets. It should be noted that the number of bins and the number of bits are equivalent to the amount of code.

[0153] When selecting from two sets of segmentation modes that result in a block being partitioned into subblocks of identical shapes and sizes, and wherein each of the sets of segmentation modes when encoded into the bitstream has an identical number of bins or an identical number of bits, the set of segmentation modes that appears first in a predetermined order of a plurality of segmentation mode sets, selected from two sets. An example of a predetermined order may be an order based on the number of segmentation modes in each segmentation mode set.

[0154] FIG. 31A and FIG. 31B illustrates an example of dividing a block into sub-blocks using a set of segmentation modes with fewer bins when encoding the segmentation modes. In this example, when the left N pixels by N pixels block is vertically divided into two subblocks, the second segmentation mode in step (2c) for the right N pixels by N pixels block is not selected. This is because in the coding method based on the segmentation mode in FIG. 31B, the second set of segmentation modes (2a, 2b, 2c) should require more bins to encode the segmentation mode compared to the first set of segmentation modes (1a, 1b).

[0155] FIG. 32A-32C illustrate an example of dividing a block into subblocks using a segmentation mode set that appears first in a predetermined order of a plurality of segmentation mode sets. In this example, when the upper 2N pixels by N/2 pixels block is vertically divided into three sub-blocks, the second segmentation mode in step (2c) for the lower 2N pixels by N/2 pixels block is not selected. This is because in the coding method based on the segmentation mode in FIG. 32B, the second set of segmentation modes (2a, 2b, 2c) has a number of bins identical to the number of bins of the first set of segmentation modes (1a, 1b, 1c, 1d), and appears after the first set of segmentation modes (1a, 1b, 1c, 1d) in a predetermined order of segmentation mode sets in FIG. 32C. The predetermined order of the set segmentation mode sets may be fixed or may be signaled in the bitstream.

[0156] FIG. 20 illustrates an example in which the second segmentation mode is not selected to segment a block of 2N pixels into N pixels as illustrated in step (2c) in Embodiment 2. As illustrated in FIG. 20, a 2N pixels by 2N pixels block (eg, a 16 pixels by 16 pixels block) may be divided into four equal subblocks of size N pixels by N pixels (eg, 8 pixels by 8 pixels) using the first partitioning method (i), similar to step (1a). In addition, the 2N pixels by 2N pixels block may also be divided into two equal 2N pixels by N pixels subblocks (eg, 16 pixels by 8 pixels) using the second partitioning method (2a), similar to step (2a). During the second partitioning method (ii), when the first segmentation mode splits the top 2N pixels by N pixels block (first block) vertically into two N pixels by N pixels subblocks, similar to step (2b), the second segmentation mode that vertically partitions the bottom block 2N pixels by N pixels (second block) by two N pixels by N pixels subblocks in step (2c) is not selected as a candidate for the segmentation mode candidate. This is due to the fact that the second segmentation mode must form subblock sizes identical to the subblock sizes in the quadratic partition from the first partition method (i).

[0157] Thus, in FIG. 20, when a first block is vertically split into two equal sub-blocks if the first segmentation mode is used, and a second block vertically adjacent to the first block is vertically split into two equal sub-blocks if the second segmentation mode is used, the second Segmentation mode is not selected as an option.

[0158] FIG. 21 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into 2N pixels as illustrated in step (2c) in Embodiment 2. As illustrated in FIG. 21, the 2N pixels by 2N pixels block can be divided into four equal N pixels by N pixels subblocks using the first partitioning method (i). Moreover, similar to step (2a), the 2N pixels by 2N pixels block may also be vertically partitioned into two equal N pixels by 2N pixels subblocks (eg, 8 pixels by 16 pixels) using the second partitioning method (ii). During the second partitioning method (ii), when the first segmentation mode partitions the left block of N pixels by 2N pixels (the first block) horizontally into two sub-blocks of N pixels by N pixels, similar to step (2b), the second segmentation mode which partitions the right block horizontally N pixels by 2N pixels (second block) by two N pixels by N pixels subblocks in step (2c) is not selected as a candidate for the segmentation mode candidate. This is due to the fact that the second segmentation mode must form subblock sizes identical to the subblock sizes in the quadratic partition from the first partition method (i).

[0159] Thus, in FIG. 21, when a first block is horizontally divided into two equal subblocks if the first segmentation mode is used, and a second block horizontally adjacent to the first block is horizontally divided into two equal subblocks if the second segmentation mode is used, the second Segmentation mode is not selected as an option.

[0160] FIG. 40 illustrates an example in which the 4Nx2N segment of FIG. 20 is divided into three at a ratio of 1:2:1, for example, Nx2N, 2Nx2N and Nx2N. Here, when the upper block is to be divided into three, the segmentation mode for dividing the lower block into three at a ratio of 1:2:1 is not selected as a possible option for the possible segmentation mode. Triple splitting can be done at a ratio other than 1:2:1. In addition, splitting by more than 3 can be performed, and even when splitting by two, the ratio may be different from 1:1, for example, 1:2 or 1:3. Although fig. 40 illustrates an example of splitting horizontally first, the same constraint may also apply when splitting vertically first.

[0161] FIG. 41 and fig. 42 illustrate an example in which the same constraint applies if the first block is a rectangle.

[0162] FIG. 43 illustrates an example of the second constraint, where the square is horizontally divided into three and further horizontally divided into two equal parts. When applying the constraint in FIG. 43, in fig. 40, a segmentation mode can be selected to split the lower 4Nx2N block into three at a ratio of 1:2:3. Information indicating which of the restrictions in FIG. 40 and limitations in FIG. 43 shall be applied, may be separately encoded in the header information. Alternatively, a constraint having a smaller amount of segment-indicating information code may be applied. For example, provided that the amount of information codes indicating the segment in case 1 and case 2 is as shown below, the division of case 1 is activated and the division of case 2 is deactivated. In other words, the limitation in FIG. 43 applies.

(Case 1) (1) After the square is horizontally divided into two, (2) each of the two rectangular blocks at the top and bottom is vertically divided into three: (1) direction information: 1 bit, segment number information: 1 bit, ( 2) (direction information: 1 bit, number of segment information: 1 bit) x 2 in total for 6 bits.

(Case 2) (1) After the square is vertically divided, (2) each of the left, center and right rectangular blocks is horizontally divided into two: (1) direction information: 1 bit, segment number information: 1 bit, (2 ) (direction information: 1 bit, number of segment information: 1 bit) x 3 for a total of 8 bits.

[0163] Alternatively, during encoding, there are cases where the corresponding segment is determined by selecting a segmentation mode in a predetermined order. For example, you can try a split into two, then try a split into three or a split into four (2 equal parts horizontally and vertically), etc. At this time, before attempting to split into three as shown in FIG. 43, a trial that starts with a split into two, similar to the examples in FIG. 40, already completed. Therefore, in a trial that starts with a split into two, the limitation in FIG. 43 applies because the segments resulting from equal division of a block horizontally and an additional vertical division of the two blocks at the top and bottom into three are already obtained. Thus, the restriction method to be selected can be determined based on a predetermined encoding scheme.

[0164] FIG. 44 illustrates an example in which, in the second segmentation mode, selectable segmentation modes for segmenting a second block in a direction identical to the direction of the first segmentation mode are limited. Here, the first segmentation mode is vertically split into three, and therefore, at this time, split into two cannot be selected as the second segmentation mode. On the other hand, halving may be selected for the vertical direction, which is a different direction from the first segmentation mode (FIG. 45).

[0165] FIG. 22 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into N pixels as illustrated in step (2c) in Embodiment 2. As illustrated in FIG. 22, a block of 2N pixels by N pixels (the value of N can be any value in the range of 8 to 128, which are integer multiples of 4, such as 16 pixels by 8 pixels) may be vertically divided into an N/2 pixels by N pixels subblock, subblock N pixels by N pixels and N/2 pixels by N pixels subblock (e.g., 4 pixels by 8 pixels subblock, 8 pixels by 8 pixels subblock, 4 pixels by 8 pixels subblock), using the first partitioning method (i), similar step (1a). In addition, the 2N pixels by N pixels block can also be divided into two N pixels by N pixels subblocks using the second partitioning method (ii), similar to step (2a). During the first partitioning method (i), the central N pixels by N pixels block may be vertically partitioned into two N/2 pixels by N pixels subblocks (eg, 4 pixels by 8 pixels) in step (1b). During the second partitioning method (ii), where the left N pixels by N pixels block (the first block) is vertically partitioned into two N/2 pixels by N pixels subblocks, similar to step (2b), the segmentation mode which vertically partitions the right N pixels block by N pixels (second block) by two sub-blocks N/2 pixels by N pixels in step (2c), is not selected as a possible option for the possible segmentation mode. This is because the segmentation mode must produce subblock sizes that are identical to the subblock sizes obtained from the first partitioning method (i), or four N/2 pixels by N pixels subblocks.

[0166] Thus, in FIG. 22, when a first block is vertically divided into two equal subblocks if the first segmentation mode is used, and a second block horizontally adjacent to the first block is vertically divided into two equal subblocks if the second segmentation mode is used, the second Segmentation mode is not selected as an option.

[0167] FIG. 23 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into N pixels as illustrated in step (2c) in Embodiment 2. As illustrated in FIG. 23, a block of N pixels by 2N pixels (the value of N can be any value in the range of 8 to 128, which are integer multiples of 4, such as 8 pixels by 16 pixels) may be divided into a subblock of N pixels by N/2 pixels, subblock N pixels by N pixels and a subblock of N pixels by N/2 pixels (for example, by a subblock of 8 pixels by 4 pixels, a subblock of 8 pixels by 8 pixels, a subblock of 8 pixels by 4 pixels) using the first partitioning method (i), similar to step ( 1a). In addition, the N pixels by 2N pixels block can also be divided into two N pixels by N pixels subblocks using the second partitioning method, similar to step (2a). During the first partitioning method (i), the central block of N pixels by N pixels may be horizontally divided into two subblocks of N pixels by N/2 pixels, similar to step (1b). During the second partitioning method (ii), when the top N pixels by N pixels (the first block) is horizontally partitioned into two N pixels by N/2 pixels subblocks in step (2b), a segmentation mode that horizontally partitions the bottom N pixels by N pixels (second block) into two sub-blocks of N pixels by N/2 pixels in step (2c), is not selected as a candidate for the candidate segmentation mode. This is because the segmentation mode must produce subblock sizes that are identical to the subblock sizes obtained from the first partitioning method (i), or four subblocks of N pixels by N/2 pixels.

[0168] Thus, in FIG. 23, when a first block is horizontally divided into two equal subblocks if the first segmentation mode is used, and a second block vertically adjacent to the first block is horizontally divided into two equal subblocks if the second segmentation mode is used, the second Segmentation mode is not selected as an option.

[0169] If it is determined that the second segmentation mode is to be selected as an option for segmenting the second block (N in S1003), the segmentation mode is selected from a plurality of segmentation modes that include the second segmentation mode as an option, in step S1004 . In step S1005, a second parameter indicating the selection result is written to the bit stream.

[0170] If it is determined that the second segmentation mode should not be selected as an option for segmenting the second block (Y in S1003), a segmentation mode different from the second segmentation mode is selected for segmenting the second block in step S1006. Here, the selected segmentation mode segments the block into subblocks with different shapes or different sizes compared to the subblocks to be formed by the second segmentation mode.

[0171] FIG. 24 illustrates an example of segmenting a block of 2N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2. As illustrated in FIG. 24, the selected segmentation mode can split the current 2N pixel by N pixel block (the bottom block in this example) into three sub-blocks, as illustrated in (c) and (f) of FIG. 24. The dimensions of the three sub-blocks may vary. For example, of three subblocks, a large subblock may have a width/height that is twice the width/height of a small subblock. Additionally, for example, the selected segmentation mode may split the current block into two subblocks with different sizes (asymmetric binary tree), as illustrated in (a), (b), (d) and (e) of FIG. 24. For example, when an asymmetric binary tree is used, a large sub-block may have a width/height that is three times the width/height of a small sub-block.

[0172] FIG. 25 illustrates an example of segmenting a block of N pixels into 2N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2. As illustrated in FIG. 25, the selected segmentation mode can divide the current block of N pixels by 2N pixels (the right block in this example) into three sub-blocks, as illustrated in (c) and (f) in FIG. 25. The dimensions of the three sub-blocks may vary. For example, of three subblocks, a large subblock may have a width/height that is twice the width/height of a small subblock. Additionally, for example, the selected segmentation mode may split the current block into two subblocks with different sizes (asymmetric binary tree), as illustrated in (a), (b), (d) and (e) of FIG. 25. For example, when an asymmetric binary tree is used, a large sub-block may have a width/height that is three times the width/height of a small sub-block.

[0173] FIG. 26 illustrates an example of segmenting a block of N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2. As illustrated in FIG. 26, the 2N pixels by N pixels block is vertically divided into two N pixels by N pixels subblocks in step (1), and the left N pixels by N pixels block is vertically divided into two N/2 pixels by N pixels subblocks in step (2). In step (3), the current block may be segmented into three sub-blocks using the segmentation mode selected for the current N pixel by N pixel block (the left block in this example), as illustrated in (c) and (f) of FIG. 26. The dimensions of the three sub-blocks may vary. For example, of three subblocks, a large subblock may have a width/height that is twice the width/height of a small subblock. Additionally, for example, the selected segmentation mode may split the current block into two subblocks with different sizes (asymmetric binary tree), as illustrated in (a), (b), (d) and (e) of FIG. 26. For example, when an asymmetric binary tree is used, a large sub-block may have a width/height that is three times the width/height of a small sub-block.

[0174] FIG. 27 illustrates an example of segmenting a block of N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2. As illustrated in FIG. 27, the N pixels by 2N pixels block is horizontally divided into two N pixels by N pixels subblocks in step (1), and the top N pixels by N pixels block is horizontally divided into two N pixels by N/2 pixels subblocks in step (2). In step (3), the current block may be segmented into three sub-blocks using the segmentation mode selected for the current N pixel by N pixel block (the bottom block in this example), as illustrated in (c) and (f) of FIG. 27. The dimensions of the three sub-blocks may vary. For example, of three subblocks, a large subblock may have a width/height that is twice the width/height of a small subblock. Additionally, for example, the selected segmentation mode may split the current block into two subblocks with different sizes (asymmetric binary tree), as illustrated in (a), (b), (d) and (e) of FIG. 27. For example, when an asymmetric binary tree is used, a large sub-block may have a width/height that is three times the width/height of a small sub-block.

[0175] FIG. 17 illustrates possible locations of a first parameter in a compressed video bit stream. As illustrated in FIG. 17, the first parameter may be in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The first parameter may indicate a method for segmenting a block into a plurality of subblocks. For example, the first parameter may include a flag to indicate whether the block should be segmented in a horizontal or vertical direction. The first parameter may also include a parameter for indicating whether the block should be segmented into two or more subblocks.

[0176] FIG. 18 illustrates possible locations of a second parameter in a compressed video bit stream. As illustrated in FIG. 18, the second parameter may be in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The second parameter may indicate a method for segmenting the block into a plurality of subblocks. For example, the second parameter may include a flag to indicate whether the block should be segmented in a horizontal or vertical direction. The second parameter may also include a parameter to indicate whether the block should be segmented into two or more subblocks. The second parameter comes after the first parameter in the bitstream, as illustrated in FIG. 19.

[0177] The first block and the second block are different blocks. The first block and the second block may be included in an identical frame. For example, the first block may be an upper adjacent block relative to the second block. In addition, for example, the first block may be a left adjacent block relative to the second block.

[0178] In step S1007, the second block is segmented into sub-blocks using the selected segmentation mode. In step S1008, the segmented blocks are encoded.

[0179] Encoder

Fig. 15 is a block diagram illustrating the structure of a video/image encoder according to Embodiment 2 or 3.

[0180] The video encoder 5000 is equipment for encoding an input video/image on a block-by-block basis so as to generate a coded output bitstream. As illustrated in FIG. 15, video encoder 5000 includes a converter 5001, a quantizer 5002, an inverse quantizer 5003, an inverse transformer 5004, a block memory 5005, a frame memory 5006, an intra prediction module 5007, an inter prediction module 5008, an entropy encoder 5009, and a segmentation determination module 5010. blocks.

[0181] The input video is input to the adder, and the summed value is output to converter 5001. Converter 5001 converts the summed values into frequency coefficients based on the block segmentation mode extracted from the block segmentation determining unit 5010, and outputs the frequency coefficients to quantizer 5002. Mode block segmentation may be related to the block segmentation mode, the block segmentation type, or the direction of block segmentation. Quantizer 5002 quantizes the input quantized coefficients and outputs the quantized values to inverse quantizer 5003 and entropy encoder 5009.

[0182] The inverse quantizer 5003 inversely quantizes the quantized values output from the quantizer 5002 and outputs the frequency coefficients to the inverse converter 5004. The inverse converter 5004 performs inverse frequency conversion on the frequency coefficients based on the block segmentation mode retrieved from the block segmentation determination unit 5010 , so as to convert the frequency coefficients into sample values of the bit stream, and outputs the sample values to the adder.

[0183] The adder adds the sample bit stream values output from the inverse converter 5004 with the predicted video/image values output from the inter/intra prediction module 5007, 5008, and outputs the summed values to a block memory 5005 or a frame memory 5006 for additional forecasting. The block segmentation determining unit 5010 collects block information from the block memory 5005 or the frame memory 5006 to retrieve the block segmentation mode and parameters associated with the block segmentation mode. Using the extracted block segmentation mode should result in the block being segmented into multiple sub-blocks. The inter/intra prediction module 5007, 5008 searches from videos/images stored in the block memory 5005 or from videos/images in the frame memory 5006 reconstructed using the block segmentation mode retrieved from the block segmentation determination module 5010, and estimates a region of video/images that is, for example, most similar to the input video/images for prediction.

[0184] Entropy encoder 5009 encodes the quantized values output from quantizer 5002, encodes parameters from block segmentation determiner 5010, and outputs a bit stream.

[0185] Decoding process

Fig. 12 illustrates a video decoding process according to Embodiment 2.

[0186] First, in step S2001, a first parameter for identifying, from a plurality of segmentation modes, a segmentation mode for segmenting the first block into subblocks is obtained by parsing from the bit stream. Using a segmentation mode should result in a block being segmented into subblocks, and using different segmentation modes may result in a block being segmented into subblocks with different shapes, either with different heights or different widths.

[0187] FIG. 28 illustrates examples of segmentation modes for segmenting a block of N pixels into N pixels in Embodiment 2. In fig. 28(a)-(h) show different segmentation modes. As illustrated in FIG. 28, using segmentation mode (a) segments a block of N pixels by N pixels (the value of N can be any value in the range of 8 to 128, which are integer multiples of 4, for example, a block of 16 pixels by 16 pixels) into two N/2 subblocks pixels by N pixels (for example, subblocks of 8 pixels by 16 pixels). Using segmentation mode (b) segments the N pixels by N pixels block into an N/4 pixels by N pixels subblock and a 3N/4 pixels by N pixels subblock (eg, into a 4 pixels by 16 pixels subblock and a 12 pixels by 16 pixels subblock). Using segmentation mode (c) segments the N pixels by N pixels block into a 3N/4 pixels by N pixels subblock and an N/4 pixels by N pixels subblock (eg, into a 12 pixels by 16 pixels subblock and a 4 pixels by 16 pixels subblock). Using segmentation mode (d) segments a block of N pixels by N pixels into a subblock of N/4 pixels by N pixels, a subblock of N/2 pixels by N pixels, and a subblock of N/4 pixels by N pixels (for example, into a subblock of 4 pixels by 16 pixels , an 8 pixel by 16 pixel subblock and a 4 pixel by 16 pixel subblock). Using segmentation mode (e) segments a block of N pixels by N pixels into two subblocks of N pixels by N/2 pixels (eg, into 16 pixels by 8 pixels subblocks). Using segmentation mode (f) segments the N pixels by N pixels block into an N pixels by N/4 pixels subblock and an N pixels by 3N/4 pixels subblock (eg, into a 16 pixels by 4 pixels subblock and a 16 pixels by 12 pixels subblock). Using segmentation mode (g) segments a block of N pixels by N pixels into a subblock of N pixels by 3N/4 pixels and a subblock of N pixels by N/4 pixels (eg, a subblock of 16 pixels by 12 pixels and a subblock of 16 pixels by 4 pixels). Using segmentation mode (h) segments a block of N pixels by N pixels into a subblock of N pixels by N/4 pixels, a subblock of N pixels by N/2 pixels, and a subblock of N pixels by N/4 pixels (for example, into a subblock of 16 pixels by 4 pixels , a 16 pixel by 8 pixel subblock and a 16 pixel by 4 pixel subblock).

[0188] Next, in step S2002, it is determined whether the first parameter identifies the first segmentation mode.

[0189] Next, in step S2003, based on at least the determination as to whether the first parameter identifies the first segmentation mode, it is determined whether the second segmentation mode should not or should be selected as an option for segmenting the second block.

[0190] Two different sets of segmentation modes can segment a block into subblocks of identical shapes and sizes. For example, as illustrated in FIG. 31A, the subblocks of (1b) and (2c) have identical shapes and sizes. The set of segmentation modes may include at least two segmentation modes. For example, a set of segmentation modes may include vertical ternary tree partitioning followed by vertical binary tree partitioning for a central subblock and no partitioning for other subblocks, as illustrated in (1a) and (1b) of FIG. 31A. Additionally, for example, another set of segmentation modes may include a vertical binary tree split followed by a vertical binary tree split for both of the subblocks, as illustrated in (2a), (2b) and (2c) of FIG. 31A. Both sets of segmentation modes should result in subblocks of identical shapes and sizes.

[0191] When choosing from two sets of segmentation modes that result in a block being divided into subblocks of identical shapes and sizes, and wherein each of the sets of segmentation modes when encoded into the bit stream has a different number of bins or a different number of bits, the set of segmentation modes that has fewer bins, or fewer bits, are selected from two sets.

[0192] When selecting from two sets of segmentation modes that result in a block being partitioned into subblocks of identical shapes and sizes, and wherein each of the sets of segmentation modes when encoded into the bitstream has an identical number of bins or an identical number of bits, the set of segmentation modes that appears first in a predetermined order of a plurality of segmentation mode sets, selected from two sets. An example of a predetermined order may be an order based on the number of segmentation modes in each segmentation mode set.

[0193] FIG. 31A and FIG. 31B illustrates an example of dividing a block into sub-blocks using a set of segmentation modes with fewer bins when encoding the segmentation modes. In this example, when the left N pixels by N pixels block is vertically divided into two subblocks, the second segmentation mode in step (2c) for the right N pixels by N pixels block is not selected. This is because in the coding method based on the segmentation mode in FIG. 31B, the second set of segmentation modes (2a, 2b, 2c) should require more bins from the segmentation mode encoding compared to the first set of segmentation modes (1a, 1b).

[0194] FIG. 32A illustrates an example of dividing a block into subblocks using a segmentation mode set that appears first in a predetermined order of a plurality of segmentation mode sets. In this example, when the upper 2N pixels by N/2 pixels block is vertically divided into three sub-blocks, the second segmentation mode in step (2c) for the lower 2N pixels by N/2 pixels block is not selected. This is because in the coding method based on the segmentation mode in FIG. 32B, the second set of segmentation modes (2a, 2b, 2c) has a number of bins identical to the number of bins of the first set of segmentation modes (1a, 1b, 1c, 1d), and appears after the first set of segmentation modes (1a, 1b, 1c, 1d) in a predetermined order of segmentation mode sets in FIG. 32C. The predetermined order of the set segmentation mode sets may be fixed or may be signaled in the bit stream.

[0195] FIG. 20 illustrates an example in which the second segmentation mode is not selected to segment a block of 2N pixels into N pixels as illustrated in step (2c) in Embodiment 1. As illustrated in FIG. 20, a 2N pixels by 2N pixels block (eg, a 16 pixels by 16 pixels block) may be divided into four equal subblocks of size N pixels by N pixels (eg, 8 pixels by 8 pixels) using the first partitioning method (i), similar to step (1a). In addition, the 2N pixels by 2N pixels block may also be divided into two equal 2N pixels by N pixels subblocks (eg, 16 pixels by 8 pixels) using the second partitioning method (2a), similar to step (2a). During the second partitioning method (ii), when the first segmentation mode splits the top 2N pixels by N pixels block (first block) vertically into two N pixels by N pixels subblocks, similar to step (2b), the second segmentation mode that vertically partitions the bottom block 2N pixels by N pixels (second block) by two N pixels by N pixels subblocks in step (2c) is not selected as a candidate for the segmentation mode candidate. This is due to the fact that the second segmentation mode must form subblock sizes identical to the subblock sizes in the quadratic partition from the first partition method (i).

[0196] Thus, in FIG. 20, when a first block is vertically divided into two equal subblocks if the first segmentation mode is used, and a second block vertically adjacent to the first block is vertically divided into two equal subblocks if the second segmentation mode is used, the second Segmentation mode is not selected as an option.

[0197] FIG. 21 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into 2N pixels as illustrated in step (2c) in Embodiment 2. As illustrated in FIG. 21, the 2N pixels by 2N pixels block can be divided into four equal N pixels by N pixels subblocks using the first partitioning method (i). In addition, similar to step (2a), the 2N pixels by 2N pixels block may also be vertically partitioned into two equal subblocks of N pixels by 2N pixels (eg, 8 pixels by 16 pixels) using the second partitioning method (ii). During the second partitioning method (ii), when the first segmentation mode partitions the left block of N pixels by 2N pixels (the first block) horizontally into two sub-blocks of N pixels by N pixels, similar to step (2b), the second segmentation mode which partitions the right block horizontally N pixels by 2N pixels (second block) by two N pixels by N pixels subblocks in step (2c) is not selected as a candidate for the segmentation mode candidate. This is due to the fact that the second segmentation mode must form subblock sizes identical to the subblock sizes in the quadratic partition from the first partition method (i).

[0198] Thus, in FIG. 21, when a first block is horizontally divided into two equal subblocks if the first segmentation mode is used, and a second block horizontally adjacent to the first block is horizontally divided into two equal subblocks if the second segmentation mode is used, the second Segmentation mode is not selected as an option.

[0199] FIG. 22 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into N pixels as illustrated in step (2c) in Embodiment 2. As illustrated in FIG. 22, a block of 2N pixels by N pixels (the value of N can be any value in the range of 8 to 128, which are integer multiples of 4, such as 16 pixels by 8 pixels) may be vertically divided into an N/2 pixels by N pixels subblock, subblock N pixels by N pixels and N/2 pixels by N pixels subblock (e.g., 4 pixels by 8 pixels subblock, 8 pixels by 8 pixels subblock, 4 pixels by 8 pixels subblock), using the first partitioning method (i), similar step (1a). In addition, the 2N pixels by N pixels block can also be divided into two N pixels by N pixels subblocks using the second partitioning method (ii), similar to step (2a). During the first partitioning method (i), the central N pixels by N pixels block may be vertically partitioned into two N/2 pixels by N pixels subblocks (eg, 4 pixels by 8 pixels) in step (1b). During the second partitioning method (ii), where the left N pixels by N pixels block (the first block) is vertically partitioned into two N/2 pixels by N pixels subblocks, similar to step (2b), the segmentation mode which vertically partitions the right N pixels block by N pixels (second block) by two sub-blocks N/2 pixels by N pixels in step (2c), is not selected as a possible option for the possible segmentation mode. This is because the segmentation mode must produce subblock sizes that are identical to the subblock sizes obtained from the first partitioning method (i), or four N/2 pixels by N pixels subblocks.

[0200] Thus, in FIG. 22, when a first block is vertically split into two equal sub-blocks if the first segmentation mode is used, and a second block horizontally adjacent to the first block is vertically split into two equal sub-blocks if the second segmentation mode is used, the second Segmentation mode is not selected as an option.

[0201] FIG. 23 illustrates an example in which the second segmentation mode is not selected to segment a block of N pixels into N pixels as illustrated in step (2c) in Embodiment 2. As illustrated in FIG. 23, a block of N pixels by 2N pixels (the value of N can be any value in the range of 8 to 128, which are integer multiples of 4, such as 8 pixels by 16 pixels) may be divided into a subblock of N pixels by N/2 pixels, subblock N pixels by N pixels and a subblock of N pixels by N/2 pixels (for example, by a subblock of 8 pixels by 4 pixels, a subblock of 8 pixels by 8 pixels, a subblock of 8 pixels by 4 pixels) using the first partitioning method (i), similar to step ( 1a). In addition, the N pixels by 2N pixels block can also be divided into two N pixels by N pixels subblocks using the second partitioning method, similar to step (2a). During the first partitioning method (i), the central block of N pixels by N pixels may be horizontally divided into two subblocks of N pixels by N/2 pixels, similar to step (1b). During the second partitioning method (ii), when the top N pixels by N pixels (the first block) is horizontally partitioned into two N pixels by N/2 pixels subblocks in step (2b), a segmentation mode that horizontally partitions the bottom N pixels by N pixels (second block) into two sub-blocks of N pixels by N/2 pixels in step (2c), is not selected as a candidate for the candidate segmentation mode. This is because the segmentation mode must produce subblock sizes that are identical to the subblock sizes obtained from the first partitioning method (i), or four subblocks of N pixels by N/2 pixels.

[0202] Thus, in FIG. 23, when a first block is horizontally divided into two equal sub-blocks if the first segmentation mode is used, and a second block vertically adjacent to the first block is horizontally divided into two equal sub-blocks if the second segmentation mode is used, the second Segmentation mode is not selected as an option.

[0203] If it is determined that the second segmentation mode is to be selected as an option for segmenting the second block (N in S2003), the second parameter is obtained by parsing from the bitstream, and the segmentation mode is selected from a plurality of segmentation modes that include a second segmentation mode as an option in step S2004.

[0204] If it is determined that the second segmentation mode should not be selected as an option for segmenting the second block (Y in S2003), a segmentation mode different from the second segmentation mode is selected for segmenting the second block in step S2005. Here, the selected segmentation mode segments the block into subblocks with different shapes or different sizes compared to the subblocks to be formed by the second segmentation mode.

[0205] FIG. 24 illustrates an example of segmenting a block of 2N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2. As illustrated in FIG. 24, the selected segmentation mode can split the current 2N pixel by N pixel block (the bottom block in this example) into three sub-blocks, as illustrated in (c) and (f) of FIG. 24. The dimensions of the three sub-blocks may vary. For example, of three subblocks, a large subblock may have a width/height that is twice the width/height of a small subblock. Additionally, for example, the selected segmentation mode may split the current block into two subblocks with different sizes (asymmetric binary tree), as illustrated in (a), (b), (d) and (e) of FIG. 24. For example, when an asymmetric binary tree is used, a large sub-block may have a width/height that is three times the width/height of a small sub-block.

[0206] FIG. 25 illustrates an example of segmenting a block of N pixels into 2N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2. As illustrated in FIG. 25, the selected segmentation mode can divide the current block of N pixels by 2N pixels (the right block in this example) into three sub-blocks, as illustrated in (c) and (f) in FIG. 25. The dimensions of the three sub-blocks may vary. For example, of three subblocks, a large subblock may have a width/height that is twice the width/height of a small subblock. Additionally, for example, the selected segmentation mode may split the current block into two subblocks with different sizes (asymmetric binary tree), as illustrated in (a), (b), (d) and (e) of FIG. 25. For example, when an asymmetric binary tree is used, a large sub-block may have a width/height that is three times the width/height of a small sub-block.

[0207] FIG. 26 illustrates an example of segmenting a block of N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2. As illustrated in FIG. 26, the 2N pixels by N pixels block is vertically divided into two N pixels by N pixels subblocks in step (1), and the left N pixels by N pixels block is vertically divided into two N/2 pixels by N pixels subblocks in step (2). In step (3), the current block may be segmented into three sub-blocks using the segmentation mode selected for the current N pixel by N pixel block (the left block in this example), as illustrated in (c) and (f) in FIG. 26. The dimensions of the three sub-blocks may vary. For example, of three subblocks, a large subblock may have a width/height that is twice the width/height of a small subblock. Additionally, for example, the selected segmentation mode may split the current block into two subblocks with different sizes (asymmetric binary tree), as illustrated in (a), (b), (d) and (e) of FIG. 26. For example, when an asymmetric binary tree is used, a large sub-block may have a width/height that is three times the width/height of a small sub-block.

[0208] FIG. 27 illustrates an example of segmenting a block of N pixels into N pixels using a segmentation mode selected when the second segmentation mode is not to be selected as illustrated in step (3) in Embodiment 2. As illustrated in FIG. 27, the N pixels by 2N pixels block is horizontally divided into two N pixels by N pixels subblocks in step (1), and the top N pixels by N pixels block is horizontally divided into two N pixels by N/2 pixels subblocks in step (2). In step (3), the current block may be segmented into three sub-blocks using the segmentation mode selected for the current N pixel by N pixel block (the bottom block in this example), as illustrated in (c) and (f) of FIG. 27. The dimensions of the three sub-blocks may vary. For example, of three subblocks, a large subblock may have a width/height that is twice the width/height of a small subblock. Additionally, for example, the selected segmentation mode may split the current block into two subblocks with different sizes (asymmetric binary tree), as illustrated in (a), (b), (d) and (e) of FIG. 27. For example, when an asymmetric binary tree is used, a large sub-block may have a width/height that is three times the width/height of a small sub-block.

[0209] FIG. 17 illustrates possible locations of a first parameter in a compressed video bit stream. As illustrated in FIG. 17, the first parameter may be in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The first parameter may indicate a method for segmenting a block into a plurality of subblocks. For example, the first parameter may include a flag to indicate whether the block should be segmented in a horizontal or vertical direction. The first parameter may also include a parameter for indicating whether the block should be segmented into two or more subblocks.

[0210] FIG. 18 illustrates possible locations of a second parameter in a compressed video bit stream. As illustrated in FIG. 18, the second parameter may be in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The second parameter may indicate a method for segmenting the block into a plurality of subblocks. For example, the second parameter may include a flag to indicate whether the block should be segmented in a horizontal or vertical direction. The second parameter may also include a parameter to indicate whether the block should be segmented into two or more subblocks. The second parameter comes after the first parameter in the bitstream, as illustrated in FIG. 19.

[0211] The first block and the second block are different blocks. The first block and the second block may be included in an identical frame. For example, the first block may be an upper adjacent block relative to the second block. In addition, for example, the first block may be a left adjacent block relative to the second block.

[0212] In step S2006, the second block is segmented into subblocks using the selected segmentation mode. In step S2007, the segmented blocks are decoded.

[0213] Decoder

Fig. 16 is a block diagram illustrating the structure of a video/image decoder according to Embodiment 2 or 3.

[0214] Video decoder 6000 is equipment for decoding an input coded bitstream on a block-by-block basis and outputting video/images. As illustrated in FIG. 16, video decoder 6000 includes an entropy decoder 6001, an inverse quantizer 6002, an inverse transformer 6003, a block memory 6004, a frame memory 6005, an intra prediction module 6006, an inter prediction module 6007, and a block segmentation determination module 6008.

[0215] The input coded bit stream is input to the entropy decoder 6001. After the input coded bit stream is input to the entropy decoder 6001, the entropy decoder 6001 decodes the input coded bit stream, outputs parameters to the block segmentation determination module 6008, and outputs the decoded values to inverse quantizer 6002.

[0216] The inverse quantizer 6002 inversely quantizes the decoded values and outputs the frequency coefficients to the inverse converter 6003. The inverse converter 6003 performs inverse frequency conversion on the frequency coefficients based on the block segmentation mode retrieved from the block segmentation determination unit 6008 to convert the frequency coefficients to sample values, and outputs the sample values to the adder. The block segmentation mode may be related to the block segmentation mode, the block segmentation type, or the block segmentation direction. The adder adds the sample values with the predicted video/image values output from the intra/inter prediction modules 6006, 6007, and outputs the summed values to a display and outputs the summed values to a block memory 6004 or a frame memory 6005 for further prediction. The block segmentation determination module 6008 collects block information from the block memory 6004 or the frame memory 6005 to retrieve the block segmentation mode using parameters decoded by the entropy decoder 6001. Use of the retrieved block segmentation mode should result in segmentation of the block into a plurality of sub-blocks. . In addition, the intra/inter-prediction module 6006, 6007 estimates the video/image area of the block to be decoded from the videos/images stored in the block memory 6004 or from the videos/images in the frame memory 6005 reconstructed using the segmentation mode into blocks retrieved from the block segmentation determination module 6008.

[0217] Embodiment 3

In the following, the encoding process and the decoding process according to Embodiment 3 will be described in detail with reference to FIGS. 13 and fig. 14. Next, the encoder and decoder according to Embodiment 3 will be described in detail with reference to FIG. 15 and fig. 16.

[0218] Encoding process

Fig. 13 illustrates a video encoding process according to Embodiment 3.

[0219] First, in step S3001, a first parameter for identifying, from a plurality of segmentation types, a segmentation type for segmenting the first block into subblocks is written into the bit stream.

[0220] Next, in step S3002, a second parameter indicating the segmentation direction is written to the bitstream. The second parameter comes after the first parameter in the bitstream. The type of segmentation, together with the direction of segmentation, can form the segmentation mode. The segmentation type specifies the number of subblocks and the ratio of segments to segment the block.

[0221] FIG. 29 illustrates examples of segmentation types and segmentation directions for segmenting a block of N pixels by N pixels in Embodiment 3. In fig. 29, (1), (2), (3) and (4) represent different segmentation types, (1a), (2a), (3a) and (4a) represent different segmentation modes from related segmentation types in the vertical direction segmentation, and (1b), (2b), (3b) and (4b) represent different segmentation modes from related segmentation types in the horizontal segmentation direction. As illustrated in FIG. 29, a block of N pixels by N pixels is segmented using the segmentation mode (1a) where it is segmented with a symmetric binary tree (ie, into two sub-blocks) in the vertical segmentation direction with a segment ratio of 1:1. A block of N pixels by N pixels is segmented using segmentation mode (1b), where it is segmented with a symmetric binary tree (ie, into two sub-blocks) in the horizontal segmentation direction with a 1:1 segment ratio. A block of N pixels by N pixels is segmented using segmentation mode (2a), where it is segmented with an asymmetric binary tree (ie, into two sub-blocks) in the vertical segmentation direction at a 1:3 segment ratio. A block of N pixels by N pixels is segmented using segmentation mode (2b), where it is segmented with an asymmetric binary tree (ie, into two sub-blocks) in the horizontal segmentation direction at a ratio of 1:3 segments. A block of N pixels by N pixels is segmented using segmentation mode (3a), where it is segmented with an asymmetric binary tree (ie, into two sub-blocks) in the vertical segmentation direction at a ratio of 3:1 segments. A block of N pixels by N pixels is segmented using the segmentation mode (3b), where it is segmented with an asymmetric binary tree (ie, into two sub-blocks) in the horizontal segmentation direction at a ratio of 3:1 segments. A block of N pixels by N pixels is segmented using the segmentation mode (4a), where it is segmented with a ternary tree (ie into three sub-blocks) in the vertical segmentation direction at a ratio of 1:2:1 segments. A block of N pixels by N pixels is segmented using the segmentation mode (4b), where it is segmented with a ternary tree (or into three sub-blocks) in the horizontal segmentation direction at a ratio of 1:2:1 segments.

[0222] FIG. 17 illustrates possible locations of a first parameter in a compressed video bit stream. As illustrated in FIG. 17, the first parameter may be in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The first parameter may indicate a method for segmenting a block into a plurality of subblocks. For example, the first parameter may include a flag to indicate whether the block should be segmented in a horizontal or vertical direction. The first parameter may also include a parameter for indicating whether the block should be segmented into two or more subblocks.

[0223] FIG. 18 illustrates possible locations of a second parameter in a compressed video bit stream. As illustrated in FIG. 18, the second parameter may be in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The second parameter may indicate a method for segmenting the block into a plurality of subblocks. For example, the second parameter may include a flag to indicate whether the block should be segmented in a horizontal or vertical direction. The second parameter may also include a parameter for indicating whether the block should be segmented into two or more subblocks. The second parameter comes after the first parameter in the bitstream, as illustrated in FIG. 19.

[0224] FIG. 30 illustrates the advantage of encoding the segmentation type before the segmentation direction compared to encoding the segmentation direction before the segmentation type. In this example, when the horizontal segmentation direction is disabled due to an unsupported size (16 pixels by 2 pixels), there is no need to encode the segmentation direction. The segmentation direction is defined as the vertical segmentation direction since the horizontal segmentation direction is deactivated in this example. Encoding the segmentation type before the segmentation direction reduces the number of encoding bits from the segmentation direction encoding compared to encoding the segmentation direction before the segmentation type.

[0225] Thus, it is possible to determine whether a block can be segmented in each of the horizontal direction and the vertical direction based on a predetermined condition for block segmentation allowed or not allowed. Thereafter, when it is determined that segmentation is possible in only one of the horizontal direction and the vertical direction, writing the segmentation direction to the bitstream may be skipped. In addition, when it is determined that this segmentation is not possible in both the horizontal direction and the vertical direction, in addition to the segmentation direction, writing the segmentation type into the bitstream may also be skipped.

[0226] The predetermined condition for enabled or disabled block segmentation is specified, for example, by the size (number of pixels) or the number of times the segmentation is performed. The condition for enabled or disabled block segmentation can be predefined in the standard specifications. In addition, the condition for enabled or disabled block segmentation may be included in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The condition for enabled or disabled block segmentation may be fixed for all blocks and may be dynamically switched according to a block property (eg, luma and chroma block) or frame property (eg, I-, P-, and B-frame).

[0227] In step S3003, the block is segmented into sub-blocks using the identified segmentation type and the indicated segmentation direction. In step S3004, the segmented blocks are encoded.

[0228] Encoder

Fig. 15 is a block diagram illustrating the structure of a video/image encoder according to Embodiment 2 or 3.

[0229] Video encoder 5000 is equipment for encoding an input video/image on a block-by-block basis so as to produce a coded output bitstream. As illustrated in FIG. 15, video encoder 5000 includes a converter 5001, a quantizer 5002, an inverse quantizer 5003, an inverse transformer 5004, a block memory 5005, a frame memory 5006, an intra prediction module 5007, an inter prediction module 5008, an entropy encoder 5009, and a segmentation determination module 5010. blocks.

[0230] The input video is input to the adder, and the summed value is output to converter 5001. Converter 5001 converts the summed values into frequency coefficients based on the block segmentation type and direction extracted from the block segmentation determination unit 5010, and outputs the frequency coefficients to quantizer 5002 The type and direction of block segmentation can be related to the block segmentation mode, the block segmentation type, or the direction of block segmentation. Quantizer 5002 quantizes the input quantized coefficients and outputs the quantized values to inverse quantizer 5003 and entropy encoder 5009.

[0231] The inverse quantizer 5003 inversely quantizes the quantized values output from the quantizer 5002 and outputs the frequency coefficients to the inverse converter 5004. The inverse converter 5004 performs inverse frequency conversion on the frequency coefficients based on the block segmentation type and direction extracted from the segmentation determination module 5010 into blocks so as to convert the frequency coefficients into sample values of the bit stream, and outputs the sample values to an adder.

[0232] The adder adds the sample bit stream values output from the inverse converter 5004 with the predicted video/image values output from the inter/intra prediction module 5007, 5008, and outputs the summed values to a block memory 5005 or a frame memory 5006 for additional forecasting. The block segmentation determining unit 5010 collects block information from the block memory 5005 or the frame memory 5006 to retrieve the block segmentation type and direction and parameters associated with the block segmentation type and direction. Using the extracted block segmentation type and direction should result in segmentation of the block into multiple subblocks. The inter/intra prediction module 5007, 5008 searches the videos/images stored in the block memory 5005 or from the videos/images in the frame memory 5006 reconstructed using the block segmentation type and direction retrieved from the block segmentation determination module 5010 blocks, and estimates the region of video/images that is, for example, most similar to the input video/images for prediction.

[0233] Entropy encoder 5009 encodes the quantized values output from quantizer 5002, encodes parameters from block segmentation determiner 5010, and outputs a bit stream.

[0234] Decoding process

Fig. 14 illustrates a video decoding process according to Embodiment 3.

[0235] First, in step S4001, a first parameter for identifying, from a plurality of segmentation types, a segmentation type for segmenting the first block into subblocks is obtained by parsing from the bit stream.

[0236] Next, in step S4002, a second parameter indicating the segmentation direction is obtained by parsing from the bitstream. The second parameter comes after the first parameter in the bitstream. The type of segmentation, together with the direction of segmentation, can form the segmentation mode. The segmentation type specifies the number of subblocks and the ratio of segments to segment the block.

[0237] FIG. 29 illustrates examples of segmentation types and segmentation directions for segmenting a block of N pixels by N pixels in Embodiment 3. In fig. 29, (1), (2), (3) and (4) represent different segmentation types, (1a), (2a), (3a) and (4a) represent different segmentation modes from related segmentation types in the vertical direction segmentation, and (1b), (2b), (3b) and (4b) represent different segmentation modes from related segmentation types in the horizontal segmentation direction. As illustrated in FIG. 29, a block of N pixels by N pixels is segmented using the segmentation mode (1a) where it is segmented with a symmetric binary tree (ie, into two sub-blocks) in the vertical segmentation direction with a segment ratio of 1:1. A block of N pixels by N pixels is segmented using segmentation mode (1b), where it is segmented with a symmetric binary tree (ie, into two sub-blocks) in the horizontal segmentation direction with a 1:1 segment ratio. A block of N pixels by N pixels is segmented using segmentation mode (2a), where it is segmented with an asymmetric binary tree (ie, into two sub-blocks) in the vertical segmentation direction at a 1:3 segment ratio. A block of N pixels by N pixels is segmented using segmentation mode (2b), where it is segmented with an asymmetric binary tree (ie, into two sub-blocks) in the horizontal segmentation direction at a ratio of 1:3 segments. A block of N pixels by N pixels is segmented using segmentation mode (3a), where it is segmented with an asymmetric binary tree (ie, into two sub-blocks) in the vertical segmentation direction at a ratio of 3:1 segments. A block of N pixels by N pixels is segmented using the segmentation mode (3b), where it is segmented with an asymmetric binary tree (ie, into two sub-blocks) in the horizontal segmentation direction at a ratio of 3:1 segments. A block of N pixels by N pixels is segmented using the segmentation mode (4a), where it is segmented with a ternary tree (ie into three sub-blocks) in the vertical segmentation direction at a ratio of 1:2:1 segments. A block of N pixels by N pixels is segmented using the segmentation mode (4b), where it is segmented with a ternary tree (or into three sub-blocks) in the horizontal segmentation direction at a ratio of 1:2:1 segments.

[0238] FIG. 17 illustrates possible locations of a first parameter in a compressed video bit stream. As illustrated in FIG. 17, the first parameter may be in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The first parameter may indicate a method for segmenting a block into a plurality of subblocks. For example, the first parameter may include an identifier of the above segmentation type. For example, the first parameter may include a flag to indicate whether the block should be segmented in a horizontal or vertical direction. The first parameter may also include a parameter for indicating whether the block should be segmented into two or more subblocks.

[0239] FIG. 18 illustrates possible locations of a second parameter in a compressed video bit stream. As illustrated in FIG. 18, the second parameter may be in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The second parameter may indicate a method for segmenting the block into a plurality of subblocks. For example, the second parameter may include a flag to indicate whether the block should be segmented in a horizontal or vertical direction. In particular, the second parameter may include a parameter for indicating the direction of segmentation. The second parameter may also include a parameter to indicate whether the block should be segmented into two or more subblocks. The second parameter comes after the first parameter in the bitstream, as illustrated in FIG. 19.

[0240] FIG. 30 illustrates the advantage of encoding the segmentation type before the segmentation direction compared to encoding the segmentation direction before the segmentation type. In this example, when the horizontal segmentation direction is disabled due to an unsupported size (16 pixels by 2 pixels), there is no need to encode the segmentation direction. The segmentation direction is defined as the vertical segmentation direction since the horizontal segmentation direction is deactivated in this example. Encoding the segmentation type before the segmentation direction reduces the number of encoding bits from the segmentation direction encoding compared to encoding the segmentation direction before the segmentation type.

[0241] Thus, it is possible to determine whether a block can be segmented in each of the horizontal direction and the vertical direction based on a predetermined condition for block segmentation allowed or not allowed. Thereafter, when it is determined that segmentation is possible in only one of the horizontal direction and the vertical direction, parsing the segmentation direction from the bitstream may be skipped. In addition, when it is determined that this segmentation is not possible in both the horizontal direction and the vertical direction, in addition to the segmentation direction, parsing the segmentation type from the bitstream may also be skipped.

[0242] The predetermined condition for enabled or disabled block segmentation is specified, for example, by the size (number of pixels) or the number of times the segmentation is performed. This condition for enabled or disabled block segmentation can be predefined in the standard specifications. In addition, the condition for enabled or disabled block segmentation may be included in a video parameter set, a sequence parameter set, a frame parameter set, a slice header, or a coding tree unit. The condition for enabled or disabled block segmentation may be fixed for all blocks and may be dynamically switched according to a block property (eg, luma and chroma block) or frame property (eg, I-, P-, and B-frame)

[0243] In step S4003, the block is segmented into sub-blocks using the identified segmentation type and the indicated segmentation direction. In step S4004, the segmented blocks are decoded.

[0244] Decoder

Fig. 16 is a block diagram illustrating the structure of a video/image decoder according to Embodiment 2 or 3.

[0245] Video decoder 6000 is equipment for decoding an input coded bitstream on a block-by-block basis and outputting video/images. As illustrated in FIG. 16, video decoder 6000 includes an entropy decoder 6001, an inverse quantizer 6002, an inverse transformer 6003, a block memory 6004, a frame memory 6005, an intra prediction module 6006, an inter prediction module 6007, and a block segmentation determination module 6008.

[0246] The input coded bit stream is input to the entropy decoder 6001. After the input coded bit stream is input to the entropy decoder 6001, the entropy decoder 6001 decodes the input coded bit stream, outputs parameters to the block segmentation determiner 6008, and outputs the decoded values to inverse quantizer 6002.

[0247] An inverse quantizer 6002 inversely quantizes the decoded values and outputs the frequency coefficients to an inverse converter 6003. The inverse converter 6003 performs inverse frequency conversion on the frequency coefficients based on the block segmentation type and direction retrieved from the block segmentation determination module 6008 to convert the frequency coefficients into sample values, and outputs the sample values to the adder. The type and direction of block segmentation may be related to the block segmentation mode, the block segmentation type, or the direction of block segmentation. The adder adds the sample values with the predicted video/image values output from the intra/inter prediction modules 6006, 6007, and outputs the summed values to a display and outputs the summed values to a block memory 6004 or a frame memory 6005 for further prediction. Block segmentation determination module 6008 collects block information from block memory 6004 or frame memory 6005 to extract block segmentation type and direction using parameters decoded by entropy decoder 6001. Using the extracted block segmentation type and direction should result in segmentation block into many subblocks. In addition, the intra/inter-prediction module 6006, 6007 estimates the video/image area of the block to be decoded from the video/images stored in the block memory 6004 or from the videos/images in the frame memory 6005 reconstructed using the type and the block segmentation direction extracted from the block segmentation determining unit 6008.

[0248] Embodiment 4

As described in each of the above-described embodiments, each functional block can typically be implemented, for example, as an MPU and a memory device. In addition, the processes performed by each of the functional blocks are typically implemented by a program execution unit such as a processor reading and executing software (program) recorded on a recording medium such as a ROM. The software may be distributed, for example, via download, and may be recorded on a recording medium such as a semiconductor storage device and distributed. It should be noted that each functional block, of course, can also be implemented as hardware (specialized circuitry).

[0249] In addition, the processing described in each of the embodiments may be implemented through integrated processing using a single piece of equipment (system), and alternatively may be implemented through decentralized processing using multiple pieces of equipment. Moreover, the processor that executes the above-described program may be a single processor or a plurality of processors. In other words, integrated processing may be performed, and alternatively, decentralized processing may be performed.

[0250] Embodiments of the present disclosure are not limited to the above exemplary embodiments; Various modifications may be made to the exemplary embodiments, the effects of which are also included within the scope of the embodiments of the present disclosure.

[0251] Next, examples of applications of the moving frame encoding method (image encoding method) and the moving frame decoding method (image decoding method) described in each of the above embodiments and a system that uses them will be described. The system is characterized as including an image encoder that uses an image encoding method, an image decoder that uses an image decoding method, and an image encoder/decoder that includes both an image encoder and an image decoder. Other configurations included in the system can be modified for each individual case.

[0252] Examples of use

Fig. 33 illustrates a general configuration of a content provision system ex100 for implementing a content distribution service. The area in which the communication service is provided is divided into cells of required sizes, and base stations ex106, ex107, ex108, ex109 and ex110, which are fixed wireless stations, are located in the corresponding cells.

[0253] In the content delivery system ex100, devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 through an Internet service provider ex102 or a communication network ex104 and base stations ex106- ex110. The content delivery system ex100 can combine and connect any combination of the above elements. The devices can communicate directly or indirectly through the telephone network or short-range communications rather than through the ex106-ex110 base stations, which are fixed wireless stations. In addition, the streaming server ex103 connects with devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114 and a smartphone ex115, for example, via the Internet ex101. The ex103 streaming server also connects, for example, to a terminal on the ex117 aircraft public access point via the ex116 satellite.

[0254] It should be noted that instead of base stations ex106-ex110, wireless access points or public access points may be used. The streaming server ex103 can connect to the communications network ex104 directly rather than via the Internet ex101 or Internet service provider ex102, and can connect to the aircraft ex117 directly rather than via a satellite ex116.

[0255] The camera ex113 is a device capable of capturing still images and video, such as a digital camera. The ex115 smartphone is a smartphone, cellular phone, or personal mobile phone system (PHS) phone that can operate according to mobile phone system standards for typical 2G, 3G, 3.9G and 4G systems, as well as 5G systems next generation.

[0256] Home appliance ex118, for example, is a refrigerator or device included in a home fuel cell cogeneration system.

[0257] In the content delivery system ex100, a terminal including an image and/or video capture function allows, for example, live streaming through a connection to a streaming server ex103, for example, through a base station ex106. When live streaming, a terminal (eg, a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or an airplane ex117) performs the encoding processing described in the above-described embodiments on the still image content or video content captured by the user via the terminal , multiplexes video data obtained through encoding and audio data obtained through encoding audio corresponding to the video, and transmits the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to one aspect of the present disclosure.

[0258] The streaming server ex103 streams the transmitted content data to clients that request the stream. Examples of the client include a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and airplane terminals ex117 that are capable of decoding the above-described encoded data. Devices that receive streaming data decode and play back the received data. In other words, the devices function as an image decoder according to one aspect of the present disclosure.

[0259] Decentralized processing

The ex103 streaming server can be implemented as multiple servers or computers that share tasks such as processing, recording, and streaming data. For example, the ex103 streaming server can be implemented as a content delivery network (CDN) that streams content over a network connecting multiple edge servers located around the world. In a CDN, an edge server physically near the client is dynamically assigned to the client. Content is cached and streamed to the edge server to reduce loading times. In the event of a specific error or connection change, for example due to a spike in traffic, data can be streamed reliably at high speeds since affected parts of the network can be avoided, for example by dividing processing among multiple edge servers or switching streaming loads to another edge server and continue streaming.

[0260] Decentralization is not limited to simply sharing processing for streaming; Encoding of the captured data can be shared between and performed via terminals, on the server side, or both. In one example, in typical encoding, processing is performed in two loops. The first loop serves to detect how complex the image is on a frame-by-frame or scene-by-scene basis, or to detect encoding load. The second circuit provides processing that maintains image quality and improves encoding efficiency. For example, it is possible to reduce the processing load of the terminals and improve the quality and efficiency of content encoding by causing the terminals to perform a first encoding loop and causing the server side that receives the content to perform a second encoding loop. In such a case, when a decoding request is received, the encoded data resulting from the first loop performed by one terminal can be received and reproduced at the other terminal in approximately real time. This allows for smooth, real-time streaming.

[0261] In another example, camera ex113 and the like. extracts the feature count from the image, compresses the data associated with the feature count as metadata, and transmits the compressed metadata to the server. For example, the server determines the significance of an object based on the number of features and changes the quantization precision accordingly to perform compression appropriate for the semantic meaning of the image. These feature counts are particularly effective in improving the accuracy and efficiency of motion vector prediction during a second pass of compression performed by the server. In addition, coding that has a relatively low processing load, such as variable length coding (VLC), can be processed by the terminal, and coding that has a relatively high processing load, such as context-adaptive binary arithmetic coding (CABAC), can be processed via the server.

[0262] In yet another example, cases arise in which multiple videos of an approximately identical scene are captured through multiple terminals, for example, in a stadium, a shopping mall, or a factory. In such a case, for example, encoding can be decentralized by dividing processing tasks among multiple terminals that capture video and, if necessary, other terminals that do not capture video, and a server on a per-unit basis. The units, for example, may be groups of pictures (GOPs), frames, or tiles resulting from dividing a frame. This allows you to reduce loading times and achieve streaming that is closer to real time.

[0263] In addition, since the videos have approximately the same scene, control and/or instruction can be performed by the server such that videos captured through the terminals can be cross-referenced. In addition, the server may receive encoded data from the terminals, change the reference relationship between data elements, or adjust or replace the frames themselves, and then perform encoding. This allows you to generate a stream with increased quality and efficiency for individual data elements.

[0264] In addition, the server may stream video data after performing transcoding to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP and can convert H.264 to H.265.

[0265] Thus, encoding may be performed by a terminal or one or more servers. Accordingly, although the device that performs the encoding is referred to as a “server” or “terminal” in the following description, some or all of the processes performed by the server may be performed by the terminal, and similarly, some or all of the processes performed by the terminal may be performed by the server . This also applies to decoding processes.

[0266] 3D Multi-View Mode

In recent years, there has been an increasing use of images or videos combined from images or videos of different scenes captured in parallel, or an identical scene captured from different angles through multiple terminals, such as the ex113 camera and/or the ex115 smartphone. Video captured by the terminals is combined, for example, based on a separately obtained relative positional relationship between terminals or regions in the video having matching feature points.

[0267] In addition to encoding two-dimensional moving frames, the server may encode the still image based on scene analysis of the moving frame automatically or at a time specified by the user, and transmit the encoded still image to the receiving terminal. In addition, when the server can obtain the relative positional relationship between video capture terminals, in addition to two-dimensional moving frames, the server can generate three-dimensional scene geometry based on video of an identical scene captured from different angles. It should be noted that the server may separately encode three-dimensional data generated from, for example, a point cloud, and may, based on the result of recognizing or tracking a person or object using the three-dimensional data, select or reconstruct and generate videos to be transmitted to the receiving terminal. from videos captured through multiple terminals.

[0268] This allows the user to enjoy the scene by freely selecting videos corresponding to the video capture terminals, and allows the user to enjoy content obtained by extracting 3D data recovered from a plurality of images or videos, videos from a selected viewpoint. Moreover, similar to video, audio can be recorded from relatively different angles, and the server can multiplex, with video, audio from a particular angle or space according to the video and transmit the result.

[0269] In recent years, content that is a composite of the real world and the virtual world, such as virtual reality (VR) and augmented reality (AR) style content, has also become popular. In the case of VR images, the server may create images from the left and right eye viewpoints and perform encoding that allows a link between the two viewpoint images, such as multi-view encoding (MVC), and alternatively, may encode the images as separate streams without a link . When images are decoded as separate streams, the streams can be synchronized during playback in such a way as to recreate the virtual 3D space according to the user's point of view.

[0270] In the case of AR images, the server overlays virtual object information existing in virtual space with camera information representing real world space based on three-dimensional position or movement from the user's perspective. The decoder can receive or store virtual object information and 3D data, generate 2D images based on movement from the user's point of view, and then generate overlay data by transparently connecting the images. Alternatively, the decoder may transmit, to the server, motion from the user's perspective in addition to requesting virtual object information, and the server may generate overlay data based on 3D data stored in the server according to the received motion, and encode and stream the generated superimposed data into the decoder. It should be noted that the overlay data includes, in addition to RGB values, an α value indicating transparency, and the server sets the α value for sections other than the object formed from the 3D data, for example, to 0, and may encode to the time when these sections are transparent. Alternatively, the server may set the background as a predefined RGB value, such as a chroma key, and generate data in which areas other than the object are set as the background.

[0271] Decoding of similarly streamed data may be performed by the client (ie, terminals), on the server side, or shared between them. In one example, one terminal may transmit a receive request to a server, the requested content may be received and decoded by the other terminal, and the decoded signal may be transmitted to a device having a display. It is possible to reproduce data with high image quality by decentralizing processing and appropriately selecting content regardless of the processing characteristics of the communication terminal itself. In yet another example, while a television, for example, receives image data that is large in size, an area of a frame, such as a tile obtained by dividing the frame, can be decoded and displayed on the personal terminal or terminals of the viewer or viewers of the television. This allows viewers to share a view with large frames, as well as each viewer to check their assigned area or explore an area in more detail with a close-up view.

[0272] In the future, both indoor and outdoor, in situations in which multiple wireless connections are possible over short, medium and long distances, it should presumably be possible to transparently receive content even when switching to data appropriate for the current connection, using a streaming system standard such as MPEG DASH. Consequently, the user can switch between real-time data with a free choice of decoder or display equipment, including not only his own terminal, but also, for example, indoor or outdoor displays. In addition, for example, based on information regarding the user's position, decoding may be performed when switching which terminal processes decoding and which terminal processes content display. This allows, while on the route to a destination, to display, on the wall of a nearby building in which a device capable of displaying content is built, or on a part of the ground, map information in motion. In addition, it is also possible to switch the bit rate of the received data based on the reachability of the encoded data over the network, for example, when the encoded data is cached on a server readily accessible from the receiving terminal, or when the encoded data is copied to an edge server in a content delivery service.

[0273] Scalable Coding

In the following, content switching will be described with reference to the scalable stream illustrated in FIG. 34, which is compression encoded through an implementation of the moving frame encoding method described in the above-described embodiments. The server may be configured in which content is switched using temporal and/or spatial stream scalability, which is achieved through partitioning and layer encoding, as illustrated in FIG. 34. It should be noted that multiple separate streams may be provided that have identical content but different quality. In other words, by determining down to which layer to decode based on internal factors such as decoder-side processing characteristics and external factors such as communication link bandwidth, the decoder side can freely switch between low-resolution content and high-resolution content. decoding resolutions. For example, in a case in which a user wants to continue watching a video at home on a device such as an Internet-connected TV that he previously watched on the ex115 smartphone while on the move, the device can simply decode the identical stream down to a different layer, reducing the load on server side.

[0274] Moreover, in addition to the configuration described above, in which scalability is achieved as a result of encoding frames per layer, and the enhancement layer is located above the base layer, the enhancement layer may include metadata, for example, based on statistical information regarding images, and the decoder side can generate high image quality content by performing super-resolution imaging of a frame in the base layer based on the metadata. Super-resolution imaging may represent an improvement in the SN ratio while maintaining resolution and/or increasing resolution. Metadata includes information to identify the linear or nonlinear filtering coefficient used in super-resolution processing, or information identifying the value of a parameter in filter, machine learning, or least squares processing used in super-resolution processing.

[0275] Alternatively, a configuration in which a frame is, for example, divided into tiles according to the semantic meaning of, for example, an object in an image, and at the decoder side, only a partial region is decoded by selecting a tile to be decoded, is also acceptable. In addition, by storing the attribute relative to the object (person, car, ball, etc.) and the position of the object in the video (coordinates in identical images) as metadata, the decoder side can identify the position of the desired object based on the metadata and determine which the tile or tiles include this object. For example, as illustrated in FIG. 35, metadata is stored using a data storage structure other than pixel data, such as an SEI message in HEVC. This metadata indicates, for example, the position, size or color of the underlying object.

[0276] In addition, metadata may be stored in multiple frame units, such as stream, sequence, or random access units. Because of this, the decoder side can obtain, for example, the time at which a particular person appears in the video, and by fitting it to frame unit information, can identify the frame in which the object is present and the position of the object in the frame.

[0277] Web page optimization

Fig. 36 illustrates an example of a web page display screen on, for example, a computer ex111. Fig. 37 illustrates an example of a web page display screen on, for example, an ex115 smartphone. As illustrated in FIG. 36 and fig. 37, a web page may include a plurality of image links that are links to image content, and the appearance of the web page differs depending on the device used to view the web page. When multiple image links are viewable on the screen, until the user explicitly selects the image link, either until the image link is approximated in the center of the screen or the entire image link fits into the screen, display hardware (decoder) displays, as image links, still images included in the content, or I-frames, displays video, such as an animated GIF using multiple still images or I-frames, for example, or receives only a base layer, and decodes and displays video.

[0278] When the image reference is selected by the user, the display equipment decodes the highest priority provision for the base layer. It should be noted that if there is information in the HTML code of the web page indicating that the content is scalable, the display equipment can decode down to the enhancement layer. In addition, to ensure real-time playback, before a selection is made, or when bandwidth is severely limited, the display equipment may reduce the delay between the time at which the initial frame is decoded and the time at which the decoded frame is displayed (i.e. i.e., the delay from the start of decoding the content until the content is displayed) by decoding and displaying only direct reference frames (I-frame, P-frame, forward reference B-frame). In addition, the display equipment may intentionally ignore the reference relationship between frames and approximately decode all B and P frames as direct reference frames and then perform normal decoding as the number of frames received in time increases.

[0279] Autonomous driving

When transmitting and receiving, in still image data or video data, such two- or three-dimensional map information for autonomous or assisted driving, the receiving terminal may receive, in addition to image data belonging to one or more layers, information such as regarding weather or roadworks as metadata, and associate the metadata with image data when decoding. It should be noted that the metadata may be assigned on a per-layer basis and, alternatively, may simply be multiplexed with the image data.

[0280] In such a case, since a car, unmanned vehicle, aircraft, etc. including the receiving terminal is mobile, the receiving terminal can transparently receive and decode when switching between base stations among the base stations ex106 to ex110 by transmission information indicating the position of the receiving terminal when requesting reception. In addition, according to the selection made by the user, the user's situation, or the connection bandwidth, the receiving terminal can dynamically select the extent to which metadata is received or the extent to which, for example, map information is updated.

[0281] Because of this, in the content delivery system ex100, the client can receive, decode and reproduce, in real time, encoded information transmitted by the user.

[0282] Streaming content from people

In the content delivery system ex100, in addition to long-lasting content with high image quality distributed by a video distribution entity, unicast or multicast streaming of short content with low image quality from a person is also possible. In addition, the popularity of such content from people is likely to increase even more. The server may first perform editing processing on the content before encoding processing to refine content from people. This can be achieved, for example, using the following configuration.

[0283] In real time, when video or image content is captured or after the content is captured and accumulated, the server performs recognition processing based on the raw or encoded data, such as capture error handling, scene retrieval processing, semantic meaning analysis, and/or or object detection processing. Thereafter, based on the result of the recognition processing, the server (either directed or automatically) edits the content, examples of which include: correction such as correction of focus and/or motion blur; Removing scenes with low priority, such as scenes that have low brightness compared to other frames or are out of focus; adjusting the edges of objects; and regulation of color tones. The server encodes the edited data based on the result of the edit. It is known that overly long videos tend to have fewer views. Accordingly, in order to maintain content within a specific duration that scales with the duration of the source video, the server can, in addition to the low priority scenes described above, automatically cut scenes with minor movement based on the result of image processing. Alternatively, the server can generate and encode a video digest based on the result of the analysis of the semantic meaning of the scene.

[0284] It should be noted that cases arise in which content from people may include content that violates copyright, moral law, book rights, etc. Such a case may lead to an undesirable situation for the creator, such as when content is shared beyond the scope intended by the creator. Accordingly, before encoding, the server, for example, can edit the images so as to, for example, blur the faces of people on the periphery of the screen or blur the inside of a house. In addition, the server may be configured to recognize faces of people other than the registered person in images to be encoded, and when such faces appear in the image, for example, apply a mosaic filter to the person's face. Alternatively, as pre-processing or post-processing for encoding, the user may specify, for copyright reasons, an area of the image including a person or a background area to be processed, and the server may process the specified area, for example, by replacing the area with another image or blur area. If the area includes a person, the person may be tracked in a moving frame, the head region may be replaced by another image as the person moves.

[0285] In addition, since there is a demand for real-time viewing of human-generated content, which often has a small data size, the decoder first takes the base layer as the highest priority and performs decoding and playback, although this may differ depending on the bandwidth . When the content is played back two or more times, for example, when the decoder receives the enhancement layer during decoding and playback of the base layer and loops playback, the decoder can play back the high image quality video including the enhancement layer. If a stream is encoded using such scalable encoding, the video may be low quality in the unselected state or at the beginning of the video, but it may offer an experience in which the quality of the images of the stream is gradually increased in an intelligent manner. This is not limited to just scalable coding; An identical experience can be offered by configuring one stream of the low quality stream played back the first time and a second stream encoded using the first stream as a reference.

[0286] Other use cases

Encoding and decoding may be performed by the LSI ex500, which is typically included in each terminal. The LSI ex500 can be configured with a single chip or multiple chips. Software for encoding and decoding moving frames may be integrated into some type of recording medium (such as a CD-ROM, floppy disk, or hard disk) that is readable, for example, by a computer ex111, and encoding and decoding may be performed using the software. In addition, when the ex114 smartphone is equipped with a camera, video data captured by the camera can be transmitted. In this case, video data is encoded using the LSI ex500 included in the ex115 smartphone.

[0287] It should be noted that the LSI ex500 may be configured to download and activate an application. In such a case, the terminal first determines whether it is compatible with the scheme used to encode the content or whether it is capable of performing a particular service. When the terminal is not compatible with the content encoding scheme, or when the terminal does not allow execution of a particular service, the terminal first downloads the codec or application software, and then receives and plays the content.

[0288] In addition to the example of a content provision system ex100 that uses the Internet ex101, at least a moving frame encoder (picture encoder) or a moving frame decoder (picture decoder) described in the above-described embodiments may be implemented in a digital broadcast system. Identical encoding processing and decoding processing can be applied to transmit and receive broadcast radio waves overlaid with multiplexed audio and video data using, for example, a satellite, even if it accommodates multicast transmission, whereas unicast transmission is simpler with the ex100 provisioning system content.

[0289] Hardware configuration

Fig. 38 illustrates the ex115 smartphone. Fig. 39 illustrates an ex115 smartphone configuration example. The ex115 smartphone includes an ex450 antenna for transmitting and receiving radio waves to/from the ex110 base station, an ex465 camera that allows video and still image capture, and an ex458 display that displays decoded data such as video captured through the ex465 camera and videos received via antenna ex450. The ex115 smartphone further includes an ex466 user interface such as a touch panel, an ex457 audio output module such as a speaker for outputting speech or other audio, an ex456 audio input module such as a microphone for audio input, an ex467 storage device capable of storing decoded data such as captured video or still images, recorded audio, received video or still images and mail messages, as well as decoded data, and the ex464 socket, which is an interface for the ex468 SIM card to authorize access to the network and various data. It should be noted that an external storage device can be used instead of the ex467 storage device.

[0290] In addition, the main controller ex460, which comprehensively controls the display ex458 and the user interface ex466, power supply circuit ex461, user interface input controller ex462, video signal processor ex455, camera interface ex463, display controller ex459, modulator/demodulator ex452, multiplexer /ex453 demultiplexer, ex454 audio processor, ex464 socket and ex467 memory are connected via ex470 bus.

[0291] When the user turns on the power button of the power supply circuit ex461, the smartphone ex115 is turned on to an operational state by supplying power to each component from the battery pack.

[0292] The smartphone ex115 performs processing, for example, for calls and data transfers, based on control performed by the main controller ex460, which includes a CPU, ROM and RAM. When making calls, the audio signal recorded by the audio input module ex456 is converted to a digital audio signal by the audio signal processor ex454, and spread spectrum processing is applied to it by the ex452 modulator/demodulator and digital-to-analog and frequency conversion processing is applied by the ex451 transceiver. and then it is transmitted through the ex450 antenna. The received data is amplified, frequency-converted and A/D converted, despread spectrum processed by the ex452 modulator/demodulator, converted to an analog audio signal by the ex454 audio processor, and then output from the ex457 audio output module. In the data transmission mode, text data, still image data or video data is transmitted by the main controller ex460 through the input controller ex462 from the user interface as a result of the operation of, for example, the main body user interface ex466, and similar transmission and reception processing is performed. In the data transmission mode, when sending video, still image or video and audio, the video signal processor ex455 compressively encodes, through the motion frame encoding method described in the above-described embodiments, the video signal stored in the storage device ex467 or the video signal input from the camera ex465 , and transmits the encoded video data to the ex453 multiplexer/demultiplexer. In addition, the audio signal processor ex454 encodes the audio signal recorded by the audio input module ex456, while the camera ex465 captures, for example, video or a still image, and transmits the encoded audio data to the multiplexer/demultiplexer ex453. The ex453 multiplexer/demultiplexer multiplexes encoded video data and encoded audio data using a predefined circuit, modulates and converts the data using the ex452 modulator/demodulator (modulator/demodulator circuit) and ex451 transceiver, and transmits the result through the ex450 antenna.

[0293] When a video attached in an email message or chat, or a video containing a link from a web page, for example, is received to decode multiplexed data received through antenna ex450, the mux/demultiplexer ex453 demultiplexes the multiplexed data to separate the multiplexed data into a video bitstream and an audio bitstream, provides encoded video data to the video processor ex455 via the ex470 synchronous bus, and provides encoded audio data to the audio processor ex454 via the ex470 synchronous bus. The video signal processor ex455 decodes the video signal using a moving frame decoding method corresponding to the moving frame encoding method described in the above-described embodiments, and the video or still image included in the associated moving frame file is displayed on the display ex458 through the display controller ex459. In addition, the audio signal processor ex454 decodes the audio signal and outputs audio from the audio output module ex457. It should be noted that as live streaming becomes more and more popular, there are cases in which audio playback may not be socially feasible depending on the user's environment. Accordingly, as an initial value, a configuration in which only video data is reproduced is preferred, i.e. No audio is played. Audio can only sync and play when input is received, such as when the user clicks on video data.

[0294] Although the smartphone ex115 is used in the above example, three implementations are possible: a transceiver terminal including both an encoder and a decoder; a transmission terminal including only an encoder; and a receiving terminal including only the decoder. Additionally, in the description of a digital broadcast system, an example is given in which multiplexed data obtained as a result of multiplexing video data, such as audio data, is received or transmitted, but the multiplexed data may be video data multiplexed with data other than audio data, such as text data. video related data. In addition, video data may be received or transmitted directly rather than multiplexed data.

[0295] Although the ex460 main controller, including a CPU, is described as controlling encoding or decoding processes, terminals often include GPUs. Accordingly, it is acceptable to have a configuration in which a large area is processed at once by using performance characteristics of the GPU through a memory shared by the CPU and the GPU, or through a memory including an address that is controlled so as to enable a wide use via CPU and GPU. This reduces encoding time, maintains the real-time nature of the stream, and reduces latency. In particular, processing related to motion estimation, deblocking filtering, sampled adaptive offset (SAO), and transform/quantization can be efficiently performed by the GPU instead of the CPU, for example, in units of frames, simultaneously.

[0296] An encoder according to an embodiment of the present disclosure may be an encoder that encodes a frame, and includes a processor and a storage device. The processor may include: a block segmentation determination module that segments a frame into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, the frame being read from a memory devices; and an encoding module that encodes the plurality of blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. When the total number of segments of the first block segmentation mode is three, the second block is a center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first block segmentation mode , the second block segmentation mode may include only the block segmentation mode indicating that the total number of segments is three.

[0297] A parameter for identifying a second block segmentation mode in an encoder according to an embodiment of the present disclosure may include a first flag that indicates whether the block is to be segmented horizontally or vertically, and need not necessarily include a second flag indicating the total number segments into which the block should be segmented.

[0298] An encoder according to an embodiment of the present disclosure may be an encoder that encodes a frame, and includes a processor and a storage device. The processor may include: a block segmentation determination module that segments a frame into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, the frame being read from a memory devices; and an encoding module that encodes the plurality of blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. When the total number of segments of the first block segmentation mode is three, the second block is a center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first block segmentation mode , the second block segmentation mode, indicating that the total number of segments is two, need not necessarily be used.

[0299] An encoder according to an embodiment of the present disclosure may be an encoder that encodes a frame, and includes: a processor and a storage device. The processor may include: a block segmentation determination module that segments a frame into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, the frame being read from a memory devices; and an encoding module that encodes the plurality of blocks. The set of block segmentation modes may include a first segmentation mode and a second block segmentation mode, each of which specifies a direction of segmentation and a total number of segments. The block segmentation determining module may restrict the use of a second block segmentation mode, which indicates that the number of segments is two.

[0300] A parameter for identifying a second block segmentation mode in an encoder according to an embodiment of the present disclosure may include a first flag that indicates whether the block should be segmented horizontally or vertically, and a second flag indicating whether the block should be segmented into two or more.

[0301] A parameter in an encoder according to an embodiment of the present disclosure may be provided in data of a series of consecutive macroblocks.

[0302] An encoder according to an embodiment of the present disclosure may be an encoder that encodes a frame, and includes: a processor; and a storage device. The processor may include: a block segmentation determining module that segments a frame into a set of blocks including a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type , wherein the frame is read from the storage device; and an encoding module that encodes the plurality of blocks. When the first set of blocks obtained by using the first set of block segmentation modes and the second set of blocks obtained by using the second set of block segmentation modes are identical, the block segmentation determining module may perform segmentation using only one of the first block segmentation mode. blocks or the second mode of segmentation into blocks.

[0303] A block segmentation determination module in an encoder according to an embodiment of the present disclosure may, based on the first code amount of the first block segmentation mode set and the second code amount of the second block segmentation mode set, perform segmentation using the smaller of the set of block segmentation modes. a first volume of codes and a second volume of codes.

[0304] A block segmentation determining module in an encoder according to an embodiment of the present disclosure may, based on a first amount of code of a first set of block segmentation modes and a second amount of code of a second set of block segmentation modes, perform segmentation using a set of block segmentation modes that appears first in a predetermined order from the first set of block segmentation modes and the second set of block segmentation modes when the first code amount and the second code amount are equal.

[0305] The decoder according to an embodiment of the present disclosure may be a decoder that decodes an encoded signal, and includes a processor and a storage device. The processor may include a block segmentation determination module that segments an encoded signal into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein the encoded signal is read from storage device; and a decoding module that decodes the plurality of blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. When the total number of segments of the first block segmentation mode is three, the second block is a center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first block segmentation mode , the second block segmentation mode may include only the block segmentation mode indicating that the total number of segments is three.

[0306] A parameter for identifying a second block segmentation mode in a decoder according to an embodiment of the present disclosure may include a first flag that indicates whether the block is to be segmented horizontally or vertically, and need not necessarily include a second flag indicating the total number segments into which the block should be segmented.

[0307] The decoder according to an embodiment of the present disclosure may be a decoder that decodes an encoded signal, and includes a processor and a storage device. The processor may include a block segmentation determination module that segments an encoded signal into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein the encoded signal is read from storage device; and a decoding module that decodes the plurality of blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. When the total number of segments of the first block segmentation mode is three, the second block is a center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first block segmentation mode , the second block segmentation mode, indicating that the total number of segments is two, need not necessarily be used.

[0308] The decoder according to an embodiment of the present disclosure may be a decoder that decodes an encoded signal, and includes a processor and a storage device. The processor may include: a block segmentation determining module that segments an encoded signal into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein the encoded signal is read from a storage device; and a decoding module that decodes the plurality of blocks. The set of block segmentation modes may include a first block segmentation mode and a second block segmentation mode, each of which specifies a direction of segmentation and a number of segments. The block segmentation determining module may restrict the use of a second block segmentation mode, which indicates that the number of segments is two.

[0309] A parameter for identifying a second block segmentation mode in a decoder according to an embodiment of the present disclosure may include a first flag that indicates whether the block should be segmented horizontally or vertically, and a second flag indicating whether the block should be segmented into two or more.

[0310] A parameter in a decoder according to an embodiment of the present disclosure may be provided in data of a series of consecutive macroblocks.

[0311] The decoder according to an embodiment of the present disclosure may be a decoder that decodes an encoded signal, and includes: a processor; and a storage device. The processor may include: a block segmentation determining module that segments the encoded signal into a set of blocks including a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type segmentation, wherein the encoded signal is read from the storage device; and a decoding module that decodes the plurality of blocks. When the first set of blocks obtained by using the first set of block segmentation modes and the second set of blocks obtained by using the second set of block segmentation modes are identical, the block segmentation determining module may perform segmentation using only one of the first block segmentation mode. blocks or the second mode of segmentation into blocks.

[0312] A block segmentation determination module in a decoder according to an embodiment of the present disclosure may, based on the first code amount of the first block segmentation mode set and the second code amount of the second block segmentation mode set, perform segmentation using the smaller of the set of block segmentation modes. a first volume of codes and a second volume of codes.

[0313] A block segmentation determining module in a decoder according to an embodiment of the present disclosure may, based on a first amount of code of a first set of block segmentation modes and a second amount of code of a second set of block segmentation modes, perform segmentation using a set of block segmentation modes that appears first in a predetermined order from the first set of block segmentation modes and the second set of block segmentation modes when the first code amount and the second code amount are equal.

[0314] An encoding method according to an embodiment of the present disclosure may include: segmenting a frame into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, wherein the frame is read from a storage device; and encoding a plurality of blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. In segmentation, when the total number of segments of the first block segmentation mode is three, the second block is the center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, the second block segmentation mode may include only a block segmentation mode indicating that the total number of segments is three.

[0315] A parameter for identifying a second block segmentation mode in the encoding method according to an embodiment of the present disclosure may include a first flag that indicates whether the block is to be segmented horizontally or vertically, and need not necessarily include a second flag indicating overall the number of segments into which the block should be segmented.

[0316] An encoding method according to an embodiment of the present disclosure may include: segmenting a frame into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, wherein the frame is read from a storage device; and encoding a plurality of blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. In segmentation, when the total number of segments of the first block segmentation mode is three, the second block is the center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, the second block segmentation mode, indicating that the total number of segments is two, need not be used.

[0317] An encoding method according to an embodiment of the present disclosure may include: segmenting a frame into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, wherein the frame is read from a storage device; and encoding a plurality of blocks. The set of block segmentation modes may include a first block segmentation mode and a second block segmentation mode, each of which specifies a direction of segmentation and a number of segments. When segmenting, the use of the second block segmentation mode, which indicates that the number of segments is two, may be limited.

[0318] The parameter for identifying a second block segmentation mode in the encoding method according to an embodiment of the present disclosure may include a first flag that indicates whether the block should be segmented horizontally or vertically, and a second flag indicating whether the block should be segmented into two or more.

[0319] A parameter in an encoding method according to an embodiment of the present disclosure may be provided in data of a series of consecutive macroblocks.

[0320] An encoding method according to an embodiment of the present disclosure may include: segmenting a frame into a set of blocks including a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies type of segmentation, wherein the frame is read from the storage device; and encoding a plurality of blocks. In segmentation, when the first set of blocks obtained by using the first set of block segmentation modes and the second set of blocks obtained by using the second set of block segmentation modes are identical, the segmentation may be performed using only one of the first block segmentation mode or the second mode of segmentation into blocks.

[0321] Segmentation in the encoding method according to an embodiment of the present disclosure may, based on the first code amount of the first block segmentation mode set and the second code amount of the second block segmentation mode set, be performed using the smaller of the first block segmentation mode set and second volume of codes.

[0322] Segmentation in the encoding method according to an embodiment of the present disclosure may, based on the first code amount of the first block segmentation mode set and the second code amount of the second block segmentation mode set, be performed using the block segmentation mode set that appears first in the preliminary a certain order of the first set of block segmentation modes and the second set of block segmentation modes when the first code amount and the second code amount are equal.

[0323] A decoding method according to an embodiment of the present disclosure may include: segmenting an encoded signal into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein coded the signal is read from the memory device; and decoding multiple blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. In segmentation, when the total number of segments of the first block segmentation mode is three, the second block is the center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, the second block segmentation mode may include only a block segmentation mode indicating that the total number of segments is three.

[0324] The parameter for identifying a second block segmentation mode in the decoding method according to an embodiment of the present disclosure may include a first flag that indicates whether the block should be segmented horizontally or vertically, and a second flag indicating whether the block should be segmented into two or more.

[0325] A decoding method according to an embodiment of the present disclosure may include: segmenting an encoded signal into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein coded the signal is read from the memory device; and decoding multiple blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. In segmentation, when the total number of segments of the first block segmentation mode is three, the second block is the center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, the second block segmentation mode, indicating that the total number of segments is two, need not necessarily be used.

[0326] A decoding method according to an embodiment of the present disclosure may include: segmenting an encoded signal into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein coded the signal is read from the memory device; and decoding multiple blocks. The set of block segmentation modes may include a first block segmentation mode and a second block segmentation mode, each of which specifies a direction of segmentation and a number of segments. When segmenting, the use of the second block segmentation mode, which indicates that the number of segments is two, may be limited.

[0327] A decoding method according to an embodiment of the present disclosure may include: segmenting an encoded signal into a set of blocks including a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies the type of segmentation, wherein the encoded signal is read from the storage device; and decoding multiple blocks. In segmentation, when the first set of blocks obtained by using the first set of block segmentation modes and the second set of blocks obtained by using the second set of block segmentation modes are identical, the segmentation may be performed using only one of the first block segmentation mode or the second mode of segmentation into blocks.

[0328] A frame compression program according to an embodiment of the present disclosure may include: segmenting a frame into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein the frame read from a storage device; and decoding multiple blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. In segmentation, when the total number of segments of the first block segmentation mode is three, the second block is the center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, the second block segmentation mode may include only a block segmentation mode indicating that the total number of segments is three.

[0329] A frame compression program according to an embodiment of the present disclosure may include: segmenting a frame into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein the frame read from a storage device; and decoding multiple blocks. The set of block segmentation modes may include a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one from blocks obtained after the first block is segmented. In segmentation, when the total number of segments of the first block segmentation mode is three, the second block is the center block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, the second block segmentation mode, indicating that the total number of segments is two, need not necessarily be used.

[0330] A frame compression program according to an embodiment of the present disclosure may include: segmenting a frame into a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, wherein the frame read from a storage device; and encoding a plurality of blocks. The set of block segmentation modes may include a first block segmentation mode and a second block segmentation mode, each of which specifies a direction of segmentation and a number of segments. When segmenting, the use of the second block segmentation mode, which indicates that the number of segments is two, may be limited.

[0331] A frame compression program according to an embodiment of the present disclosure may include: segmenting a frame into a set of blocks including a plurality of blocks, using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies the type of segmentation, wherein the frame is read from the storage device; and encoding a plurality of blocks. In segmentation, when the first set of blocks obtained by using the first set of block segmentation modes and the second set of blocks obtained by using the second set of block segmentation modes are identical, the segmentation may be performed using only one of the first block segmentation mode or the second mode of segmentation into blocks.

Industrial applicability

[0332] The present disclosure can be used in multimedia encoding/decoding and, in particular, in an image and video encoder/decoder that uses block encoding/decoding.

Links with numbers on drawings

[0333] 100 - encoder

102 - partitioning module

104 - subtractor

106, 5001 - converter

108, 5002 - quantizer

110, 5009 - entropy encoder

112, 5003, 6002 - inverse quantizer

114, 5004, 6003 - inverse converter

116 - adder

118, 5005, 6004 - block storage device

120 - contour filter

122, 5006, 6005 - frame memory

124, 5007, 6006 - internal forecasting module

126, 5008, 6007 - mutual prediction module

128 - prediction controller

200 - decoder

202, 6001 - entropy decoder

204 - inverse quantizer

206 - inverse converter

208 - adder

210 - block storage device

212 - contour filter

214 - frame memory

216 - internal forecasting module

218 - mutual prediction module

220 - prediction controller

5000 - video encoder

5010, 6008 - module for determining segmentation into blocks

6000 - video decoder

Position 1. An encoder that encodes a frame, the encoder comprising:

CPU; And

Memory device,

the processor includes:

a block segmentation determining module that segments a frame into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, the frame being read from a storage device; And

an encoding module that encodes multiple blocks,

wherein the set of block segmentation modes includes a first segmentation mode that specifies the segmentation direction and the total number of segments for segmenting the first block, and a second block segmentation mode that specifies the segmentation direction and the total number of segments for segmentation of the second block, which is one of the blocks obtained after the first block is segmented, and

when the total number of segments of the first block segmentation mode is three, the second block is a central block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first block segmentation mode , wherein the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three.

Statement 2. The encoder according to Statement 1, wherein the parameter for identifying the second block segmentation mode includes a first flag that indicates whether the block is to be segmented horizontally or vertically and does not include a second flag indicating the total number of segments per which block is to be segmented.

Position 3. A decoder that decodes the encoded signal, the decoder comprising:

CPU; And

Memory device,

the processor includes:

a block segmentation determining module that segments the encoded signal into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, wherein the encoded signal is read from a storage device; And

a decoding module that decodes a plurality of blocks,

wherein the set of block segmentation modes includes a first segmentation mode that specifies the segmentation direction and the total number of segments for segmenting the first block, and a second block segmentation mode that specifies the segmentation direction and the total number of segments for segmentation of the second block, which is one of the blocks obtained after the first block is segmented, and

when the total number of segments of the first block segmentation mode is three, the second block is a central block among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first block segmentation mode , wherein the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three.

Statement 4. The decoder according to Position 3, wherein the parameter for identifying the second block segmentation mode includes a first flag that indicates whether the block is to be segmented horizontally or vertically and does not include a second flag indicating the total number of segments per which block is to be segmented.

Provision 5. An encoding method containing the steps of:

segmenting the frame into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, and the frame is read from the storage device; And

encode many blocks, while:

the set of block segmentation modes includes a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one of blocks received after the first block is segmented, and

in segmentation, when the total number of segments of the first block segmentation mode is three, the second block is a central block from among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, wherein the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three.

Clause 6. A decoding method comprising the steps of:

segmenting the encoded signal into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, and the encoded signal is read from a storage device; And

decode many blocks, while:

the set of block segmentation modes includes a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one of blocks received after the first block is segmented, and

in segmentation, when the total number of segments of the first block segmentation mode is three, the second block is a central block from among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, wherein the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three.

Regulation 7. Frame compression program containing:

segmenting the frame into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a segmentation type, the frame being read from a storage device; And

decoding multiple blocks, while:

the set of block segmentation modes includes a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one of blocks received after the first block is segmented, and

in segmentation, when the total number of segments of the first block segmentation mode is three, the second block is a central block from among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, wherein the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three.

Claims (6)

A non-transitory computer-readable medium on which software is stored which, when executed by a decoder processor, configures the decoder to receive a bit stream, wherein the bit stream includes an encoded signal and a parameter in accordance with which the decoder is further configured to perform: segmenting the encoded signal into a plurality of blocks using a set of block segmentation modes obtained by combining one or more block segmentation modes, each of which specifies a type of segmentation, and the encoded signal is read from memory; And decoding multiple blocks, while the set of block segmentation modes includes a first segmentation mode that specifies a segmentation direction and a total number of segments for segmenting a first block, and a second block segmentation mode that specifies a segmentation direction and a total number of segments for segmentation of a second block that is one of blocks received after the first block is segmented, in segmentation, when the total number of segments of the first block segmentation mode is three, the second block is a central block from among the blocks obtained after the first block is segmented, and the segmentation direction for the second block segmentation mode is identical to the segmentation direction for the first mode block segmentation, wherein the second block segmentation mode includes only the block segmentation mode indicating that the total number of segments is three, and said parameter for identifying the second block segmentation mode includes a first flag that indicates whether the block is to be segmented horizontally or vertically and does not include a second flag indicating the total number of segments into which the block is to be segmented.

Images