KR20190116067A - Method and apparatus for inter predection using reference frame generabed based on deep-learning - Google Patents

Abstract

A method and apparatus for inter prediction using a reference frame generated based on deep learning are disclosed. A reference frame is selected, and a virtual reference frame is generated based on the selected reference frame. A reference picture list is configured to include the generated virtual reference frame. Inter prediction on a target block is performed based on the virtual reference frame. The virtual reference frame may be generated based on a deep learning network structure, and may be generated based on video interpolation and/or video interpolation using the selected reference frame.

Description

TECHNICAL AND APPARATUS FOR INTER PREDECTION USING REFERENCE FRAME GENERABED BASED ON DEEP-LEARNING}

The following embodiments are related to a video decoding method, a decoding device, an encoding method, and an encoding device. The decoding method for performing inter prediction using a reference frame generated based on deep learning, a decoding device, an encoding method, and an encoding device. It is about.

Through the continuous development of the information and telecommunications industry, broadcasting services having high definition (HD) resolution have spread worldwide. This proliferation has resulted in many users becoming accustomed to high resolution, high quality images and / or video.

In order to satisfy users' demand for high image quality, many organizations are spurring the development of next generation video devices. Users are interested in Ultra High Definition (UHD) TVs, which have four times the resolution of FHD TVs, as well as High Definition TV (HDTV) and Full HD (FHD) TVs. This has increased, and as the interest increases, an image encoding / decoding technique for an image having higher resolution and image quality is required.

An image encoding / decoding apparatus and method include an inter prediction technique, an intra prediction technique, an entropy encoding technique, etc. in order to perform encoding / decoding of high resolution and high quality images. Can be used. The inter prediction technique may be a technique of predicting a value of a pixel included in a target picture by using a previous picture temporally and / or a later picture temporally. An intra prediction technique may be a technique of predicting a value of a pixel included in a target picture by using information of a pixel in the target picture. The entropy encoding technique may be a technique of allocating a short code to a symbol having a high appearance frequency and a long code to a symbol having a low appearance frequency.

Various inter prediction techniques and intra prediction techniques have been developed for more accurate prediction.

An embodiment may provide an encoding apparatus, an encoding method, a decoding apparatus, and a decoding method that perform inter prediction on a target block based on a virtual reference frame.

An embodiment may provide an encoding device, an encoding method, a decoding device, and a decoding method for generating a virtual reference frame based on a deep learning network structure.

An embodiment can provide an encoding device, an encoding method, a decoding device, and a decoding method for generating a virtual reference frame based on video interpolation and / or video interpolation using a selected reference frame.

In one side, selecting a reference frame; Generating a virtual reference frame based on the selected reference frame; And performing inter prediction based on the virtual reference frame.

The selected reference frame may be plural.

The virtual reference frame may be generated based on a deep learning network structure.

The virtual reference frame may be generated using a generative adversarial network (GAN) structure.

The virtual reference frame may be generated using an Adaptive Convolution Network (ACN).

The virtual reference frame may be generated through interpolation using frames predicted by network structures.

The virtual reference frame may be generated based on video interpolation using the selected reference frame.

Video prediction by optical flow, ACN or Long Short Term Memory (LSTM) may be used for the video interpolation.

The virtual reference frame may be generated based on video extrapolation using the selected reference frame.

The decoding method may further include constructing a reference picture list based on the virtual reference frame.

A reference frame specified among reference frames in a decoded picture buffer (DPB) may be replaced with the virtual reference frame.

And the inter prediction mode of the inter prediction is an advanced motion vector prediction (AMVP) mode.

The inter prediction mode of the inter prediction may be a merge mode or a skip mode.

The selected reference frame may be a reference frame closest to the distance from the target frame in the reverse or forward direction among the reference frames included in the reference picture list.

When one reference frame is selected in both directions, the first difference and the second difference may be the same.

The first difference may be a difference between a picture order count (POC) of the selected reference frame in the reverse direction and a POC of the target frame.

The second difference may be a difference between the POC of the target frame and the POC of the selected reference frame in the forward direction.

The selected reference frame may be a reference frame compressed with the smallest quantization parameter (QP) among the reference frames in the decoded picture buffer (DPB).

Reference frame specific information indicating the selected reference frame generated for generation of the virtual reference frame may be signaled for the specified unit.

The reference frame may be selected based on a temporal identifier of the reference frame.

In another aspect, the method comprising: selecting a reference frame; Generating a virtual reference frame based on the selected reference frame; And performing inter prediction based on the virtual reference frame.

In another aspect, the method comprising: selecting a reference frame; Generating a virtual reference frame based on the selected reference frame; And performing inter prediction based on the virtual reference frame.

Provided are an encoding device, an encoding method, a decoding device, and a decoding method for performing inter prediction on a target block based on a virtual reference frame.

Provided are an encoding apparatus, an encoding method, a decoding apparatus, and a decoding method for generating a virtual reference frame based on a deep learning network structure.

Provided are an encoding apparatus, an encoding method, a decoding apparatus, and a decoding method for generating a virtual reference frame based on video interpolation and / or video interpolation using a selected reference frame.

1 is a block diagram illustrating a configuration of an encoding apparatus according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a decoding apparatus according to an embodiment of the present invention.
3 is a diagram schematically illustrating a division structure of an image when encoding and decoding an image.
4 is a diagram illustrating a form of a prediction unit PU that a coding unit CU may include.
FIG. 5 is a diagram illustrating a form of a transform unit (TU) that may be included in a coding unit (CU).
6 illustrates partitioning of a block according to an example.
7 is a diagram for explaining an embodiment of an intra prediction process.
8 is a diagram for describing a position of a reference sample used in an intra prediction process.
9 is a diagram for explaining an embodiment of an inter prediction process.
10 illustrates spatial candidates according to an example.
11 illustrates an addition order of spatial information of motion candidates to a merge list according to an example.
12 illustrates a process of transform and quantization according to an example.
13 illustrates diagonal scanning according to an example.
14 illustrates horizontal scanning according to an example.
15 illustrates vertical scanning according to an example.
16 is a structural diagram of an encoding apparatus according to an embodiment.
17 is a structural diagram of a decoding apparatus according to an embodiment.
18 illustrates an operation of a convolutional layer according to an example.
19 illustrates an operation of a pulling layer according to an example.
20 illustrates an operation of a deconvolution layer according to an example.
21 illustrates an operation of an unpooling layer according to an example.
22 illustrates an operation of a lelu layer according to an example.
23 shows an automatic encoder according to an example.
24 illustrates a convolution encoder and a convolution decoder according to an example.
25 illustrates a configuration of a generator of a GAN according to an example.
26 illustrates a configuration of a discriminator of a GAN according to an example.
27 shows a structure of an RNN according to an example.
28 illustrates a structure of a convolutional LSTM neural network according to an example.
29 shows a structure of an ACN according to an example.
30 illustrates a structure of an adaptive separable convolution according to an example.
31 is a flow diagram of generation-encoding and generation-decoding according to one embodiment.
32 is a flowchart of an inter prediction method, according to an exemplary embodiment.
33 illustrates a structure of a hierarchical B frame according to an example.
34 shows generation of a reference frame using interpolation through a process of generation-coding and generation-decoding.
35 is a diagram illustrating a process of generating a reference frame using interpolation of a video and encoding and decoding a video using a reference frame according to an example.
36 illustrates a process of generating a reference frame using extrapolation of a video and encoding and decoding a video using a reference frame, according to an example.
37 illustrates a configuration of a reference picture list of a virtual reference frame when bidirectional prediction is used according to an example.
38 is a flowchart of a method of searching for a motion vector candidate in AMVP mode according to an example.
39 is a flowchart of a method of searching for motion vector candidates in an AMVP mode according to an example.
40 is a flowchart of another method of searching for motion vector candidates in an AMVP mode according to an example.
41 is a flowchart of a method of searching for a temporal motion vector candidate according to a reference frame index of a temporal motion vector candidate in a merge mode and a skip mode according to an example.
42 is a flowchart of a method for searching for a motion vector that does not consider a motion vector of a temporal neighboring block as a motion vector candidate when the temporal motion vector candidate refers to a virtual reference frame according to an embodiment.
43 is a flowchart of a method of predicting a target block and a method of generating a bitstream, according to an embodiment.
44 is a flowchart of a method of predicting a target block using a bitstream, according to an embodiment.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

DETAILED DESCRIPTION For the following detailed description of exemplary embodiments, reference is made to the accompanying drawings that show, by way of illustration, specific embodiments. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be embodied in other embodiments without departing from the spirit and scope of the invention in connection with one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiments. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the exemplary embodiments, if properly described, is defined only by the appended claims, along with the full scope of equivalents to which such claims are entitled.

Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects. Shape and size of the elements in the drawings may be exaggerated for clarity.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is said to be "connected" or "connected" to another component, the above two components may be directly connected to or connected to each other, It is to be understood that other components may exist in the middle of the two components. When a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that no other component exists between the two components.

The components shown in the embodiments of the present invention are independently shown to represent different characteristic functions, and do not mean that each component is composed of separate hardware or one software unit. In other words, each component is listed as each component for convenience of description, and at least two of each component is combined into one component, or one component is divided into a plurality of components to provide a function. Combined and separate embodiments of each of these components, which can be carried out, are also included within the scope of the present invention without departing from the spirit of the invention.

In addition, the description "including" a specific configuration in the exemplary embodiments does not exclude a configuration other than the specific configuration described above, the additional configuration is the implementation of the exemplary embodiments or the technical spirit of the exemplary embodiments. It can be included in the range.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present invention, the terms "comprise" or "having" and the like are intended to indicate that there exists a feature, number, step, operation, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof. In other words, the description "include" a specific configuration in the present invention does not exclude a configuration other than the configuration, it means that additional configuration may be included in the scope of the technical spirit of the present invention or the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the embodiments. In describing the embodiments, when it is determined that the detailed description of the related well-known configuration or function may obscure the subject matter of the present specification, the detailed description thereof will be omitted. In addition, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

In the following description, an image may mean one picture constituting a video and may represent a video itself. For example, "encoding and / or decoding of an image" may mean "encoding and / or decoding of a video" and may mean "encoding and / or decoding of one of images constituting the video." It may be.

Hereinafter, the terms "video" and "motion picture" may be used interchangeably and may be used interchangeably.

Hereinafter, the target image may be an encoding target image that is a target of encoding and / or a decoding target image that is a target of decoding. The target image may be an input image input to the encoding apparatus or may be an input image input to the decoding apparatus.

Hereinafter, the terms "image", "picture", "frame" and "screen" may be used in the same sense and may be used interchangeably.

Hereinafter, the target block may be an encoding target block that is a target of encoding and / or a decoding target block that is a target of decoding. In addition, the target block may be a current block that is a target of current encoding and / or decoding. For example, the terms "target block" and "current block" may be used interchangeably and may be used interchangeably.

In the following, the terms “block” and “unit” may be used interchangeably and may be used interchangeably. Or “block” may indicate a particular unit.

In the following, the terms “region” and “segment” may be used interchangeably.

In the following, the specific signal may be a signal representing a specific block. For example, the original signal may be a signal representing a target block. The prediction signal may be a signal representing a prediction block. The residual signal may be a signal representing a residual block.

In embodiments, each of the specified information, data, flags and elements, attributes, etc. may have a value. The value "0" of information, data, flags and elements, attributes, etc. may represent a logical false or first predefined value. In other words, the value "0", false, logical false and the first predefined value can be used interchangeably. The value "1" of information, data, flags and elements, attributes, etc. may represent logical true or second predefined values. In other words, the value "1", true, logical true and the second predefined value can be used interchangeably.

When a variable such as i or j is used to indicate a row, column, or index, the value of i may be an integer of 0 or more and may be an integer of 1 or more. In other words, in embodiments, rows, columns, indexes, etc. may be counted from zero and counted from one.

In the following, terms used in the embodiments are described.

Encoder: Refers to a device that performs encoding.

Decoder: Refers to an apparatus for performing decoding.

Unit: A unit may represent a unit of encoding and decoding of an image. The terms "unit" and "block" may be used interchangeably and may be used interchangeably.

The unit may be an M × N array of samples. M and N may each be a positive integer. A unit can often mean an array of two-dimensional samples.

In encoding and decoding of an image, a unit may be an area generated by division of one image. In other words, the unit may be a specified area within one image. One image may be divided into a plurality of units. Alternatively, the unit may mean the divided portion when an image is divided into subdivided portions, and encoding or decoding is performed on the divided portion.

In encoding and decoding of an image, predefined processing for a unit may be performed according to the type of the unit.

Depending on the function, the type of unit may be a macro unit, a coding unit (CU), a prediction unit (PU), a residual unit, a transform unit (TU), or the like. Can be classified as. Alternatively, depending on the function, the unit may be a block, a macroblock, a coding tree unit, a coding tree block, a coding unit, a coding block, a prediction unit. It may mean a (Prediction Unit), a prediction block, a residual unit, a residual block, a transform unit, a transform block, and the like.

A unit may refer to information including a luma component block, a corresponding chroma component block, and a syntax element for each block, to refer to the block separately.

The size and shape of the unit may vary. In addition, the unit may have various sizes and various shapes. In particular, the shape of the unit may include not only square but also geometric figures that can be expressed in two dimensions such as rectangles, trapezoids, triangles, and pentagons.

In addition, the unit information may include at least one of a type of a unit, a size of a unit, a depth of a unit, a coding order of a unit, a decoding order of a unit, and the like. For example, the type of unit may refer to one of a CU, a PU, a residual unit, a TU, and the like.

One unit can be further divided into sub-units having a smaller size than the unit.

Depth: Depth may refer to the degree of division of a unit. In addition, the unit depth may indicate the level at which the unit exists when the unit is represented in a tree structure.

The unit division information may comprise a depth regarding the depth of the unit. Depth may indicate the number and / or degree of unit division.

In the tree structure, the root node has the shallowest depth and the leaf node has the deepest depth.

One unit may be divided into a plurality of sub-units hierarchically with depth information based on a tree structure. In other words, the unit and the lower unit generated by the division of the unit may correspond to the node and the child node of the node, respectively. Each divided subunit may have a depth. Since the depth indicates the number and / or degree of division of the unit, the division information of the lower unit may include information about the size of the lower unit.

In the tree structure, the highest node may correspond to the first unit that is not split. The highest node may be called the root node. In addition, the highest node may have a minimum depth value. At this time, the highest node may have a depth of level 0.

A node with a depth of level 1 may represent a unit created as the first unit is divided once. A node with a depth of level 2 may represent a unit created as the first unit is split twice.

A node with a depth of level n may represent a unit generated as the first unit is divided n times.

The leaf node may be the lowest node or may be a node that cannot be further divided. The depth of the leaf node may be at the maximum level. For example, the predefined value of the maximum level may be three.

QT depth may indicate depth for quad division. The BT depth may represent the depth for binary division. The TT depth may represent the depth for the ternary split.

Sample: A sample may be a base unit constituting a block. The sample may be represented as values from 0 to 2 ^Bd- 1 depending on the bit depth Bd.

The sample may be a pixel or pixel value.

In the following, the terms "pixel", "pixel" and "sample" may be used in the same sense and may be used interchangeably.

Coding Tree Unit (CTU): A CTU may consist of one luma component (Y) coding tree block and two chroma component (Cb, Cr) coding tree blocks associated with the luma component coding tree block. have. In addition, the CTU may mean that the block and the syntax element for each block of the blocks.

Each coding tree unit is a quad tree (QT), a binary tree (BT), a ternary tree (TT), etc. to form sub-units such as a coding unit, a prediction unit and a transform unit. It may be partitioned using one or more partitioning schemes. In addition, each coding tree unit may be partitioned using a MultiType Tree (MTT) using one or more partitioning schemes.

-CTU can be used as a term for referring to a pixel block which is a processing unit in a decoding and encoding process of an image, as in the division of an input image.

Coding Tree Block (CTB): A coding tree block may be used as a term for referring to any one of a Y coding tree block, a Cb coding tree block, and a Cr coding tree block.

Neighbor block: The neighbor block may mean a block adjacent to the target block. The neighboring block may mean a restored neighboring block.

In the following, the terms “neighbor block” and “adjacent block” may be used in the same sense and may be used interchangeably.

Spatial neighbor block: The spatial neighbor block may be a block spatially adjacent to the target block. The neighboring block may include spatial neighboring blocks.

The target block and the spatial neighboring blocks can be included in the target picture.

The spatial neighboring block may mean a block in which a boundary of the target block abuts or a block located within a predetermined distance from the target block.

The spatial neighboring block may mean a block adjacent to a vertex of the target block. The block adjacent to the vertex of the target block may be a block vertically adjacent to a neighboring block horizontally adjacent to the target block or a block horizontally adjacent to a neighboring block vertically adjacent to the target block.

Temporal neighbor block: The temporal neighbor block may be a block temporally adjacent to the target block. The neighboring block may include a temporal neighboring block.

The temporal neighboring block may comprise a co-located block (col block).

The call block may be a block in a co-located picture (col picture). The position in the call picture of the call block may correspond to the position in the target picture of the target block. Alternatively, the position in the call picture of the call block may be the same as the position in the target picture of the target block. The call picture may be a picture included in the reference picture list.

The temporal neighboring block may be a block temporally adjacent to the spatial neighboring block of the target block.

Prediction unit: This may mean a base unit for prediction such as inter prediction, intra prediction, inter compensation, intra compensation, and motion compensation.

One prediction unit may be divided into a plurality of partitions or lower prediction units having a smaller size. The plurality of partitions may also be the base unit in performing prediction or compensation. The partition generated by the partitioning of the prediction unit may also be the prediction unit.

Prediction unit partition: A prediction unit partition may mean a form in which a prediction unit is divided.

Reconstructed neighboring unit: The reconstructed neighboring unit may be a unit that is already decoded and reconstructed in the neighbor of the target unit.

The reconstructed neighboring unit may be a spatial neighboring unit or a temporal neighboring unit to the target unit.

The reconstructed spatial neighboring unit may be a unit in the target picture and already reconstructed through encoding and / or decoding.

The reconstructed temporal neighboring unit may be a unit in the reference picture and a unit already reconstructed through encoding and / or decoding. The position in the reference image of the reconstructed temporal neighboring unit may be the same as the position in the target picture of the target unit or may correspond to the position in the target picture of the target unit.

Parameter set: The parameter set may correspond to header information among structures in the bitstream. For example, the parameter set may be a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS) and an adaptation parameter set (APS). And the like.

In addition, the parameter set may include slice header information and tile header information.

Rate-distortion optimization: The encoding apparatus uses a combination of the size of the coding unit, the prediction mode, the size of the prediction unit, the motion information, and the size of the transform unit to provide high coding efficiency. Distortion optimization can be used.

The rate-distortion optimization method can calculate the rate-distortion cost of each combination in order to select the optimal combination among the above combinations. Rate-distortion cost can be calculated using Equation 1 below. In general, a combination in which the rate-distortion cost is minimized may be selected as an optimal combination in the rate-distortion optimization scheme.

[Equation 1]

D may represent distortion. D may be the mean square error of the squares of difference values between the original transform coefficients and the reconstructed transform coefficients in the transform unit.

-R can represent the rate R may indicate a bit rate using the associated context information.

λ may represent the Lagrangian multiplier. R may include not only coding parameter information such as prediction mode, motion information, coded block flag, etc., but also bits generated by encoding of transform coefficients.

The encoding apparatus may perform processes such as inter prediction, intra prediction, transformation, quantization, entropy encoding, inverse quantization, and / or inverse transformation to calculate accurate D and R. These processes can greatly increase the complexity in the encoding apparatus.

Bitstream: A bitstream may mean a string of bits including encoded image information.

Parameter set: The parameter set may correspond to header information among structures in the bitstream.

The parameter set may comprise at least one of a video parameter set, a sequence parameter set, a picture parameter set and an adaptation parameter set. In addition, the parameter set may include information of a slice header and information of a tile header.

Parsing: Parsing may mean entropy decoding a bitstream to determine a value of a syntax element. Alternatively, parsing may refer to entropy decoding itself.

A symbol: may mean at least one of syntax elements, coding parameters, transform coefficients, etc. of the encoding target unit and / or decoding target unit. In addition, the symbol may mean an object of entropy encoding or a result of entropy decoding.

Reference picture: The reference picture may mean an image referenced by a unit for inter prediction or motion compensation. Alternatively, the reference picture may be an image including a reference unit referenced by the target unit for inter prediction or motion compensation.

Hereinafter, the terms "reference picture" and "reference picture" may be used in the same sense and may be used interchangeably.

Reference picture list: The reference picture list may be a list including one or more reference pictures used for inter prediction or motion compensation.

The types of reference picture lists are List Combined (LC), List 0 (List 0; L0), List 1 (List 1; L1), List 2 (List 2; L2), and List 3 (List 3; L3). ) And the like.

One or more reference picture lists may be used for inter prediction.

Inter prediction indicator: The inter prediction indicator may indicate the direction of inter prediction for the target unit. The inter prediction may be one of unidirectional prediction and bidirectional prediction. Alternatively, the inter prediction indicator may indicate the number of reference pictures used when generating the prediction unit of the target unit. Alternatively, the inter prediction indicator may mean the number of prediction blocks used for inter prediction or motion compensation for the target unit.

Reference picture index: The reference picture index may be an index indicating a specific reference picture in the reference picture list.

Picture order count (POC): The POC of a picture may indicate the display order of the picture.

Motion Vector (MV): The motion vector may be a two-dimensional vector used in inter prediction or motion compensation. The motion vector may mean an offset between the target image and the reference image.

For example, MV can be expressed in the form (mv _x , mv _y ). mv _x may represent a horizontal component and mv _y may represent a vertical component.

Search range: The search range may be a two-dimensional area in which a search for MV is performed during inter prediction. For example, the size of the search region may be M × N. M and N may each be a positive integer.

Motion vector candidate: A motion vector candidate may mean a motion vector of a block that is a prediction candidate or a block that is a prediction candidate when predicting a motion vector.

The motion vector candidate may be included in the motion vector candidate list.

Motion vector candidate list: A motion vector candidate list may mean a list constructed using one or more motion vector candidates.

Motion vector candidate index: The motion vector candidate index may mean an indicator indicating a motion vector candidate in the motion vector candidate list. Alternatively, the motion vector candidate index may be an index of a motion vector predictor.

Motion information: The motion information includes not only a motion vector, a reference picture index and an inter prediction indicator, but also reference picture list information, a reference picture, a motion vector candidate, a motion vector candidate index, a merge candidate, a merge index, and the like. It may mean information including at least one.

Merge candidate list: The merge candidate list may mean a list constructed using merge candidates.

Merge candidate: The merge candidate may mean a spatial merge candidate, a temporal merge candidate, a combined merge candidate, a combined bi-prediction merge candidate, a zero merge candidate, and the like. The merge candidate may include motion type information such as prediction type information, a reference picture index for each list, and a motion vector.

Merge index: The merge index may be an indicator indicating a merge candidate in the merge candidate list.

The merge index may indicate a reconstructed unit that induced a merge candidate among spatially adjacent reconstructed units and reconstructed units temporally adjacent to the target unit.

The merge index may indicate at least one of motion information of the merge candidate.

Transform unit: A transform unit may be a basic unit in residual signal encoding and / or residual signal decoding such as transform, inverse transform, quantization, inverse quantization, transform coefficient encoding, transform coefficient decoding, and the like. One transform unit can be divided into a plurality of transform units of smaller size.

Scaling: Scaling may refer to a process of multiplying a transform coefficient level by a factor.

As a result of scaling for the transform coefficient level, a transform coefficient can be generated. Scaling may be referred to as dequantization.

Quantization Parameter (QP): The quantization parameter may mean a value used when generating a transform coefficient level for a transform coefficient in quantization. Alternatively, the quantization parameter may mean a value used when generating transform coefficients by scaling transform coefficient levels in inverse quantization. Alternatively, the quantization parameter may be a value mapped to a quantization step size.

Delta quantization parameter: The delta quantization parameter refers to a difference value between the predicted quantization parameter and the quantization parameter of the target unit.

Scan: A scan may refer to a method of ordering coefficients within a unit, block, or matrix. For example, sorting a two-dimensional array into a one-dimensional array may be referred to as a scan. Alternatively, arranging the one-dimensional array in the form of a two-dimensional array may also be referred to as a scan or inverse scan.

Transform coefficient: The transform coefficient may be a coefficient value generated by performing a transform in the encoding apparatus. Alternatively, the transform coefficient may be a coefficient value generated by performing at least one of entropy decoding and inverse quantization in the decoding apparatus.

The quantized level or quantized transform coefficient level generated by applying quantization to the transform coefficient or the residual signal may also be included in the meaning of the transform coefficient.

Quantized level: A quantized level may mean a value generated by performing quantization on a transform coefficient or a residual signal in an encoding apparatus. Alternatively, the quantized level may mean a value that is an object of inverse quantization in performing inverse quantization in the decoding apparatus.

The quantized transform coefficient level resulting from the transform and quantization may also be included in the meaning of the quantized level.

Non-zero transform coefficient: A non-zero transform coefficient may mean a transform coefficient having a nonzero value or a transform coefficient level having a nonzero value. Alternatively, the non-zero transform coefficient may mean a transform coefficient whose value is not zero or a transform coefficient level whose value is not zero.

Quantization Matrix: A quantization matrix may refer to a matrix used in a quantization process or an inverse quantization process in order to improve the subjective quality or the objective quality of an image. The quantization matrix may also be called a scaling list.

Quantization matrix coefficient: The quantization matrix coefficient may mean each element in the quantization matrix. Quantization matrix coefficients may also be referred to as matrix coefficients.

Default matrix: The default matrix may be a quantization matrix predefined in the encoding apparatus and the decoding apparatus.

Non-default Matrix: The non-default matrix may be a quantization matrix that is not predefined in the encoding apparatus and the decoding apparatus. The non-default matrix may be signaled from the encoding device to the decoding device.

Most Probable Mode (MPM): The MPM may indicate an intra prediction mode that is likely to be used for intra prediction of a target block.

The encoding apparatus and the decoding apparatus may determine one or more MPMs based on coding parameters related to the target block and attributes of an entity related to the target block.

The encoding apparatus and the decoding apparatus may determine one or more MPMs based on the intra prediction mode of the reference block. There may be a plurality of reference blocks. The plurality of reference blocks may include a spatial neighboring block adjacent to the left side of the target block and a spatial neighboring block adjacent to the top of the target block. In other words, one or more different MPMs may be determined according to which intra prediction modes are used for the reference blocks.

One or more MPMs may be determined in the same manner in the encoding apparatus and the decoding apparatus. In other words, the encoding apparatus and the decoding apparatus may share an MPM list including the same one or more MPMs.

MPM List: The MPM List may be a list including one or more MPMs. The number of one or more MPMs in the MPM list may be predefined.

MPM indicator: The MPM indicator may indicate the MPM used for intra prediction of the target block among one or more MPMs in the MPM list. For example, the MPM indicator may be an index into the MPM list.

Since the MPM list is determined in the same manner in the encoding apparatus and the decoding apparatus, the MPM list itself may not need to be transmitted from the encoding apparatus to the decoding apparatus.

The MPM indicator may be signaled from the encoding device to the decoding device. As the MPM indicator is signaled, the decoding apparatus may determine the MPM to be used for intra prediction for the target block among the MPMs in the MPM list.

MPM usage indicator: The MPM usage indicator may indicate whether the MPM usage mode is used for prediction for the target block. The MPM usage mode may be a mode that uses the MPM list to determine the MPM to be used for intra prediction for the target block.

The MPM usage indicator may be signaled from the encoding apparatus to the decoding apparatus.

Signaling: Signaling may indicate that information is transmitted from an encoding device to a decoding device. Or, signaling may mean including information in a bitstream or a recording medium. The information signaled by the encoding apparatus may be used by the decoding apparatus.

The encoding apparatus may generate encoded information by performing encoding on the signaled information. The encoded information may be transmitted from the encoding apparatus to the decoding apparatus. The decoding apparatus may obtain information by decoding the transmitted encoded information. Here, the encoding may be entropy encoding, and the decoding may be entropy decoding.

1 is a block diagram illustrating a configuration of an encoding apparatus according to an embodiment of the present invention.

The encoding apparatus 100 may be an encoder, a video encoding apparatus, or an image encoding apparatus. The video may include one or more images. The encoding apparatus 100 may sequentially encode one or more images of the video.

Referring to FIG. 1, the encoding apparatus 100 may include an inter prediction unit 110, an intra prediction unit 120, a switch 115, a subtractor 125, a transform unit 130, a quantization unit 140, and entropy encoding. The unit 150 may include an inverse quantization unit 160, an inverse transform unit 170, an adder 175, a filter unit 180, and a reference picture buffer 190.

The encoding apparatus 100 may perform encoding on the target image using an intra mode and / or an inter mode.

In addition, the encoding apparatus 100 may generate a bitstream including encoding information through encoding of the target image, and may output the generated bitstream. The generated bitstream can be stored in a computer readable recording medium and can be streamed via wired / wireless transmission media.

As an prediction mode, when an intra mode is used, the switch 115 may be switched to intra. When the inter mode is used as the prediction mode, the switch 115 may be switched to the inter.

The encoding apparatus 100 may generate a prediction block for the target block. In addition, after the prediction block is generated, the encoding apparatus 100 may encode a residual of the target block and the prediction block.

When the prediction mode is the intra mode, the intra prediction unit 120 may use a pixel of an already encoded / decoded block in the neighbor of the target block as a reference sample. The intra prediction unit 120 may perform spatial prediction on the target block by using the reference sample, and generate prediction samples on the target block through spatial prediction.

The inter predictor 110 may include a motion predictor and a motion compensator.

When the prediction mode is the inter mode, the motion predictor may search an area that best matches the target block from the reference image in the motion prediction process, and derive a motion vector for the target block and the searched area using the searched area. can do.

The reference picture may be stored in the reference picture buffer 190 and may be stored in the reference picture buffer 190 when encoding and / or decoding of the reference picture is processed.

The reference picture may be a reconstructed picture, and the reference picture buffer 190 may be referred to as a decoded picture buffer (DPB).

The motion compensator may generate a prediction block for the target block by performing motion compensation using the motion vector. Here, the motion vector may be a two-dimensional vector used for inter prediction. In addition, the motion vector may indicate an offset between the target image and the reference image.

If the motion predictor and the motion compensator have non-integer values, the motion predictor and the motion compensator may generate a prediction block by applying an interpolation filter to a portion of the reference image. In order to perform inter prediction or motion compensation, methods of motion prediction and motion compensation of a PU included in a CU based on a CU include a skip mode, a merge mode, and an advanced motion vector prediction. Whether it is a prediction (AMVP) mode or a current picture reference mode may be determined, and inter prediction or motion compensation may be performed according to each mode.

The subtractor 125 may generate a residual block that is a difference between the target block and the prediction block. The residual block may be referred to as a residual signal.

The residual signal may mean a difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming or quantizing the difference between the original signal and the prediction signal, or by transforming and quantizing. The residual block may be a residual signal on a block basis.

The transform unit 130 may generate transform coefficients by performing transform on the residual block, and output the generated transform coefficients. Here, the transform coefficient may be a coefficient value generated by performing transform on the residual block.

The transformation unit 130 may use one of a plurality of predefined transformation methods in performing the transformation.

A plurality of predefined transformation methods may include Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and Karhunen-Loeve Transform (KLT) based transformations. have.

The transform method used for transforming the residual block may be determined according to at least one of the coding parameters for the target block and / or the neighboring block. For example, the conversion method may be determined based on at least one of the inter prediction mode for the PU, the intra prediction mode for the PU, the size of the TU, and the shape of the TU. Alternatively, transformation information indicating a transformation method may be signaled from the encoding apparatus 100 to the decoding apparatus 200.

When the transform skip mode is applied, the transform unit 130 may omit the transform on the residual block.

By applying quantization to the transform coefficients, a quantized transform coefficient level or quantized level can be generated. In the following embodiments, the quantized transform coefficient level and the quantized level may also be referred to as transform coefficients.

The quantization unit 140 may generate a quantized transform coefficient level (that is, a quantized level or a quantized coefficient) by quantizing the transform coefficient according to the quantization parameter. The quantization unit 140 may output the generated quantized transform coefficient level. In this case, the quantization unit 140 may quantize the transform coefficients using the quantization matrix.

The entropy encoder 150 may generate a bitstream by performing entropy encoding according to a probability distribution based on the values calculated by the quantizer 140 and / or the coding parameter values calculated in the encoding process. . The entropy encoder 150 may output the generated bitstream.

The entropy encoder 150 may perform entropy encoding on information about pixels of an image and information for decoding an image. For example, the information for decoding the image may include a syntax element.

When entropy coding is applied, a small number of bits may be allocated to a symbol having a high occurrence probability, and a large number of bits may be allocated to a symbol having a low occurrence probability. As the symbol is represented through this assignment, the size of the bitstring for the symbols to be encoded may be reduced. Therefore, compression performance of image encoding may be improved through entropy encoding.

Also, the entropy encoder 150 may use exponential golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding for entropy encoding. A coding method such as Arithmetic Coding (CABAC) may be used. For example, the entropy encoder 150 may perform entropy coding using a variable length coding (VLC) table. For example, the entropy encoder 150 may derive a binarization method for the target symbol. Also, the entropy encoder 150 may derive a probability model of the target symbol / bin. The entropy encoder 150 may perform arithmetic coding using the derived binarization method, probability model, and context model.

The entropy encoder 150 may change the coefficient of the form of the two-dimensional block into a one-dimensional vector through a transform coefficient scanning method to encode the quantized transform coefficient level.

The coding parameter may be information required for encoding and / or decoding. The coding parameter may include information encoded by the encoding apparatus 100 and transferred from the encoding apparatus 100 to the decoding apparatus, and may include information that may be inferred in the encoding or decoding process. For example, there is a syntax element as information transmitted to the decoding apparatus.

The coding parameter may include information derived from an encoding process or a decoding process, as well as information (or a flag, an index, etc.) encoded by the encoding apparatus and signaled from the encoding apparatus to the decoding apparatus, as the syntax element. have. In addition, the coding parameter may include information required for encoding or decoding an image. For example, the size of the unit / block, the depth of the unit / block, the partition information of the unit / block, the partition structure of the unit / block, the information indicating whether the unit / block is divided into quad tree form, the unit / block is binary Information indicating whether the tree is split, whether the binary tree is split (horizontal or vertical), the binary tree is split (symmetric split or asymmetric split), or whether the unit / block is split into ternary trees Information to indicate, the split direction in the form of a ternary tree (horizontal or vertical), the split form in the form of a ternary tree (such as symmetrical splitting or asymmetric splitting), information indicating whether the unit / block is split into a compound tree, and the composite tree Combination and direction of division of (such as landscape or portrait), prediction method (intra prediction or inter prediction), intra prediction mode / direction, reference sample filtering method Prediction block filtering method, prediction block boundary filtering method, filter tab of filtering, filter coefficient of filtering, inter prediction mode, motion information, motion vector, reference picture index, inter prediction direction, inter prediction indicator, reference picture list, reference picture , POC, motion vector predictor, motion vector prediction candidate, motion vector candidate list, information indicating whether to use merge mode, information indicating whether to use merge candidate, merge candidate list, skip mode, interpolation filter Type, filter tab of the interpolation filter, filter coefficients of the interpolation filter, motion vector magnitude, motion vector representation accuracy, transform type, transform size, information indicating whether or not to use a first-order transform, whether additional (secondary) transforms are used Information indicating, 1st transform selection information (or 1st transformation index), 2nd transformation selection information (or 2nd transformation index), residual Information indicating the presence or absence of a signal, a coded block pattern, a coded block flag, a quantization parameter, a quantization matrix, information about an intra-loop filter, whether to apply an intra-loop filter Information about the coefficients of the intra-loop filter, the filter tab of the intra-loop, the shape / form of the intra loop filter, information indicating whether or not the deblocking filter is applied, the deblocking filter coefficients, the deblocking filter Tap, deblocking filter strength, deblocking filter shape / shape, information indicating whether to apply adaptive sample offset, adaptive sample offset value, adaptive sample offset category, adaptive sample offset type, adaptive loop-in ( in-loop) filter, adaptive in-loop filter coefficients, adaptive in-loop filter tab, adaptive in-loop filter shape / shape, binarization / debinarization method, context model, Context Model Determination Method, Context Model Update Method, Whether to Perform Regular Mode, Bypass Mode, Context Bean, Bypass Bin, Transform Coefficient, Transform Coefficient Level, Transform Coefficient Level Scanning Method, Display / Output of Image Order, slice identification information, slice type, slice partition information, tile identification information, tile type, tile partition information, picture type, bit depth, information on luma signals, and information on chroma signals, a combined form Or statistics may be included in the coding parameters. The prediction method may indicate one prediction mode of an intra prediction mode and an inter prediction mode.

The primary transform selection information may indicate a primary transform applied to the target block.

The secondary transform selection information may indicate a secondary transform applied to the target block.

The residual signal may represent a difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming the difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming and quantizing the difference between the original signal and the prediction signal. The residual block may be a residual signal for the block.

Here, signaling a flag or an index means that the encoding apparatus 100 converts an entropy coded flag or an entropy coded index generated by performing entropy encoding on a flag or an index into a bitstream. In the decoding apparatus 200, the decoding apparatus 200 may mean obtaining an flag or an index by performing entropy decoding on an entropy coded flag or an entropy coded index extracted from the bitstream. .

Since encoding through inter prediction is performed by the encoding apparatus 100, the encoded target image may be used as a reference image with respect to other image (s) to be processed later. Accordingly, the encoding apparatus 100 may reconstruct or decode the encoded target image and store the reconstructed or decoded image in the reference picture buffer 190 as a reference image. Inverse quantization and inverse transform on the encoded target image may be processed for decoding.

The quantized level may be inversely quantized in the inverse quantization unit 160 and inversely transformed in the inverse transformer 170. The inverse quantization unit 160 may generate inverse quantized coefficients by performing inverse quantization on the quantized level. The inverse transform unit 170 may generate the reconstructed residual block by performing an inverse transform on the inverse quantized coefficients. In other words, the reconstructed residual block may be an inverse quantized and inverse transformed coefficient.

The inverse quantized and inverse transformed coefficients may be summed with the prediction block via the adder 175, and a reconstructed block may be generated by adding the inverse quantized and / or inverse transformed coefficients with the prediction block. Here, the inverse quantized and / or inversely transformed coefficient may refer to a coefficient on which at least one or more of dequantization and inverse-transformation have been performed, and may mean a reconstructed residual block.

The reconstructed block may pass through the filter unit 180. The filter unit 180 may include at least one of a deblocking filter, a sample adaptive offset (SAO), an adaptive loop filter (ALF), and a non local filter (NLF). One or more may be applied to the reconstructed block or reconstructed picture. The filter unit 180 may be referred to as an in-loop filter.

The deblocking filter may remove block distortion generated at boundaries between blocks. To determine whether to apply the deblocking filter, it may be determined whether to apply the deblocking filter to the target block based on the pixel (s) included in some columns or rows included in the block.

When the deblocking filter is applied to the target block, the filter applied may vary depending on the strength of the required deblocking filtering. In other words, a filter determined according to the strength of deblocking filtering among different filters may be applied to the target block. When the deblocking filter is applied to the target block, one of a strong filter and a weak filter may be applied to the target block according to the strength of the required deblocking filtering.

In addition, when the vertical direction filtering and the horizontal direction filtering are performed on the target block, the horizontal direction filtering and the vertical direction filtering may be processed in parallel.

The SAO may add an appropriate offset to the pixel value of the pixel to compensate for coding errors. The SAO may perform correction using an offset on a difference between an original image and a deblocked image in units of pixels for the deblocked image. In order to perform offset correction on an image, a method of dividing pixels included in an image into a predetermined number of regions, determining a region to be offset from the divided regions, and applying an offset to the determined region may be used. In addition, a method of applying an offset in consideration of edge information of each pixel of the image may be used.

The ALF may perform filtering based on a value obtained by comparing the reconstructed image with the original image. After dividing the pixels included in the image into predetermined groups, a filter to be applied to each divided group may be determined, and filtering may be performed for each group. For the luma signal, information related to whether to apply the adaptive loop filter may be signaled for each CU. The shape and filter coefficients of the ALF to be applied to each block may vary from block to block. Alternatively, a fixed form of ALF may be applied to the block, regardless of the features of the block.

The non-local filter may perform filtering based on reconstructed blocks similar to the target block. A region similar to the target block may be selected in the reconstructed image, and filtering of the target block may be performed using statistical properties of the selected similar region. Information related to whether to apply the non-local filter may be signaled for the CU. In addition, the shapes and filter coefficients of the non-local filter to be applied to the blocks may be different from block to block.

The reconstructed block or the reconstructed image that has passed through the filter unit 180 may be stored in the reference picture buffer 190. The reconstructed block that has passed through the filter unit 180 may be part of a reference picture. In other words, the reference picture may be a reconstructed picture composed of reconstructed blocks that have passed through the filter unit 180. The stored reference picture can then be used for inter prediction.

2 is a block diagram illustrating a configuration of a decoding apparatus according to an embodiment of the present invention.

The decoding apparatus 200 may be a decoder, a video decoding apparatus, or an image decoding apparatus.

2, the decoding apparatus 200 may include an entropy decoder 210, an inverse quantizer 220, an inverse transformer 230, an intra predictor 240, an inter predictor 250, and a switch 245. , An adder 255, a filter unit 260, and a reference picture buffer 270.

The decoding apparatus 200 may receive a bitstream output from the encoding apparatus 100. The decoding apparatus 200 may receive a bitstream stored in a computer readable recording medium, and may receive a bitstream streamed through a wired / wireless transmission medium.

The decoding apparatus 200 may perform intra mode and / or inter mode decoding on the bitstream. In addition, the decoding apparatus 200 may generate a reconstructed image or a decoded image through decoding, and output the generated reconstructed image or the decoded image.

For example, switching to the intra mode or the inter mode according to the prediction mode used for decoding may be performed by the switch 245. When the prediction mode used for decoding is an intra mode, the switch 245 may be switched to intra. When the prediction mode used for decoding is an inter mode, the switch 245 may be switched to inter.

The decoding apparatus 200 may obtain a reconstructed residual block by decoding the input bitstream, and generate a prediction block. When the reconstructed residual block and the prediction block are obtained, the decoding apparatus 200 may generate the reconstructed block to be decoded by adding the reconstructed residual block and the prediction block.

The entropy decoder 210 may generate symbols by performing entropy decoding on the bitstream based on a probability distribution on the bitstream. The generated symbols may include symbols in the form of quantized transform coefficient levels (ie, quantized levels or quantized coefficients). Here, the entropy decoding method may be similar to the entropy encoding method described above. For example, the entropy decoding method may be an inverse process of the above-described entropy encoding method.

The entropy decoder 210 may change a coefficient in the form of a one-dimensional vector into a form of a two-dimensional block through a transform coefficient scanning method in order to decode the quantized transform coefficient level.

For example, the coefficients can be changed into a two-dimensional block form by scanning the coefficients of the block using a right top diagonal scan. Or, it may be determined which scan of the upper right diagonal scan, the vertical scan and the horizontal scan will be used according to the size of the block and / or the intra prediction mode.

The quantized coefficient may be inverse quantized by the inverse quantization unit 220. The inverse quantization unit 220 may generate inverse quantized coefficients by performing inverse quantization on the quantized coefficients. In addition, the inverse quantized coefficient may be inversely transformed by the inverse transformer 230. The inverse transform unit 230 may generate the reconstructed residual block by performing an inverse transform on the inverse quantized coefficients. As a result of inverse quantization and inverse transformation on the quantized coefficients, a reconstructed residual block may be generated. In this case, the inverse quantization unit 220 may apply a quantization matrix to the quantized coefficients in generating the reconstructed residual block.

When the intra mode is used, the intra predictor 240 may generate the prediction block by performing spatial prediction using pixel values of the already decoded block of the neighbor of the target block.

The inter predictor 250 may include a motion compensator. Alternatively, the inter predictor 250 may be referred to as a motion compensator.

When the inter mode is used, the motion compensator may generate a prediction block by performing motion compensation using a motion vector and a reference picture stored in the reference picture buffer 270.

When the motion vector has a non-integer value, the motion compensator may apply an interpolation filter to a portion of the reference image, and generate a prediction block using the reference image to which the interpolation filter is applied. The motion compensation unit may determine which of the skip mode, the merge mode, the AMVP mode, and the current picture reference mode is a motion compensation method used for the PU included in the CU based on the CU to perform motion compensation. According to the present invention, motion compensation may be performed.

The reconstructed residual block and prediction block may be added via adder 255. The adder 255 may generate the reconstructed block by adding the reconstructed residual block and the predictive block.

The reconstructed block may pass through the filter unit 260. The filter unit 260 may apply at least one of the deblocking filter, the SAO, the ALF, and the non-local filter to the reconstructed block or the reconstructed image. The reconstructed image may be a picture including the reconstructed block.

The reconstructed image that has passed through the filter unit 260 may be output by the encoding apparatus 100 and may be used by the encoding apparatus 100.

The reconstructed image that has passed through the filter unit 260 may be stored as a reference picture in the reference picture buffer 270. The reconstructed block that has passed through the filter unit 260 may be part of the reference picture. In other words, the reference picture may be an image composed of reconstructed blocks that have passed through the filter unit 260. The stored reference picture can then be used for inter prediction.

The reference picture may be a reconstructed picture, and the reference picture buffer 270 may be referred to as a decoded picture buffer (DPB).

3 is a diagram schematically illustrating a division structure of an image when encoding and decoding an image.

3 may schematically illustrate an example in which one unit is divided into a plurality of sub-units.

In order to efficiently divide an image, a coding unit (CU) may be used in encoding and decoding. A unit may be a term that collectively refers to 1) a block including image samples and 2) a syntax element. For example, "division of a unit" may mean "division of a block corresponding to a unit".

The CU may be used as a base unit of image encoding / decoding. In addition, the CU may be used as a unit to which one selected mode of intra mode and inter mode is applied in image encoding / decoding. In other words, in image encoding / decoding, it may be determined which mode of intra mode and inter mode is applied to each CU.

In addition, a CU may be a base unit in encoding / decoding of prediction, transform, quantization, inverse transform, inverse quantization, and transform coefficients.

Referring to FIG. 3, the image 300 may be sequentially divided in units of a largest coding unit (LCU). For each LCU, the partition structure can be determined. Here, the LCU may be used as the same meaning as a coding tree unit (CTU).

The division of the unit may mean division of a block corresponding to the unit. The block division information may include depth information regarding a depth of a unit. The depth information may indicate the number and / or degree of division of the unit. One unit may be divided into sub-units hierarchically with depth information based on a tree structure. Each divided subunit may have depth information. Depth information may be information indicating the size of a CU. Depth information may be stored for each CU.

Each CU may have depth information. If a CU is split, the CUs created by splitting may have a depth increased by one from the depth of the split CU.

The partition structure may mean a distribution of a CU in the LCU 310 for efficiently encoding an image. This distribution may be determined according to whether to divide one CU into a plurality of CUs. The number of partitioned CUs may be two or more positive integers including 2, 4, 8, 16, and the like. The horizontal size and the vertical size of the CU generated by the split may be smaller than the horizontal size and the vertical size of the CU before the split, depending on the number of CUs created by the split.

The partitioned CU may be recursively divided into a plurality of CUs in the same manner. By recursive partitioning, the size of at least one of the horizontal size and vertical size of the divided CU can be reduced compared to at least one of the horizontal size and vertical size of the CU before splitting.

Partitioning of a CU can be done recursively up to a predefined depth or a predefined size. For example, the depth of the CU may have a value of 0 to 3. The size of the CU may be from 64x64 to 8x8 depending on the depth of the CU.

For example, the depth of the LCU may be zero, and the depth of the smallest coding unit (SCU) may be a predefined maximum depth. Here, the LCU may be a CU having a maximum coding unit size as described above, and the SCU may be a CU having a minimum coding unit size.

The division may begin from the LCU 310, and the depth of the CU may increase by one each time the division reduces the horizontal and / or vertical sizes of the CU.

For example, for each depth, a CU that is not divided may have a size of 2N × 2N. In addition, in the case of a partitioned CU, a CU of 2N × 2N size may be divided into four CUs having an N × N size. The size of N can be reduced by half for every 1 increase in depth.

Referring to FIG. 3, an LCU having a depth of 0 may be 64x64 pixels or 64x64 block. 0 may be the minimum depth. An SCU of depth 3 may be 8x8 pixels or 8x8 block. 3 may be the maximum depth. In this case, a CU of a 64x64 block, which is an LCU, may be represented by depth 0. A CU of a 32x32 block may be represented by depth 1. A CU of a 16x16 block may be represented by depth 2. A CU of an 8x8 block, which is an SCU, may be represented by depth 3.

Information on whether a CU is split may be expressed through split information of the CU. The split information may be 1 bit of information. All CUs except the SCU may include partition information. For example, the value of partition information of a CU that is not divided may be 0, and the value of partition information of a CU that is divided may be 1.

For example, when one CU is divided into four CUs, the horizontal size and vertical size of each CU of the four CUs generated by the split are each half the horizontal size and half the vertical size of the CU before splitting. Can be. When a 32x32 size CU is divided into four CUs, sizes of the divided four CUs may be 16x16. When one CU is divided into four CUs, it may be said that the CU is divided into quad-tree shapes.

For example, when one CU is divided into two CUs, the horizontal size or vertical size of each CU of the two CUs generated by the split is half the horizontal size or half the vertical size of the CU before splitting, respectively. Can be. When a 32x32 size CU is vertically divided into two CUs, sizes of the divided two CUs may be 16x32. When a 32x32 size CU is horizontally divided into two CUs, sizes of the two divided CUs may be 32x16. When one CU is divided into two CUs, it can be said that the CU is divided into a binary-tree.

In the LCU 310 of FIG. 3, both quad-tree splitting and binary-tree splitting are applied.

In the encoding apparatus 100, a 64x64 coding tree unit (CTU) may be split into a plurality of smaller CUs by a recursive quad-tree structure. One CU may be divided into four CUs having the same sizes. CUs may be recursively split, and each CU may have a quad tree structure.

Through recursive partitioning for a CU, an optimal partitioning method can be selected that produces the smallest rate-distortion ratio.

4 is a diagram illustrating a form of a prediction unit PU that a coding unit CU may include.

A CU that is no longer split among CUs split from the LCU may be split into one or more prediction units (PUs).

The PU may be a basic unit for prediction. The PU may be encoded and decoded in any one of a skip mode, an inter mode, and an intra mode. PU may be divided into various types according to each mode. For example, the target block described above with reference to FIG. 1 and the target block described above with reference to FIG. 2 may be a PU.

A CU may not be divided into PUs. If the CU is not divided into PUs, the size of the CU and the size of the PU may be the same.

In skip mode, there may be no partition in the CU. In the skip mode, 2N × 2N mode 410 having the same size of PU and CU without splitting may be supported.

In the inter mode, eight divided forms in a CU may be supported. For example, in the inter mode, 2Nx2N mode 410, 2NxN mode 415, Nx2N mode 420, NxN mode 425, 2NxnU mode 430, 2NxnD mode 435, nLx2N mode 440, and nRx2N Mode 445 may be supported.

In intra mode, 2Nx2N mode 410 and NxN mode 425 may be supported.

In the 2Nx2N mode 410, a PU having a size of 2Nx2N may be encoded. A PU having a size of 2N × 2N may mean a PU having a size equal to the size of a CU. For example, a PU having a size of 2N × 2N may have a size of 64 × 64, 32 × 32, 16 × 16, or 8 × 8.

In the NxN mode 425, a PU having a size of NxN may be encoded.

For example, in intra prediction, when the size of the PU is 8x8, four divided PUs may be encoded. The size of the partitioned PU may be 4 × 4.

When the PU is encoded by the intra mode, the PU may be encoded using one intra prediction mode among the plurality of intra prediction modes. For example, High Efficiency Video Coding (HEVC) technology can provide 35 intra prediction modes, and the PU can be coded in one of the 35 intra prediction modes.

Which of the 2Nx2N mode 410 and NxN mode 425 is to be coded may be determined by the rate-distortion cost.

The encoding apparatus 100 may perform an encoding operation on a PU having a size of 2N × 2N. Here, the encoding operation may be to encode the PU in each of a plurality of intra prediction modes that the encoding apparatus 100 may use. Through the coding operation, an optimal intra prediction mode for a 2N × 2N size PU may be derived. The optimal intra prediction mode may be an intra prediction mode that generates a minimum rate-distortion cost for encoding a 2N × 2N size PU among a plurality of intra prediction modes that can be used by the encoding apparatus 100.

In addition, the encoding apparatus 100 may sequentially perform encoding operations on each PU of the PUs divided by N × N. Here, the encoding operation may be to encode the PU in each of a plurality of intra prediction modes that the encoding apparatus 100 may use. Through the coding operation, an optimal intra prediction mode for a N × N size PU may be derived. The optimal intra prediction mode may be an intra prediction mode that generates a minimum rate-distortion cost for encoding of a PU of an N × N size among a plurality of intra prediction modes that can be used by the encoding apparatus 100.

The encoding apparatus 100 may determine which of 2Nx2N size PU and NxN size PU to encode based on a comparison of the rate-distortion cost of the 2Nx2N size PU and the rate-distortion costs of the NxN size PUs.

One CU may be divided into one or more PUs, and a PU may also be divided into a plurality of PUs.

For example, when one PU is divided into four PUs, the horizontal size and vertical size of each PU of the four PUs generated by the division are each half of the horizontal size and half of the vertical size of the PU before splitting. Can be. When a 32x32 sized PU is divided into four PUs, the sizes of the divided four PUs may be 16x16. When one PU is divided into four PUs, it can be said that the PU is divided into quad-tree shapes.

For example, when one PU is divided into two PUs, the horizontal size or vertical size of each PU of the two PUs generated by the split is half the horizontal size or half the vertical size of the PU before splitting, respectively. Can be. When a 32x32 sized PU is vertically divided into two PUs, the sizes of the divided two PUs may be 16x32. When a 32x32 sized PU is horizontally divided into two PUs, the sizes of the two divided PUs may be 32x16. When one PU is divided into two PUs, it can be said that the PU is divided into a binary-tree form.

FIG. 5 is a diagram illustrating a form of a transform unit (TU) that may be included in a coding unit (CU).

A transform unit (TU) may be a basic unit used for a process of transform, quantization, inverse transform, inverse quantization, entropy encoding, and entropy decoding in a CU.

The TU may have a square shape or a rectangular shape. The shape of the TU may be determined depending on the size and / or shape of the CU.

Of the CUs partitioned from the LCU, a CU that is no longer split into CUs may be split into one or more TUs. In this case, the partition structure of the TU may be a quad-tree structure. For example, as shown in FIG. 5, one CU 510 may be divided one or more times according to the quad-tree structure. Through division, one CU 510 may be configured with TUs of various sizes.

If a CU is split more than once, the CU can be considered to be split recursively. Through division, one CU may be composed of TUs having various sizes.

Alternatively, one CU may be divided into one or more TUs based on the number of vertical lines and / or horizontal lines that divide the CU.

The CU may be divided into symmetrical TUs and may be divided into asymmetrical TUs. For splitting into asymmetric TUs, information about the size and / or shape of the TU may be signaled from the encoding apparatus 100 to the decoding apparatus 200. Alternatively, the size and / or shape of the TU may be derived from information about the size and / or shape of the CU.

A CU may not be divided into TUs. If the CU is not divided into TUs, the size of the CU and the size of the TU may be the same.

One CU may be divided into one or more TUs, and the TU may also be divided into a plurality of TUs.

For example, if one TU is divided into four TUs, the horizontal size and vertical size of each TU of the four TUs generated by the division are each half the horizontal size and half the vertical size of the TU before the splitting. Can be. When a 32x32 sized TU is divided into four TUs, the sizes of the divided four TUs may be 16x16. When one TU is divided into four TUs, it can be said that the TU is divided into quad-tree shapes.

For example, when one TU is divided into two TUs, the horizontal size or vertical size of each TU of the two TUs generated by the division is half the horizontal size or half the vertical size of the TU before the splitting, respectively. Can be. When a 32x32 sized TU is vertically divided into two TUs, the sizes of the divided two TUs may be 16x32. When a 32x32 sized TU is horizontally divided into two TUs, the sizes of the two divided TUs may be 32x16. When one TU is divided into two TUs, it can be said that the TU is divided into a binary-tree form.

The CU may be partitioned in other ways than shown in FIG. 5.

For example, one CU may be divided into three CUs. The horizontal size or vertical size of the divided three CUs may be 1/4, 1/2, and 1/4 of the horizontal size or vertical size of the CU before splitting, respectively.

For example, when a 32x32 size CU is vertically divided into three CUs, sizes of the divided three CUs may be 8x32, 16x32, and 8x32. As such, when one CU is divided into three CUs, the CU may be regarded as being divided into a ternary tree.

The illustrated division of the quad tree, division of the binary tree, and division of the ternary tree may be applied for division of the CU, and a plurality of division schemes may be combined together to be used for division of the CU. . In this case, a case in which a plurality of partitioning methods are used in combination may be referred to as partitioning of a complex tree.

6 illustrates partitioning of a block according to an example.

In the process of encoding and / or decoding an image, a target block may be divided as illustrated in FIG. 6.

In order to divide the target block, an indicator indicating the split information may be signaled from the encoding apparatus 100 to the decoding apparatus 200. The partitioning information may be information indicating how the target block is divided.

The splitting information is divided into a flag (hereinafter referred to as "split_flag"), a quad-binary flag (hereinafter referred to as "QB_flag"), a quad tree flag (hereinafter referred to as "quadtree_flag"), a binary tree flag (hereinafter referred to as "binarytree_flag" And a binary type flag (hereinafter, denoted as "Btype_flag").

split_flag may be a flag indicating whether a block is split. For example, a value of split_flag 1 may indicate that the block is split. A value of 0 of split_flag may represent that the block is not split.

The QB_flag may be a flag indicating whether the block is divided into a quad tree form and a binary tree form. For example, a value 0 of QB_flag may indicate that the block is divided into quad tree shapes. A value of 1 of QB_flag may indicate that the block is divided into a binary tree. Alternatively, the value 0 of QB_flag may represent that the block is divided into a binary tree. A value of 1 of QB_flag may indicate that the block is divided into quad tree shapes.

quadtree_flag may be a flag indicating whether a block is divided into quad tree shapes. For example, a value of quadtree_flag 1 may indicate that a block is divided into quad tree shapes. A value of zero of the quadtree_flag may represent that the block is not divided into quadtrees.

The binarytree_flag may be a flag indicating whether a block is divided into a binary tree. For example, a value of 1 of binarytree_flag may indicate that a block is divided into a binary tree. A value of 0 of binarytree_flag may represent that the block is not divided into a binary tree.

The Btype_flag may be a flag indicating whether the block is split into a vertical split or a horizontal split when the block is split into a binary tree. For example, the value 0 of Btype_flag may indicate that the block is divided in the horizontal direction. A value of 1 of Btype_flag may indicate that the block is divided in the vertical direction. Alternatively, the value 0 of Btype_flag may indicate that the block is divided in the vertical direction. A value of 1 of Btype_flag may indicate that the block is divided in the horizontal direction.

For example, the split information for the block of FIG. 6 may be derived by signaling at least one of quadtree_flag, binarytree_flag, and Btype_flag as shown in Table 1 below.

TABLE 1

For example, the split information for the block of FIG. 6 may be derived by signaling at least one of split_flag, QB_flag, and Btype_flag as shown in Table 2 below.

TABLE 2

The partitioning method may be limited to quad trees only, or only to binary trees, depending on the size and / or shape of the block. When this restriction is applied, the split_flag may be a flag indicating whether to split into a quad tree or a flag indicating whether to split into a binary tree. The size and shape of the block may be derived according to the depth information of the block, and the depth information may be signaled from the encoding apparatus 100 to the decoding apparatus 200.

If the size of the block falls within the specified range, only quad tree type partitioning may be possible. For example, the specified range may be defined by at least one of the maximum block size and the minimum block size that can only be divided in quad tree form.

Information representing the maximum block size and / or the minimum block size that can be divided only by the quart tree form may be signaled from the encoding apparatus 100 to the decoding apparatus 200 through the bitstream. In addition, such information may be signaled for at least one unit of video, sequence, picture, and slice (or segment).

Alternatively, the maximum block size and / or the minimum block size may be a fixed size predefined by the encoding apparatus 100 and the decoding apparatus 200. For example, if the block size is 64x64 or more and 256x256 or less, only quad tree-type partitioning may be possible. In this case, split_flag may be a flag indicating whether to split into quad tree form.

If the size of the block falls within the specified range, only partitioning in the form of a binary tree may be possible. Here, for example, the specified range may be defined by at least one of the maximum block size and the minimum block size that can be divided only in the form of a binary tree.

Information representing a maximum block size and / or a minimum block size that can be divided only in a binary tree form may be signaled from the encoding apparatus 100 to the decoding apparatus 200 through a bitstream. In addition, such information may be signaled for at least one unit of a sequence, a picture, and a slice (or segment).

Alternatively, the maximum block size and / or the minimum block size may be a fixed size predefined by the encoding apparatus 100 and the decoding apparatus 200. For example, if the size of the block is greater than or equal to 8x8 and less than or equal to 16x16, only binary tree partitioning may be possible. In this case, split_flag may be a flag indicating whether to split into a binary tree.

Partitioning of blocks may be limited by previous partitioning. For example, when a block is divided into a binary tree to generate a plurality of divided blocks, each divided block may be further divided into a binary tree only.

The above-described indicator may not be signaled if the horizontal size or vertical size of the divided block corresponds to a size that can no longer be divided.

7 is a diagram for explaining an embodiment of an intra prediction process.

Arrows outward from the center of the graph of FIG. 7 may indicate prediction directions of intra prediction modes. In addition, the number displayed near the arrow may represent an example of a mode value allocated to the intra prediction mode or the prediction direction of the intra prediction mode.

Intra encoding and / or decoding may be performed using reference samples of units in neighboring units of the target block. The neighboring block may be a neighboring rebuilt block. For example, intra encoding and / or decoding may be performed using a coding parameter or a value of a reference sample included in a neighboring reconstructed block.

The encoding apparatus 100 and / or the decoding apparatus 200 may generate the prediction block by performing intra prediction on the target block based on the information of the sample in the target image. When performing intra prediction, the encoding apparatus 100 and / or the decoding apparatus 200 may generate a prediction block for the target block by performing intra prediction based on information of a sample in the target image. When performing intra prediction, the encoding apparatus 100 and / or the decoding apparatus 200 may perform directional prediction and / or non-directional prediction based on at least one reconstructed reference sample.

The prediction block may mean a block generated as a result of performing intra prediction. The prediction block may correspond to at least one of a CU, a PU, and a TU.

The unit of a prediction block may be the size of at least one of a CU, a PU, and a TU. The prediction block may have a square shape, having a size of 2N × 2N or a size of N × N. The size of NxN may include 4x4, 8x8, 16x16, 32x32 and 64x64.

Alternatively, the prediction block may be a block in the form of a square having a size of 2x2, 4x4, 8x8, 16x16, 32x32, or 64x64, or a block of a rectangular shape having a size of 2x8, 4x8, 2x16, 4x16, and 8x16. have.

Intra prediction may be performed according to an intra prediction mode for a target block. The number of intra prediction modes that the target block may have may be a predetermined fixed value or may be a value determined differently according to the properties of the prediction block. For example, the attributes of the prediction block may include the size of the prediction block and the type of the prediction block.

For example, the number of intra prediction modes may be fixed to 35 regardless of the size of the prediction block. Or, for example, the number of intra prediction modes may be 3, 5, 9, 17, 34, 35, 36, or the like.

The intra prediction mode may be a non-directional mode or a directional mode. For example, the intra prediction mode may include two non-directional modes and 33 directional modes as shown in FIG. 7.

Two non-directional modes may include a DC mode and a planar mode.

The directional modes may be prediction modes with a specific direction or a specific angle.

The intra prediction mode may be represented by at least one of a mode number, a mode value, and a mode angle. The number of intra prediction modes may be M pieces. M may be one or more. In other words, the intra prediction mode may be M pieces including the number of non-directional modes and the number of directional modes.

The number of intra prediction modes may be fixed to M regardless of the size and / or color component of the block. For example, the number of intra prediction modes may be fixed to one of 35 or 67, regardless of the size of the block.

Alternatively, the number of intra prediction modes may differ depending on the size of the block and / or the type of color component.

For example, as the size of a block increases, the number of intra prediction modes may increase. Alternatively, as the size of the block increases, the number of intra prediction modes may decrease. When the size of the block is 4x4 or 8x8, the number of intra prediction modes may be 67. When the size of the block is 16x16, the number of intra prediction modes may be 35. When the size of the block is 32x32, the number of intra prediction modes may be 19. When the size of the block is 64x64, the number of intra prediction modes may be 7.

For example, the number of intra prediction modes may vary depending on whether the color component is a luma signal or a chroma signal. Alternatively, the number of intra prediction modes of the luma component block may be greater than the number of intra prediction modes of the chroma component block.

For example, in the vertical mode having a mode value of 26, prediction may be performed in the vertical direction based on the pixel value of the reference sample. For example, in the horizontal mode having a mode value of 10, prediction may be performed in the horizontal direction based on the pixel value of the reference sample.

Even in the directional mode other than the above-described mode, the encoding apparatus 100 and the decoding apparatus 200 may perform intra prediction on the target unit using the reference sample according to the angle corresponding to the directional mode.

The intra prediction mode located on the right side of the vertical mode may be referred to as a vertical right mode. The intra prediction mode located at the bottom of the horizontal mode may be referred to as a horizontal-below mode. For example, in FIG. 7, intra prediction modes in which the mode value is one of 27, 28, 29, 30, 31, 32, 33, and 34 may be vertical right modes 613. Intra prediction modes with a mode value of one of 2, 3, 4, 5, 6, 7, 8, and 9 may be horizontal bottom modes 616.

The non-directional mode may include a DC mode and a planar mode. For example, the mode value of the DC mode may be 1. The mode value of the planner mode may be zero.

The directional mode may include an angular mode. Among the plurality of intra prediction modes, a mode other than the DC mode and the planner mode may be a directional mode.

When the intra prediction mode is the DC mode, the prediction block may be generated based on an average of pixel values of the plurality of reference samples. For example, the value of a pixel of the prediction block may be determined based on an average of pixel values of the plurality of reference samples.

The number of intra prediction modes described above and the mode value of each intra prediction modes may be exemplary only. The number of intra prediction modes described above and the mode value of each intra prediction modes may be defined differently according to an embodiment, implementation, and / or need.

Checking whether samples included in the reconstructed neighboring block can be used as a reference sample of the target block to perform intra prediction on the target block may be performed. If there are samples of the neighboring block that are not available as reference samples of the target block, the values generated by copying and / or interpolation using at least one sample value of the samples included in the restored neighboring block are It can be replaced with sample values of samples that are not available as reference samples. If the value generated by copying and / or interpolation is replaced with the sample value of the sample, the sample can be used as a reference sample of the target block.

In intra prediction, a filter may be applied to at least one of a reference sample or a prediction sample based on at least one of an intra prediction mode and a size of a target block.

The type of filter applied to at least one of the reference sample or the prediction sample may vary according to at least one of the intra prediction mode of the target block, the size of the target block, and the shape of the target block. Types of filters may be classified according to one or more of the number of filter taps, the value of filter coefficients, and the filter strength.

When the intra prediction mode is the planner mode, in generating the prediction block of the target block, the top reference sample of the target sample, the left reference sample of the target sample, and the right top reference sample of the target block, according to the position in the prediction block of the prediction target sample, And a sample value of the prediction target sample using the weighted sum of the lower left reference samples of the target block.

When the intra prediction mode is the DC mode, in generating the prediction block of the target block, an average value of the top reference samples and the left reference samples of the target block may be used. In addition, filtering using the values of the reference samples may be performed on the specified rows or specified columns in the target block. The specified rows may be one or more top rows adjacent to the reference sample. The specified columns may be one or more left columns adjacent to the reference sample.

When the intra prediction mode is the directional mode, the prediction block may be generated using an upper reference sample, a left reference sample, a right upper reference sample, and / or a lower left reference sample.

Real unit interpolation may be performed to generate the above-described prediction sample.

The intra prediction mode of the target block may be predicted from the intra prediction mode of the neighboring block of the target block, and information used for prediction may be entropy encoded / decoded.

For example, if the intra prediction modes of the target block and the neighboring block are the same, it may be signaled that the intra prediction modes of the target block and the neighboring block are the same using a predefined flag.

For example, an indicator indicating the same intra prediction mode as the intra prediction mode of the target block among the intra prediction modes of the plurality of neighboring blocks may be signaled.

If the intra prediction modes of the target block and the neighboring block are different from each other, information of the intra prediction mode of the target block may be encoded and / or decoded using entropy encoding and / or decoding.

8 is a diagram for describing a position of a reference sample used in an intra prediction process.

8 shows the location of a reference sample used for intra prediction of a target block. Referring to FIG. 8, the reconstructed reference samples used for intra prediction of the target block include lower-left reference samples 831, left reference samples 833, and upper-above- left corner reference sample 835, upper reference samples 837, upper right reference samples 839, and the like.

For example, the left reference samples 833 may refer to a reconstructed reference pixel adjacent to the left side of the target block. The top reference samples 837 may refer to a reconstructed reference pixel adjacent to the top of the target block. The upper left corner reference sample 835 may refer to a reconstructed reference pixel located at the upper left corner of the target block. Also, the lower left reference samples 831 may refer to a reference sample located at the bottom of the left sample line among samples positioned on the same line as the left sample line composed of the left reference samples 833. The upper right reference samples 839 may refer to reference samples positioned to the right of the upper pixel line among samples positioned on the same line as the upper sample line composed of the upper reference samples 837.

When the size of the target block is N × N, the lower left reference samples 831, the left reference samples 833, the upper reference samples 837, and the upper right reference samples 839 may each be N pieces.

The prediction block may be generated through intra prediction on the target block. Generation of the predictive block may include determining a value of pixels of the predictive block. The size of the target block and the prediction block may be the same.

The reference sample used for intra prediction of the target block may vary according to the intra prediction mode of the target block. The direction of the intra prediction mode may indicate a dependency relationship between the reference samples and the pixels of the prediction block. For example, the value of the specified reference sample can be used as the value of the specified one or more pixels of the prediction block. In this case, the specified one or more specified pixels of the specified reference sample and prediction block may be samples and pixels designated by a straight line in the direction of the intra prediction mode. In other words, the value of the specified reference sample may be copied to the value of the pixel located in the reverse direction of the intra prediction mode. Alternatively, the pixel value of the prediction block may be a value of a reference sample located in the direction of the intra prediction mode based on the position of the pixel.

For example, when the intra prediction mode of the target block is a vertical mode having a mode value of 26, the upper reference samples 837 may be used for intra prediction. When the intra prediction mode is the vertical mode, the value of a pixel of the prediction block may be a value of a reference sample located vertically above the position of the pixel. Thus, the top reference samples 837 adjacent to the top of the target block can be used for intra prediction. In addition, the values of the pixels of one row of the prediction block may be the same as the values of the top reference samples 837.

For example, when the intra prediction mode of the target block is a horizontal mode having a mode value of 10, the left reference samples 833 may be used for intra prediction. When the intra prediction mode is the horizontal mode, the pixel value of the prediction block may be a value of a reference sample located horizontally on the left side with respect to the pixel. Thus, left reference samples 833 adjacent to the target block to the left may be used for intra prediction. In addition, the values of the pixels of one column of the prediction block may be the same as the values of the left reference samples 833.

For example, when the mode value of the intra prediction mode of the target block is 18, at least some of the left reference samples 833, the upper left corner reference sample 835, and at least some intra prediction of the top reference samples 837 are included. Can be used. When the mode value of the intra prediction mode is 18, the value of the pixel of the prediction block may be the value of the reference sample positioned diagonally on the upper left side with respect to the pixel.

In addition, when an intra prediction mode having a mode value of 27, 28, 29, 30, 31, 32, 33, or 34 is used, at least some of the upper right reference samples 839 may be used for intra prediction.

In addition, when an intra prediction mode having a mode value of 2, 3, 4, 5, 6, 7, 8, or 9 is used, at least some of the lower left reference samples 831 may be used for intra prediction.

In addition, when an intra prediction mode having a mode value of 11 to 25 is used, the upper left corner reference sample 835 may be used for intra prediction.

The reference sample used to determine the pixel value of one pixel of the prediction block may be one, or may be two or more.

As described above, the pixel value of the pixel of the prediction block may be determined according to the position of the reference sample indicated by the position of the pixel and the direction of the intra prediction mode. If the position of the reference sample indicated by the position of the pixel and the direction of the intra prediction mode is an integer position, the value of one reference sample indicated by the integer position may be used to determine the pixel value of the pixel of the prediction block.

If the position of the reference sample indicated by the position of the pixel and the direction of the intra prediction mode is not an integer position, an interpolated reference sample may be generated based on the two reference samples closest to the position of the reference sample. have. The value of the interpolated reference sample can be used to determine the pixel value of the pixel of the prediction block. In other words, when the position of the reference sample indicated by the position of the pixel of the prediction block and the direction of the intra prediction mode indicates between the two reference samples, an interpolated value is generated based on the values of the two samples. Can be.

The prediction block generated by the prediction may not be the same as the original target block. In other words, there may be a prediction error that is a difference between the target block and the prediction block, and the prediction error may exist between the pixels of the target block and the pixels of the prediction block.

Hereinafter, the terms "difference", "error" and "residual" may be used in the same sense and may be used interchangeably.

For example, in the case of directional intra prediction, the greater the distance between the pixel and the reference sample of the prediction block, the larger prediction error may occur. Discontinuity may occur between the prediction block and the neighboring block generated by such a prediction error.

Filtering on the prediction block may be used to reduce the prediction error. The filtering may be to adaptively apply a filter to a region that is considered to have a large prediction error in the prediction block. For example, an area considered to have a large prediction error may be a boundary of a prediction block. In addition, according to the intra prediction mode, an area considered to have a large prediction error among the prediction blocks may be different, and characteristics of the filter may be different.

9 is a diagram for explaining an embodiment of an inter prediction process.

The rectangle illustrated in FIG. 9 may represent an image (or a picture). In addition, arrows in FIG. 9 may indicate prediction directions. That is, the image may be encoded and / or decoded according to the prediction direction.

Each picture may be classified into an I picture (Intra Picture), a P picture (Uni-prediction Picture), and a B picture (Bi-prediction Picture). Each picture may be encoded and / or decoded according to an encoding type of each picture.

If the target image to be encoded is an I picture, the target image may be encoded using data in the image itself without inter prediction referring to another image. For example, an I picture can only be encoded with intra prediction.

When the target image is a P picture, the target image may be encoded through inter prediction using only a reference picture existing in one direction. Here, the unidirectional may be forward or reverse.

When the target picture is a B picture, the target picture may be encoded through inter prediction using reference pictures existing in both directions or inter prediction using reference pictures existing in one of the forward and reverse directions. Here, the bidirectional can be forward and reverse.

P pictures and B pictures that are encoded and / or decoded using the reference picture may be regarded as an image using inter prediction.

In the following, inter prediction in inter mode according to an embodiment is described in detail.

Inter prediction may be performed using motion information.

In the inter mode, the encoding apparatus 100 may perform inter prediction and / or motion compensation on a target block. The decoding apparatus 200 may perform inter prediction and / or motion compensation corresponding to inter prediction and / or motion compensation in the encoding apparatus 100 with respect to the target block.

The motion information on the target block may be derived during inter prediction by each of the encoding apparatus 100 and the decoding apparatus 200. The motion information may be derived using motion information of the restored neighboring block, motion information of the call block, and / or motion information of a block adjacent to the call block.

For example, the encoding apparatus 100 or the decoding apparatus 200 uses the motion information of the spatial candidate and / or the temporal candidate as the motion information of the target block to perform prediction and / or motion compensation. Can be done. The target block may mean a PU and / or a PU partition.

The spatial candidate may be a reconstructed block spatially adjacent to the target block.

The temporal candidate may be a reconstructed block corresponding to a target block in a collocated picture (col picture).

In inter prediction, the encoding apparatus 100 and the decoding apparatus 200 may improve encoding efficiency and decoding efficiency by using motion information of spatial candidates and / or temporal candidates. The motion information of the spatial candidate may be referred to as spatial motion information. The motion information of the temporal candidate may be referred to as temporal motion information.

Hereinafter, the motion information of the spatial candidate may be motion information of the PU including the spatial candidate. The motion information of the temporal candidate may be motion information of the PU including the temporal candidate. The motion information of the candidate block may be motion information of the PU including the candidate block.

Inter prediction may be performed using a reference picture.

The reference picture may be at least one of a previous picture of the target picture or a subsequent picture of the target picture. The reference picture may mean an image used for prediction of the target block.

In inter prediction, an area within a reference picture can be specified by using a reference picture index (or refIdx) indicating a reference picture, a motion vector to be described later, and the like. Here, the specified region in the reference picture may represent a reference block.

Inter prediction may select a reference picture, and may select a reference block corresponding to the target block within the reference picture. In addition, inter prediction may generate a prediction block for a target block using the selected reference block.

The motion information may be derived during inter prediction by each of the encoding apparatus 100 and the decoding apparatus 200.

The spatial candidate may be 1) present in the target picture, 2) already reconstructed through encoding and / or decoding, and 3) adjacent to the target block or located at the corner of the target block. The block located at the corner of the target block may be a block vertically adjacent to a neighboring block horizontally adjacent to the target block or a block horizontally adjacent to a neighboring block vertically adjacent to the target block. "Block located at the corner of the target block" may have the same meaning as "block adjacent to the corner of the target block". The "block located at the corner of the target block" may be included in the "block adjacent to the target block".

For example, a spatial candidate may be a reconstructed block located to the left of the target block, a reconstructed block located to the top of the target block, a reconstructed block located at the lower left corner of the target block, or a top right corner of the target block. It may be a reconstructed block or a reconstructed block located at the upper left corner of the target block.

Each of the encoding apparatus 100 and the decoding apparatus 200 may identify a block that exists at a position spatially corresponding to the target block in the col picture. The position of the target block in the target picture and the position of the identified block in the call picture may correspond to each other.

Each of the encoding apparatus 100 and the decoding apparatus 200 may determine a coll block existing at a predetermined relative position with respect to the identified block as a temporal candidate. The predefined relative position may be a position inside and / or outside of the identified block.

For example, the call block may include a first call block and a second call block. When the coordinates of the identified block are (xP, yP) and the size of the identified block is (nPSW, nPSH), the first call block may be a block located at coordinates (xP + nPSW, yP + nPSH). The second call block may be a block located at coordinates (xP + (nPSW >> 1), yP + (nPSH >> 1)). The second call block can optionally be used if the first call block is unavailable.

The motion vector of the target block may be determined based on the motion vector of the call block. Each of the encoding apparatus 100 and the decoding apparatus 200 may scale a motion vector of a call block. The scaled motion vector of the call block can be used as the motion vector of the target block. In addition, the motion vector of the motion information of the temporal candidate stored in the list may be a scaled motion vector.

The ratio of the motion vector of the target block and the motion vector of the call block may be equal to the ratio of the first temporal distance and the second temporal distance. The first temporal distance may be a distance between the reference picture and the target picture of the target block. The second temporal distance may be a distance between the reference picture and the call picture of the call block.

The derivation method of the motion information may vary according to the inter prediction mode of the target block. For example, as the inter prediction mode applied for inter prediction, there may be an enhanced motion vector predictor (AMVP) mode, a merge mode and a skip mode, a current picture reference mode, and the like. have. The merge mode may be referred to as a motion merge mode. In the following, each of the modes is described in detail.

1) AMVP Mode

When the AMVP mode is used, the encoding apparatus 100 may search for a similar block in the neighbor of the target block. The encoding apparatus 100 may obtain the prediction block by performing prediction on the target block using the retrieved motion information of the similar block. The encoding apparatus 100 may encode a residual block that is a difference between the target block and the prediction block.

1-1) Preparation of predictive motion vector candidate list

When the AMVP mode is used as the prediction mode, each of the encoding apparatus 100 and the decoding apparatus 200 may generate a predicted motion vector candidate list using the motion vector of the spatial candidate, the motion vector of the temporal candidate, and the zero vector. have. The predictive motion vector candidate list may include one or more predictive motion vector candidates. At least one of the motion vector of the spatial candidate, the motion vector of the temporal candidate, and the zero vector may be determined and used as the predictive motion vector candidate.

In the following, the terms “predictive motion vector (candidate)” and “motion vector (candidate)” may be used in the same sense and may be used interchangeably.

Hereinafter, the terms "predictive motion vector candidate" and "AMVP candidate" may be used in the same sense and may be used interchangeably.

Hereinafter, the terms "predictive motion vector candidate list" and "AMVP candidate list" may be used in the same sense and may be used interchangeably.

The spatial candidate may include the reconstructed spatial neighboring block. In other words, the motion vector of the reconstructed neighboring block may be referred to as a spatial prediction motion vector candidate.

The temporal candidate may include a call block and a block adjacent to the call block. In other words, the motion vector of the call block or the motion vector of the block adjacent to the call block may be referred to as a temporal prediction motion vector candidate.

The zero vector may be a (0, 0) motion vector.

The predictive motion vector candidate may be a motion vector predictor for prediction of the motion vector. Also, in the encoding apparatus 100, the predicted motion vector candidate may be a motion vector initial search position.

1-2) Searching for Motion Vectors Using Predictive Motion Vector Candidate List

The encoding apparatus 100 may determine a motion vector to be used for encoding a target block within a search range using the predictive motion vector candidate list. Also, the encoding apparatus 100 may determine a prediction motion vector candidate to be used as a prediction motion vector of the target block among the prediction motion vector candidates of the prediction motion vector candidate list.

The motion vector to be used for encoding the target block may be a motion vector that can be encoded at a minimum cost.

In addition, the encoding apparatus 100 may determine whether to use the AMVP mode in encoding the target block.

1-3) Transmission of Inter Prediction Information

The encoding apparatus 100 may generate a bitstream including inter prediction information required for inter prediction. The decoding apparatus 200 may perform inter prediction on the target block by using inter prediction information of the bitstream.

The inter prediction information includes 1) mode information indicating whether the AMVP mode is used, 2) a predicted motion vector index, 3) a motion vector difference (MVD), 4) a reference direction, and 5) a reference picture index. can do.

In the following, the terms "predictive motion vector index" and "AMVP index" may be used in the same sense and may be used interchangeably.

In addition, the inter prediction information may include a residual signal.

When the mode information indicates that the AMVP mode is used, the decoding apparatus 200 may obtain a predicted motion vector index, a motion vector difference, a reference direction, and a reference picture index from the bitstream through entropy decoding.

The prediction motion vector index may indicate a prediction motion vector candidate used for prediction of a target block among prediction motion vector candidates included in the prediction motion vector candidate list.

1-4) Inter Prediction in AMVP Mode Using Inter Prediction Information

The decoding apparatus 200 may derive the predictive motion vector candidate using the predictive motion vector candidate list, and determine the motion information of the target block based on the derived predicted motion vector candidate.

The decoding apparatus 200 may determine the motion vector candidate for the target block from among the prediction motion vector candidates included in the prediction motion vector candidate list using the prediction motion vector index. The decoding apparatus 200 may select a prediction motion vector candidate indicated by the prediction motion vector index from among prediction motion vector candidates included in the prediction motion vector candidate list as the prediction motion vector of the target block.

The motion vector to be actually used for inter prediction of the target block may not match the prediction motion vector. The motion vector to be actually used for inter prediction of the target block and MVD may be used to indicate the difference between the predicted motion vector. The encoding apparatus 100 may derive a predictive motion vector similar to the motion vector actually used for inter prediction of the target block in order to use the MVD of the smallest possible size.

The MVD may be a difference between the motion vector and the predicted motion vector of the target block. The encoding apparatus 100 may calculate an MVD and entropy encode the MVD.

The MVD may be transmitted from the encoding apparatus 100 to the decoding apparatus 200 through a bitstream. The decoding apparatus 200 may decode the received MVD. The decoding apparatus 200 may derive the motion vector of the target block by adding the decoded MVD and the predictive motion vector. In other words, the motion vector of the target block derived from the decoding apparatus 200 may be the sum of the entropy decoded MVD and the motion vector candidate.

The reference direction may point to the reference picture list used for prediction of the target block. For example, the reference direction may point to one of the reference picture list L0 and the reference picture list L1.

The reference direction only points to the reference picture list used for prediction of the target block, but may not indicate that the directions of the reference pictures are limited in the forward direction or the backward direction. In other words, each of the reference picture list L0 and the reference picture list L1 may include pictures in the forward and / or reverse direction.

That the reference direction is uni-direction may mean that one reference picture list is used. The bi-direction of the reference direction may mean that two reference picture lists are used. That is to say, the reference direction may indicate that only the reference picture list L0 is used, that only the reference picture list L1 is used and one of the two reference picture lists.

The reference picture index may indicate a reference picture used for prediction of a target block among reference pictures of the reference picture list. The reference picture index may be entropy encoded by the encoding apparatus 100. The entropy coded reference picture index may be signaled from the encoding apparatus 100 to the decoding apparatus 200 through a bitstream.

When two reference picture lists are used for prediction of the target block. One reference picture index and one motion vector may be used for each reference picture list. In addition, when two reference picture lists are used for prediction of the target block, two prediction blocks may be specified for the target block. For example, the (final) prediction block of the target block may be generated through an average or weighted-sum of two prediction blocks for the target block.

The motion vector of the target block may be derived by the prediction motion vector index, the MVD, the reference direction, and the reference picture index.

The decoding apparatus 200 may generate a prediction block for the target block based on the derived motion vector and the reference picture index. For example, the prediction block may be a reference block indicated by the derived motion vector in the reference picture indicated by the reference picture index.

By encoding the predicted motion vector index and the MVD without encoding the motion vector itself of the target block, the amount of bits transmitted from the encoding device 100 to the decoding device 200 may be reduced, and encoding efficiency may be improved.

The motion information of the neighboring block reconstructed with respect to the target block may be used. In a particular inter prediction mode, the encoding apparatus 100 may not separately encode motion information about the target block. The motion information of the target block is not encoded, and other information capable of deriving the motion information of the target block through the motion information of the reconstructed neighboring block may be encoded instead. As other information is encoded instead, the amount of bits transmitted to the decoding apparatus 200 may be reduced, and encoding efficiency may be improved.

For example, the inter prediction mode in which the motion information of the target block is not directly encoded may include a skip mode and / or a merge mode. In this case, the encoding apparatus 100 and the decoding apparatus 200 may use an identifier and / or an index indicating which motion information of which unit among the reconstructed neighboring units is used as the motion information of the target unit.

2) merge mode

Merge is a method of deriving the motion information of the target block. Merge may mean merging of motions for a plurality of blocks. Merge may mean applying motion information of one block to other blocks. In other words, the merge mode may mean a mode in which the motion information of the target block is derived from the motion information of the neighboring block.

When the merge mode is used, the encoding apparatus 100 may predict the motion information of the target block by using the motion information of the spatial candidate and / or the motion information of the temporal candidate. The spatial candidate may include a reconstructed spatial neighboring block spatially adjacent to the target block. The spatial neighboring block may include a left neighboring block and a top neighboring block. Temporal candidates may include call blocks. The terms “spatial candidate” and “spatial merge candidate” may be used interchangeably and may be used interchangeably. The terms "temporal candidate" and "temporary merge candidate" may be used interchangeably and may be used interchangeably.

The encoding apparatus 100 may obtain a prediction block through prediction. The encoding apparatus 100 may encode a residual block that is a difference between a target block and a prediction block.

2-1) Creating a merge candidate list

When the merge mode is used, each of the encoding apparatus 100 and the decoding apparatus 200 may generate the merge candidate list using the motion information of the spatial candidate and / or the motion information of the temporal candidate. The motion information may include 1) a motion vector, 2) a reference picture index, and 3) a reference direction. The reference direction may be unidirectional or bidirectional.

The merge candidate list may include merge candidates. The merge candidate may be motion information. In other words, the merge candidate list may be a list in which motion information is stored.

The merge candidates may be motion information such as temporal candidates and / or spatial candidates. In addition, the merge candidate list may include a new merge candidate generated by a combination of merge candidates already present in the merge candidate list. In other words, the merge candidate list may include new motion information generated by a combination of motion information already present in the merge candidate list.

Merge candidates may be specified modes that derive inter prediction information. The merge candidate may be information indicating a specified mode for deriving inter prediction information. Inter prediction information of the target block may be derived according to the specified mode indicated by the merge candidate. In this case, the specified mode may include a process of deriving a series of inter prediction information. This specified mode may be an inter prediction information derivation mode or a motion information derivation mode.

Inter prediction information of the target block may be derived according to a mode indicated by the merge candidate selected by the merge index among the merge candidates in the merge candidate list.

For example, the motion information derivation modes in the merge candidate list may be at least one of 1) motion information derivation mode in sub-block units and 2) affine motion information derivation mode.

In addition, the merge candidate list may include motion information of the zero vector. The zero vector may be referred to as a zero merge candidate.

In other words, the motion information in the merge candidate list may include: 1) motion information of a spatial candidate, 2) motion information of a temporal candidate, 3) motion information generated by a combination of motion information already present in the merge candidate list, and 4) zero vector. It may be at least one of.

The motion information may include 1) a motion vector, 2) a reference picture index, and 3) a reference direction. The reference direction may be referred to as an inter prediction indicator. The reference direction may be unidirectional or bidirectional. The unidirectional reference direction may indicate L0 prediction or L1 prediction.

The merge candidate list may be generated before prediction by the merge mode is performed.

The number of merge candidates in the merge candidate list may be predefined. The encoding apparatus 100 and the decoding apparatus 200 may add the merge candidates to the merge candidate list according to a predefined method and a predefined rank so that the merge candidate list has a predetermined number of merge candidates. The merge candidate list of the encoding apparatus 100 and the merge candidate list of the decoding apparatus 200 may be identical through the predefined scheme and the predefined ranking.

Merge may be applied in a CU unit or a PU unit. When merging is performed in a CU unit or a PU unit, the encoding apparatus 100 may transmit a bitstream including predefined information to the decoding apparatus 200. For example, the predefined information may include 1) information indicating whether or not to perform merge for each block partition, and 2) any block among blocks that are spatial candidates and / or temporal candidates for the target block. It may include information about whether it is.

2-2) Searching for Motion Vectors Using Merge Candidate Lists

The encoding apparatus 100 may determine a merge candidate to be used for encoding the target block. For example, the encoding apparatus 100 may perform predictions on the target block by using merge candidates of the merge candidate list and generate residual blocks for the merge candidates. The encoding apparatus 100 may use a merge candidate for the encoding of the target block, which requires a minimum cost in prediction and encoding of the residual block.

In addition, the encoding apparatus 100 may determine whether to use the merge mode in encoding the target block.

2-3) Inter prediction information transmission

The encoding apparatus 100 may generate a bitstream including inter prediction information required for inter prediction. The encoding apparatus 100 may generate entropy encoded inter prediction information by performing entropy encoding on the inter prediction information, and may transmit a bitstream including the entropy encoded inter prediction information to the decoding apparatus 200. Entropy encoded inter prediction information may be signaled from the encoding apparatus 100 to the decoding apparatus 200 through the bitstream.

The decoding apparatus 200 may perform inter prediction on the target block by using inter prediction information of the bitstream.

The inter prediction information may include 1) mode information indicating whether to use the merge mode and 2) the merge index.

In addition, the inter prediction information may include a residual signal.

The decoding apparatus 200 may obtain the merge index from the bitstream only when the mode information indicates that the merge mode is used.

The mode information may be a merge flag. The unit of the mode information may be a block. The information on the block may include mode information, and the mode information may indicate whether a merge mode is applied to the block.

The merge index may indicate a merge candidate used for prediction of the target block among merge candidates included in the merge candidate list. Alternatively, the merge index may indicate which one of neighboring blocks spatially or temporally adjacent to the target block is performed.

The encoding apparatus 100 may select a merge candidate having the highest encoding performance among merge candidates included in the merge candidate list, and set a merge index value to indicate the selected merge candidate.

2-4) Inter prediction in merge mode using inter prediction information

The decoding apparatus 200 may perform prediction on the target block by using the merge candidate indicated by the merge index among the merge candidates included in the merge candidate list.

The motion vector of the target block may be specified by the motion vector of the merge candidate indicated by the merge index, the reference picture index, and the reference direction.

3) Skip Mode

The skip mode may be a mode in which the motion information of the spatial candidate or the motion information of the temporal candidate is applied to the target block as it is. Also, the skip mode may be a mode that does not use the residual signal. In other words, when the skip mode is used, the reconstructed block may be a prediction block.

The difference between the merge mode and the skip mode may be whether to transmit or use the residual signal. In other words, the skip mode may be similar to the merge mode except that no residual signal is transmitted or used.

When the skip mode is used, the encoding apparatus 100 transmits, to the decoding apparatus 200, information indicating which of the blocks that are spatial candidates or temporal candidates is used as the motion information of the target block. Can transmit The encoding apparatus 100 may generate entropy encoded information by performing entropy encoding on the information, and may signal the entropy encoded information to the decoding apparatus 200 through a bitstream.

In addition, when the skip mode is used, the encoding apparatus 100 may not transmit other syntax element information such as MVD to the decoding apparatus 200. For example, when used with a skip mode, the encoding apparatus 100 may not signal a syntax element regarding at least one of an MVD, a coded block flag, and a transform coefficient level to the decoding apparatus 200.

3-1) Creation of merge candidate list

Skip mode can also use the merge candidate list. In other words, the merge candidate list can be used in both merge mode and skip mode. In this regard, the merge candidate list may be named "skip candidate list" or "merge / skip candidate list."

Alternatively, the skip mode may use a separate candidate list different from the merge mode. In this case, in the following description, the merge candidate list and the merge candidate may be replaced with the skip candidate list and the skip candidate, respectively.

The merge candidate list may be generated before the prediction by the skip mode is performed.

3-2) Motion Vector Search Using Merge Candidate List

The encoding apparatus 100 may determine a merge candidate to be used for encoding the target block. For example, the encoding apparatus 100 may perform predictions on the target block by using merge candidates of the merge candidate list. The encoding apparatus 100 may use a merge candidate that requires a minimum cost in prediction for encoding a target block.

In addition, the encoding apparatus 100 may determine whether to use the skip mode in encoding the target block.

3-3) Inter prediction information transmission

The encoding apparatus 100 may generate a bitstream including inter prediction information required for inter prediction. The decoding apparatus 200 may perform inter prediction on the target block by using inter prediction information of the bitstream.

The inter prediction information may include 1) mode information indicating whether to use a skip mode and 2) a skip index.

The skip index may be the same as the merge index described above.

When the skip mode is used, the target block may be encoded without a residual signal. The inter prediction information may not include the residual signal. Or, the bitstream may not include the residual signal.

The decoding apparatus 200 may obtain the skip index from the bitstream only when the mode information indicates that the skip mode is used. As described above, the merge index and the skip index may be the same. The decoding apparatus 200 may obtain the skip index from the bitstream only when the mode information indicates that the merge mode or the skip mode is used.

The skip index may indicate a merge candidate used for prediction of the target block among merge candidates included in the merge candidate list.

3-4) Inter Prediction in Skip Mode Using Inter Prediction Information

The decoding apparatus 200 may perform prediction on the target block by using the merge candidate indicated by the skip index among the merge candidates included in the merge candidate list.

The motion vector of the target block may be specified by the motion vector of the merge candidate indicated by the skip index, the reference picture index, and the reference direction.

4) Current picture reference mode

The current picture reference mode may refer to a prediction mode that uses a pre-restored region in the target picture to which the target block belongs.

A motion vector may be used to specify the pre-restored region. Whether the target block is encoded in the current picture reference mode may be determined using the reference picture index of the target block.

A flag or index indicating whether the target block is a block encoded in the current picture reference mode may be signaled from the encoding device 100 to the decoding device 200. Alternatively, whether the target block is a block encoded in the current picture reference mode may be inferred through the reference picture index of the target block.

When the target block is encoded in the current picture reference mode, the target picture may exist at a fixed position or any position within the reference picture list for the target block.

For example, the fixed position may be the position where the value of the reference picture index is 0 or the last position.

When the target picture exists at an arbitrary position in the reference picture list, a separate reference picture index indicating this arbitrary position may be signaled from the encoding apparatus 100 to the decoding apparatus 200.

In the above-described AMVP mode, merge mode, and skip mode, the motion information to be used for prediction of the target block among the motion information in the list may be specified through an index to the list.

In order to improve encoding efficiency, the encoding apparatus 100 may signal only an index of an element causing a minimum cost in inter prediction of a target block among elements of a list. The encoding apparatus 100 may encode the index and may signal the encoded index.

Therefore, the aforementioned lists (that is, the prediction motion vector candidate list and the merge candidate list) may be derived in the same manner based on the same data in the encoding apparatus 100 and the decoding apparatus 200. Here, the same data may include the reconstructed picture and the reconstructed block. Also, to specify elements by index, the order of the elements in the list may have to be constant.

10 illustrates spatial candidates according to an example.

In FIG. 10, the positions of the spatial candidates are shown.

The large block in the middle may represent the target block. Five small blocks may represent spatial candidates.

The coordinates of the target block may be (xP, yP), and the size of the target block may be (nPSW, nPSH).

The spatial candidate A ₀ may be a block adjacent to the lower left corner of the target block. A ₀ may be a block occupying a pixel of coordinates (xP − 1, yP + nPSH + 1).

The spatial candidate A ₁ may be a block adjacent to the left side of the target block. A ₁ may be the lowest block among blocks adjacent to the left side of the target block. Alternatively, A ₁ may be a block adjacent to the top of A ₀ . A ₁ may be a block occupying a pixel of coordinates (xP-1, yP + nPSH).

The spatial candidate B ₀ may be a block adjacent to the upper right corner of the target block. B ₀ may be a block occupying a pixel of coordinates (xP + nPSW + 1, yP-1).

The spatial candidate B ₁ may be a block adjacent to the top of the target block. B ₁ may be the rightmost block among blocks adjacent to the top of the target block. Alternatively, B ₁ may be a block adjacent to the left side of B ₀ . B ₁ may be a block occupying a pixel of coordinates (xP + nPSW, yP-1).

The spatial candidate B ₂ may be a block adjacent to the upper left corner of the target block. B ₂ may be a block occupying a pixel of coordinates (xP-1, yP-1).

Judgment of the availability of spatial and temporal candidates

In order to include the motion information of the spatial candidate or the motion information of the temporal candidate in the list, it is determined whether the motion information of the spatial candidate or the motion information of the temporal candidate is available.

In the following, the candidate block may include a spatial candidate and a temporal candidate.

For example, the above determination may be made by sequentially applying steps 1) to 4) below.

Step 1) If the PU including the candidate block is outside the boundary of the picture, the availability of the candidate block may be set to false. "Availability is set to false" may mean the same as "set to unavailable".

Step 2) If the PU containing the candidate block is outside the boundary of the slice, the availability of the candidate block may be set to false. If the target block and the candidate block are located in different slices, the availability of the candidate block may be set to false.

Step 3) If the PU containing the candidate block is outside the boundary of the tile, the availability of the candidate block may be set to false. If the target block and the candidate block are located in different tiles, the availability of the candidate block may be set to false.

Step 4) If the prediction mode of the PU including the candidate block is an intra prediction mode, the availability of the candidate block may be set to false. If the PU including the candidate block does not use inter prediction, the availability of the candidate block may be set to false.

11 illustrates an addition order of spatial information of motion candidates to a merge list according to an example.

As shown in FIG. 11, in adding motion information of spatial candidates to a merge list, an order of A ₁ , B ₁ , B ₀ , A _0, and B ₂ may be used. That is, motion information of available spatial candidates may be added to the merge list in the order of A ₁ , B ₁ , B ₀ , A _0, and B ₂ .

Derivation method of merge list in merge mode and skip mode

As described above, the maximum number of merge candidates in the merge list may be set. The maximum number set is indicated by N. The set number may be transmitted from the encoding apparatus 100 to the decoding apparatus 200. The slice header of the slice may include N. In other words, the maximum number of merge candidates of the merge list for the target block of the slice may be set by the slice header. For example, by default, the value of N may be five.

The motion information (ie, merge candidate) may be added to the merge list in the order of steps 1) to 4) below.

Step 1) Available spatial candidates among the spatial candidates may be added to the merge list. The motion information of the available spatial candidates may be added to the merge list in the order shown in FIG. 10. In this case, when the motion information of the available spatial candidates overlaps with other motion information already existing in the merge list, the motion information may not be added to the merge list. Checking whether it overlaps with other motion information present in the list may be abbreviated as "redundancy check".

The added motion information may be up to N pieces.

Step 2) If the number of motion information in the merge list is smaller than N and a temporal candidate is available, the motion information of the temporal candidate may be added to the merge list. At this time, if the motion information of the available temporal candidate overlaps with other motion information already existing in the merge list, the motion information may not be added to the merge list.

Step 3) If the number of motion information in the merge list is less than N and the type of the target slice is "B", the combined motion information generated by the combined bi-prediction is added to the merge list. Can be.

The target slice may be a slice including the target block.

The combined motion information may be a combination of L0 motion information and L1 motion information. The L0 motion information may be motion information referring only to the reference picture list L0. The L1 motion information may be motion information referring only to the reference picture list L1.

Within the merge list, the L0 motion information may be one or more. Also, within the merge list, there may be one or more L1 motion information.

The combined motion information may be one or more. Which L0 motion information and which L1 motion information among one or more L0 motion information and one or more L1 motion information are used in generating the combined motion information may be defined. One or more combined motion information may be generated in a predefined order by combined bidirectional prediction using a pair of different motion information in the merge list. One of the pairs of different motion information may be L0 motion information and the other may be L1 motion information.

For example, the combined motion information added first may be a combination of L0 motion information having a merge index of 0 and L1 motion information having a merge index of 1. If the motion information having the merge index of 0 is not the L0 motion information or the motion information having the merge index of 1 is not the L1 motion information, the combined motion information may not be generated and added. Next, the additional motion information may be a combination of L0 motion information having a merge index of 1 and L1 motion information having a merge index of 0. The following specific combinations may follow other combinations in the field of video encoding / decoding.

In this case, when the combined motion information is overlapped with other motion information already existing in the merge list, the combined motion information may not be added to the merge list.

Step 4) If the number of motion information in the merge list is smaller than N, zero vector motion information may be added to the merge list.

The zero vector motion information may be motion information in which the motion vector is a zero vector.

The zero vector motion information may be one or more. The reference picture indices of the one or more zero vector motion information may be different from each other. For example, the value of the reference picture index of the first zero vector motion information may be zero. The value of the reference picture index of the second zero vector motion information may be one.

The number of zero vector motion information may be equal to the number of reference pictures in the reference picture list.

The reference direction of the zero vector motion information may be bidirectional. Both motion vectors may be zero vectors. The number of zero vector motion information may be smaller than the number of reference pictures in the reference picture list L0 and the number of reference pictures in the reference picture list L1. Alternatively, when the number of reference pictures in the reference picture list L0 and the number of reference pictures in the reference picture list L1 are different from each other, a unidirectional reference direction may be used for the reference picture index that can be applied to only one reference picture list.

The encoding apparatus 100 and / or the decoding apparatus 200 may sequentially add zero vector motion information to the merge list while changing the reference picture index.

When the zero vector motion information overlaps with other motion information already existing in the merge list, the zero vector motion information may not be added to the merge list.

The order of steps 1) to 4) described above is merely exemplary, and the order between the steps may be interchanged. In addition, some of the steps may be omitted depending on predefined conditions.

Derivation Method of Predictive Motion Vector Candidate List in AMVP Mode

The maximum number of predicted motion vector candidates in the predicted motion vector candidate list may be predefined. The predefined maximum number is denoted by N. For example, the predefined maximum number may be two.

The motion information (ie, the predicted motion vector candidate) may be added to the predicted motion vector candidate list in the order of steps 1) to 3) below.

Step 1) Available spatial candidates of the spatial candidates may be added to the predicted motion vector candidate list. Spatial candidates may include a first spatial candidate and a second spatial candidate.

The first spatial candidate may be one of A ₀ , A ₁ , scaled A _0, and scaled A ₁ . The second spatial candidate may be one of B ₀ , B ₁ , B ₂ , scaled B ₀ , scaled B _1, and scaled B ₂ .

The motion information of the available spatial candidates may be added to the predicted motion vector candidate list in the order of the first spatial candidate and the second spatial candidate. At this time, if the motion information of the available spatial candidates overlaps with other motion information already existing in the predicted motion vector candidate list, the motion information may not be added to the predicted motion vector candidate list. In other words, when the value of N is 2, if the motion information of the second spatial candidate is the same as the motion information of the first spatial candidate, the motion information of the second spatial candidate may not be added to the predicted motion vector candidate list.

The added motion information may be up to N pieces.

Step 2) If the number of motion information in the predicted motion vector candidate list is smaller than N and a temporal candidate is available, motion information of the temporal candidate may be added to the predicted motion vector candidate list. At this time, if the motion information of the available temporal candidate overlaps with other motion information already existing in the predicted motion vector candidate list, the motion information may not be added to the predicted motion vector candidate list.

Step 3) If the number of motion information in the predicted motion vector candidate list is smaller than N, zero vector motion information may be added to the predicted motion vector candidate list.

The zero vector motion information may be one or more. The reference picture indices of the one or more zero vector motion information may be different from each other.

The encoding apparatus 100 and / or the decoding apparatus 200 may sequentially add zero vector motion information to the predicted motion vector candidate list while changing the reference picture index.

When zero vector motion information overlaps with other motion information already existing in the predicted motion vector candidate list, the zero vector motion information may not be added to the predicted motion vector candidate list.

The description of the zero vector motion information described above with respect to the merge list may also be applied to the zero vector motion information. Duplicate explanations are omitted.

The order of steps 1) to 3) described above is merely illustrative, and the order between the steps may be interchanged. In addition, some of the steps may be omitted depending on predefined conditions.

12 illustrates a process of transform and quantization according to an example.

As illustrated in FIG. 12, a quantized level may be generated by performing a transform and / or quantization process on the residual signal.

The residual signal may be generated by the difference between the original block and the prediction block. Here, the prediction block may be a block generated by intra prediction or inter prediction.

The residual signal may be converted into the frequency domain through a conversion process that is part of the quantization process.

Transform kernels used for the transformation may include various DCT kernels, such as the Discrete Cosine Transform (DCT) type 2 (DCT-II), and the Discrete Sine Transform (DST) kernel. .

Such transform kernels may perform a separable transform or a 2D (2D) non-separable transform on the residual signal. The separable transform may be a transform that performs a 1D (1D) transformation on the residual signal in each of the horizontal direction and the vertical direction.

DCT types and DST types that are adaptively used for 1D conversion may include DCT-V, DCT-VIII, DST-I, and DST-VII in addition to DCT-II as shown in Tables 3 and 4, respectively, below. have.

TABLE 3

TABLE 4

As indicated in Tables 3 and 4, a transform set may be used in deriving a DCT type or a DST type to be used for the transformation. Each transform set may include a plurality of transform candidates. Each conversion candidate may be a DCT type or a DST type.

Table 5 below shows an example of a transform set applied to the horizontal direction and a transform set applied to the vertical direction according to the intra prediction mode.

TABLE 5

In Table 5, the number of the vertical direction transform set and the number of the horizontal direction transform set applied to the horizontal direction of the residual signal according to the intra prediction mode of the target block are displayed.

As illustrated in Table 5, transform sets applied to the horizontal direction and the vertical direction may be defined according to the intra prediction mode of the target block. The encoding apparatus 100 may perform transform and inverse transform on the residual signal by using a transform included in a transform set corresponding to the intra prediction mode of the target block. In addition, the decoding apparatus 200 may perform inverse transform on the residual signal using a transform included in a transform set corresponding to the intra prediction mode of the target block.

In this transform and inverse transform, the transform set applied to the residual signal may be determined as illustrated in Tables 3 and 4 and may not be signaled. The transformation indication information may be signaled from the encoding apparatus 100 to the decoding apparatus 200. The transform indication information may be information indicating which transform candidate among a plurality of transform candidates included in a transform set applied to the residual signal is used.

For example, when the size of the target block is 64x64 or less, three sets of transforms may be configured as in the example of Table 4 according to the intra prediction mode. The optimal transform method can be selected from among all nine multiple transform methods due to the combination of three transforms in the horizontal direction and three transforms in the vertical direction. The encoding efficiency may be improved by encoding and / or decoding the residual signal using such an optimal conversion method.

In this case, for at least one of the vertical transform and the horizontal transform, information on which transform among transforms belonging to the transform set is used may be entropy encoded and / or decoded. Truncated unary binarization may be used for encoding and / or decoding such information.

A method using various transforms as described above may be applied to a residual signal generated by intra prediction or inter prediction.

The transform may include at least one of a primary transform and a secondary transform. The transform coefficients may be generated by performing the primary transform on the residual signal, and the secondary transform coefficients may be generated by performing the secondary transform on the transform coefficients.

The primary transform can be named primary. In addition, the primary transform may be referred to as an adaptive multiple transform (AMT). The AMT may mean that different transformations are applied to each of the 1D directions (ie, the vertical direction and the horizontal direction) as described above.

The secondary transformation may be a transformation for improving the energy concentration of the transformation coefficients generated by the primary transformation. Secondary transforms can be separable transforms or non-separable transforms like primary transforms. The non-separable transform can be a Non-Separable Secondary Transform (NSST).

The primary transformation may be performed using at least one of a plurality of predefined transformation methods. For example, the predefined plurality of transformation methods may include a discrete cosine transform (DCT), a discrete sine transform (DST), and a karhunen-loeve transform (KLT) based transform. It may include.

In addition, the primary transform may be a transform having various types according to a kernel function defining a DCT or a DST.

For example, the primary transform may include transforms such as DCT-2, DCT-5, DCT-7, DST-1 and DST-8 according to the transform kernels set forth in Table 6 below. In Table 6, various transform types and transform kernel functions for Multiple Transform Selection (MTS) are illustrated.

MTS may mean that a combination of one or more DCT and / or DST conversion kernels is selected for transforming the residual signal in the horizontal and / or vertical direction.

TABLE 6

In Table 6, i and j may be an integer value of 0 or more and N-1 or less.

A secondary transform may be performed on the transform coefficients generated by performing the primary transform.

As in the primary transform, a transform set can be defined in the secondary transform. Methods for deriving and / or determining a transform set as described above may be applied to the secondary transform as well as the primary transform.

The primary transform and the secondary transform can be determined for the specified object.

For example, the first order and second order transforms can be applied to one or more signal components of a luma component and a chroma component. Whether to apply the first transform and / or the second transform may be determined according to at least one of coding parameters for the target block and / or the neighboring block. For example, whether to apply the primary transform and / or the secondary transform may be determined by the size and / or shape of the target block.

In the encoding apparatus 100 and the decoding apparatus 200, transformation information indicating a transformation method used for a target may be derived by using specified information.

For example, the transformation information may include an index of the transformation to be used for the primary and / or secondary transformation. Alternatively, the transform information may indicate that the primary transform and / or the secondary transform are not used.

For example, when the target of the primary transform and the secondary transform is the target block, the transform method (s) applied to the primary transform and / or secondary transform indicated by the transform information is applied to the target block and / or the neighboring block. It may be determined according to at least one of the coding parameters for.

Alternatively, the transform information for the specified object may be signaled from the encoding apparatus 100 to the decoding apparatus 200.

For example, whether a primary transform is used, an index indicating a primary transform, whether a secondary transform is used, and an index indicating a secondary transform may be derived as transform information in the decoding apparatus 200 for one CU. have. Alternatively, transformation information indicating whether a primary transform is used, an index indicating a primary transform, whether a secondary transform is used, and an index indicating a secondary transform may be signaled for one CU.

Quantized transform coefficients (ie, quantized levels) may be generated by performing quantization on the result or residual signal generated by performing the primary and / or secondary transform.

13 illustrates diagonal scanning according to an example.

14 illustrates horizontal scanning according to an example.

15 illustrates vertical scanning according to an example.

The quantized transform coefficients may be scanned according to at least one of (up-right) diagonal scanning, vertical scanning, and horizontal scanning, according to at least one of intra prediction mode, block size, and block shape. The block may be a transform unit.

Each scanning can start at a specified start point and end at a specified end point.

For example, the quantized transform coefficients may be changed to a one-dimensional vector form by scanning the coefficients of the block using the diagonal scanning of FIG. 13. Alternatively, the horizontal scanning of FIG. 14 or the vertical scanning of FIG. 15 may be used instead of diagonal scanning depending on the size of the block and / or the intra prediction mode.

Vertical scanning may be a two-dimensional block shape coefficient scanning in the column direction. Horizontal scanning may be scanning two-dimensional block shape coefficients in a row direction.

In other words, depending on the size of the block and / or the inter prediction mode, which of the diagonal scanning, the vertical scanning, and the horizontal scanning may be used may be determined.

As illustrated in FIGS. 13, 14, and 15, the quantized transform coefficients may be scanned along a diagonal direction, a horizontal direction, or a vertical direction.

The quantized transform coefficients may be represented in a block form. The block may include a plurality of sub blocks. Each sub block may be defined according to a minimum block size or a minimum block shape.

In scanning, a scanning order according to the type or direction of scanning may first be applied to the sub blocks. In addition, the scanning order according to the scanning direction may be applied to the quantized transform coefficients in the subblock.

For example, as shown in FIGS. 13, 14, and 15, when the size of the target block is 8x8, the transform coefficients quantized by the first-order transform, the second-order transform, and the quantization of the residual signal of the target block are Can be generated. Subsequently, one scanning order of three scanning orders may be applied to four 4x4 subblocks, and quantized transform coefficients may be scanned according to the scanning order for each 4x4 subblock.

The scanned quantized transform coefficients can be entropy coded and the bitstream can include entropy coded quantized transform coefficients.

The decoding apparatus 200 may generate quantized transform coefficients through entropy decoding on the bitstream. The quantized transform coefficients may be aligned in the form of a two-dimensional block through inverse scanning. In this case, at least one of the upper right diagonal scan, the vertical scan, and the horizontal scan may be performed as a method of reverse scanning.

In the decoding apparatus 200, inverse quantization may be performed on the quantized transform coefficients. Depending on whether the second inverse transform is performed, the second inverse transform may be performed on the result generated by the inverse quantization. Further, depending on whether the first inverse transform is performed, the first inverse transform may be performed on the result generated by the second inverse transform. The reconstructed residual signal may be generated by performing the first order inverse on the result generated by the second inverse transform.

16 is a structural diagram of an encoding apparatus according to an embodiment.

The encoding apparatus 1600 may correspond to the encoding apparatus 100 described above.

The encoding apparatus 1600 may include a processor 1610, a memory 1630, a user interface (UI) input device 1650, a UI output device 1660, and storage that communicate with each other through a bus 1690. (1640). In addition, the encoding apparatus 1600 may further include a communication unit 1620 connected to the network 1699.

The processor 1610 may be a semiconductor device that executes processing instructions stored in the central processing unit (CPU), the memory 1630, or the storage 1640. The processor 1610 may be at least one hardware processor.

The processor 1610 may generate and process a signal, data, or information input to the encoding apparatus 1600, output from the encoding apparatus 1600, or used in the encoding apparatus 1600, and may include a signal, Inspection, comparison, and judgment related to data or information can be performed. In other words, in an embodiment, generation and processing of data or information, and inspection, comparison, and determination related to the data or information may be performed by the processor 1610.

The processor 1610 may include an inter predictor 110, an intra predictor 120, a switch 115, a subtractor 125, a transformer 130, a quantizer 140, an entropy encoder 150, and inverse quantization. The unit 160, an inverse transform unit 170, an adder 175, a filter unit 180, and a reference picture buffer 190 may be included.

The inter predictor 110, the intra predictor 120, the switch 115, the subtractor 125, the transformer 130, the quantizer 140, the entropy encoder 150, the inverse quantizer 160, At least some of the inverse transform unit 170, the adder 175, the filter unit 180, and the reference picture buffer 190 may be program modules, and may communicate with an external device or system. The program modules may be included in the encoding apparatus 1600 in the form of an operating system, an application program module, and other program modules.

The program modules may be physically stored on various known storage devices. In addition, at least some of these program modules may be stored in a remote storage device that can communicate with the encoding device 1600.

Program modules perform routines or subroutines, programs, objects, components, and data to perform functions or operations, or to implement abstract data types, according to one embodiment. Data structures and the like, but is not limited thereto.

The program modules may be composed of instructions or codes performed by at least one processor of the encoding apparatus 1600.

The processor 1610 may include an inter predictor 110, an intra predictor 120, a switch 115, a subtractor 125, a transformer 130, a quantizer 140, an entropy encoder 150, and inverse quantization. Instructions or codes of the unit 160, the inverse transform unit 170, the adder 175, the filter unit 180, and the reference picture buffer 190 may be executed.

The storage can represent the memory 1630 and / or the storage 1640. Memory 1630 and storage 1640 may be various forms of volatile or nonvolatile storage media. For example, the memory 1630 may include at least one of a ROM 1631 and a RAM 1632.

The storage unit may store data or information used for the operation of the encoding apparatus 1600. In an embodiment, data or information included in the encoding apparatus 1600 may be stored in the storage.

For example, the storage unit may store a picture, a block, a list, motion information, inter prediction information, a bitstream, and the like.

The encoding apparatus 1600 may be implemented in a computer system including a recording medium that may be read by a computer.

The recording medium may store at least one module required for the encoding apparatus 1600 to operate. The memory 1630 may store at least one module, and the at least one module may be configured to be executed by the processor 1610.

Functions related to communication of data or information of the encoding apparatus 1600 may be performed through the communication unit 1620.

For example, the communication unit 1620 may transmit the bitstream to the decoding device 1700, which will be described later.

17 is a structural diagram of a decoding apparatus according to an embodiment.

The decoding device 1700 may correspond to the decoding device 200 described above.

The decoding apparatus 1700 may include a processor 1710, a memory 1730, a user interface (UI) input device 1750, a UI output device 1760, and storage that communicate with each other through a bus 1790. 1740. In addition, the decryption apparatus 1700 may further include a communication unit 1720 connected to the network 1799.

The processor 1710 may be a semiconductor device that executes processing instructions stored in the central processing unit (CPU), the memory 1730, or the storage 1740. The processor 1710 may be at least one hardware processor.

The processor 1710 may generate and process a signal, data, or information input to the decoding apparatus 1700, output from the decoding apparatus 1700, or used in the decoding apparatus 1700, and may include a signal, Inspection, comparison, and judgment related to data or information can be performed. In other words, in an embodiment, generation and processing of data or information, and inspection, comparison, and determination related to the data or information may be performed by the processor 1710.

The processor 1710 includes an entropy decoder 210, an inverse quantizer 220, an inverse transformer 230, an intra predictor 240, an inter predictor 250, a switch 245, an adder 255, and a filter. The unit 260 and the reference picture buffer 270 may be included.

Entropy decoder 210, inverse quantizer 220, inverse transformer 230, intra predictor 240, inter predictor 250, switch 245, adder 255, filter 260, and At least some of the reference picture buffer 270 may be program modules and may communicate with an external device or system. The program modules may be included in the decryption apparatus 1700 in the form of an operating system, an application program module, and other program modules.

The program modules may be physically stored on various known storage devices. In addition, at least some of these program modules may be stored in a remote storage device that can communicate with the decryption apparatus 1700.

Program modules perform routines or subroutines, programs, objects, components, and data to perform functions or operations, or to implement abstract data types, according to one embodiment. Data structures and the like, but is not limited thereto.

The program modules may be composed of instructions or codes performed by at least one processor of the decoding apparatus 1700.

The processor 1710 includes an entropy decoder 210, an inverse quantizer 220, an inverse transformer 230, an intra predictor 240, an inter predictor 250, a switch 245, an adder 255, and a filter. Instructions or codes of the unit 260 and the reference picture buffer 270 may be executed.

The storage unit can represent memory 1730 and / or storage 1740. The memory 1730 and the storage 1740 may be various types of volatile or nonvolatile storage media. For example, the memory 1730 may include at least one of a ROM 1731 and a RAM 1732.

The storage unit may store data or information used for the operation of the decoding apparatus 1700. In an embodiment, data or information included in the decryption apparatus 1700 may be stored in the storage.

For example, the storage unit may store a picture, a block, a list, motion information, inter prediction information, a bitstream, and the like.

The decoding device 1700 may be implemented in a computer system including a recording medium that can be read by a computer.

The recording medium may store at least one module required for the decoding apparatus 1700 to operate. The memory 1730 may store at least one module, and the at least one module may be configured to be executed by the processor 1710.

Functions related to communication of data or information of the decoding apparatus 1700 may be performed through the communication unit 1720.

For example, the communication unit 1720 may receive a bitstream from the encoding apparatus 1600.

Convolution Neural Networks (CNN)

CNN may refer to a network including a plurality of layers consisting of a convolution layer and a pooling layer. Filtering may be performed on the image input by the convolutional layer, and a feature map may be extracted as a result of the filtering. The extracted feature map can be used as input to the next layer. This process can be performed continuously for the layers.

As learning is performed, the network reacts to simple structures in the image, such as edges, at lower layers, and as the layers are deeper, textures and objects You can learn the characteristics of response to parts.

Convolution Layer

18 illustrates an operation of a convolutional layer according to an example.

The convolutional layer may perform filtering on the input frame and output a feature map as a result of the filtering. The feature map can be used as input to the next layer. With this structure, the input frame can be processed continuously by a plurality of layers.

In the convolution layer, the kernel may mean a filter that performs a convolution operation or filtering. The size of the kernel may be referred to as kernel size or filter size. Operational parameters constituting the kernel may also be referred to as weights, kernel parameters or filter parameters.

In the convolutional layer, different kinds of filters can be used for one input. In this case, a process in which one filter processes an input may be referred to as a convolution channel.

As shown in FIG. 18, the convolutional layer may reduce samples by the size of the kernel into one sample. In FIG. 18, the size of the illustrated kernel may be 3 × 3. That is, in FIG. 18, a convolution operation is performed by a filter having a kernel size of 3 × 3 in one channel.

In FIG. 18, an operation may be performed on a rectangle having a dark border in the input image. In this case, the window may mean an operation area such as a rectangle having a dark border. The window may move one space from the upper left of the frame to the lower right of the frame, and the size of the movement may be adjusted.

Stride and padding may be used for filters of convolution operations.

The stride may refer to the size of the movement. The value of the stride illustrated in FIG. 18 may be 1. When the value of stride is 2, operations may be performed on windows spaced at intervals of two columns.

Padding may be to make the input image larger, and may mean filling in specific values at the top, bottom, left, and right sides of the input image.

Pooling Layer

19 illustrates an operation of a pulling layer according to an example.

Pulling may refer to subsampling of a feature map obtained by an operation in a convolutional layer.

As shown in FIG. 19, the pooling layer may select a representative sample for samples of a specified size passing through the pooling layer.

In pooling, the size of the stride and the size of the window can generally be the same.

Pooling may include max pooling and average pooling.

Maximum pooling may be selecting a sample having a maximum value among samples of a specified size as a representative sample. For example, for samples of 2 × 2, the maximum value of the samples may be selected as the representative sample.

Average pooling may be to set the average value of samples of a specified size as a representative sample.

The pulling layer illustrated in FIG. 19 may perform maximum pulling. For example, the pooling layer may select one sample from among samples of a window having a size of 2 × 2. With this selection, the width and length of the output from the pooling layer can be half the width and length of the input to the pooling layer.

As illustrated in FIG. 19, the size of the stride and the window may both be set to two. For example, when values of the size of [h, w, n] are input to the pooling layer, the size of the values output through the pooling layer may be [h / 2, w / 2, n].

Deconvolution layer

20 illustrates an operation of a deconvolution layer according to an example.

The deconvolution layer may perform calculation in a direction opposite to that of the calculation of the convolution layer. Except for the orientation, the computation of the convolutional layer and the computation of the deconvolutional layer may be considered identical.

The deconvolution layer may perform a convolution operation on the input feature map and output a frame through the convolution operation.

The size of the output frame may vary depending on the value of the stride. For example, when the value of the stride is 1, the horizontal size and the vertical size of the frame may be equal to the horizontal size and the vertical size of the feature map. When the value of the stride is 2, the horizontal size and the vertical size of the frame may be 1/2 of the horizontal size and the vertical size of the feature map.

Unpooling layer

21 illustrates an operation of an unpooling layer according to an example.

The un-pooling layer may proceed up-sampling in a direction opposite to the direction of pulling in the pooling layer. The unpooling layer, as opposed to the pooling layer, may perform a function of enlarging the dimension. In other words, the unpooling layer, in contrast to the pooling layer, can magnify a sample passing through the unpooling layer to samples of a specified size. For example, a sample passing through the un-pooling layer can be magnified to samples of a 2x2 window.

For example, when values of the size of [h, w, n] are input to the unpooling layer, the size of the values output through the unpooling layer may be [h * 2, w * 2, n]. have.

Nonlinear operation layer

22 illustrates an operation of a lelu layer according to an example.

An example of values input to a relu layer is illustrated on the left side of FIG. 22, and an example of values output from the relu layer is illustrated on the right side of FIG. 22.

The real layer may perform a nonlinear operation as shown in FIG. 22. In embodiments, the relu layer may be replaced with a nonlinear computational layer.

The lelu layer may generate output values by applying a transfer function to input values.

The magnitudes of the values input to the relu layer and the magnitudes of the values output from the relu layer may be the same. In other words, the magnitudes of the values passing through the relu layer may not change.

Auto encoder

23 shows an automatic encoder according to an example.

The automatic encoder may have a structure as shown in FIG. 23 and may be widely used for unsupervised learning.

A convolution encoder and a convolution decoder can be derived from the automatic encoder.

According to the structure of the automatic encoder, the dimensions of the input and the dimensions of the output may be the same. The purpose of an automatic encoder may be to perform a learning on f () such that f (X) = X. X may be an input value. In other words, the purpose of the automatic encoder may be to approximate the output prediction value X 'to the input value X.

The automatic encoder may include an encoder and a decoder. The encoder can provide a code or latent variable as an output value for the input value X. The code can be used as a feature vector for the input value X. The code can be input to the decoder. The decoder may output the predicted value X 'formed from the code.

Convolutional Encoder and Convolutional Decoder

24 illustrates a convolution encoder and a convolution decoder according to an example.

The structures of the convolutional encoder and the convolutional decoder may consist of a pair of convolutional layers and deconvolutional layers. Convolutional encoders and convolutional decoders can provide inputs, feature vectors, and outputs similarly to automatic encoders.

The convolution encoder may include a convolutional layer and a pooling layer. The input to the convolutional encoder may be a frame, and the output from the convolutional encoder may be a feature-map.

The convolution decoder may include a deconvolution layer and an un-pooling layer. The input to the convolutional decoder can be a feature map and the output from the convolutional decoder can be a (reconstructed) frame.

The features of the convolutional encoder and the convolutional decoder may reflect the characteristics of the convolution. According to this reflection, the convolutional encoder and the convolutional decoder may have a smaller learning weight. Convolutional encoders and convolutional decoders may be particularly useful when the operation is performed under such purposes as optical flow and counter edges for the output frame.

The convolutional encoder can reduce the dimension by utilizing convolution and pooling, and generate a feature vector from the frame. The feature vector may be generated at the output of the convolutional encoder.

The feature vector may be a vector representing a feature of the original signal at a lower dimension than the dimension of the original signal.

The convolution decoder can reconstruct the frame from the feature vector utilizing deconvolution and un-pooling.

Genetic Adversarial Network (GAN)

25 illustrates a configuration of a generator of a GAN according to an example.

26 illustrates a configuration of a discriminator of a GAN according to an example.

The GAN may include opposing-pairs of a generator for generating an image and a discriminator for separating the generated image from an actual image. The generator may generate an image similar to the real image by learning a probability distribution of the real image. The discriminator can learn to discriminate between the actual image and the generated image. The GAN composed of the generator and the discriminator may operate while simultaneously performing discrimination and generation. Here, the determination may be determination of similar images and dissimilar images. Generation may be to generate a similar image.

Since the generator must generate an image that is similar to the original image but cannot be distinguished by the discriminator, it is possible to combine a plurality of loss functions with different purposes, and to minimize the value of the combined loss function. I can learn.

In other words, the GAN may be a network structure designed for generating a signal that cannot be discriminated by the discriminator whether the generator is a generated signal or an original signal.

In FIG. 25, a process of generating an image representing a number from a random signal by a generator is schematically illustrated.

The network of generators may consist of an input layer, one or more hidden layers and an output layer.

The input of the generator may be a random signal z ⁽ⁱ⁾ and the output of the generator may be a fake image F ⁽ⁱ⁾ .

A target o _k may be set for the generator, and learning of the generator may be performed according to the set purpose. The target may represent an objective function.

For example, the objective function of the GAN may be defined as in Equation 2 or Equation 3 below.

[Formula 2]

[Equation 3]

In FIG. 26, a process of determining whether an input image is a fake image or a real image when a fake image or a real image is input to the discriminator is schematically illustrated.

The network of discriminators may comprise an input layer, one or more hidden layers and an output layer. In the output layer, a result value of 1 (true) or 0 (false) may be output.

The input of the discriminator may be a fake image F ⁽ⁱ⁾ or a real image x ⁽ⁱ⁾ .

The output of the discriminator may be 1 (true) or 0 (false). 1 may indicate that the input image is determined to be an actual image. 0 may indicate that the input image is determined to be a fake image. In other words, the discriminator may be trained to discriminate the input image into one of an actual image and a fake image.

The target o _k may be set for the discriminator, and learning of the discriminator may be performed according to the set purpose.

Recurrent Neural Network (RNN)

27 shows a structure of an RNN according to an example.

The structure of the RNN is shown on the left side of FIG. 27, and the structure of the unfolded RNN is shown on the right side. In FIG. 27, s may represent latent variables. x may be an input value. o may be an output value. U, V, and W may be weights that are subject to learning.

A general neural network may be referred to as a feed-forward neural network. In a feed-forward neural network, the input data may pass only once in nodes in the neural network while the operation is performed from the input layer to the hidden layer through the hidden layer. On the other hand, the RNN may have a structure in which a result output from the hidden layer is input back to the hidden layer.

In the RNN, the data inputted in the past and the data input in the past may be used for learning at the same time. In RNN, the output at time t-1 may also affect the output at time t. In other words, the RNN may be a structure that learns that the latent variable s at a past point in time affects the output at a future point in time.

According to this feature, the RNN can be used to learn time series information and can be used to analyze time series data.

The hidden vector at the present time can be computed as follows. That is to say, the latent variables at the current time t in s _t may be computed as shown in equation 4 below.

[Equation 4]

May be a nonlinear function, such as a sigmoid function. s _t can provide the output o _t with weight V and can be stored in memory for calculation of s _{t + 1} .

In Equation 4, if the value of U is large, the determination may be made based on the input value at the present time. If the value of W is large, a judgment can be made based on the information stored.

Long Short Term Memory (LSTM)

28 illustrates a structure of a convolutional LSTM neural network according to an example.

As described above, in the RNN, the data currently input and the data input in the past may be used for learning at the same time.

In the course of learning the RNN, a vanishing gradient problem with respect to data input in the past may occur as time passes. LSTM can be used to overcome the disappearing slope problem. The structure of the LSTM allows the slope of the error to pass back in time in the neural network. In other words, the structure of the LSTM may allow the data previously entered into the neural network to affect the current output of the neural network more consistently or significantly.

The structure of the LSTM may be composed of cells to which a plurality of gates are attached. The cell may change or store information. Learning may be performed on weights of gates attached to cells, and learning performance may be improved as learning about weights is performed for each cell.

The value or weight of the gate connected to the cell may determine how much value is stored in the cell, when to output information from the cell, and when to delete data in the cell.

Learning about the weight of each gate of the LSTM may be performed through the same principle as in Equation 5 below.

[Equation 5]

Video generation and prediction

1. Video Interpolation

Video interpolation may be a method of predicting a target frame using previous frames of the target frame and subsequent frames of the target frame among the frames of the video. The target frame may be a current frame. For example, the interpolation of the video, when given a frame x _{t + 1} of frame x _t-1 and time t + 1 at time t-1, may be to estimate the frame x _t at time t . Frame x _t may be defined as shown in Equation 6 below.

[Equation 6]

x _t ∈ R ^{w ㅧ h ㅧ c}

t may represent time. In other words, x _t may represent a frame at time t of the frames of the video. w may represent the horizontal length of the frame. h may represent the vertical length of the frame. c may represent a color dimension of the frame.

2. Video addition

The video extrapolation may be a method of predicting a future frame using previous frames of the current frame and the current frame among the frames of the video. For example, the extrapolation of the video may be to produce frames at times n + 1 to m, given the frames at times 0 to n. n and m may be integers and m may be greater than n.

3. Video generation technique based on deep learning

Below we describe some techniques for generating frames of video based on the deep learning model. Techniques to be described below may be applied to video interpolation and / or video interpolation depending on the order and direction of frames being input.

3.1 Prediction through the generation of optical flow

The optical flow may refer to a motion vector for a pixel representing a motion of a pixel occurring between frames. The video may be generated through a deep learning structure that generates an optical flow that estimates the movement of the pixel.

Interpolation may be performed using the two frames and the optical flow to generate an intermediate frame located between the two frames. Here, the two frames may be a frame at time t-1 and a frame at time t + 1, and the intermediate frame may be a frame at current time t.

In addition, extrapolation may be performed to generate a frame located at the left or the right of the two frames using the two frames and the optical flow. Through extrapolation, future frames can be predicted.

3.2 Prediction via Adaptive Convolution Network (ACN)

29 shows a structure of an ACN according to an example.

In the above-described methods, the process of two steps may be performed. In other words, in the above-described methods, 1) prediction for a feature appearing between frames may be performed, and 2) interpolation between pixels may be performed using the obtained feature.

The ACN can perform kernel learning using CNN, and can perform frame prediction and pixel interpolation (or extrapolation) at once through end-to-end learning.

The processor may generate a predictive frame using a separable structure or a voxel flow of the adaptive CNN. In the generation of the prediction frame, the frame X _n at the current time point may be generated using the previous frames x _n-1 , x _n-2 , x _n-3, etc. stored in the DPB, and at the current time point The frame X _n of may be used as the prediction frame.

Adaptive separable convolution, described below, may be an example of a modification of the ACN.

30 illustrates a structure of an adaptive separable convolution according to an example.

As frames x _n-1 , x _n-2, and x _n-3, etc. are input to the convolutional encoder and the convolutional decoder, learning about the convolution filter kernel K can be performed, whereby frame x _n Can be predicted.

3.3 Video Prediction Using LSTM

Video generation may be performed by utilizing LSTM for the structure of the RNN for learning time series information. The convolutional LSTM neural network may be one of these applications.

The convolutional LSTM neural network described above with reference to FIG. 28 can predict the feature vector on a time-series.

In convolution LSTM, the connection between the input and the hidden vector can be replaced with a convolution filter, as shown in FIG. This substitution allows the convolutional LSTM to perform learning on fewer parameters than the existing LSTM, and better reflect local characteristics in this learning.

Generation-encoding and generation-decoding

31 is a flow diagram of generation-encoding and generation-decoding according to one embodiment.

In an embodiment, a method of generating a virtual reference frame using video interpolation and video interpolation based on deep learning, and using the virtual reference frame for encoding and / or decoding is described.

In an embodiment, the process of generating a feature vector from an input video frame using a deep learning model is named generation-encoding. In addition, a module that performs generation encoding is called a generation-encoder.

Also, in an embodiment, the process of reconstructing video from the feature vector is named generation-decoding. In addition, a module that performs generation-decoding is called a generation-decoder.

Interpolation, extrapolation, and other generation methods for an image signal may be performed through the generation- encoder and the generation-decoder. The video signal generation encoder and the video signal generation decoder may be used in the process of video encoding and / or decoding in a video codec such as the encoding apparatus 100 and the decoding apparatus 200.

The input of the generator-encoder may be a residual frame. The generator-coder may generate a feature vector of the input residual frame. The residual frame may be the difference between the two frames. Alternatively, the generation-encoder may generate a residual frame for two input frames.

The generation-decoder may generate a future frame by adding the residual frame to a previously reconstructed frame. The generator-decoder may generate the summed feature vector by summing the feature vector of the previously reconstructed frame and the predicted feature vector of the residual frame. The generator-decoder may generate a future frame using the combined feature vector.

A video consisting of frames from time 0 to time t- 1 is expressed in Equation 7 below. It can be defined as.

[Formula 7]

Residual Video for May be defined as in Equation 8 below.

Equation 8

The target frame for time n , The prediction frame at the current time point n through the generation-coding and the generation-decoding by the steps 3110 to 3150 below. Can be generated.

In steps 3110-3150 below, the generation-encoding can include steps 3110, 3120, and 3130. Generation-decryption may include steps 3140 and 3150.

In step 3110, the generation-encoder is a target frame Feature Vector Can be generated. Feature vector May be generated as shown in Equation 9 below.

Equation 9

May represent a first convolutional neural network (CNN) of the generator-encoder.

Generate-Encoder Target Frame Is the output of the first CNN by inputting the first CNN. Can be generated.

Claim 1 CNN may generate a feature vector for the target frame x _t at time t. The size of the target frame x _t may be [h, w, c].

The first CNN of the generator-encoder may include a convolutional layer, a pooling layer, and a nelu layer. The convolutional layer, the pulling layer, and the nilu layer may be plural.

In step 3120, the generation-encoder is a residual frame Can be generated. Residual frame May be generated as shown in Equation 10 below.

Equation 10

In other words, the generation-encoder is two frames And The residual frame by calculating the difference between Can be obtained. Residual frame Silver target frame And previous frame of target frame It can be the difference between.

Or, the generation-encoder is the target frame Residual frame using motion prediction using motion vector for Can be obtained.

In step 3130, the generation-encoder is a residual frame Feature Vector Can be generated. Feature vector May be generated as shown in Equation 11 below.

[Equation 11]

May represent the second CNN of the generator-encoder.

Generate-Encoder Residual Frame Is the output of the second CNN by inputting the second CNN. Can be generated.

The CNN 2 can generate the feature vectors for the residual frame y _t at time t. The magnitude of the residual frame y _t may be [h, w, c].

First CNN And second CNN The hyper-parameters of may be different. Hyper-parameters may include 1) number of convolution layers, pooling layers and real layers, 2) positions, 3) arrays, and 4) kernel sizes.

In step 3140, the generation-decoder is a residual frame Predicted feature vector for Can be generated. Generating-Decoder Residual Frame Feature Vector Residual Frames Using Predicted feature vector for Can be generated. Predicted feature vector May be generated as shown in Equation 12 below.

Equation 12

May be a convolutional Long Short Term Memory (LSTM) neural network that predicts feature vectors on a time-series.

In other words, the residual frame Predicted feature vector for The residual frame Feature Vector May be generated by the input LSTM neural network.

Predicted feature vector May be that a feature vector at time n + 1 is predicted with respect to the residual frame. In other words, the generation-decoder is a residual frame outputted from the generation encoder. Feature Vector Predicted feature vector for next time n +1 Can be generated.

In one embodiment, step 3140 may be performed by a generator-encoder. In such a case, generation decoding may include step 3150.

Generation of a virtual reference frame using video interpolation and video extrapolation based on deep learning, and a method of using the generated virtual reference frame for encoding and decoding video.

Encoding and decoding of the target block and / or the target frame may be performed through inter prediction based on the virtual reference frame. Hereinafter, the virtual reference frame of the target frame may mean a virtual reference frame of the target block included in the target frame.

In encoding and decoding a target frame, a virtual reference frame may be generated, and virtual reference frame usage information indicating whether inter prediction based on the virtual reference frame is used may be signaled. For example, the virtual reference frame usage information may be a flag indicating whether a virtual reference frame is generated and whether inter prediction based on the virtual reference frame is used as one of "true" and "false".

In generation of a virtual reference frame and inter prediction based on the virtual reference frame, a virtual reference frame generation method indicator used to distinguish a generation method for the virtual reference frame may be signaled.

In the information such as the virtual reference frame usage information and the virtual reference frame generation method indicator, the information includes the level of the SPS, the level of the PPS, the level of the VPS, the level of the supplemental enhancement information (SEI) message, and the slice header. It may be signaled for one or more levels, such as level and level of CTU.

32 is a flowchart of an inter prediction method, according to an exemplary embodiment.

The inter prediction method may be performed by the encoding apparatus 1600 and / or the decoding apparatus 1700.

Hereinafter, the processing unit may correspond to the processing unit 1610 of the encoding apparatus 1600 and / or the processing unit 1710 of the decoding apparatus 1700.

For example, the encoding apparatus 1600 may perform the inter prediction method of the embodiment to compare the efficiencies of the plurality of prediction methods for the target block of the target frame, and to generate a reconstructed block for the target block. The inter prediction method of the embodiment may be performed.

The target block may be a block that is a target of encoding and / or decoding, and the target frame may be a frame including the target block.

In one embodiment, the target of inter prediction may be a target block. The target block may be a CU, or the target block may be at least one of a CTB, a CU, a PU, a TU, a sub block, a block having a specified block size, and a block within a predetermined range of block sizes. Alternatively, the target block may represent a unit of coding. Alternatively, the target block may represent a specified area within the target picture.

For example, the decoding apparatus 1700 may perform the inter prediction method of the embodiment to generate a reconstructed block for the target block.

As mentioned above, in the embodiments, below, the terms "image", "picture", "frame" and "screen" may be used in the same sense and may be mutually exclusive. Can be used interchangeably.

In operation 3210, the processor may select a reference frame. The processor may select a reference frame for inter prediction of an embodiment among existing reference frames generated according to the above-described embodiment.

The processor may select a plurality of reference frames. For example, the selected plurality of reference frames can be used as input for generating a virtual reference frame.

The existing reference frame may be a picture used as a reference frame of the target picture among the reconstructed pictures stored in the DPB for encoding and / or decoding of video.

In operation 3220, the processor may generate a virtual reference frame based on the selected reference frame.

The processor may select a deep learning network structure that may be used in the generation of the virtual reference frame in generating the virtual reference frame. In other words, the virtual reference frame may be generated based on the deep learning network structure.

In addition, the processor may select video interpolation and / or video interpolation according to a time point of the selected reference frame in generating the virtual reference frame. In other words, the virtual reference frame may be generated based on video interpolation using the selected reference frame. The virtual reference frame may be generated based on video extrapolation using the selected reference frame.

In operation 3230, the processor may construct a reference picture list based on the virtual reference frame. The processor may include the virtual reference frame in the reference picture list.

In operation 3240, the processor may perform inter prediction based on the virtual reference frame.

The processor may perform inter prediction on the target block using the virtual reference frame according to the inter prediction mode selected for the target block.

The inter prediction mode of inter prediction may be an AMVP mode, a merge mode, or a skip mode.

Inter prediction may be bidirectional prediction.

The processor may generate the prediction block for the target block by performing inter prediction on the target block based on the virtual reference frame.

In operation 3250, the processor may reconstruct the target frame based on a result of the inter prediction.

The processor may generate a reconstructed residual block for the target block.

The processor may generate a reconstructed block for the target block based on the prediction block and the reconstructed residual block. The reconstructed target frame may include a reconstructed block.

Operation of the above described steps 3210, 3220, 3230, 3240 and 3250 is described in more detail below.

1. Select an existing reference frame

33 illustrates a structure of a hierarchical B frame according to an example.

In FIG. 33, the vertical bar may represent a frame. Arrows between the frames may indicate a reference relationship between the frames. The number below the frame may indicate the POC of the frame. T ₀ , T ₁ , T _2, or T ₃ below the POC of the frame may indicate a temporal identifier (ID) of the frame.

In operation 3210, in selecting an existing reference frame for generating a virtual reference frame, one or more reference frames among reference frames included in the plurality of reference picture lists may be selected. For example, the plurality of reference picture lists may include reference picture list 0 and reference picture list 1.

In one embodiment, the processor may select the reference frame using one of the following schemes or a combination of one or more schemes.

The reference frame closest to the distance from the target frame in the backward direction and / or the forward direction among the reference frames may be selected. The distance between the frames can be determined by the POC. In other words, a reference frame having a POC having the smallest difference from the POC of the target frame in the reverse and / or forward direction among the reference frames may be selected. In other words, the selected reference frame may be a reference frame closest to the distance from the target frame in the reverse or forward direction among the reference frames included in the reference picture list.

If one reference frame is selected in both directions, the difference between the POC of the backward reference frame and the POC of the target frame is equal to the difference between the POC of the target frame and the POC of the forward reference frame. The reference frame can be selected. In other words, when one reference frame is selected in both directions, the first difference and the second difference may be the same. Here, the first difference may be a difference between the POC of the reversely selected reference frame and the POC of the target frame. The second difference may be a difference between the POC of the target frame and the POC of the selected reference frame in the forward direction.

Reference frame specific information indicating the selected reference frame used for generation of the virtual frame may be signaled for the specified unit. For example, the specified unit may be a slice or a frame.

The reference frame may be selected based on whether the reference frame has been designated as a collocated picture.

For example, the reference frame specific information may instruct to select a reference frame designated as a co-located picture.

For example, the reference frame specific information may instruct to select a reference frame that is not designated as the collocated picture.

The reference frame may be selected based on the temporal identifier of the reference frame.

For example, the reference frame specific information may indicate to select a reference frame having the smallest temporal ID.

For example, the reference frame specific information may indicate to select a reference frame having the largest temporal identifier.

In the structure of a hierarchical B frame, among the reference frames having a large temporal identifier, the reference frame closest to the distance from the target frame can be selected. Alternatively, a reference frame having a POC having the smallest difference from the POC of the target frame among the reference frames having the largest temporal identifier may be selected.

2. Generation of Virtual Reference Frames According to Network Structure

2.1 Creating a Virtual Reference Frame Using a GAN

As described above with reference to FIGS. 25 and 26, the GAN may include opposing-pairs of a generator that generates an image and a discriminator that separates the generated image from the actual image. Can be. The generator may generate an image similar to the real image by learning a probability distribution of the real image. The discriminator can learn to discriminate between the actual image and the generated image. The GAN composed of the generator and the discriminator may operate while simultaneously performing discrimination and generation. Here, the determination may be determination of similar images and dissimilar images. Generation may be generation of similar images.

In one embodiment, at step 3220, the processor may generate a virtual reference frame using the GAN.

2.2 Prediction of Virtual Reference Frames via ACN

A process of performing interpolation for pixels (x, y) of an output frame using the ACN described above with reference to FIG. 29 is illustrated.

As frames are input to the ACN, the learning of the kernel function K may be performed in the ACN. The virtual reference frame can be predicted by learning.

According to FIG. 29, a virtual reference frame generated by interpolation using two reference frames I ₁ and I ₂ may be output. The reference frames I ₁ and I ₂ may be the reference frames selected in step 3210.

In one embodiment, at step 3220, the processor may generate a virtual reference frame using the ACN.

2.3 Prediction of Virtual Reference Frames with Long Short Term Memory (LSTM)

34 shows generation of a reference frame using interpolation through a process of generation-coding and generation-decoding.

In FIG. 34, a process of generation-encoding and generation-decoding using the structure of the convolutional encoder and the convolutional decoder is illustrated, and generation of interpolation and reference frames through the processes of generation-encoding and generation-decoding are illustrated.

Interpolation may be performed to generate a virtual reference frame by a convolutional encoder and a network based on LSTM.

Learning on bidirectional videos may be performed by the same network structures. The input video may be input in a direction in which a prediction about the frame at the target view is made using the frame at a future view with respect to the network structures, and the frame at the target view using the frame at the past view. It may be input in the direction in which the prediction is made. In other words, the input video may be input to the network structure in the direction of predicting the frame of view t using the frame at view t + 1, and the direction of predicting the frame of view t using the frame at view t-1. Can be entered into the network structure.

In one embodiment, at step 3220, the processor may generate a virtual frame through interpolation using frames predicted by the network structures.

In one embodiment, at step 3220, the processor may generate a virtual reference frame using interpolation through a process of generation-encoding and generation-decoding using the structure of the convolutional encoder and the convolutional decoder.

3. Creation of a Virtual Reference Frame According to the Start of Existing Reference Frame

3.1 Generation of virtual reference frames using interpolation based on deep learning

35 is a diagram illustrating a process of generating a reference frame using interpolation of a video and encoding and decoding a video using a reference frame according to an example.

In Figure 35, the signal at the past time point And signals at a future point in time Virtual reference frames by interpolation using This shows the process that is created.

Hereinafter, the term "signal" may refer to a frame and may also mean "signal indicating a frame".

T may represent a transform. Q may represent quantization. E may represent entropy encoding. I ⁻¹ may indicate entropy decoding. T ⁻¹ may represent an inverse-transform. Q ⁻¹ may indicate dequantization.

In FIG. 35, the signal is generated through generation-encoding and generation-decoding using the reconstructed signal. The reconstructed signal is a decoded frame And decoded frames It may be a signal of. The generated signal is a virtual reference frame Can be represented.

The reconstructed signal may be selected for generation of the signal through generation-encoding and generation-decoding. After the reconstructed signal is selected, interpolation may be performed as an example of a method of generating a virtual reference frame using the deep learning model.

Video prediction by optical flow, ACN or LSTM may be used for interpolation.

In the encoding apparatus 100 and the decoding apparatus 200, the reference frame of the past viewpoint is shown. And future reference frames Virtual reference frames by interpolation using Can be generated, virtual reference frame May be used for inter prediction on the target frame that is the current frame.

In the encoding apparatus 100, after the inter prediction, the residual signal is based on the prediction signal generated by the inter prediction. Can be obtained. Residual signal As transform, quantization, and entropy encoding are applied to, coded information about a target frame may be generated. The encoded information may be signaled to the decoding apparatus 200 from the encoding apparatus 100.

In the decoding apparatus 200, as the entropy decoding, inverse quantization, and inverse transformation are applied to the encoded information on the target frame, the residual signal Can be generated. Residual signal And virtual reference frames Based on the target frame can be reconstructed.

Reference frames of past viewpoints and reference frames of future viewpoints used for interpolation may be selected in different ways depending on the temporal prediction structure of the video. For example, in a random access scheme of HEVC, a hierarchical B frame structure as described above with reference to FIG. 33 may be used.

3.2 Creating Virtual Reference Frames Using Extrapolation Based on Deep Learning

36 illustrates a process of generating a reference frame using extrapolation of a video and encoding and decoding a video using a reference frame, according to an example.

36 shows signals at a past point in time. And Virtual reference frame by extrapolation using This shows the process that is created.

In FIG. 36, the signal is generated through generation-encoding and generation-decoding using the reconstructed signal. The reconstructed signal is a decoded frame And decoded frames It may be a signal of. The generated signal is a virtual reference frame Can be represented.

The reconstructed signal may be selected for generation of the signal through generation-encoding and generation-decoding. After the reconstructed signal is selected, extrapolation may be performed as an example of a method of generating a virtual reference frame using the deep learning model.

For extrapolation, video prediction by optical flow, ACN or LSTM may be used.

In the encoding apparatus 100 and the decoding apparatus 200, reference frames of past viewpoints And Virtual reference frame by extrapolation using Can be generated, virtual reference frame May be used for inter prediction on the target frame that is the current frame.

In the encoding apparatus 100, after the inter prediction, the residual signal is based on the prediction signal generated by the inter prediction. Can be obtained. Residual signal As transform, quantization, and entropy encoding are applied to, coded information about a target frame may be generated. The encoded information may be transmitted from the encoding apparatus 100 to the decoding apparatus 200.

In the decoding apparatus 200, as the entropy decoding, inverse quantization, and inverse transformation are applied to the encoded information on the target frame, the residual signal Can be generated. Residual signal And virtual reference frames Based on the target frame can be reconstructed.

The above-described extrapolation may be applied to a low delay B structure or a low delay P structure of HEVC. A generalized B / P prediction structure may be used for the prediction of the frame of video.

For reconstruction, reconstructed reference frames of past viewpoints may be selected, and reconstructed reference frames stored in the reference picture list may be selected in the following manner.

A reference frame having a POC having the smallest difference from the POC of the target frame among the reference frames in the DPB may be selected.

A reference frame compressed with the smallest QP of the reference frames in the DPB can be selected.

4. Structure of Reference Picture List

37 illustrates a configuration of a reference picture list of a virtual reference frame when bidirectional prediction is used according to an example.

When inter prediction is performed based on the virtual reference frame, as described above in operation 3230, the reference picture list may be configured based on the virtual reference frame.

For the construction of the reference picture list, the step of adding the generated virtual reference frame to the DPB may be preceded. When the generated virtual reference frame is added to the DPB, the specified reference frame among the reference frames in the DPB may be replaced with the virtual reference frame. The alternative manner may be one of the following manners.

The target frame in the DPB may be replaced with a virtual reference frame. In this case, the target frame in the DPB may be updated in a specific unit according to the process of encoding and / or decoding. The specified unit may be a block, slice or frame. The block can be a PU, CU or CTU.

Other specified reference frames in the DPB except the target frame may be replaced with virtual reference frames. In this case, the POC of the replaced virtual reference frame may be a predefined value. Alternatively, the POC of the replaced virtual reference frame may be a value derived based on coding parameters related to inter prediction such as inter prediction information. For example, coding parameters related to inter prediction may be 1) coding parameters of a target block, 2) coding parameters of a target frame, or 3) inter prediction information.

Alternatively, the generated virtual reference frame may be added to an additional picture buffer (APB) other than the DPB, and reference frames in the APB may be used for constructing the reference picture list. The APB may store one or more virtual reference frames for encoding and / or decoding the target frame.

In order to use the generated virtual reference frame, the virtual reference frame may need to be included in the reference picture list. In the following, a method of constructing a reference picture list for using a virtual reference frame through the reference picture list will be described in detail. Hereinafter, the reference picture list may be plural. For example, the reference picture list below may mean at least one of reference picture list 0 and reference picture list 1.

4.1 Replace existing reference frames in reference picture list with virtual reference frames

The reference picture list may be composed of reference frames in the DPB. A reference picture list is constructed, and some of the reference frames in the reference picture list may be replaced with virtual reference frames in the APB. At least one of the following schemes may be used for the above replacement.

The reference frame furthest from the target frame among the reference frames in the constructed reference picture list may be replaced with the virtual reference frame in the APB.

A reference frame having the largest reference picture index among the reference frames in the constructed reference picture list may be replaced with a virtual reference frame in the APB. For example, the reference picture index may be named "ref_pic_idx".

A reference frame having the smallest reference picture index among the reference frames in the constructed reference picture list may be replaced with the virtual reference frame in the APB.

The reference frame used for generation of the virtual reference frame among the reference frames in the configured reference picture list may be replaced with the virtual reference frame in the APB.

All of the reference frames in the constructed reference picture list can be deleted from the reference picture list, and the reference frames can be replaced with virtual reference frames in the APB.

4.2 Expanding the size of the reference picture list and adding virtual reference frames

The reference picture list may be composed of reference frames in the DPB. After the reference picture list is constructed by the reference frames in the DPB, the size of the reference picture list may be enlarged. As the size of the reference picture list is enlarged, a virtual reference frame in the APB may be added to the reference picture list having the enlarged size.

For this addition, the number of reference frames that the reference picture list may have may increase by the number of virtual reference frames in the APB. The number of reference frames that the reference picture list may have may be named "ref_pics_active".

The reference picture index of the virtual reference frame added to the reference picture list may be determined by at least one of the following methods.

The reference picture index of the added virtual reference frame may be greater than the reference picture indexes of existing reference frames in the reference picture list. One or more reference picture indices, which are larger than the reference picture indices of existing reference frames in the reference picture list, may be assigned to one or more virtual reference frames added to the reference picture list, respectively.

The reference picture index of the virtual reference frame to be added may be the smallest value. One or more reference picture indices sequentially increasing by one from the smallest value may be assigned to one or more virtual reference frames added to the reference picture list, respectively. Reference picture indices of existing reference frames of the reference picture list may increase by the number of added virtual reference frames.

4.3 Constructing a Reference Picture List Using Separate Picture Buffers

In using the reference picture list for inter prediction, a picture buffer indicator (IDC) for a reference frame may be used.

IDC may indicate a buffer containing a reference frame. The IDC may refer to one of the DPB and APB. If the IDC of the reference frame indicates a DPB, the reference frame may be a reference frame in the DPB. If the IDC of the reference frame indicates the APB, the reference frame may be a virtual reference frame in the APB.

IDC may be signaled for each reference frame in the reference picture list. Or, for each reference frame in the reference picture list, IDC may be derived based on other coding parameters. For example, the coding parameter may include the coding parameter of the neighboring block of the target block.

4.4 Using Additional Reference Picture Lists

The reference picture list may be composed of reference frames in the DPB. A plurality of reference picture lists such as reference picture list 0 and reference picture list 1 may be configured with reference frames in the DPB, and an additional reference picture list may be configured with virtual reference frames in the APB. For example, the additional reference picture list may be composed of reference picture list 2.

In this case, the inter prediction indicator may be used to distinguish PRED_L0, PRED_L1, PRED_L2, PRED_BI_L0_L1, PRED_BI_L0_L2, and PRED_BI_L1_L2. The inter prediction indicator may be named "inter_pred_idc".

PRED_L0 may indicate unidirectional inter prediction using reference picture list 0. FIG.

PRED_L1 may indicate unidirectional inter prediction using reference picture list 1. FIG.

PRED_2 may indicate unidirectional inter prediction using reference picture list 2. FIG.

PRED_BI_L0_L1 may indicate bidirectional inter prediction using reference picture list 0 and reference picture list 1.

PRED_BI_L0_L2 may indicate bidirectional inter prediction using reference picture list 0 and reference picture list 2.

PRED_BI_L1_L2 may indicate bidirectional inter prediction using reference picture list 1 and reference picture list 2.

Referring again to FIG. 37, in FIG. 37, the reference frame is shown as a rectangle. The first number in the rectangle may indicate the number of the reference picture list. The second number in the rectangle may indicate the reference picture index of the reference frame. For example, "[0] [2]" in the rectangle may indicate that the reference frame is a frame in reference picture list 0, and that the reference picture index of the reference frame is two.

In FIG. 37, the virtual reference frame is shown as a square filled with a gray interior.

The generated virtual reference frame may be added to the APB which is an additional picture buffer other than the DPB. In the process of constructing the reference picture list, reference picture list 0 and reference picture list 1 may be composed of reference frames in the DPB. After this configuration, a virtual reference frame in the APB may be added to each of reference picture list 0 and reference picture list 1. At this time, the reference picture index of the added virtual reference frame is illustrated as two.

In other words, in bidirectional prediction, a virtual reference frame may be added to the last indices of reference picture list 0 and reference picture list 1, respectively.

The reference picture indices of the generated virtual reference frame and the existing reference frames may coincide with each other in the plurality of reference picture lists. For example, as shown in FIG. 37, existing n reference frames (in the DPB) may be added as the 1 st to n th reference frames to each reference picture list of the plurality of reference picture lists, and in the APB The virtual reference frame may be added as the n + 1th reference frame to each reference picture list of the plurality of reference picture lists.

5. Inter Prediction Using Virtual Reference Frames

Hereinafter, a method of using a virtual reference frame in a specified inter prediction mode will be described.

5.1 Changing AMVP Mode to Use Virtual Reference Frames

38 is a flowchart of a method of searching for a motion vector candidate in AMVP mode according to an example.

When the inter prediction mode for the target block is the AMVP mode, the search for the motion vector candidate may be performed as illustrated in FIG. 38.

In searching for a motion vector candidate, it may be considered whether the generation principle of the reference frame referenced by the target block and the generation principle of the reference frame of the motion vector candidate match. In other words, the motion vector candidate may be inserted into the motion vector candidate list based on the type of the reference frame of the motion vector candidate. In inserting the motion vector candidate into the motion vector candidate list, it may be determined whether the type of the reference frame of the motion vector candidate is a virtual reference frame.

Hereinafter, the non-virtual reference frame may mean an existing reference frame that is not generated by the method of generating a virtual reference frame of the embodiment.

Whether the type of the reference frame of the motion vector candidate is a virtual reference frame or a non-virtual reference frame may be derived based on the reference picture list and the reference frame index of the reference frame.

In operation 3810, it may be determined whether the reference frame index of the target block points to the virtual reference frame.

If the reference frame index of the target block points to the virtual reference frame, step 3820 may be performed.

If the reference frame index of the target block does not point to a virtual reference frame, step 3830 may be performed.

In step 3820, it may be determined whether the reference frame index of the motion vector candidate points to the virtual reference frame.

In an embodiment, the motion vector candidate to be searched may be a motion vector candidate of a spatial neighboring block or a motion vector candidate of a temporal neighboring block. The spatial neighboring block may be a block spatially neighboring the target block. The temporal neighboring block may be a block temporally neighboring the target block.

If the reference frame index of the motion vector candidate points to the virtual reference frame, step 3840 may be performed.

If the reference frame index of the motion vector candidate does not point to a virtual reference frame, step 3850 may be performed.

In operation 3830, it may be determined whether a reference frame index of the motion vector candidate indicates a virtual reference frame.

If the reference frame index of the motion vector candidate points to the virtual reference frame, step 3850 may be performed.

If the reference frame index of the motion vector candidate does not point to a virtual reference frame, step 3860 may be performed.

In operation 3840, when the reference frame of the target block is a virtual reference frame and the reference frame of the motion vector candidate is a virtual reference frame, one of the following operations may be performed. Or, if the reference frame index of the target block points to the virtual reference frame and the reference frame index of the motion vector candidate points to the virtual reference frame, one of the following operations may be performed.

A motion vector candidate may be selected. In other words, the motion vector candidate may be added to the motion vector candidate list.

The motion vector candidate may be set to a motion vector (0, 0). The motion vector (0, 0) may be a zero vector. A motion vector candidate set to a motion vector (0, 0) may be selected. A motion vector candidate set to a motion vector (0, 0) may be added to the motion vector candidate list. Hereinafter, selection of the motion vector candidate may mean that the motion vector candidate is added to or added to the motion vector candidate list. Hereinafter, setting the motion vector candidate to the motion vector (0, 0) may mean that the motion vector (0, 0) is used as the motion vector candidate.

In step 3850, if one of the reference frame of the target block and the reference frame of the motion vector candidate is a virtual reference frame and the other is a non-virtual reference frame, one of the following operations may be performed. Alternatively, if one of the reference frame index of the target block and the reference frame index of the motion vector candidate points to the virtual reference frame and the other points to the non-virtual reference frame, one of the following operations may be performed.

The motion vector candidate may be set to a motion vector (0, 0). The motion vector (0, 0) may be a zero vector. A motion vector candidate set to a motion vector (0, 0) may be added to the motion vector candidate list.

The motion vector candidate may not be added to the motion vector list. At this time, the search for the next motion vector candidate may be performed.

In step 3860, if the reference frame of the target block is a non-virtual reference frame and the reference frame of the motion vector candidate is a non-virtual reference frame, a search may be performed for a motion vector candidate that does not use a virtual reference frame. have. Alternatively, if the reference frame index of the target block points to a non-virtual reference frame and the reference frame index of the motion vector candidate points to a non-virtual reference frame, processing for a motion vector candidate not using a virtual reference frame may be performed. .

For example, searching for a motion vector candidate not using a virtual reference frame may mean searching for a motion vector candidate according to HEVC or another embodiment described above.

In one embodiment, when a virtual reference frame is used, the order of searches for the spatial neighboring block and the temporal neighboring block may be changed.

In addition, in one embodiment, when the reference frame of the target block is a virtual reference frame, the motion vector (0, 0) may be inserted in the motion vector candidate list first.

39 is a flowchart of a method of searching for motion vector candidates in an AMVP mode according to an example.

In FIG. 39, the spatial motion vector candidate may be searched before the temporal motion vector candidate.

In step 3910, the motion vector (0, 0) may be selected as a motion vector candidate.

The motion vector (0, 0) may be added to the motion vector candidate list as a motion vector candidate.

In other words, in the embodiment, the motion vector (0, 0) may be preferentially added to the motion vector candidate list.

In step 3920, the spatial motion vector candidate may be searched. The searched spatial motion vector candidate may use a virtual reference frame. There may be a plurality of spatial motion vector candidates.

The search for the motion vector candidate may mean the steps 3810, 3820, 3830, 3840, 3850, and 3860 described above with reference to FIG. 38. In other words, step 3920 may include steps 3810, 3820, 3830, 3840, 3850, and 3860 described above. In this case, the motion vector candidates in steps 3810, 3820, 3830, 3840, 3850, and 3860 may be spatial motion vector candidates.

In step 3930, it may be checked whether the number of the selected motion vector candidates is smaller than the predefined number.

Here, the selected motion vector candidates may mean motion vector candidates in a motion vector candidate list. The predefined number may be the maximum number of motion vector candidates that may be included in the motion vector candidate list.

If the number of the selected motion vector candidates is smaller than the predefined number, step 3940 may be performed. If the number of selected motion vector candidates is not smaller than the predefined number, the procedure may end.

In other words, the search for the temporal motion vector candidate may be selectively performed when the motion vector candidate list is not completely filled with the zero vector and the spatial motion vector candidates.

In step 3940, a temporal motion vector candidate may be searched. The temporal motion vector candidate to be searched may use a virtual reference frame. The temporal motion vector candidate may be plural.

The search for the motion vector candidate may mean the steps 3810, 3820, 3830, 3840, 3850, and 3860 described above with reference to FIG. 38. In other words, step 3940 may include the steps 3810, 3820, 3830, 3840, 3850, and 3860 described above. In this case, the motion vector candidates in steps 3810, 3820, 3830, 3840, 3850, and 3860 may be temporal motion vector candidates.

In an embodiment, in adding the motion vector candidate to the motion vector candidate list, the motion vector (0, 0) may be preferentially added to the motion vector candidate list, and the priorities of the spatial motion vector candidates are determined by the temporal motion vector candidates. May be higher than priorities.

40 is a flowchart of another method of searching for motion vector candidates in an AMVP mode according to an example.

In FIG. 40, a temporal motion vector candidate may be searched before a spatial motion vector candidate.

In step 4010, a temporal motion vector candidate may be searched. The temporal motion vector candidate to be searched may use a virtual reference frame. The temporal motion vector candidate may be plural.

The search for the motion vector candidate may mean the steps 3810, 3820, 3830, 3840, 3850, and 3860 described above with reference to FIG. 38. In other words, step 4010 may include the steps 3810, 3820, 3830, 3840, 3850, and 3860 described above. In this case, the motion vector candidates in steps 3810, 3820, 3830, 3840, 3850, and 3860 may be temporal motion vector candidates.

In step 4020, it may be checked whether the number of the selected motion vector candidates is smaller than the predefined number.

Here, the selected motion vector candidates may mean motion vector candidates in a motion vector candidate list. The predefined number may be the maximum number of motion vector candidates in the motion vector candidate list.

If the number of selected motion vector candidates is smaller than the predefined number, step 4030 may be performed. If the number of selected motion vector candidates is not smaller than the predefined number, the procedure may end.

In other words, the search for the spatial motion vector candidate may be selectively performed when the motion vector candidate list is not completely filled with the temporal motion vector candidates.

In step 4030, the spatial motion vector candidate may be searched. The searched spatial motion vector candidate may use a virtual reference frame. There may be a plurality of spatial motion vector candidates.

The search for the motion vector candidate may mean the steps 3810, 3820, 3830, 3840, 3850, and 3860 described above with reference to FIG. 38. In other words, step 4030 may include the steps 3810, 3820, 3830, 3840, 3850, and 3860 described above. In this case, the motion vector candidates in steps 3810, 3820, 3830, 3840, 3850, and 3860 may be spatial motion vector candidates.

At 4040, it may be checked whether the number of selected motion vector candidates is smaller than the predefined number.

Here, the selected motion vector candidates may mean motion vector candidates in a motion vector candidate list. The predefined number may be the maximum number of motion vector candidates in the motion vector candidate list.

If the number of the selected motion vector candidates is smaller than the predefined number, step 4050 may be performed. If the number of selected motion vector candidates is not smaller than the predefined number, the procedure may end.

In other words, the selection of the motion vector (0, 0) may be selectively performed when the motion vector candidate list is not completely filled with the temporal motion vector candidates and the spatial motion vector candidates.

In step 4050, the motion vector (0, 0) may be selected as a motion vector candidate.

The motion vector (0, 0) may be added to the motion vector candidate list as a motion vector candidate.

In an embodiment, in adding the motion vector candidate to the motion vector candidate list, the priorities of the spatial motion vector candidates may be higher than the priorities of the temporal motion vector candidates. After the spatial motion vector candidates and the temporal motion vector candidates are added to the motion vector candidate list, if the number of motion vector candidates in the motion vector candidate list is smaller than the maximum number of motion vector candidates that the motion vector candidate list can include, The motion vector (0, 0) can then be added to the motion vector candidate list.

The changed order of searching for the motion vector candidate described with reference to FIGS. 39 and 40 may be selectively performed only when the reference frame of the target block is a virtual reference frame. For example, the order of searching for one or more motion vector candidates for the AMVP mode may be determined based on whether the reference frame of the target block is a virtual reference frame.

5.2 Changing the Merge Mode and Skip Mode to Use Virtual Reference Frames

When the inter prediction mode for the target block is the merge mode or the skip mode, the search for the motion vector candidate may be changed and performed as described below.

5.2.1 Searching for Motion Vector Candidates in Temporal Neighboring Blocks

In an embodiment, the motion vector candidate list can be constructed based on whether the reference frame in which the co-located block of the target block exists is a virtual reference frame and a non-virtual reference frame. Here, the collocated block may be a collocated PU. Alternatively, the collocated block can be another block of the embodiments described above.

In addition, the motion vector candidate list may be constructed based on whether the reference frame of the motion vector candidate is a virtual reference frame or a non-virtual reference frame.

In one embodiment, if the reference frame in which the collocated block exists is a virtual reference frame and the reference frame index of the temporal motion vector candidate points to the virtual reference frame, the temporal motion vector candidate is set to the motion vector (0, 0). In addition, a temporal motion vector candidate set as a motion vector (0, 0) may be added to the motion vector candidate list. In addition, the virtual reference frame for the target frame may be used in the merge mode or the skip mode.

In one embodiment, if the reference frame in which the collocated block exists is a virtual reference frame, or if the reference frame index of the temporal motion vector candidate points to the virtual reference frame, the temporal motion vector candidate may be set to the motion vector (0, 0). In addition, a temporal motion vector candidate set as a motion vector (0, 0) may be added to the motion vector candidate list. In addition, the virtual reference frame for the target frame may be used in the merge mode or the skip mode.

In one embodiment, if the reference frame in which the collocated block is present is a virtual reference frame and the reference frame index of the temporal motion vector candidate points to a non-virtual reference frame, one of the following operations may be performed.

A temporal motion vector candidate may be set as a motion vector (0, 0) and a temporal motion vector candidate set as a motion vector (0, 0) may be added to the motion vector candidate list. The virtual reference frame for the target frame may be used in the merge mode or the skip mode.

A temporal motion vector candidate may not be added to the motion vector candidate list.

In one embodiment, if the reference frame in which the collocated block is present is a non-virtual reference frame and the reference frame index of the temporal motion vector candidate points to a virtual reference frame, one of the following operations may be performed.

A temporal motion vector candidate may be set as a motion vector (0, 0) and a temporal motion vector candidate set as a motion vector (0, 0) may be added to the motion vector candidate list. The virtual reference frame for the target frame may be used in the merge mode or the skip mode.

A temporal motion vector candidate may not be added to the motion vector candidate list.

41 is a flowchart of a method of searching for a temporal motion vector candidate according to a reference frame index of a temporal motion vector candidate in a merge mode and a skip mode according to an example.

In step 4110, it may be checked whether a spatial motion vector candidate is searched for. In other words, it may be checked whether the motion vector candidate that is the search target is a spatial motion vector candidate.

If a spatial motion vector candidate is found, step 4120 may be performed.

Step 4130 may be performed when a temporal motion vector candidate is searched instead of the spatial motion vector candidate.

In step 4120, a search may be performed for a spatial motion vector candidate that does not use a virtual reference frame. For example, searching for a spatial motion vector candidate not using a virtual reference frame may mean searching for a spatial motion vector candidate according to HEVC or another embodiment described above.

In operation 4130, it may be determined whether the reference frame index of the target block indicates the virtual reference frame.

If the reference frame index of the target block points to the virtual reference frame, step 4140 may be performed.

If the reference frame index of the target block does not point to the virtual reference frame, step 4150 may be performed.

In step 4140, it may be determined whether the reference frame index of the collocated block points to the virtual reference frame.

If the reference frame index of the collocated block points to the virtual reference frame, step 4160 may be performed.

If the reference frame index of the collocated block does not point to a virtual reference frame, step 4170 may be performed.

In step 4150, it may be determined whether the reference frame index of the collocated block points to the virtual reference frame.

If the reference frame index of the collocated block points to the virtual reference frame, step 4170 may be performed.

If the reference frame index of the collocated block does not point to a virtual reference frame, step 4180 may be performed.

In operation 4160, when the reference frame of the target block is a virtual reference frame and the reference frame of the collocated block is a virtual reference frame, one of the following operations may be performed. Alternatively, if the reference frame index of the target block points to the virtual reference frame and the reference frame index of the collocated block points to the virtual reference frame, one of the following operations may be performed.

A temporal motion vector candidate can be selected. In other words, a temporal motion vector candidate may be added to the motion vector candidate list.

A temporal motion vector candidate may be set to a motion vector (0, 0). The motion vector (0, 0) may be a zero vector. A temporal motion vector candidate set to a motion vector (0, 0) may be selected. A temporal motion vector candidate set to a motion vector (0, 0) may be added to the motion vector candidate list.

In step 4170, if one of the reference frame of the target block and the reference frame of the collocated block is a virtual reference frame and the other is a non-virtual reference frame, one of the following operations may be performed. Alternatively, if one of the reference frame index of the target block and the reference frame index of the collocated block points to the virtual reference frame and the other points to the non-virtual reference frame, one of the following operations may be performed.

A temporal motion vector candidate may be set to a motion vector (0, 0). The motion vector (0, 0) may be a zero vector. A temporal motion vector candidate set to a motion vector (0, 0) may be added to the motion vector candidate list.

A temporal motion vector candidate may not be added to the motion vector list. At this time, the search for the next motion vector candidate may be performed.

In step 4180, if a reference frame of the target block is a non-virtual reference frame and the reference frame of the collocated block is a non-virtual reference frame, a search is performed for a temporal motion vector candidate that does not use a virtual reference frame. Can be. Or, if the reference frame index of the target block points to a non-virtual reference frame and the reference frame index of the collocated block points to a non-virtual reference frame, processing may be performed on a temporal motion vector candidate that does not use a virtual reference frame. have.

For example, the search for a temporal motion vector candidate that does not use a virtual reference frame may mean a search for a temporal motion vector candidate according to HEVC or another embodiment described above.

In one embodiment, when a temporal motion vector candidate refers to a virtual reference frame, the motion vector of the temporal neighboring block may not be considered as a motion vector candidate.

In one embodiment, when a temporal motion vector candidate refers to a virtual reference frame, the motion vector of the temporal neighboring block may be considered as a motion vector candidate preferentially.

42 is a flowchart of a method for searching for a motion vector that does not consider a motion vector of a temporal neighboring block as a motion vector candidate when the temporal motion vector candidate refers to a virtual reference frame according to an embodiment.

In step 4210, the spatial motion vector candidate may be searched. The searched spatial motion vector candidate may use a virtual reference frame. There may be a plurality of spatial motion vector candidates.

The search for the motion vector candidate may mean the steps 3810, 3820, 3830, 3840, 3850, and 3860 described above with reference to FIG. 38. In other words, step 4210 may include the steps 3810, 3820, 3830, 3840, 3850 and 3860 described above. In this case, the motion vector candidates in steps 3810, 3820, 3830, 3840, 3850, and 3860 may be spatial motion vector candidates.

In step 4220, a predefined number of spatial motion vector candidates may be selected.

For example, the number of spatial motion vector candidates selected in step 4220 may be a predefined number.

For example, step 4220 may be repeated until a predefined number of spatial motion vector candidates are selected.

For example, the spatial motion vector candidates selected in step 4220 may be adjusted to a predefined number.

For example, the predefined number may be four.

In step 4230, the motion vector (0, 0) may be selected as the motion vector candidate.

The motion vector (0, 0) may be added to the motion vector candidate list as a motion vector candidate.

In operation 4240, it may be determined whether the inter prediction on the target block is bidirectional prediction.

If the inter prediction for the target block is bidirectional prediction, step 4250 may be performed.

If the inter prediction for the target block is not the bidirectional prediction, the procedure may end.

At 4250, motion vector candidates for bidirectional prediction may be searched.

5.2.2 Searching for motion vector candidates in spatial neighboring blocks

One or more of the following processes may be performed on the motion vector candidate of the spatial neighboring block of the target block.

When the motion vector of the spatial neighboring block of the target block refers to the virtual reference frame, the target block may be encoded by the same encoding scheme as that for other blocks referring to the non-virtual reference frame.

When the motion vector of the spatial neighboring block of the target block refers to the virtual reference frame, the motion vector candidate of the spatial neighboring block may be set to the motion vector (0, 0).

When the motion vector of the spatial neighboring block of the target block refers to the virtual reference frame, the target block may be encoded in the skip mode using the motion vector candidate of the spatial neighboring block.

When the motion vector of the spatial neighboring block of the target block refers to the virtual reference frame, the motion vector of the spatial neighboring block is not considered as a motion vector candidate, and a search for the next neighboring block may be performed.

43 is a flowchart of a method of predicting a target block and a method of generating a bitstream, according to an embodiment.

The prediction method and the bitstream generation method of the target block according to the embodiment may be performed by the encoding apparatus 1600. The embodiment may be part of an encoding method or a video encoding method of a target block.

In operation 4310, the processor 1610 may select a reference frame. The processor may select a reference frame for inter prediction of an embodiment among existing reference frames generated according to the above-described embodiment. Existing reference frames may be non-virtual reference frames.

Step 4310 may correspond to step 3210 described above with reference to FIG. 32.

In operation 4320, the processor 1610 may generate a virtual reference frame based on the selected reference frame.

Step 4320 may correspond to step 3220 described above with reference to FIG. 32.

In operation 4330, the processor 1610 may construct a reference picture list based on the virtual reference frame. The processor 1610 may include the virtual reference frame in the reference picture list.

Step 4330 may correspond to step 3230 described above with reference to FIG. 32.

In operation 4340, the processor 1610 may perform inter prediction based on a virtual reference frame.

Step 4340 may correspond to step 3240 described above with reference to FIG. 32.

Information about the encoded target block may be generated by performing inter prediction on the target block.

A prediction block may be generated by inter prediction on the target block, and a residual block that is a difference between the target block and the prediction block may be generated. The transform and quantization may be applied to the residual block to generate information about the encoded target block.

The information about the encoded target block may include transformed and quantized coefficients for the target block. Also, the information about the encoded target block may include coding parameters for the target block.

In operation 4350, the processor 1610 may generate a bitstream.

The bitstream may include information about the encoded target block.

The bitstream may include prediction information. The prediction information may include information about inter prediction of the above-described target block. In addition, the prediction information may include information about the aforementioned virtual reference frame. The information about the inter prediction of the target block may include coding parameters related to the target block and / or sub-block and the like for the inter prediction described in the embodiments. The information about inter prediction may include the above-described inter prediction information.

Prediction information may be generated at step 4350, or at least partially generated at steps 4310, 4320, 4330, and 4340.

The processor 1610 may store the generated bitstream in the storage 1640. Alternatively, the communication unit 1620 may transmit the bitstream to the decoding device 1700.

The processor 1610 may perform entropy encoding on the prediction information, and may generate a bitstream including the entropy-coded prediction information.

The embodiment may be combined with the operation of the encoding apparatus 100 described above with reference to FIG. 1. For example, operations of steps 4310, 4320, 4330, and 4340 may be performed by the inter predictor 110. Operations of step 4350 may be performed by the entropy encoder 150. In addition, operations performed in other components of the encoding apparatus 100 may be performed before, after, and after steps 4310, 4320, 4330, 4340, and 4350. .

44 is a flowchart of a method of predicting a target block using a bitstream, according to an embodiment.

The prediction method of the target block using the bitstream of the embodiment may be performed by the decoding apparatus 1700. An embodiment may be part of a decoding method or a video decoding method of a target block.

In operation 4410, the communicator 1720 may acquire a bitstream. The communication unit 1720 may receive a bitstream from the encoding apparatus 1600.

The bitstream may include information about the encoded target block.

The information about the encoded target block may include transformed and quantized coefficients for the target block. The information about the encoded target block may include coding parameters for the target block.

The bitstream may include prediction information. The prediction information may include information about inter prediction of the above-described target block. In addition, the prediction information may include information about the aforementioned virtual reference frame. The information about the inter prediction of the target block may include coding parameters related to the target block and / or sub-block and the like for the inter prediction described in the embodiments. The information about inter prediction may include the above-described inter prediction information.

The processor 1710 may store the obtained bitstream in the storage 1740.

The processor 1710 may obtain prediction information from the bitstream. The processor 1710 may obtain prediction information by performing entropy decoding on the entropy-coded prediction information of the bitstream.

In operation 4420, the processor 1710 may select a reference frame. The processor may select a reference frame for inter prediction of an embodiment among existing reference frames generated according to the above-described embodiment. Existing reference frames may be non-virtual reference frames.

Step 4420 may correspond to step 3210 described above with reference to FIG. 32.

In operation 4430, the processor 1710 may generate a virtual reference frame based on the selected reference frame.

Step 4430 may correspond to step 3220 described above with reference to FIG. 32.

In operation 4440, the processor 1710 may configure a reference picture list based on the virtual reference frame. The processor 1710 may include the virtual reference frame in the reference picture list.

Step 4440 may correspond to step 3230 described above with reference to FIG. 32.

In operation 4450, the processor 1710 may perform inter prediction based on the virtual reference frame.

Step 4450 may correspond to step 3240 described above with reference to FIG. 32. In operation 4450, the prediction block may be generated by performing inter prediction on the target block.

In operation 4440, the processor 1710 may reconstruct the target frame based on a result of the inter prediction.

The processor 1710 may generate a reconstructed residual block for the target block.

The processor 1710 may generate a reconstructed block for the target block based on the prediction block and the reconstructed residual block. The reconstructed target frame may include a reconstructed block.

Step 4460 may correspond to step 3250 described above with reference to FIG. 32.

The embodiment may be combined with the operation of the decoding apparatus 200 described above with reference to FIG. 2. For example, operations of step 4410 may be performed by the entropy decoder 210. The operations of steps 4420, 4430, 4440, and 4450 may be performed by the inter prediction unit 250. Operations of step 4460 may be performed by adder 255. In addition, operations performed in other components of the decoding apparatus 200 before, after, and between steps 4410, 4440, 444, 444, 444, and 4460. This can be done.

In the above-described embodiments, the methods are described based on a flowchart as a series of steps or units, but the present invention is not limited to the order of the steps, and any steps may occur in a different order or at the same time than the other steps described above. Can be. Also, one of ordinary skill in the art appreciates that the steps shown in the flowcharts are not exclusive, that other steps may be included, or that one or more steps in the flowcharts may be deleted without affecting the scope of the present invention. I can understand.

Embodiments according to the present invention described above may be implemented in the form of program instructions that may be executed by various computer components, and may be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or may be known and available to those skilled in the computer software arts.

Computer-readable recording media may include information used in embodiments according to the present invention. For example, a computer readable recording medium may include a bitstream, and the bitstream may include information described in embodiments according to the present invention.

Computer-readable recording media can include non-transitory computer-readable medium.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the process according to the invention, and vice versa.

Although the present invention has been described by specific embodiments such as specific components and the like, but the embodiments and the drawings are provided to assist in a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations can be made from these descriptions.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the appended claims, fall within the scope of the spirit of the present invention. I will say.

Claims (20)

Selecting a reference frame;
Generating a virtual reference frame based on the selected reference frame; And
Performing inter prediction based on the virtual reference frame
It includes, the decoding method. The method of claim 1,
And the selected reference frame is plural. The method of claim 1,
And the virtual reference frame is generated based on a deep learning network structure. The method of claim 1,
And the virtual reference frame is generated using a Genetic Adversarial Network (GAN) structure. The method of claim 1,
And the virtual reference frame is generated using an Adaptive Convolution Network (ACN). The method of claim 1,
Wherein the virtual reference frame is generated through interpolation using frames predicted by network structures. The method of claim 1,
And the virtual reference frame is generated based on video interpolation using the selected reference frame. The method of claim 7, wherein
And video prediction by optical flow, ACN or Long Short Term Memory (LSTM) is used for the video interpolation. The method of claim 1,
And the virtual reference frame is generated based on video extrapolation using the selected reference frame. The method of claim 1,
Constructing a reference picture list based on the virtual reference frame
Further comprising, the decoding method. The method of claim 10,
A reference frame specified among reference frames in a decoded picture buffer (DPB) is replaced with the virtual reference frame. The method of claim 1,
And the inter prediction mode of the inter prediction is an advanced motion vector prediction (AMVP) mode. The method of claim 1,
And the inter prediction mode of the inter prediction is a merge mode or a skip mode. The method of claim 1,
And the selected reference frame is a reference frame closest to the distance from the target frame in the reverse or forward direction among the reference frames included in the reference picture list. The method of claim 1,
When one reference frame is selected in both directions, the first difference and the second difference are the same,
The first difference is a difference between a picture order count (POC) of the selected reference frame in a reverse direction and a POC of a target frame,
And the second difference is a difference between a POC of the target frame and a POC of the selected reference frame in the forward direction. The method of claim 1,
Wherein the selected reference frame is a reference frame compressed with the smallest quantization parameter (QP) of reference frames in a decoded picture buffer (DPB). The method of claim 1,
And reference frame specific information indicating the selected reference frame generated for generation of the virtual reference frame is signaled for the specified unit. The method of claim 1,
And the reference frame is selected based on a temporal identifier of the reference frame. Selecting a reference frame;
Generating a virtual reference frame based on the selected reference frame; And
Performing inter prediction based on the virtual reference frame
The encoding method comprising a. Selecting a reference frame;
Generating a virtual reference frame based on the selected reference frame; And
Performing inter prediction based on the virtual reference frame
Including, the inter prediction method.