WO2020180159A1

WO2020180159A1 - Image encoding/decoding method and apparatus, and method for transmitting bitstream

Info

Publication number: WO2020180159A1
Application number: PCT/KR2020/003196
Authority: WO
Inventors: 장형문; 박내리; 남정학
Original assignee: 엘지전자 주식회사
Priority date: 2019-03-06
Filing date: 2020-03-06
Publication date: 2020-09-10

Abstract

An image encoding/decoding method and apparatus are provided. A method by which an image decoding device decodes an image, according to the present disclosure, comprises the steps of: determining whether a prediction mode of the current block is a merge mode with motion vector difference (MMVD) mode; deriving a merge candidate list for the current block when the prediction mode of the current block is determined as the MMVD mode; deriving a predicted motion vector for the current block by using the merge candidate list; deriving a motion vector difference for the current block; setting the resolution of the predicted motion vector; deriving a motion vector for the current block by using the predicted motion vector and the motion vector difference; and obtaining a prediction block for the current block by using the motion vector for the current block, wherein the resolution of the predicted motion vector can be set on the basis of the resolution of the motion vector difference.

Description

Video encoding/decoding method, apparatus, and method for transmitting bitstream

The present disclosure relates to an image encoding/decoding method, an apparatus, and a method of transmitting a bitstream, and more particularly, a method, apparatus and a method for encoding/decoding an image using a merge mode with motion vector differences (MMVD) mode. It relates to a method of transmitting a bitstream generated by the disclosed video encoding method/apparatus.

Recently, demand for high-resolution and high-quality images such as high definition (HD) images and ultra high definition (UHD) images is increasing in various fields. As the image data becomes high-resolution and high-quality, the amount of information or bits to be transmitted increases relative to the existing image data. An increase in the amount of information or bits to be transmitted causes an increase in transmission cost and storage cost.

Accordingly, there is a need for a highly efficient image compression technology for effectively transmitting, storing, and reproducing information of high-resolution and high-quality images.

An object of the present disclosure is to provide an image encoding/decoding method and apparatus with improved encoding/decoding efficiency.

Another object of the present disclosure is to provide a method and apparatus for encoding/decoding an image using a Merge mode with Motion Vector differences (MMVD) mode.

In addition, an object of the present disclosure is to provide a method and apparatus for encoding/decoding an inter-predicted image using an MMVD mode.

In addition, an object of the present disclosure is to provide a method and apparatus for determining a resolution of a motion vector using information related to an MMVD mode.

In addition, an object of the present disclosure is to provide a method for transmitting a bitstream generated by an image encoding method or apparatus according to the present disclosure.

Further, an object of the present disclosure is to provide a recording medium storing a bitstream generated by an image encoding method or apparatus according to the present disclosure.

In addition, an object of the present disclosure is to provide a recording medium storing a bitstream that is received and decoded by an image decoding apparatus according to the present disclosure and used for image restoration.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present disclosure belongs from the following description. I will be able to.

According to an image encoding/decoding method according to an aspect of the present disclosure, motion information for a current block may be derived using an MMVD mode. Specifically, the resolution of the motion vector or the prediction motion vector of the current block may be determined based on the resolution of the motion vector difference. As the resolution of the predicted motion vector or the motion vector is determined based on the resolution of the motion vector difference, the interpolation or rounding process is omitted, thereby simplifying the encoding/decoding process using the MMVD mode and increasing the encoding efficiency. Can be.

An image decoding method according to an aspect of the present disclosure includes determining whether a prediction mode of a current block is a merge mode with motion vector difference (MMVD) mode, and when the prediction mode of the current block is determined to be an MMVD mode, the Deriving a merge candidate list for the current block, deriving a predicted motion vector for the current block using the merge candidate list, deriving motion vector differences for the current block, Setting a resolution of the predicted motion vector, deriving a motion vector for the current block using the predicted motion vector and the motion vector difference, and for the current block using a motion vector for the current block Obtaining a prediction block, wherein the resolution of the prediction motion vector may be set based on the resolution of the difference in the motion vector.

In the video decoding method of the present disclosure, further comprising the step of decoding motion vector difference information for the current block, wherein the motion vector difference information includes size information for the motion vector difference and a direction for the motion vector difference It may include at least one of information and resolution information on the motion vector difference.

In the video decoding method of the present disclosure, the resolution information on the motion vector difference may indicate whether the motion vector difference of the current block uses an integer resolution.

In the video decoding method of the present disclosure, the size of the motion vector difference is derived based on a plurality of tables, and the table used to induce the size of the motion vector difference is based on resolution information on the motion vector difference. Can be selected as

In the image decoding method of the present disclosure, the size information on the motion vector difference may indicate one of values of a table used to induce the motion vector difference.

In the video decoding method of the present disclosure, the direction information on the motion vector difference is one of (+,0), (-,0), (0,+) and (0,-) It can be indicated that it is expressed as a set of fragrance components.

In the video decoding method of the present disclosure, only a predetermined number of candidates among candidates in the merge candidate list may be used for derivation of the predicted motion vector.

In the video decoding method of the present disclosure, setting the resolution of the predicted motion vector may include setting the resolution of the predicted motion vector by rounding the predicted motion vector.

In the image decoding method of the present disclosure, the rounding may be performed based on a first shifting value determined using resolution information on the motion vector difference.

In the image decoding method of the present disclosure, the rounding is performed by applying a left shift operation and a right shift operation to the predicted motion vector, and a shifting value used for the left shift operation and the right shift operation is the first shift Can be determined by value.

An image decoding apparatus according to another aspect of the present disclosure includes a memory and at least one processor, wherein the at least one processor determines whether a prediction mode of a current block is a merge mode with motion vector difference (MMVD) mode, When it is determined that the prediction mode of the current block is the MMVD mode, a merge candidate list for the current block is derived, a predicted motion vector for the current block is derived using the merge candidate list, and Induce motion vector differences, set the resolution of the predicted motion vector, derive a motion vector for the current block using the predicted motion vector and the motion vector difference, and A prediction block for the current block is obtained using a motion vector, and the resolution of the prediction motion vector may be set based on the resolution of the difference in the motion vector.

An image encoding method according to another aspect of the present disclosure includes determining a prediction mode of a current block as a Merge mode with Motion Vector Difference (MMVD) mode, deriving a merge candidate list for the current block, and the merge candidate Deriving a predicted motion vector for the current block using a list, deriving motion vector differences for the current block based on the predicted motion vector, and setting a resolution of the predicted motion vector And information encoding motion vector difference information for the current block, wherein the resolution of the predicted motion vector may be set based on the resolution of the motion vector difference.

In the video encoding method of the present disclosure, the motion vector difference information includes resolution information for the motion vector difference, and the resolution information for the motion vector difference uses an integer resolution for a motion vector difference of the current block. You can indicate whether or not.

A computer-readable recording medium according to another aspect of the present disclosure may store a bitstream generated by the image encoding method or image encoding apparatus of the present disclosure.

Features briefly summarized above with respect to the present disclosure are only exemplary aspects of the detailed description of the present disclosure described below, and do not limit the scope of the present disclosure.

According to the present disclosure, an image encoding/decoding method and apparatus with improved encoding/decoding efficiency may be provided.

In addition, according to the present disclosure, a method and apparatus for encoding/decoding an image using a Merge mode with Motion Vector differences (MMVD) mode may be provided.

Also, according to the present disclosure, a method and apparatus for encoding/decoding an inter-predicted image using an MMVD mode may be provided.

In addition, the present disclosure may provide a method and apparatus for determining a resolution of a motion vector using a syntax element related to an MMVD mode.

In addition, according to the present disclosure, a method for transmitting a bitstream generated by an image encoding method or apparatus according to the present disclosure may be provided.

Further, according to the present disclosure, a recording medium storing a bitstream generated by an image encoding method or apparatus according to the present disclosure may be provided.

In addition, according to the present disclosure, a recording medium may be provided that stores a bitstream that is received and decoded by the image decoding apparatus according to the present disclosure and used for image restoration.

The effects that can be obtained in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

1 is a diagram schematically illustrating a video coding system to which an embodiment according to the present disclosure can be applied.

2 is a diagram schematically illustrating an image encoding apparatus to which an embodiment according to the present disclosure can be applied.

3 is a diagram schematically illustrating an image decoding apparatus to which an embodiment according to the present disclosure can be applied.

4 is a flowchart illustrating a process of generating a prediction block (predicted block) of a current block by performing intra prediction.

5 is a diagram for explaining a decoding method using the MMVD mode.

6 is a diagram for explaining an encoding method using an MMVD mode.

7 is a diagram for describing a method of determining a prediction mode of a current block as an MMVD mode.

8 is a diagram illustrating a structure of a bitstream of motion vector difference information.

9 is a diagram for explaining a method of inducing a motion vector difference.

10 is a diagram for describing an image decoding method according to an embodiment of the present disclosure.

11 is a diagram for describing an image encoding method according to an embodiment of the present disclosure.

12 is a diagram for describing a motion compensation method according to another embodiment of the present disclosure.

13 is a diagram for describing a motion estimation method according to still another embodiment of the present disclosure.

14 is a diagram illustrating a content streaming system to which an embodiment of the present disclosure can be applied.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the embodiments. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

In describing an embodiment of the present disclosure, when it is determined that a detailed description of a known configuration or function may obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In addition, parts not related to the description of the present disclosure in the drawings are omitted, and similar reference numerals are attached to similar parts.

In the present disclosure, when a component is said to be "connected", "coupled" or "connected" with another component, it is not only a direct connection relationship, but an indirect connection relationship in which another component exists in the middle It can also include. In addition, when a certain component "includes" or "have" another component, it means that other components may be further included rather than excluding other components unless otherwise stated. .

In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of the components unless otherwise stated. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment is a first component in another embodiment. It can also be called.

In the present disclosure, components that are distinguished from each other are intended to clearly describe each feature, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to be formed in one hardware or software unit, or one component may be distributed in a plurality of hardware or software units. Therefore, even if not stated otherwise, such integrated or distributed embodiments are also included in the scope of the present disclosure.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, an embodiment consisting of a subset of components described in an embodiment is also included in the scope of the present disclosure. In addition, embodiments including other elements in addition to the elements described in the various embodiments are included in the scope of the present disclosure.

The present disclosure relates to encoding and decoding of an image, and terms used in the present disclosure may have a common meaning commonly used in the technical field to which the present disclosure belongs unless newly defined in the present disclosure.

In the present disclosure, a “picture” generally refers to a unit representing one image in a specific time period, and a slice/tile is a coding unit constituting a part of a picture, and one picture is one It may be composed of more than one slice/tile. In addition, a slice/tile may include one or more coding tree units (CTU).

In the present disclosure, "pixel" or "pel" may mean a minimum unit constituting one picture (or image). Also, as a term corresponding to a pixel, "sample" may be used. A sample may generally represent a pixel or a value of a pixel, may represent only a pixel/pixel value of a luma component, or may represent only a pixel/pixel value of a chroma component.

In the present disclosure, "unit" may represent a basic unit of image processing. The unit may include at least one of a specific area of a picture and information related to the corresponding area. The unit may be used interchangeably with terms such as "sample array", "block", or "area" depending on the case. In general, the MxN block may include samples (or sample arrays) consisting of M columns and N rows, or a set (or array) of transform coefficients.

In the present disclosure, "current block" may mean one of "current coding block", "current coding unit", "coding object block", "decoding object block", or "processing object block". When prediction is performed, “current block” may mean “current prediction block” or “prediction target block”. When transformation (inverse transformation)/quantization (inverse quantization) is performed, "current block" may mean "current transform block" or "transform target block". When filtering is performed, “current block” may mean “block to be filtered”.

In the present disclosure, "/" and "," may be interpreted as "and/or". For example, "A/B" and "A, B" may be interpreted as "A and/or B". In addition, "A/B/C" and "A, B, C" may mean "at least one of A, B and/or C".

In the present disclosure, "or" may be interpreted as "and/or". For example, "A or B" may mean 1) only "A", 2) only "B", or 3) "A and B". Or, in the present disclosure, "or" may mean "additionally or alternatively".

비디오 코딩 시스템 개요Video coding system overview

1 shows a video coding system according to this disclosure.

A video coding system according to an embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 may transmit the encoded video and/or image information or data in a file or streaming format to the decoding device 20 through a digital storage medium or a network.

The encoding apparatus 10 according to an embodiment may include a video source generator 11, an encoder 12, and a transmission unit 13. The decoding apparatus 20 according to an embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoder 12 may be referred to as a video/image encoder, and the decoder 22 may be referred to as a video/image decoder. The transmission unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may include a display unit, and the display unit may be configured as a separate device or an external component.

The video source generator 11 may acquire a video/image through a process of capturing, synthesizing, or generating a video/image. The video source generator 11 may include a video/image capturing device and/or a video/image generating device. The video/image capture device may include, for example, one or more cameras, a video/image archive including previously captured video/images, and the like. The video/image generating device may include, for example, a computer, a tablet and a smartphone, and may (electronically) generate a video/image. For example, a virtual video/image may be generated through a computer or the like, and in this case, a video/image capturing process may be substituted as a process of generating related data.

The encoder 12 may encode an input video/image. The encoder 12 may perform a series of procedures such as prediction, transformation, and quantization for compression and encoding efficiency. The encoder 12 may output encoded data (coded video/image information) in a bitstream format.

The transmission unit 13 may transmit the encoded video/image information or data output in the form of a bitstream to the receiving unit 21 of the decoding apparatus 20 through a digital storage medium or a network in a file or streaming form. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit 13 may include an element for generating a media file through a predetermined file format, and may include an element for transmission through a broadcast/communication network. The receiving unit 21 may extract/receive the bitstream from the storage medium or network and transmit it to the decoding unit 22.

The decoder 22 may decode the video/image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoder 12.

The rendering unit 23 may render the decoded video/image. The rendered video/image may be displayed through the display unit.

영상 부호화 장치 개요Video encoding device overview

As shown in FIG. 2, the image encoding apparatus 100 includes an image segmentation unit 110, a subtraction unit 115, a transform unit 120, a quantization unit 130, an inverse quantization unit 140, and an inverse transform unit ( 150), an addition unit 155, a filtering unit 160, a memory 170, an inter prediction unit 180, an intra prediction unit 185, and an entropy encoding unit 190. The inter prediction unit 180 and the intra prediction unit 185 may be collectively referred to as a “prediction unit”. The transform unit 120, the quantization unit 130, the inverse quantization unit 140, and the inverse transform unit 150 may be included in a residual processing unit. The residual processing unit may further include a subtraction unit 115.

All or at least some of the plurality of constituent units constituting the image encoding apparatus 100 may be implemented as one hardware component (eg, an encoder or a processor) according to embodiments. In addition, the memory 170 may include a decoded picture buffer (DPB), and may be implemented by a digital storage medium.

The image splitter 110 may divide an input image (or picture, frame) input to the image encoding apparatus 100 into one or more processing units. For example, the processing unit may be referred to as a coding unit (CU). The coding unit is a coding tree unit (CTU) or a largest coding unit (LCU) recursively according to a QT/BT/TT (Quad-tree/binary-tree/ternary-tree) structure ( It can be obtained by dividing recursively. For example, one coding unit may be divided into a plurality of coding units of a deeper depth based on a quad tree structure, a binary tree structure, and/or a ternary tree structure. For the division of the coding unit, a quad tree structure may be applied first, and a binary tree structure and/or a ternary tree structure may be applied later. The coding procedure according to the present disclosure may be performed based on the final coding unit that is no longer divided. The largest coding unit may be directly used as the final coding unit, or a coding unit of a lower depth obtained by dividing the largest coding unit may be used as the final cornet unit. Here, the coding procedure may include a procedure such as prediction, transformation, and/or restoration described later. As another example, the processing unit of the coding procedure may be a prediction unit (PU) or a transform unit (TU). Each of the prediction unit and the transform unit may be divided or partitioned from the final coding unit. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for inducing a transform coefficient and/or a unit for inducing a residual signal from the transform coefficient.

The prediction unit (inter prediction unit 180 or intra prediction unit 185) performs prediction on a block to be processed (current block), and generates a predicted block including prediction samples for the current block. Can be generated. The prediction unit may determine whether intra prediction or inter prediction is applied in units of the current block or CU. The prediction unit may generate various information on prediction of the current block and transmit it to the entropy encoding unit 190. The information on prediction may be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

The intra prediction unit 185 may predict the current block by referring to samples in the current picture. The referenced samples may be located in a neighborhood of the current block or may be located away from each other according to an intra prediction mode and/or an intra prediction technique. The intra prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional mode may include, for example, a DC mode and a planar mode (Planar mode). The directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of detail of the prediction direction. However, this is an example, and more or less directional prediction modes may be used depending on the setting. The intra prediction unit 185 may determine a prediction mode applied to the current block by using the prediction mode applied to the neighboring block.

The inter prediction unit 180 may derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on the reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, motion information may be predicted in units of blocks, subblocks, or samples based on a correlation between motion information between a neighboring block and a current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring block may include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different from each other. The temporal neighboring block may be referred to as a collocated reference block, a collocated CU (colCU), or the like. A reference picture including the temporal neighboring block may be referred to as a collocated picture (colPic). For example, the inter prediction unit 180 constructs a motion information candidate list based on neighboring blocks, and provides information indicating which candidate is used to derive a motion vector and/or a reference picture index of the current block. Can be generated. Inter prediction may be performed based on various prediction modes. For example, in the case of a skip mode and a merge mode, the inter prediction unit 180 may use motion information of a neighboring block as motion information of a current block. In the case of the skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, motion vectors of neighboring blocks are used as motion vector predictors, and indicators for motion vector difference and motion vector predictors ( indicator) to signal the motion vector of the current block. The motion vector difference may mean a difference between a motion vector of a current block and a motion vector predictor.

The prediction unit may generate a prediction signal based on various prediction methods and/or prediction techniques to be described later. For example, the prediction unit may apply intra prediction or inter prediction for prediction of the current block, and may simultaneously apply intra prediction and inter prediction. A prediction method in which intra prediction and inter prediction are applied simultaneously for prediction of a current block may be called combined inter and intra prediction (CIIP). Also, the prediction unit may perform intra block copy (IBC) for prediction of the current block. The intra block copy may be used for content image/movie coding such as games, such as, for example, screen content coding (SCC). IBC is a method of predicting a current block by using a reference block in a current picture at a distance from the current block by a predetermined distance. When IBC is applied, the position of the reference block in the current picture may be encoded as a vector (block vector) corresponding to the predetermined distance.

The prediction signal generated through the prediction unit may be used to generate a reconstructed signal or may be used to generate a residual signal. The subtraction unit 115 subtracts the prediction signal (predicted block, prediction sample array) output from the prediction unit from the input image signal (original block, original sample array), and subtracts a residual signal (remaining block, residual sample array). ) Can be created. The generated residual signal may be transmitted to the converter 120.

The transform unit 120 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transformation technique uses at least one of DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Can include. Here, GBT refers to the transformation obtained from this graph when the relationship information between pixels is expressed in a graph. CNT refers to a transform obtained based on generating a prediction signal using all previously reconstructed pixels. The conversion process may be applied to a block of pixels having the same size of a square, or may be applied to a block of a variable size other than a square.

The quantization unit 130 may quantize the transform coefficients and transmit the quantization to the entropy encoding unit 190. The entropy encoding unit 190 may encode a quantized signal (information on quantized transform coefficients) and output it as a bitstream. The information on the quantized transform coefficients may be called residual information. The quantization unit 130 may rearrange the quantized transform coefficients in the form of a block into a one-dimensional vector form based on a coefficient scan order, and the quantized transform coefficients in the form of the one-dimensional vector It is also possible to generate information about transform coefficients.

The entropy encoding unit 190 may perform various encoding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoding unit 190 may encode together or separately information necessary for video/image restoration (eg, values of syntax elements) in addition to quantized transform coefficients. The encoded information (eg, encoded video/video information) may be transmitted or stored in a bitstream format in units of network abstraction layer (NAL) units. The video/video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/video information may further include general constraint information. The signaling information, transmitted information, and/or syntax elements mentioned in the present disclosure may be encoded through the above-described encoding procedure and included in the bitstream.

The bitstream may be transmitted through a network or may be stored in a digital storage medium. Here, the network may include a broadcasting network and/or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmission unit (not shown) for transmitting the signal output from the entropy encoding unit 190 and/or a storage unit (not shown) for storing may be provided as an inner/outer element of the image encoding apparatus 100, or transmission The unit may be provided as a component of the entropy encoding unit 190.

The quantized transform coefficients output from the quantization unit 130 may be used to generate a residual signal. For example, a residual signal (residual block or residual samples) may be restored by applying inverse quantization and inverse transform to the quantized transform coefficients through the inverse quantization unit 140 and the inverse transform unit 150.

The addition unit 155 adds the reconstructed residual signal to the prediction signal output from the inter prediction unit 180 or the intra prediction unit 185 to obtain a reconstructed signal (a reconstructed picture, a reconstructed block, and a reconstructed sample array). Can be generated. When there is no residual for a block to be processed, such as when the skip mode is applied, the predicted block may be used as a reconstructed block. The addition unit 155 may be referred to as a restoration unit or a restoration block generation unit. The generated reconstructed signal may be used for intra prediction of the next processing target block in the current picture, and may be used for inter prediction of the next picture through filtering as described later.

Meanwhile, luma mapping with chroma scaling (LMCS) may be applied in the picture encoding process as described later.

The filtering unit 160 may apply filtering to the reconstructed signal to improve subjective/objective image quality. For example, the filtering unit 160 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and the modified reconstructed picture may be converted to the memory 170, specifically, the DPB of the memory 170. Can be saved on. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, and the like. The filtering unit 160 may generate a variety of filtering information and transmit it to the entropy encoding unit 190 as described later in the description of each filtering method. The filtering information may be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

The modified reconstructed picture transmitted to the memory 170 may be used as a reference picture in the inter prediction unit 180. When inter prediction is applied through this, the image encoding apparatus 100 may avoid prediction mismatch between the image encoding apparatus 100 and the image decoding apparatus, and may improve encoding efficiency.

The DPB in the memory 170 may store a reconstructed picture modified to be used as a reference picture in the inter prediction unit 180. The memory 170 may store motion information of a block from which motion information in a current picture is derived (or encoded) and/or motion information of blocks in a picture that have already been reconstructed. The stored motion information may be transmitted to the inter prediction unit 180 to be used as motion information of spatial neighboring blocks or motion information of temporal neighboring blocks. The memory 170 may store reconstructed samples of reconstructed blocks in the current picture, and may transmit the reconstructed samples to the intra prediction unit 185.

영상 복호화 장치 개요Overview of video decoding device

As shown in FIG. 3, the image decoding apparatus 200 includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an addition unit 235, a filtering unit 240, and a memory 250. ), an inter prediction unit 260 and an intra prediction unit 265 may be included. The inter prediction unit 260 and the intra prediction unit 265 may be collectively referred to as a “prediction unit”. The inverse quantization unit 220 and the inverse transform unit 230 may be included in the residual processing unit.

All or at least some of the plurality of constituent units constituting the image decoding apparatus 200 may be implemented as one hardware component (eg, a decoder or a processor) according to embodiments. Also, the memory 170 may include a DPB and may be implemented by a digital storage medium.

The image decoding apparatus 200 receiving a bitstream including video/image information may reconstruct an image by performing a process corresponding to the process performed by the image encoding apparatus 100 of FIG. 1. For example, the image decoding apparatus 200 may perform decoding using a processing unit applied in the image encoding apparatus. Thus, the processing unit of decoding may be, for example, a coding unit. The coding unit may be a coding tree unit or may be obtained by dividing the largest coding unit. In addition, the reconstructed image signal decoded and output through the image decoding apparatus 200 may be reproduced through a reproduction device (not shown).

The image decoding apparatus 200 may receive a signal output from the image encoding apparatus of FIG. 1 in the form of a bitstream. The received signal may be decoded through the entropy decoding unit 210. For example, the entropy decoding unit 210 may parse the bitstream to derive information (eg, video/video information) necessary for image restoration (or picture restoration). The video/video information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/video information may further include general constraint information. The image decoding apparatus may additionally use information on the parameter set and/or the general restriction information to decode an image. The signaling information, received information, and/or syntax elements mentioned in the present disclosure may be obtained from the bitstream by being decoded through the decoding procedure. For example, the entropy decoding unit 210 decodes information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and a value of a syntax element required for image restoration, a quantized value of a transform coefficient related to a residual. Can be printed. In more detail, the CABAC entropy decoding method receives a bin corresponding to each syntax element in a bitstream, and includes information on the syntax element to be decoded, decoding information of the neighboring block and the block to be decoded, or information of the symbol/bin decoded in the previous step. The context model is determined by using and, according to the determined context model, the probability of occurrence of bins is predicted to perform arithmetic decoding of the bins to generate a symbol corresponding to the value of each syntax element. I can. In this case, the CABAC entropy decoding method may update the context model using information of the decoded symbol/bin for the context model of the next symbol/bin after the context model is determined. Among the information decoded by the entropy decoding unit 210, information on prediction is provided to the prediction unit (inter prediction unit 260 and intra prediction unit 265), and the register on which entropy decoding is performed by the entropy decoding unit 210 Dual values, that is, quantized transform coefficients and related parameter information may be input to the inverse quantization unit 220. In addition, information about filtering among information decoded by the entropy decoding unit 210 may be provided to the filtering unit 240. Meanwhile, a receiving unit (not shown) for receiving a signal output from the image encoding device may be additionally provided as an inner/outer element of the image decoding device 200, or the receiving unit is provided as a component of the entropy decoding unit 210 It could be.

Meanwhile, the video decoding apparatus according to the present disclosure may be referred to as a video/video/picture decoding apparatus. The video decoding apparatus may include an information decoder (video/video/picture information decoder) and/or a sample decoder (video/video/picture sample decoder). The information decoder may include an entropy decoding unit 210, and the sample decoder includes an inverse quantization unit 220, an inverse transform unit 230, an addition unit 235, a filtering unit 240, a memory 250, It may include at least one of the inter prediction unit 260 and the intra prediction unit 265.

The inverse quantization unit 220 may inverse quantize the quantized transform coefficients and output transform coefficients. The inverse quantization unit 220 may rearrange the quantized transform coefficients into a two-dimensional block shape. In this case, the rearrangement may be performed based on a coefficient scan order performed by the image encoding apparatus. The inverse quantization unit 220 may perform inverse quantization on quantized transform coefficients by using a quantization parameter (eg, quantization step size information) and obtain transform coefficients.

The inverse transform unit 230 may inverse transform the transform coefficients to obtain a residual signal (residual block, residual sample array).

The prediction unit may perform prediction on the current block and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 210, and determine a specific intra/inter prediction mode (prediction technique). I can.

It is the same as described in the description of the prediction unit of the video encoding apparatus 100 that the prediction unit can generate the prediction signal based on various prediction methods (techniques) described later.

The intra prediction unit 265 may predict the current block by referring to samples in the current picture. The description of the intra prediction unit 185 may be equally applied to the intra prediction unit 265.

The inter prediction unit 260 may derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on the reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, motion information may be predicted in units of blocks, subblocks, or samples based on a correlation between motion information between a neighboring block and a current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring block may include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. For example, the inter prediction unit 260 may construct a motion information candidate list based on neighboring blocks, and derive a motion vector and/or a reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes (techniques), and the information about the prediction may include information indicating a mode (technique) of inter prediction for the current block.

The addition unit 235 is reconstructed by adding the obtained residual signal to the prediction signal (predicted block, prediction sample array) output from the prediction unit (including the inter prediction unit 260 and/or the intra prediction unit 265). Signals (restored pictures, reconstructed blocks, reconstructed sample arrays) can be generated. The description of the addition unit 155 may be equally applied to the addition unit 235.

Meanwhile, luma mapping with chroma scaling (LMCS) may be applied in the picture decoding process as described later.

The filtering unit 240 may apply filtering to the reconstructed signal to improve subjective/objective image quality. For example, the filtering unit 240 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and the modified reconstructed picture may be converted to the memory 250, specifically the DPB of the memory 250. Can be saved on. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, and the like.

The (modified) reconstructed picture stored in the DPB of the memory 250 may be used as a reference picture in the inter prediction unit 260. The memory 250 may store motion information of a block from which motion information in a current picture is derived (or decoded) and/or motion information of blocks in a picture that have already been reconstructed. The stored motion information may be transmitted to the inter prediction unit 260 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 250 may store reconstructed samples of reconstructed blocks in the current picture, and may be transmitted to the intra prediction unit 265.

In this specification, the embodiments described in the filtering unit 160, the inter prediction unit 180, and the intra prediction unit 185 of the encoding apparatus 100 are, respectively, the filtering unit 240 and the inter prediction unit of the image decoding apparatus 200. The same or corresponding to the prediction unit 260 and the intra prediction unit 265 may be applied.

인터 예측 개요Inter prediction overview

4 is a flowchart illustrating a process of generating a prediction block (predicted block) of a current block by performing inter prediction.

The process illustrated in FIG. 4 may be performed by the inter prediction unit 180 of FIG. 2 and/or the inter prediction unit 260 of FIG. 3.

The apparatus for encoding/decoding an image may derive a prediction sample by performing inter prediction in block units. Inter prediction may mean a prediction technique derived by a method dependent on data elements of picture(s) other than the current picture. When inter prediction is applied to the current block, a prediction block for the current block may be derived based on a reference block specified by a motion vector on a reference picture.

At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, motion information of the current block may be derived based on the correlation of motion information between the neighboring block and the current block, and motion information in units of blocks, sub-blocks, or samples. Can be induced. In this case, the motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction type information. Here, the inter prediction type information may mean directional information of inter prediction. The inter prediction type information may indicate that the current block is predicted using one of L0 prediction, L1 prediction, and Bi prediction.

When inter prediction is applied to the current block, the neighboring blocks of the current block may include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. In this case, the reference picture including the reference block for the current block and the reference picture including the temporal neighboring block may be the same or different. The temporal neighboring block may be referred to as a collocated reference block, a co-located coding unit (colCU), and the like, and a reference picture including the temporal neighboring block may be referred to as a collocated picture (colPic). I can.

Meanwhile, a motion information candidate list may be configured based on neighboring blocks of the current block. In this case, a flag or index information indicating which candidate is used to derive a motion vector and/or a reference picture index of the current block is provided. Can be signaled.

The motion information may include L0 motion information and/or L1 motion information according to the inter prediction type. The motion vector in the L0 direction may be defined as an L0 motion vector or MVL0, and the motion vector in the L1 direction may be defined as an L1 motion vector or MVL1. The prediction based on the L0 motion vector may be defined as L0 prediction, the prediction based on the L1 motion vector may be defined as L1 prediction, and the prediction based on both the L0 motion vector and the L1 motion vector is paired (Bi) prediction. Can be defined as Here, the motion vector L0 may mean a motion vector associated with the reference picture list L0, and the motion vector L1 may mean a motion vector associated with the reference picture list L1.

The reference picture list L0 may include pictures prior to the current picture in output order as reference pictures, and the reference picture list L1 may include pictures after the current picture in output order. In this case, previous pictures may be defined as forward (reference) pictures, and subsequent pictures may be defined as backward (reference) pictures. Meanwhile, the reference picture list L0 may further include pictures after the output order than the current picture. In this case, previous pictures in the reference picture list L0 may be indexed first, and pictures afterwards may be indexed next. The reference picture list L1 may further include previous pictures in output order than the current picture. In this case, in the reference picture list L1, subsequent pictures may be indexed first, and previous pictures may be indexed next. Here, the output order may correspond to a picture order count (POC) order.

In the video encoding apparatus, the determination of whether to perform inter prediction of FIG. 4 on the current block may be performed based on rate-distortion optimization (RDO). For example, when inter prediction is more efficient than intra prediction, the prediction mode for the current block may be determined as the inter prediction mode. However, in the video encoding apparatus, a method of determining a prediction mode for a current block is not limited to the above example.

In the image decoding apparatus, determination of whether to perform inter prediction of FIG. 4 on the current block may be performed based on information on the prediction mode of the current block. The information on the prediction mode of the current block may be information indicating whether the prediction mode of the current block is an inter prediction mode or an intra prediction mode. Information about the prediction mode of the current block may be explicitly signaled through a bitstream, or may be implicitly derived based on an encoding parameter about the current block.

Referring to FIG. 4, in an image encoding/decoding method according to an embodiment of the present disclosure, determining an inter prediction technique (S410), and deriving motion information for a current block based on the determined inter prediction technique (S420). ) And/or performing inter prediction using the derived motion information (S430).

When it is determined that inter prediction is to be performed on the current block, the apparatus for encoding/decoding an image may determine at least one inter prediction technique that can be used for inter prediction. In the following specification, a prediction technique that can be used to perform inter prediction is defined as an inter prediction technique. For example, the inter prediction technique may mean one of an encoding/decoding method used to derive motion information for a current block and/or an encoding/decoding method used to derive a prediction sample.

That is, based on the determined inter prediction technique, at least one of steps S420 and S430 of FIG. 4 may be performed.

In step S410, an inter prediction technique of the current block may be determined. For example, the inter prediction technique for the current block is a merge mode, merge skip mode, AMVP mode (Advanced Motion Vector Prediction mode), affine mode, and subblock-based merge mode. merge mode), AMVR mode (Adaptive Motion Vector Resolution mode), HMVP mode (History-based Motion Vector Predictor mode), Pair-wise average merge mode, MMVD mode (Merge mode with Motion Vector Differences mode) , DMVR mode (Decoder side Motion Vector Refinement mode), CIIP mode (Combined Inter and Intra Prediction mode), and GPM (Geometric Partitioning mode) may be determined at least one of. In step S420, motion information on the current block may be derived based on at least one inter prediction technique determined in step S410.

For example, when the inter prediction scheme of the current block is determined as the merge mode, motion information of the current block is not signaled directly, but may be derived using motion information of neighboring blocks. The video encoding apparatus may signal flag information indicating that a merge mode has been used and a merge index indicating which prediction blocks are used. Meanwhile, the video decoding apparatus may derive motion information of the current block using signaled flag information and merge index.

The image encoding apparatus may determine a merge candidate block used to induce motion information of a current block in order to perform a merge mode. As an example, up to five merge candidate blocks may be used. As another example, the maximum number of merge candidate blocks may be signaled through a slice header or a tile group header. The image encoding apparatus may generate a merge candidate list and select a merge candidate block having the lowest cost among them as a final merge candidate block. Meanwhile, the image decoding apparatus may generate the same merge candidate list as the image encoding apparatus, and select a final merge candidate block by using the signaled merge index.

As another example, when the inter prediction technique of the current block is determined as the skip mode, motion information of the current block may be derived in the same manner as when the merge mode is applied to the current block. However, when the skip mode is applied to the current block, the residual signal for the corresponding block may be omitted. In this case, the derived prediction samples can be directly used as reconstructed samples for the current block.

As another example, when the inter prediction method of the current block is determined as the AMVP mode, the image encoding/decoding apparatus predicts a motion vector by using a motion vector of a reconstructed spatial neighboring block and/or a motion vector corresponding to a temporal neighboring block. A motion vector predictor (MVP) candidate list can be generated. A motion vector predictor candidate may be derived using the generated motion vector predictor candidate list. Motion information of the current block may be determined based on the derived motion vector predictor candidate.

The image encoding apparatus may calculate a motion vector difference (MVD) by using a difference between a motion vector of a current block and a motion vector predictor candidate, and entropy-encode the MVD. In addition, the image encoding apparatus may generate a bitstream by entropy encoding the motion vector predictor candidate index. The motion vector predictor candidate index may indicate an optimal motion vector predictor candidate from among motion vector predictor candidates included in the motion vector candidate predictor list. On the other hand, the image decoding apparatus entropy-decodes the motion vector candidate predictor index from the bitstream, and uses the entropy-decoded motion vector predictor candidate index from among the motion vector predictor candidates included in the motion vector predictor candidate list. Can select a motion vector predictor candidate of. Also, the decoding apparatus may derive the motion vector of the current block through the sum of the entropy-decoded MVD and the motion vector predictor candidate.

In step S430, inter prediction on the current block may be performed using the motion information derived in step S420. That is, inter prediction is performed on the current block, and a prediction block for the current block may be derived. The prediction block may include prediction samples (prediction sample array) of the current block. When the motion vector of the current block has fractional resolution, an interpolation procedure for the prediction block may be performed. Through the interpolation procedure, prediction samples of the current block may be derived based on the reference samples of the fractional resolution unit. Meanwhile, when bi-prediction is applied, prediction samples for the current block may be derived through a weighted sum or weighted average of prediction samples derived based on L0 prediction and prediction samples derived based on L1 prediction.

Merge mode with Motion Vector Differences (MMVD) mode overview

In order to increase the accuracy of motion vectors derived through merge mode or skip mode, MMVD mode can be applied to the current block. That is, when the inter prediction technique of the current block is determined as the merge mode or the skip mode, the MMVD mode may be additionally applied to the current block. When the MMVD mode is applied, motion vectors derived from neighboring blocks may be more precisely corrected, and thus image encoding efficiency may be improved.

5 is a diagram for explaining an image decoding method using an MMVD mode.

5, the image decoding method using the MMVD mode includes determining whether the prediction mode of the current block is the MMVD mode (S510). When the prediction mode for the current block is determined as the MMVD mode, the current block is Deriving a merge candidate list for the current block (S520), deriving a predicted motion vector for the current block using the merge candidate list (S530), and deriving motion vector differences for the current block ( S540) and/or the step of deriving a motion vector of the current block by using the predicted motion vector and the motion vector difference (S550). In this case, the motion vector difference for the current block may be derived based on motion vector difference information signaled from the image encoding apparatus.

6 is a diagram for describing an image encoding method using an MMVD mode.

Referring to FIG. 6, the method of encoding an image using the MMVD mode includes determining a prediction mode of a current block as an MMVD mode (S610), deriving a merge candidate list for the current block (S620), and a merge candidate list. Deriving a predicted motion vector for the current block by using (S630), deriving motion vector differences for the current block based on the predicted motion vector (S640) and/or motion vector difference information It may include an encoding step (S650). At this time, motion vector difference information for the current block may be determined based on the derived motion vector difference.

Hereinafter, a video encoding/decoding method using the MMVD mode will be described in detail.

First, the apparatus for encoding an image may determine a prediction mode of the current block as the MMVD mode (S510), and may encode information indicating that the prediction mode of the current block is the MMVD mode. Meanwhile, the image decoding apparatus may determine whether the prediction mode of the current block is the MMVD mode based on the signaled information (S610).

7 is a diagram for describing a method of determining/determining a prediction mode of a current block as an MMVD mode.

For example, according to the flowchart of FIG. 7, the prediction mode of the current block may be determined as the MMVD mode. The syntax elements disclosed in FIG. 7 are examples, and the scope of the present disclosure is not limited by the names of the illustrated syntax elements. In addition, specific values of the syntax elements in the following embodiments are examples, and the scope of the present disclosure is not limited by the following description. Each syntax element can be expressed as an arbitrary value (for example, a first value or a second value), and a value of 1 when a specific syntax element has a value of 0, and a value of 0 when a specific syntax element has a value of 1. It can also be expressed as having In addition, the image decoding apparatus may decode information on an inter prediction mode based on the flowchart of FIG. 7 and determine an inter prediction method for a current block based on this. On the other hand, the image encoding apparatus may encode information on an inter prediction mode applied to the current block based on FIG. 7.

First, it may be determined whether the skip mode is applied to the current block by the CU_SKIP_FLAG value. For example, when the value of CU_SKIP_FLAG is 1, the skip mode is applied to the current block, and when the value of CU_SKIP_FLAG is 0, the skip mode may not be applied to the current block.

When the skip mode is applied to the current block, it may be determined whether the MMVD mode is applied to the current block by the MMVD_FLAG value. If the MMVD_FLAG value is 1, the MMVD mode may be applied to the current block. When the MMVD mode is applied to the current block, MMVD_IDX for inducing a motion vector difference may be additionally encoded/decoded.

On the other hand, when the value of MMVD_FLAG is 0, the MMVD mode may not be applied to the current block. When the MMVD mode is not applied to the current block, it may be determined whether the subblock-based skip mode is applied to the current block by the value of SB_MRG_FLAG. For example, when the SB_MRG_FLAG value is 1, a subblock-based skip mode may be applied to the current block. When the subblock-based skip mode is applied to the current block, MRG_IDX for deriving a motion vector for each subblock may be additionally encoded/decoded. On the other hand, when the SB_MRG_FLAG value is 0, the subblock-based skip mode may not be applied to the current block.

When the subblock-based skip mode is not applied to the current block, the triangulation skip mode may be applied to the current block. In the following specification, the triangulation mode may mean the aforementioned GPM. For example, whether or not a triangular skip mode is applied to the current block may be determined by the TRI_FLAG value, and MRG_IDX for deriving a motion vector according to the triangulation skip mode may be additionally encoded/decoded.

Meanwhile, when the skip mode is not applied to the current block (CU_SKIP_FLAG==0), the prediction mode of the current block may be determined as one of an intra prediction mode and an inter prediction mode according to the PRED_MODE value. When the prediction mode of the current block is determined as the inter prediction mode, whether the merge mode is applied to the current block may be determined based on the MRG_FLAG value. For example, when the MRG_FLAG value is 0, the AMVP mode may be applied to the current block. On the other hand, when the MRG_FLAG value is 1, the merge mode may be applied to the current block.

When the merge mode is applied to the current block, it may be determined whether the MMVD mode is applied to the current block by the MMVD_FLAG value. If the MMVD_FLAG value is 1, the MMVD mode may be applied to the current block. When the MMVD mode is applied to the current block, MMVD_IDX for inducing a motion vector difference may be additionally encoded/decoded.

On the other hand, when the value of MMVD_FLAG is 0, the MMVD mode may not be applied to the current block. When the MMVD mode is not applied to the current block, it may be determined whether or not the subblock-based merge mode is applied to the current block by the value of SB_MRG_FLAG. For example, when the SB_MRG_FLAG value is 1, a subblock based merge mode may be applied to the current block. When the subblock-based merge mode is applied to the current block, MRG_IDX for inducing a motion vector for each subblock may be additionally encoded/decoded. On the other hand, when the SB_MRG_FLAG value is 0, the subblock based merge mode may not be applied to the current block.

When the sub-block-based merge mode is not applied to the current block, it may be determined whether the intra-inter combining merge mode is applied to the current block by the MHINTRA_FLAG value. For example, when the MHINTRA_FLAG value is 1, the intra-inter combining merge mode may be applied to the current block. When the intra-inter combining merge mode is applied to the current block, MHINTRA_MODE for inducing the intra mode of the intra-inter combining merge mode may be additionally encoded/decoded. On the other hand, when the MHINTRA_FLAG value is 0, the intra-inter combining merge mode may not be applied to the current block. In the following specification, the intra-inter combining merge mode may mean the aforementioned CIIP mode.

When the intra-inter combining merge mode is not applied to the current block, it may be determined whether or not the triangular division merge mode is applied to the current block by the TRI_FLAG value. For example, when the TRI_FLAG value is 1, the triangulation merge mode may be applied to the current block. When the triangulation merge mode is applied to the current block, MRG_IDX for deriving a motion vector according to the triangulation merge mode may be additionally encoded/decoded. On the other hand, when the TRI_FLAG value is 0, the triangulation merge mode may not be applied to the current block.

When the triangulation merge mode is not applied to the current block, the general merge mode may be applied to the current block. When the general merge mode is applied to the current block, MRG_IDX for determining a merge candidate according to the general merge mode may be additionally encoded/decoded. That is, the general merge mode may be applied to the current block only when at least one of the MMVD mode, the subblock-based merge mode, the intra-inter combination merge mode, and the triangular division merge mode is not performed for the current block.

Referring to a portion indicated by a gray shade in FIG. 7, when the skip mode or merge mode is performed on the current block, the MMVD mode may be applied to the current block.

When the MMVD mode is applied to the current block, the image encoding apparatus may encode information indicating that the MMVD mode is applied to the current block. Meanwhile, the image decoding apparatus may determine whether the MMVD mode is applied to the current block based on the information signaled from the image encoding apparatus. For example, information signaled from the image encoding apparatus to the image decoding apparatus may be expressed as at least one of a syntax element mmvd_flag or mmvd_merge_flag. When mmvd_flag has the first value, the MMVD mode may be applied to the current block. On the other hand, when mmvd_flag has a second value, the MMVD mode may not be applied to the current block.

Next, the video encoding/decoding apparatus may derive a merge candidate list for the current block (S520 and S620). The apparatus for encoding/decoding an image may derive a merge candidate list for the current block by using at least one of conventional merge candidate list construction methods.

Next, the apparatus for encoding/decoding an image may derive a predicted motion vector for the current block (S530 and S630), and may derive a motion vector difference for the current block (S530 and S630).

The image encoding apparatus may derive and encode motion vector difference information used to induce a motion vector difference of a current block. Meanwhile, the image decoding apparatus may decode motion vector difference information signaled from the image encoding apparatus, and derive a predicted motion vector and a motion vector difference for the current block based on this. As an example, the motion vector difference information may include at least one of merge candidate indication information, size information on the motion vector difference, and direction information. Here, the merge candidate indication information may be information indicating a merge candidate used to induce a prediction motion vector of a current block among merge candidates according to the merge candidate list.

8 is a diagram illustrating a structure of a bitstream of motion vector difference information. Referring to FIG. 8, motion vector difference information to a current block may be encoded/decoded only when the MMVD mode is applied to the current block. The syntax element mmvd_flag may be an indicator indicating whether to apply the MMVD mode. For example, when mmvd_flag indicates the first value, the MMVD mode may be applied to the current block. Meanwhile, when mmvd_flag indicates the second value, the MMVD mode may not be applied to the current block. As another example, unlike the illustration of FIG. 8, as described above, the syntax element mmvd_flag may also be referred to as mmvd_merge_flag.

When the prediction mode of the current block is determined as the MMVD mode (mmvd_flag==1), at least one of the syntax elements mmvd_merge_flag, mmvd_distance_idx, and mmvd_direction_idx may be encoded/decoded.

The syntax element mmvd_merge_flag may be an indicator indicating one candidate among candidates in the merge candidate list. For example, mmvd_merge_flag may correspond to the above-described merge candidate indication information. The syntax element mmvd_merge_flag may also be named mmvd_cand_flag. For example, mmvd_merge_flag may indicate one of a first candidate and a second candidate in the merge candidate list. The maximum number of available merge candidates of the current block may be determined according to whether the MMVD mode is applied to the current block (mmvd_flag value). That is, when the MMVD mode is applied to the current block, only the first and second candidates among the candidates in the merge candidate list may be used for derivation of the predicted motion vector.

Meanwhile, the merge index according to the general merge mode may be named as a syntax element merge_idx. For example, when MMVD is applied to the current block, the value of merge_idx may be set as the value of mmvd_merge_flag. That is, the merge candidate for deriving the prediction motion vector of the current block may be determined by the merge_idx set to mmvd_merge_flag.

The motion vector difference can be signaled using a direction component and a magnitude component. In the following embodiments, MmvdSign may mean a direction component of a motion vector difference, and MmvdDistance may mean a magnitude component of a motion vector difference. The direction component and magnitude component of the motion vector difference can mean the direction component and the actual vector size value of the motion vector difference, as well as the information used to derive the direction component and the actual vector size value of the motion vector difference. I can. Meanwhile, MmvdOffset may mean an actual vector size value for a motion vector difference.

The syntax element mmvd_direction_idx may indicate a direction component (MmvdSign) of a motion vector difference. In this case, mmvd_direction_idx may indicate that the motion vector difference is expressed using one of a direction component set of (+,0), (-,0), (0,+), and (0,-).

The syntax element mmvd_distance_idx may indicate a size component (MmvdDistance) of a motion vector difference. As an example, mmvd_distance_idx may indicate that the residual motion vector has one of 1, 2, 4, 8, 16, 32, 64, 128, 256, and 512 size components.

Meanwhile, the motion vector difference information may additionally include resolution information on the motion vector difference. The resolution information on the motion vector difference may be information indicating whether the motion vector difference of the current block uses an integer resolution.

For example, the resolution information on the motion vector difference may be named as a syntax element fpel_mmvd_enabled_flag. Resolution information about the motion vector difference may be signaled at at least one of a sequence level, a picture level, a tile level, a tile group level, and a slice level. For example, the resolution information on the motion vector difference may be named sps_fpel_mmvd_enabled_flag, tile_group_fpel_mmvd_enabled_flag, ph_fpel_mmvd_enabled_flag, and the like according to the signaled level.

For example, when sps_fpel_mmvd_enabled_flag is signaled as a first value, at least one of tile_group_fpel_mmvd_enabled_flag or ph_fpel_mmvd_enabled_flag may be signaled. Meanwhile, when sps_fpel_mmvd_enabled_flag is signaled as the second value, tile_group_fpel_mmvd_enabled_flag and ph_fpel_mmvd_enabled_flag may not be signaled. When the values of tile_group_fpel_mmvd_enabled_flag and ph_fpel_mmvd_enabled_flag are not separately signaled, the values of tile_group_fpel_mmvd_enabled_flag and ph_fpel_mmvd_enabled_flag may be set to second values, respectively. Hereinafter, the first value and the second value may mean '1' and '0', respectively, and conversely, may mean '0' and '1'. Hereinafter, tile_group_fpel_mmvd_enabled_flag and ph_fpel_mmvd_enabled_flag may be used as syntax elements indicating the same condition.

In this case, the motion vector difference may be derived based on at least one of the above-described syntax elements mmvd_distance_idx, mmvd_direction_idx, and tile_group_fpel_mmvd_enabled_flag.

As an example, the magnitude component of the motion vector difference may be determined according to Table 1 below.

[Table 1]

Referring to Table 1, mmvd_distance_idx may have values from 0 to 7. For example, MmvdDistance is one of (1, 2, 4, 8, 16, 32, 64, 128) or (4, 8, 16, 32, 64, 128, 256, 512) according to the value indicated by mmvd_distance_idx and tile_group_fpel_mmvd_enabled_flag ) Can be one of. That is, the magnitude component of the motion vector difference may be derived using a plurality of tables based on the resolution information of the motion vector difference.

For example, when tile_group_fpel_mmvd_enabled_flag has a first value, MmvdDistance may be determined as one of (1, 2, 4, 8, 16, 32, 64, 128). Meanwhile, when tile_group_fpel_mmvd_enabled_flag has a second value, MmvdDistance may be determined as one of (4, 8, 16, 32, 64, 128, 256, 512). For example, when mmvd_distance_idx is 3 and tile_group_fpel_mmvd_enabled_flag is 1, the size component of the motion vector difference may be determined as 32.

Meanwhile, the direction component of the motion vector difference may be determined according to Table 2 below.

[Table 2]

Referring to Table 2, the direction component (MmvdSign) of the motion vector difference may be derived using mmvd_direction_idx. In this case, mmvd_direction_idx may indicate that the motion vector difference is expressed using one of a direction component set of (+,0), (-,0), (0,+), and (0,-).

An actual vector value (MmvdOffset) of the motion vector difference may be derived based on the direction component (MmvdSign) and the magnitude component (MmvdDistance) of the derived motion vector difference. As an example, the vector value of the motion vector difference (MmvdOffset) may be derived according to Equation 1 below. Hereinafter, MmvdOffset[　x0　][　y0　][　0　]　 and MmvdOffset[　x0　][　y0　][　1　] may mean the x-axis value and y-axis value of the motion vector difference, respectively.

[Equation 1]

MmvdOffset[ x0 ][ y0 ][ 0] = (MmvdDistance[ x0 ][ y0] << 2) * MmvdSign[ x0 ][ y0 ][0]

MmvdOffset[ x0 ][ y0 ][ 1] = (MmvdDistance[ x0 ][ y0] << 2) * MmvdSign[ x0 ][ y0 ][1]

At least one of the L0 motion vector difference and the L1 motion vector difference for the current block may be derived based on the actual vector value (MmvdOffset) of the derived motion vector difference. The L0 motion vector difference and the L1 motion vector difference for the current block may be derived based on motion information of the merge candidate used to derive the predicted motion vector of the current block. That is, the L0 motion vector difference and the L1 motion vector difference may be determined based on motion information of the merge candidate determined using mmvd_merge_flag.

First, it may be determined whether the merge candidate of the current block has bidirectional motion information. When it is determined that the merge candidate has bidirectional motion information, the POC difference between the current picture and the L0 reference picture and the POC difference between the current picture and the L1 reference picture may be calculated. In this case, when the POC difference between the current picture and the L0 reference picture and the POC difference between the current picture and the L1 reference picture are the same, the L0 motion vector difference and the L1 motion vector difference may be derived to the same value using MmvdOffset.

Meanwhile, when the POC difference between the current picture and the L0 reference picture and the POC difference between the current picture and the L1 reference picture are not the same, the L0 motion vector difference and the L1 motion vector difference are different using the POC difference and MmvdOffset between the pictures. It can be derived by value.

As an example, when the POC difference between the current picture and the L0 reference picture is greater than the POC difference between the current picture and the L1 reference picture, the L0 motion vector difference may be derived first by using MmvdOffset. Next, the L1 motion vector difference may be derived by scaling the L0 motion vector difference based on the POC difference between the current picture and the L0 reference picture and the size difference of the POC difference between the current picture and the L1 reference picture.

When the POC difference between the current picture and the L0 reference picture is smaller than the POC difference between the current picture and the L1 reference picture, the L1 motion vector difference may be derived first by using MmvdOffset. Next, the L1 motion vector difference may be derived by scaling the L1 motion vector difference based on the POC difference between the current picture and the L0 reference picture and the size difference of the POC difference between the current picture and the L1 reference picture.

When the absolute value of the POC difference between the current picture and the L0 reference picture and the POC difference between the current picture and the L1 reference picture is the same but the sign is different, first, one of the L0 motion vector difference or the L1 motion vector difference may be derived using MmvdOffset. . Next, the motion vector difference in the remaining prediction directions may be derived by changing the sign of the motion vector difference in the first derived prediction direction.

Meanwhile, when the merge candidate of the current block has only unidirectional motion information, only the motion vector difference in the prediction direction in which the motion information exists may be derived using MmvdOffset. On the other hand, a motion vector difference in a prediction direction in which no motion information exists may be derived as a zero vector. For example, when the determined merge candidate has only L0 motion information, the L0 motion vector difference may be derived using MmvdOffset, and the L1 motion vector difference may be derived as (0,0).

9 is a diagram for explaining a method of inducing a motion vector difference.

The syntax element mmvd_direction_idx indicates that the motion vector difference has one of (+,0), (-,0), (0,+) and (0,-) direction component set, so that the derived motion vectors are shown in FIG. 9 It may be determined in a cross shape based on the center coordinate of. That is, the motion vector difference may have a non-zero value only for one of the x-axis and y-axis.

In addition, since the L0 motion vector difference and the L1 motion vector difference are derived based on the same MmvdOffset, there may be an association between them. For example, according to the POC difference value between the current picture and the L0 reference picture and the POC difference between the current picture and the L1 reference picture, the L0 motion vector difference and the L1 motion vector difference may have the same value. In addition, the remaining motion vector difference may have a value in which one motion vector difference is mirrored and/or scaled.

Next, the image decoding apparatus may derive the motion vector of the current block by using the difference between the predicted motion vector and the motion vector (S550). Meanwhile, the image encoding apparatus may encode motion vector difference information based on the derived predicted motion vector and the motion vector difference (S650). The motion vector of the current block may be derived through the sum of the difference between the predicted motion vector derived in step S530 and the motion vector derived in step S540. More specifically, the L0 motion vector of the current block can be derived from the sum of the L0 predicted motion vector and the L0 motion vector difference of the current block, and the L1 motion vector of the current block is the L1 predicted motion vector and the L1 motion vector difference of the current block. It can be derived from the sum of

Hereinafter, a method of changing or setting the resolution of the predicted motion vector and/or the motion vector using the above-described resolution information on the motion vector difference will be described.

Referring to FIG. 10, in the video decoding method according to an embodiment of the present disclosure, determining whether a prediction mode of a current block is an MMVD mode (S1010). When a prediction mode of a current block is an MMVD mode, a current block Deriving a merge candidate list for (S1020), deriving a predicted motion vector for the current block using the merge candidate list (S1030), deriving motion vector differences for the current block (S1040), determining the resolution of the derived predicted motion vector (S1050), deriving a motion vector of the current block using the predicted motion vector and the motion vector difference (S1060), and/or using the derived motion vector Thus, it may include obtaining a prediction block for the current block (S1070). In this case, the resolution of the predicted motion vector may be determined based on the resolution of the motion vector difference.

Referring to FIG. 11, the method of encoding an image using the MMVD mode includes determining a prediction mode of a current block as an MMVD mode (S1110), deriving a merge candidate list for the current block (S1120), and a merge candidate list. Deriving a predicted motion vector for the current block by using (S1130), inducing motion vector differences for the current block based on the predicted motion vector (S1140), and determining the resolution of the predicted motion vector It may include determining (S1150) and/or encoding motion vector difference information for the derived motion vector difference (S1160). In this case, the resolution of the predicted motion vector may be determined based on the resolution of the motion vector difference.

A description of the steps excluding steps S1050 and S1150 is the same as the encoding/decoding method using the MMVD mode described with reference to FIGS. 5 and 6, and thus a description thereof is omitted.

Since the resolution of the motion vector difference is determined based on the resolution information on the above-described motion vector difference, the resolution of the predicted motion vector used to induce the motion vector by adding it to the motion vector difference needs to be matched with the resolution of the motion vector difference. have. According to an embodiment of the present disclosure, since resolutions of a predicted motion vector and a motion vector difference may be matched, encoding efficiency may be increased. For example, when the resolution of the motion vector difference and the resolution of the motion vector are unified by an integer resolution, the resolution of the motion vector is also determined as an integer resolution, so that an interpolation process in motion compensation may be omitted.

For example, when the resolution of the motion vector difference is determined to be an integer resolution, the image encoding/decoding apparatus may set or change the resolution of the predicted motion vector to an integer resolution by rounding the predicted motion vector. That is, when the syntax element tile_group_fpel_mmvd_enabled_flag has a value of 1, the video encoding/decoding apparatus may set or change the resolution of the predicted motion vector for the current block as an integer resolution. On the other hand, when the syntax element tile_group_fpel_mmvd_enabled_flag has a value of 0, the video encoding/decoding apparatus may not set or change the resolution of the predicted motion vector for the current block as an integer resolution.

The motion vector rounding process described above may mean an operation in which a vector value less than a certain number of digits is discarded by performing left shift and right shift operations on an actual value of the motion vector. As values less than a certain number of digits are discarded, the resolution of the motion vector may be changed from fractional resolution to integer resolution. In this case, the left shifting value and the right shifting value used to perform rounding may be determined by the tile_group_fpel_mmvd_enabled_flag value. For example, the left shift value and the right shift value may be determined according to Equation 2 below.

[Equation 2]

MmvdMvShift = (tile_group_fpel_mmvd_enabled_flag) << 2

leftShift = MmvdMvShift, rightShift = MmvdMvShift

In the following description, MmvdMvShift of Equation 2 may be referred to as a first shifting value.

According to another embodiment of the present disclosure, the apparatus for encoding/decoding an image may perform motion compensation on a current block in consideration of a resolution of a motion vector difference. The embodiment of FIG. 12 may be an embodiment in which the resolution of the motion vector difference is determined as an integer resolution.

When the resolution of the motion vector difference has an integer resolution (S1210), the image encoding/decoding apparatus may determine whether the resolution of the motion vector derived from the sum of the predicted motion vector and the motion vector difference is an integer resolution (S1220). . When the resolution of the motion vector is not the integer resolution, the image encoding/decoding apparatus may perform interpolation (S1230) to perform motion compensation on the current block in units of fractional resolution (S1240). On the other hand, when the resolution of the motion vector is an integer resolution, motion compensation is performed without a separate interpolation process (S1240), and a prediction sample for the current block may be derived.

According to the present embodiment, when the resolution of a motion vector is an integer resolution, since there is no need to perform a separate interpolation process, encoding efficiency can be increased.

13 is a diagram for describing a motion estimation method according to another embodiment of the present disclosure.

According to another embodiment of the present disclosure, the apparatus for encoding an image may perform motion estimation on a current block in consideration of a resolution of a motion vector difference. Referring to FIG. 13, the image encoding apparatus may generate a merge candidate list (S1310) and perform motion estimation for each candidate (S1320) to derive an optimal prediction sample. As a result of motion estimation, a merge candidate capable of inducing an optimal prediction sample may be determined, and a merge candidate index indicating the determined candidate may be encoded.

The image encoding apparatus may determine the resolution of the motion vector difference of each merge candidate (S1330). For example, when the resolution of a motion vector difference of a specific candidate is an integer resolution (tile_group_fpel_mmvd_enabled_flag==1), rounding may be performed on the predicted motion vector of the corresponding merge candidate (S1340). Since the rounding process of the predicted motion vector is the same as that described with reference to FIG. 11, the description is omitted here. According to the rounding process, the predicted motion vector may be set to have integer resolution. The image encoding apparatus may perform integer-unit motion estimation (S1360) on the current block by using the predicted motion vector and the motion vector difference set to the integer resolution.

On the other hand, when the resolution of the motion vector difference of a specific candidate is not an integer resolution (tile_group_fpel_mmvd_enabled_flag==0), interpolation (S1350) for the reference picture may be performed. The image encoding apparatus may perform fractional motion estimation (S1370) on the current block by using the predicted motion vector and the motion vector difference set to the fractional resolution.

According to the present embodiment, the image encoding apparatus may selectively perform a rounding process and/or an interpolation process depending on whether a motion vector difference of a merge candidate has an integer resolution, and thus, image encoding efficiency may be increased.

The exemplary methods of the present disclosure are expressed as a series of operations for clarity of description, but this is not intended to limit the order in which steps are performed, and each step may be performed simultaneously or in a different order if necessary. In order to implement the method according to the present disclosure, the illustrative steps may include additional steps, other steps may be included excluding some steps, or may include additional other steps excluding some steps.

In the present disclosure, an image encoding apparatus or an image decoding apparatus performing a predetermined operation (step) may perform an operation (step) of confirming an execution condition or situation of the operation (step). For example, when it is described that a predetermined operation is performed when a predetermined condition is satisfied, the video encoding apparatus or the video decoding apparatus performs an operation to check whether the predetermined condition is satisfied, and then performs the predetermined operation. I can.

Various embodiments of the present disclosure are not intended to list all possible combinations, but to describe representative aspects of the present disclosure, and matters described in the various embodiments may be applied independently or may be applied in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general purpose It may be implemented by a processor (general processor), a controller, a microcontroller, a microprocessor, or the like.

In addition, the image decoding device and the image encoding device to which the embodiment of the present disclosure is applied include a multimedia broadcasting transmission/reception device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video chat device, and a real-time communication device such as video communication. , Mobile streaming devices, storage media, camcorders, video-on-demand (VoD) service providers, OTT video (Over the top video) devices, Internet streaming service providers, three-dimensional (3D) video devices, video telephony video devices, and medical use. It may be included in a video device or the like, and may be used to process a video signal or a data signal. For example, an OTT video (Over the top video) device may include a game console, a Blu-ray player, an Internet-connected TV, a home theater system, a smartphone, a tablet PC, and a digital video recorder (DVR).

12 is a diagram illustrating a content streaming system to which an embodiment of the present disclosure can be applied.

As shown in FIG. 12, the content streaming system to which the embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage device, a user device, and a multimedia input device.

The encoding server serves to generate a bitstream by compressing content input from multimedia input devices such as smartphones, cameras, camcorders, etc. into digital data, and transmits it to the streaming server. As another example, when multimedia input devices such as smartphones, cameras, camcorders, etc. directly generate bitstreams, the encoding server may be omitted.

The bitstream may be generated by an image encoding method and/or an image encoding apparatus to which an embodiment of the present disclosure is applied, and the streaming server may temporarily store the bitstream in a process of transmitting or receiving the bitstream.

The streaming server may transmit multimedia data to a user device based on a user request through a web server, and the web server may serve as an intermediary informing the user of a service. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server may transmit multimedia data to the user. In this case, the content streaming system may include a separate control server, and in this case, the control server may play a role of controlling a command/response between devices in the content streaming system.

The streaming server may receive content from a media storage and/or encoding server. For example, when content is received from the encoding server, the content may be received in real time. In this case, in order to provide a smooth streaming service, the streaming server may store the bitstream for a predetermined time.

Examples of the user device include a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, a slate PC, and Tablet PC, ultrabook, wearable device, for example, smartwatch, smart glass, head mounted display (HMD)), digital TV, desktop There may be computers, digital signage, etc.

Each server in the content streaming system may be operated as a distributed server, and in this case, data received from each server may be distributedly processed.

The scope of the present disclosure is software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause an operation according to the method of various embodiments to be executed on a device or computer, and such software or It includes a non-transitory computer-readable medium (non-transitory computer-readable medium) which stores instructions and the like and is executable on a device or a computer.

An embodiment according to the present disclosure may be used to encode/decode an image.

Claims

A video decoding method performed by a video decoding apparatus, the video decoding method comprising:

Determining whether a prediction mode of the current block is a merge mode with motion vector difference (MMVD) mode;

If it is determined that the prediction mode of the current block is the MMVD mode, deriving a merge candidate list for the current block;

Deriving a predicted motion vector for the current block by using the merge candidate list;

Deriving motion vector differences for the current block;

Setting a resolution of the predicted motion vector;

Deriving a motion vector for the current block by using the predicted motion vector and the motion vector difference; And

Including the step of obtaining a prediction block for the current block by using the motion vector for the current block,

The resolution of the predicted motion vector is set based on the resolution of the difference of the motion vector.
The method of claim 1,

Further comprising the step of decoding motion vector difference information for the current block,

The motion vector difference information, characterized in that it comprises at least one of size information on the motion vector difference, direction information on the motion vector difference, and resolution information on the motion vector difference.
The method of claim 2,

The resolution information on the motion vector difference indicates whether or not the motion vector difference of the current block uses an integer resolution.
The method of claim 2,

The magnitude of the motion vector difference is derived based on a plurality of tables,

A table used to derive the magnitude of the motion vector difference, characterized in that it is selected based on resolution information on the motion vector difference.
The method of claim 4,

The image decoding method, characterized in that the size information on the motion vector difference indicates one of values of a table used to induce the motion vector difference.
The method of claim 2,

The direction information on the motion vector difference indicates that the motion vector difference is expressed as a set of direction components of (+,0), (-,0), (0,+) and (0,-). Characterized in, the video decoding method.
The method of claim 1,

A video decoding method, characterized in that only a preset number of candidates among candidates in the merge candidate list is used for deriving the predicted motion vector.
The method of claim 2,

The step of setting the resolution of the predicted motion vector,

And setting the resolution of the predicted motion vector by rounding the predicted motion vector.
The method of claim 8,

The rounding is performed based on a first shifting value determined using resolution information for the motion vector difference.
The method of claim 9,

The rounding is performed by applying a left shift operation and a right shift operation to the predicted motion vector,

The image decoding method, characterized in that the shifting value used for the left shift operation and the right shift operation is determined as the first shifting value.
An image decoding apparatus comprising a memory and at least one processor,

The at least one processor

It is determined whether the prediction mode of the current block is the Merge mode with Motion Vector Difference (MMVD) mode,

When it is determined that the prediction mode of the current block is the MMVD mode, a merge candidate list for the current block is derived,

Derive a predicted motion vector for the current block using the merge candidate list,

Induce motion vector differences for the current block,

Setting the resolution of the predicted motion vector,

Derive a motion vector for the current block using the predicted motion vector and the motion vector difference,

Obtaining a prediction block for the current block using a motion vector for the current block,

The image decoding apparatus, characterized in that the resolution of the predicted motion vector is set based on the resolution of the motion vector difference.
An image encoding method performed by an image encoding apparatus, wherein the image encoding method comprises:

Determining a prediction mode of the current block as a merge mode with motion vector difference (MMVD) mode;

Deriving a merge candidate list for the current block;

Deriving a predicted motion vector for the current block by using the merge candidate list;

Inducing motion vector differences for the current block based on the predicted motion vector;

Setting a resolution of the predicted motion vector; And

Further comprising information for encoding motion vector difference information for the current block,

The resolution of the predicted motion vector is set based on the resolution of the difference of the motion vector.
The method of claim 12,

The motion vector difference information includes resolution information on the motion vector difference,

The resolution information on the motion vector difference indicates whether or not the motion vector difference of the current block uses an integer resolution.
A method of transmitting a bitstream generated by the video encoding method of claim 12.