RU2790325C2 - Method and device for inter-frame prediction based on the fusion mode - Google Patents

Abstract

FIELD: imaging technology.

SUBSTANCE: invention relates to a method and device for inter-frame prediction. The result is achieved by compiling a list of candidates for the fusion of the current block, obtaining information about the movement of the current block from the list of candidates for the fusion and performing an inter-frame prediction of the current block using movement information, and the list of candidates for the fusion contains at least one of the spatial candidate for the fusion, a temporary candidate for the fusion or a combined candidate for the fusion, and a combined candidate for fusion is obtained by combining n candidates for fusion belonging to the list of candidates for fusion, moreover, information about the movement of a combined candidate for fusion is obtained by means of a weighted average of information about the movement of the first candidate for fusion and information about the movement of the second candidate for fusion.

EFFECT: increase in the accuracy of motion information by using not only a spatial/temporal fusion candidate, but also a combined fusion candidate.

13 cl, 15 dwg, 9 tbl

Description

BACKGROUND OF THE INVENTION

The technical field to which the invention belongs

The present invention relates to a method and apparatus for inter-picture prediction.

State of the art

Recently, the demand for high-definition and high-quality video such as high-definition (HD) video and ultra-high-definition (UHD) video in various applications has been increasing, and accordingly, a high-performance video compression technology has been discussed.

As for the video compression technology, there are various technologies, such as inter frame prediction technology for predicting a pixel value included in the current image from an image before or after the current image, intra frame prediction technology for predicting a pixel value included in the current image using pixel information in the current image, an entropy encoding technique for assigning a short code to a value having a high occurrence rate and assigning a long code to a value having a low occurrence rate, etc., and video data can be efficiently compressed and transmitted or stored using such technology video compression.

SUMMARY OF THE INVENTION

[Technical issue]

One of the objectives of the present invention is to provide a method and apparatus for inter-picture prediction.

One of the objectives of the present invention is to provide a method and apparatus for compiling a list of merger candidates.

One of the objectives of the present invention is to provide a method and apparatus for compensating for motion in subblock elements.

One of the objectives of the present invention is to provide a method and apparatus for determining an affine candidate.

One of the objectives of the present invention is to provide a method and apparatus for inter-picture prediction according to a 360-degree video projection format.

[The solution of the problem]

The video encoding/decoding method and apparatus according to the present invention can list the current block merge candidates, obtain motion information of the current block from the merge candidate list, and perform inter-frame prediction of the current block using the motion information.

In the video encoding/decoding method and apparatus according to the present invention, the merge candidate list may comprise at least one of a spatial merge candidate, a temporal merge candidate, or a combined merge candidate.

In the video encoding/decoding method and apparatus according to the present invention, a combined merge candidate can be obtained by combining n merge candidates belonging to a merge candidate list.

In the video encoding/decoding method and apparatus according to the present invention, the n merge candidates may be merge candidates corresponding to indices 0 to (n-1) in the merge candidate list.

In the video encoding/decoding method and apparatus according to the present invention, the combined merge candidate index may be larger than the temporary merge candidate index.

In the video encoding/decoding method and apparatus according to the present invention, n merge candidates may comprise a first merge candidate and a second merge candidate, and motion information of the combined merge candidate can be obtained considering the prediction direction of the first merge candidate and the prediction direction second merger candidate.

In the video encoding/decoding method and apparatus according to the present invention, the motion information of the combined merge candidate can be obtained by a weighted average of the motion information of the first merge candidate and the motion information of the second merge candidate.

In the video encoding/decoding method and apparatus according to the present invention, the weighted average may be any of [1:1], [1:2], [1:3], or [2:3].

The video encoding/decoding method and apparatus according to the present invention can generate a candidate list for predicting motion information of the current block, obtain the current block's checkpoint vector based on the candidate list and the candidate index, obtain the current block's motion vector based on the current block's checkpoint vector, and perform inter-frame prediction on the current block using the motion vector.

In the video encoding/decoding device according to the present invention, the candidate list may contain a plurality of affine candidates.

In the video encoding/decoding apparatus of the present invention, affine candidates may include at least one of a spatial candidate, a temporal candidate, or a tuned candidate.

In the video encoding/decoding device according to the present invention, the motion vector of the current block can be obtained in subblock elements of the current block.

In the video encoding/decoding device according to the present invention, a spatial candidate can be determined based on whether a current block boundary is in contact with a code tree block boundary (CTU boundary).

In the video encoding/decoding apparatus of the present invention, a tuned candidate may be determined based on a combination of at least two checkpoint vectors that correspond to respective corners of the current block.

In the video encoding/decoding method and apparatus of the present invention, when the inter prediction reference area comprises a reference image boundary or a boundary between discontinuous surfaces, all or some of the pixels in the reference area can be obtained using correlated area data.

[Advantageous Effects of the Invention]

According to the present invention, the accuracy of motion information can be improved by using not only a spatial/temporal merge candidate, but also a combined merge candidate.

According to the present invention, video encoding/decoding performance can be improved by inter-picture prediction based on an affine model.

According to the present invention, the prediction accuracy can be improved by inter-picture prediction in sub-block elements.

According to the present invention, the performance of inter prediction encoding/decoding can be improved by efficiently determining an affine candidate.

According to the present invention, the performance of inter prediction coding can be improved by adjusting the reference plot in view of the correlation.

BRIEF DESCRIPTION OF THE DRAWINGS

In FIG. 1 is a block diagram of a video encoding apparatus according to one embodiment of the present invention.

In FIG. 2 is a block diagram of a video decoding apparatus according to one embodiment of the present invention.

In FIG. 3 is an exemplary diagram depicting a block tree form.

In FIG. 4 is an exemplary diagram depicting various types of blocks that can be obtained from a block splitting section according to the present invention.

In FIG. 5 shows a block splitting process according to one embodiment of the present invention.

In FIG. 6 shows a method for performing inter prediction based on a block-based merge mode as an embodiment to which the present invention is applied.

In FIG. 7 shows an inter-picture prediction method based on an affine model as an embodiment to which the present invention is applied.

Fig. 8 relates to a method for deriving an affine candidate from space/temporal neighbor block checkpoint vectors as an embodiment to which the present invention is applied.

In FIG. 9 shows a method for obtaining a tuned candidate based on a combination of motion vectors of space/time neighbor blocks as an embodiment to which the present invention is applied.

In FIG. 10 is a diagram showing a sub-block-based temporal candidate motion information acquisition method as an embodiment to which the present invention is applied.

In FIG. 11 shows an ERP projection format inter-picture prediction method as an embodiment to which the present invention is applied.

In FIG. 12 and 15 show an inter-picture prediction method in CMP projection format as an embodiment to which the present invention is applied.

DETAILED DESCRIPTION

The video encoding/decoding method and apparatus according to the present invention can list the current block merge candidates, obtain motion information of the current block from the merge candidate list, and perform inter-frame prediction of the current block using the motion information.

In the video encoding/decoding method and apparatus according to the present invention, the merge candidate list may comprise at least one of a spatial merge candidate, a temporal merge candidate, or a combined merge candidate.

In the video encoding/decoding method and apparatus according to the present invention, a combined merge candidate can be obtained by combining n merge candidates belonging to a merge candidate list.

In the video encoding/decoding method and apparatus according to the present invention, the n merge candidates may be merge candidates corresponding to indices 0 to (n-1) in the merge candidate list.

In the video encoding/decoding method and apparatus according to the present invention, the combined merge candidate index may be larger than the temporary merge candidate index.

In the video encoding/decoding method and apparatus according to the present invention, n merge candidates may comprise a first merge candidate and a second merge candidate, and motion information of the combined merge candidate can be obtained considering the prediction direction of the first merge candidate and the prediction direction second merger candidate.

In the video encoding/decoding method and apparatus according to the present invention, the motion information of the combined merge candidate can be obtained by a weighted average of the motion information of the first merge candidate and the motion information of the second merge candidate.

In the video encoding/decoding method and apparatus according to the present invention, the weighted average may be any of [1:1], [1:2], [1:3], or [2:3].

The video encoding/decoding method and apparatus according to the present invention can generate a candidate list for predicting motion information of the current block, obtain the current block's checkpoint vector based on the candidate list and the candidate index, obtain the current block's motion vector based on the current block's checkpoint vector, and perform inter-frame prediction on the current block using the motion vector.

In the video encoding/decoding device according to the present invention, the candidate list may contain a plurality of affine candidates.

In the video encoding/decoding apparatus of the present invention, affine candidates may include at least one of a spatial candidate, a temporal candidate, or a tuned candidate.

In the video encoding/decoding device according to the present invention, the motion vector of the current block can be obtained in subblock elements of the current block.

In the video encoding/decoding device according to the present invention, a spatial candidate can be determined based on whether a current block boundary is in contact with a code tree block boundary (CTU boundary).

In the video encoding/decoding apparatus of the present invention, a tuned candidate may be determined based on a combination of at least two checkpoint vectors that correspond to respective corners of the current block.

In the video encoding/decoding method and apparatus of the present invention, when the inter prediction reference area comprises a reference image boundary or a boundary between discontinuous surfaces, all or some of the pixels in the reference area can be obtained using correlated area data.

[Embodiment]

Various modifications may be made to the present invention and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail. However, it should be understood that the present invention is not limited to these specific embodiments, but is to include all alterations, equivalents, and substitutions that are within the spirit and scope of the present invention.

Terms such as first, second, A, and B may be used to describe various components, but components should not be limited to such terms. These terms are only used to distinguish one component from another component. For example, without deviating from the scope of the present invention, the first component may be referred to as the second component, and likewise the second component may be referred to as the first component. The term "and/or" includes a combination of a plurality of related listed objects or any of a plurality of related listed objects.

When a component is referred to as "coupled" or "coupled" to another component, the component may be directly coupled to or connected to the other component. However, it should be understood that another component may be present in the middle. On the other hand, when a component is referred to as "directly coupled" or "directly coupled" to another component, it should be understood that there is no other component in the middle.

The terms used in this application are used only to describe specific embodiments and are not intended to limit the present invention. The singular forms include the plural forms unless the context clearly indicates otherwise. In this application, a term such as "include" or "have" is intended to mean the presence of the features, numbers, steps, acts, components, parts, or combinations thereof described in the description, and it should be understood that this term does not exclude the possibility of the presence or adding one or more other features or numbers, steps, actions, components, parts, or combinations thereof.

Unless otherwise indicated, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person skilled in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary are to be interpreted as corresponding to the meanings of the relevant technology and not to be interpreted as ideal or overly formal meanings unless they are explicitly defined in this application.

The video encoding and decoding devices may be user terminals such as a personal computer (PC), laptop, personal digital assistant (PDA), portable media player (PMP), PlayStation Portable (PSP), wireless communication terminal, smartphone, TV, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, head-mounted display (HMD), and smart glasses, or server terminals such as an application server and a service server, and may include various devices equipped with a communication device, such as a communication modem, for communicating with various wired/wireless communication devices or networks, a memory device for storing various programs and data for encoding or decoding video, or for performing intra-frame or inter-frame prediction for encoding or decoding, a processor to execute the program to execute and calculation and control operations, etc. In addition, video encoded as a bitstream with a video encoding device can be transmitted to a device for real-time or non-real-time video decoding via a wired or wireless network such as Internet, wireless local area network, wireless LAN, WiBro network or mobile network, or through various communication interfaces such as cable and universal serial bus (USB), decoded by the video decoding device and restored and played back as video.

In addition, video encoded as a bitstream by a video encoding device can be transferred from the encoding device to the decoding device via a computer-readable storage medium.

The above-described video encoding device and video decoding device may be separate devices, respectively. However, the devices may be implemented as a single video encoding/decoding device according to the implementation. In this case, some components of the video encoding device are essentially the same technical elements as some components of the video decoding device, and can be implemented with at least the same design or perform at least the same function as some components. video decoding devices.

Therefore, redundant descriptions of the respective technical elements will be omitted in the detailed description of the following technical elements and their working principles.

In addition, since the video decoding apparatus corresponds to a computing device that applies the video encoding method performed by the video encoding apparatus to decoding, the following description will focus on the video encoding apparatus.

The computing device may include a storage device that stores a program or software module that implements the video encoding method and/or the video decoding method, and a processor associated with the storage device for executing the program. In addition, a video encoding device may be referred to as an encoder, and a video decoding device may be referred to as a decoder.

Typically, a video may contain a sequence of still pictures, and these still pictures may be classified into group of pictures (GOP) elements, and each still picture may be called a picture. In this case, the picture may represent one of a frame or a field in a progressive signal or an interlaced signal, and the video may be expressed as a "frame" when encoding/decoding is performed on a frame-by-frame basis, and expressed as a "field" when encoding /decoding is done based on the fields. The present invention discusses and describes a progressive signal. However, the present invention can also be applied to an interlaced signal. In a higher level concept, there may be elements such as GOPs and sequences, and each image may be broken down into defined areas such as layers, tiles, and blocks. In addition, one GOP may contain elements such as an I picture, a P picture, and a B picture. /decoded by performing a process such as motion estimation and motion compensation using a reference picture. In general, the I picture and the P picture can be used as reference pictures in the case of the P picture, and the I picture and the P picture can be used as reference pictures in the case of the B picture. However, the above definition can be changed by encoding/decoding setting.

Here, an image is referred to for encoding/decoding as a reference picture, and a block or pixel is referred to as a reference block or a reference pixel. In addition, the reference data may be not only a pixel value in the spatial domain, but also a coefficient value in the frequency domain and various types of encoding/decoding information generated and determined during the encoding/decoding process. Examples thereof may correspond to information related to intra-frame prediction or motion related information in the prediction section, information related to transformation in the transformation section/inverse transformation section, information related to quantization in the quantization section/inverse quantization section, information related to with encoding/decoding (context information) in the encoding section/decoding section, filter related information in the filtering section in the loop, etc.

The smallest element that makes up a video can be a pixel, and the number of bits used to represent one pixel is called bit depth. In general, the bit depth may be 8 bits, and according to the encoding settings, bit depths greater than 8 bits may be supported. As the bit depth, at least one bit depth according to the color space may be supported. Further, at least one color space may be provided according to the video color format. One or more images having a certain size or one or more images having different sizes may be provided according to the color format. For example, in the case of YCbCr 4:2:0, one luminance component (Y in this example) and two color difference components (Cb/Cr in this example) may be provided. In this example, the component ratio of the color difference components and the luminance component may be a 1:2 ratio in width and height. As another example, in the case of 4:4:4, the width and height may be the same in aspect ratio. In the case of having one or more color spaces, as in the example above, the image can be split into the corresponding color spaces.

In the present invention, description will be given based on some color spaces (Y in this example) of some color formats (YCbCr in this example), and the same or similar application (setting depending on the specific color space) can be applied to other color spaces ( Cb and Cr in this example) according to the color format. However, some differences can be made to each color space (regardless of the setting for a particular color space). In other words, a setting dependent on each color space can mean having a setting proportional to or dependent on the component ratio of each component (e.g., 4:2:0, 4:2:2, 4:4:4, etc.). .), and an independent setting for each color space may mean that there is only a setting for the corresponding color space, which does not take into account or does not depend on the component ratio of each component. In the present invention, depending on the encoder/decoder, some configurations may have independent or dependent settings.

The configuration information or syntax element required in the video encoding process can be defined at the element level, such as video, sequence, image, layer, tile, block, etc., which is included in the bitstream in elements, such as set video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), layer header, tile header, block header, etc., and is transmitted to the decoder, and the decoder can perform analysis in the same layer elements for recovering the setting information transmitted from the encoder, and using the recovered setting information in a video decoding process. In addition, related information can be transmitted as a bitstream in the form of additional optimization information (SEI) or metadata, and it can be parsed and used. Each option set has a unique identifier (ID) value, and the bottom option set can have an ID value of the top option set to refer to. For example, the lower parameter set may refer to the information of the upper parameter set having a matching identifier value among one or more upper parameter sets. Among the examples of the various elements mentioned above, when one element includes one or more other elements, the corresponding element may be referred to as the top element, and the included element may be referred to as the bottom element.

The setting information generated in an element may contain content for independent setting for each element, or contain content for setting depending on the previous, next, or top element, etc. In this case, the dependent setting can be understood as indicating the setting information of the corresponding element as flag information indicating that the setting of the previous, next or top element is respected (eg, 1-bit flag, setting is respected in case of 1 and not respected in case of 0). The description of setting information in the present invention will focus on an example of independent setting. However, an example of adding or replacing content based on a relationship dependent on the setting information of the previous or next element relative to the current element, or the top element, may be included.

In FIG. 1 is a block diagram of a video encoding apparatus according to one embodiment of the present invention. In FIG. 2 is a block diagram of a video decoding apparatus according to one embodiment of the present invention.

As shown in FIG. 1, the video encoding apparatus may comprise a prediction section, a subtraction section, a transformation section, a quantization section, an inverse quantization section, an inverse transform section, an adder section, an in-loop filtering section, a memory, and/or an encoding section. Some of the above components may not necessarily be included, some or all of the components may be optionally included depending on the implementation, and some additional components not shown in the figure may be included.

As shown in FIG. 2, the video decoding apparatus may comprise a decoding section, a prediction section, an inverse quantization section, an inverse transform section, an addition section, an in-loop filtering section, and/or a storage device. Some of the above components may not necessarily be included, some or all of the components may be optionally included depending on the implementation, and some additional components not shown in the figure may be included.

The video encoding device and the video decoding device may be separate devices, respectively, or may be implemented as a single video encoding/decoding device according to an implementation. In this case, some components of the video encoding device are essentially the same technical elements as some components of the video decoding device, and can be implemented with at least the same design or perform at least the same function as some components. video decoding devices. Therefore, redundant descriptions of the respective technical elements will be omitted in the detailed description of the following technical elements and their working principles. Since the video decoding device corresponds to a computing device that applies the video encoding method performed by the video encoding device to decoding, the following description will focus on the video encoding device. A video encoding device may be referred to as an encoder and a video decoding device may be referred to as a decoder.

The prediction section may include an intra prediction section that performs intra prediction and an inter prediction section that performs inter prediction. In intra prediction, the intra prediction mode may be determined by setting a block pixel adjacent to the current block as a reference pixel, and the prediction block may be generated using the intra prediction mode. In inter-picture prediction, a prediction block may be generated by determining motion information of the current block using one or more reference videos and performing motion compensation using the motion information. One of intra prediction and inter prediction used for the current block (coding unit or prediction unit) may be determined, and specific information (eg, intra prediction mode, motion vector, reference video, etc.) may be determined according to each method. predictions. In this case, the processing element in which prediction is performed and the processing element in which the prediction method and specific content are determined can be determined according to the encoding/decoding setting. For example, a prediction method, a prediction mode, etc. are defined in a prediction element (or coding element), and prediction is performed in a prediction block element (or coding element or transform element).

The subtract section generates a residual block by subtracting the prediction block from the current block. That is, the subtraction section calculates the difference between the pixel value of each pixel in the current block to be encoded and the predicted pixel value of each pixel in the prediction block, which is generated by the prediction section, to generate a residual block, which is a block residual signal. .

The conversion section can convert a signal belonging to the spatial domain into a signal belonging to the frequency domain, and at this time, the signal resulting from the conversion process is called a converted coefficient. For example, a transform block having a transformed coefficient can be obtained by transforming a residual block having a residual signal received from the subtractor section, and the received signal is determined according to the encoding setting and is not limited to the residual signal.

The transform section may transform the residual block using a transform scheme such as Hadamard transform, DST (Discrete Sine Transform), or DCT (Discrete Cosine Transform) based transform. However, the conversion scheme is not limited to this, and various conversion schemes obtained by improving and modifying this conversion scheme can be used.

The conversion can be performed in the horizontal/vertical direction. For example, using a basis vector in the transformation, a full 2D transformation can be performed by performing a 1D transformation in the horizontal direction and performing a 1D transformation in the vertical direction, thereby converting a pixel value in the spatial domain to a pixel value in the frequency domain.

In addition, the transformation section can transmit information necessary for generating a transformation block to the encoding section, whereby the information is encoded, include the corresponding information in the bitstream, and transmit the information to the decoder, and the decoding section of the decoder can analyze the information and use the information for the inverse transformation process. .

The quantization section may quantize the input signal, and at this time, the signal resulting from the quantization process is called the quantized coefficient. For example, a quantization block having a quantized coefficient can be obtained by quantizing a residual block having a residual transformed coefficient received from the transform section, and an input signal is determined according to an encoding setting that is not limited to the residual transformed coefficient.

The quantization section may quantize the transformed residual block using a quantization scheme such as uniform threshold quantization with a dead zone and a weighted quantization matrix. However, the quantization scheme is not limited to this, and various quantization schemes obtained by improving and modifying this quantization scheme can be used.

The quantization section may transmit the information necessary to generate the quantization block to the coding section, whereby the information is encoded, include the corresponding information in the bitstream, and transmit the information to the decoder, and the decoder section of the decoder may analyze the information and use the information for the inverse quantization process.

In the above example, the description has been made on the assumption that the residual block is subjected to transformation and quantization by a transformation section and a quantization section. However, the residual signal of the residual block may be transformed to generate a residual block having a transformed coefficient, and the quantization process may not be performed. Alternatively, the residual signal of the residual block may not be converted to a transformed coefficient, and only the quantization process may be performed. Alternatively, neither the transform process nor the quantization process may be performed. The determination may be made according to the setting of the encoder.

The coding section may scan the quantized coefficient, the transformed coefficient, the residual signal, etc. of the generated residual block according to at least one scan order (e.g., zigzag scan, vertical scan, horizontal scan, etc.) to generate a sequence of quantized coefficients, a sequence of transformed coefficients or a sequence of signals; and performing encoding using at least one entropy coding scheme. In this case, the scan order information may be determined according to the encoding setting (eg, encoding mode, prediction mode, etc.), and related information may be implicitly determined or explicitly generated. For example, one of the plurality of scan orders may be selected according to the intra-prediction mode. In this case, the scanning pattern can be set to one of various patterns such as zigzag pattern, diagonal lines, and raster.

In addition, encoding data including encoding information transmitted from each component may be generated and output as a bitstream, which may be implemented by a multiplexer (MUX). In this case, encoding may be performed using a method such as exponential Golomb, context adaptive variable length coding (CAVLC), or context adaptive binary arithmetic coding (CABAC) as the coding scheme. However, the coding scheme is not limited to this, and various coding schemes obtained by improving and modifying this coding scheme can be used.

When performing entropy encoding (considering CABAC in this example) on residual block data and a syntax element such as information generated during the encoding/decoding process, the entropy encoding apparatus may comprise a binarizer, a context simulator, and a binary arithmetic encoder . In this case, the binary arithmetic encoder may comprise a normal coding engine and a bypass coding engine. In this case, the engine for normal coding may be a process performed on the context modeler, and the engine for bypass coding may be a process performed independently of the context modeler.

Since the syntax element input to the entropy encoding device may not be a binary value when the syntax elements are not binary values, the binarizer can binarize the syntax elements and output a binary number string containing 0 or 1. In this case, a binary number is a bit containing 0 or 1, and it can be encoded using a binary arithmetic encoder. In this case, one of the normal encoding engine or the bypass encoding engine may be selected based on the occurrence probability of 0 and 1, which may be determined according to the encoding/decoding setting. The engine for bypass encoding can be used when the syntax element is data in which the frequency of occurrence of 0 and 1 is the same, and the engine for normal encoding can be used in other cases that can be referenced when the subsequent engine for normal encoding is performed via context modeling ( or contextual information updates).

In this case, context is information about the occurrence probability of a binary number, and context modeling is a process of estimating the probability of a binary number required for binary arithmetic encoding using a binary number that is obtained by binarization as an input value. To estimate the probability, information about the syntax element of the binary number, the index, which is the position of the binary number in the string of binary numbers, the probability that the binary number is included in a block adjacent to the block, etc., can be used, and for this can be used by at least one context table. For example, a plurality of context tables may be used as information for some flags according to a combination of whether the flags of neighboring blocks are used or not.

When performing binarization of a syntax element, various methods can be used. For example, the methods can be divided into fixed-length binarization and variable-length binarization. In the case of variable length binarization, unary binarization (truncated unary binarization), Rice binarization (truncated Rice binarization), exponential Golomb binarization of order k, truncated binarization, etc. In addition, signed binarization or unsigned binarization can be performed according to the value range of the syntax element. The binarization process for the syntax element carried out in the present invention can be performed, including not only the binarization mentioned in the above example, but also other additional binarization methods.

The inverse quantization section and the inverse transform section may be implemented by reversing the processes in the transform section and the quantization section. For example, the inverse quantization section may inversely quantize the quantized transformed coefficient generated by the quantization section, and the inverse transformation section may inversely transform the inversely quantized transformed coefficient to generate a reconstructed residual block.

The addition section reconstructs the current block by adding the prediction block and the reconstructed residual block. The reconstructed block may be stored in a storage device and used as reference data (prediction section, filter element, etc.).

The in-loop filtering section may include at least one post-processing filtering process such as a deblocking filter, sample adaptive shift (SAO), and adaptive loop filter (ALF). The deblocking filter can remove block distortion that occurs at the boundary between blocks in the reconstructed video. ALF can perform filtering based on the value obtained by comparing the reconstructed video and the input video. More specifically, after filtering a block with a deblock filter, filtering can be performed based on the value obtained by comparing the reconstructed video and the input video. Alternatively, after filtering the block with SAO, filtering can be performed based on a value obtained by comparing the reconstructed video and the input video.

The storage device may store the reconstructed block or image. The reconstructed block or image stored in the memory may be provided to a prediction section that performs intra prediction or inter prediction. In more detail, a storage space in the form of a bitstream queue compressed by an encoder can be placed and processed as a coded picture buffer (CPB), and a space for storing decoded video in picture elements can be placed and processed as a decoded picture buffer (DPB). In the case of the CPB, decoding items are stored according to the decoding order, the decoding operation is emulated in the encoder, the bitstream compressed during the emulation process can be stored, the bitstream output from the CPB is restored by the decoding process, the reconstructed video is stored in the DPB, and images stored in the DPB may be referred to in a subsequent video encoding and decoding process.

The decoding section may be implemented by performing the process in the encoding section in reverse order. For example, a sequence of quantized coefficients, a sequence of transformed coefficients, or a sequence of signals may be received from a bitstream and decoded, and decoding data including decoding information may be parsed and transmitted to each component.

A block splitting section may be included in a video encoding apparatus and a video decoding apparatus according to embodiments of the present invention. A base coding element may mean a base (or start) element for prediction, transformation, quantization, etc. in a video encoding/decoding process. In this case, the coding unit may contain one luminance coding unit and two color difference coding units according to the color format (YCbCr in this example), and the size of each block may be determined according to the color format. In the example to be described later, the description is given on a block basis (luminance components in this example). In this case, the block is assumed to be an element that can be obtained after each element is defined, and the description is given with the understanding that similar settings can be applied to other types of blocks.

The block splitting section may be configured with respect to each component of the video encoding apparatus and the decoding apparatus, and the size and shape of the block may be determined through this process. In this case, the tuned block may be defined differently depending on the component, and may correspond to a prediction block in the case of a prediction section, a transform block in the case of a transform section, and a quantization block in the case of a quantization section. The present invention is not limited to this, and block elements according to other components can be additionally defined. The size and shape of a block can be determined by the length of the block in the horizontal and vertical directions.

In the block splitting section, a block can be expressed as M×N, and the maximum and minimum values of each block can be obtained within a range. For example, when the block shape supports a square, and the maximum block value is set to 256x256 and the minimum block value is set to 8x8, a block having a size of 2mx2m can be obtained (in this example, m is an integer from 3 to 8 , for example, 8x8, 16x16, 32x32, 64x64, 128x128 or 256x256), a block having a size of 2mx2m (in this example, m is an integer from 4 to 128), or a block having size m×m (in this example, m is an integer from 8 to 256). Alternatively, in the case where the block shape supports square and rectangle, and has the same range as in the above example, a block having a size of 2m×2n can be obtained (in this example, m and n are integers from 3 to 8, for example, 8×8, 8×16, 16×8, 16×16, 16×32, 32×16, 32×32, 32×64, 64×32, 64×64, 64×128, 128×64, 128× 128, 128x256, 256x128, or 256x256, assuming that the width-to-height ratio is a maximum of 2:1, and there may be no restriction on the width-to-height ratio, or there may be a maximum ratio value depending on the encoding setting / decoding). Alternatively, you can get a block having a size of 2m×2n (in this example, m and n are integers from 4 to 128). Alternatively, you can get a block having a size m×n (in this example, m and n are integers from 8 to 256).

In the block splitting section, the input block can receive one or more blocks. For example, the input block may be the output without change, or may be obtained by splitting into two or more sub-blocks. In this case, the number of sub-blocks may be 2, 3, 4, etc., and may be determined according to the partition scheme (partition type). The shape of the sub-block can be derived based on the block before splitting (in this example, the input block), and different shapes can be derived according to the setting of splitting the blocks.

In the present invention, the description is given based on a tree-based partitioning scheme. However, the present invention is not limited thereto.

Received blocks can be defined according to encoding/decoding settings (eg, block type, splitting scheme, splitting setting, etc.). For example, a block having a size of 2m×2n can be obtained as a coding block, a block having a size of 2m×2n or m×n can be obtained as a prediction block, and a block having a size of 2m×2n can be obtained as conversion block. Information about block size, range, etc. (for example, information related to index and multiplier, etc.) can be generated based on the setting.

The range (defined as the maximum value and minimum value in this example) can be defined according to the block type. In addition, for some blocks, the block range information may be explicitly generated, and for some blocks, the block range information may be implicitly defined. For example, the associated information may be explicitly generated in the coding and transform block, and the associated information may be implicitly processed in the prediction block.

In the explicit case, at least one piece of range information may be generated. For example, in the case of a coding block, the maximum value and minimum value information may be generated as range information. Alternatively, the information may be generated based on the difference between a maximum value and a preset minimum value (e.g., 8) (e.g., generated based on a setting, value information of the difference between the maximum value and minimum value indices, etc.). In addition, information on a plurality of ranges for the length in the horizontal and vertical direction of the rectangular block can be generated.

In the case of the implicit view, the range information can be obtained based on the encoding/decoding setting (eg, block type, partitioning scheme, partitioning setting, etc.). For example, in the case of a prediction block, information about the maximum value and minimum value can be obtained as a group of candidates (in this example, M×N and m/2×n/2), which can be obtained by adjusting the division of the prediction block (for example, quadtree split + split depth 0) in a coding block (for example, the maximum coding block size is M×N and the minimum coding block size is m×n), which is the top element.

The size and shape of the initial (or initial) block of the block splitting section can be determined from the top element. A base coding block may be a source block in the case of a coding block, an coding block may be a source block in the case of a prediction block, and an encoding block or a prediction block may be a source block in the case of a transform block, which may be customized according to the encoding/decoding setting.

For example, when the coding mode is intra frame, the prediction block may be the top element of the transform block, and when the coding mode is inter frame, the prediction block may be an independent element of the transform block. The original block can be split into smaller blocks as the initial element of the split. When the optimal size and shape is determined according to the partition of each block, the block can be determined as the source block of the lower element. For example, the block may be a coding block in the first case, and may be a prediction block or a transformation block in the latter case (sub-element). When the source block of the bottom element is determined as in the above example, a splitting process can be performed to find a block having the optimal size and shape as the top element.

Finally, the block splitting section may divide the base coding unit (or the largest coding unit) into at least one coding unit (or bottom coding unit). In addition, the coding element may be split into at least one prediction element, and may be split into at least one transform element. A coding unit may be split into at least one coding unit, a coding unit may be split into at least one prediction unit, and may be split into at least one transform unit. The prediction element may be split into at least one prediction block, and the transform element may be split into at least one transform block.

As in the above example, when a block having an optimal size and shape is found through the mode determination process, mode information (eg, split information, etc.) for that block can be generated. The mode information may be included in the bitstream along with the information generated from the component to which the block belongs (eg, prediction-related information, transformation-related information, etc.) and transmitted to the decoder, and may be parsed into element of the same level in the decoder and used in the video decoding process.

In the example described later, a partitioning scheme will be described, and the description will be made on the basis that the original block has a square shape. However, in the case of a rectangular shape, the same or similar application is allowed.

In FIG. 3 is an exemplary diagram depicting a block tree form.

Examples of obtaining blocks are shown, in which "a" depicts one 2N × 2N block that is not split, "b" depicts two 2N × N blocks obtained through a partial split flag (horizontal split by the binary tree method in this example), "c" depicts two N × 2N blocks obtained by the partial split flag (vertical binary tree split in this example) and “d” depicts four N × N blocks obtained by the partial split flag (four quadtree splits in this example). The shape of the resulting block can be determined according to the type of tree used for splitting. For example, when quadtree partitioning is performed, the resulting candidate blocks may be "a" and "d" blocks. When binary tree splitting is performed, the resulting candidate blocks may be "a", "b", and "c" blocks. In the case of a quadtree, one split flag is supported. Block "a" can be obtained when the split flag is "0", and block "b" can be obtained when the split flag is "1". In the case of a binary tree, a plurality of splitting flags are supported, one of which may be a flag indicating whether splitting is performed and one of which may be a flag indicating whether splitting is performed horizontally or vertically. In the case of a quadtree, a basic tree-based partitioning scheme may be used. In addition, a tree-based partitioning scheme (binary tree in this example) can be included in a tree-based partitioning scheme. When the flag enabling additional tree splitting is enabled implicitly or explicitly, a plurality of tree splitting operations can be performed. Tree-based partitioning may be a technique that provides a recursive partitioning. That is, the broken block can be set again as a source block to perform tree-based splitting, which can be determined according to splitting settings such as a splitting range and a valid splitting depth. This schema may be an example of a hierarchical partitioning schema.

In FIG. 4 is an exemplary diagram depicting various types of blocks that can be obtained from a block splitting section according to the present invention.

As shown in FIG. 4, blocks "a" to "s" can be obtained according to the split setting and split pattern, and an additional block shape that is not shown may be possible.

As an example (1), an asymmetric split may be allowed for tree-based partitioning. For example, in the case of a binary tree, blocks such as blocks "b" and "c" or blocks such as blocks "b" to "g" may be possible. When the asymmetric split enabling flag is explicitly disabled or implicitly according to the encoding/decoding setting, the resulting candidate block may be "b" or "c" blocks, and when the asymmetric split enabling flag is enabled, the resulting candidate block may represent blocks "b", "d" and "e" (horizontal split in this example) or blocks "c", "f" and "g" (vertical split in this example).

In the above example, the description is given considering that the ratio of the lengths of the left block and the right block or the top block and the bottom block in an asymmetric split is 1:3 or 3:1. However, the present invention is not limited to this, and there may be a group of candidates having a different ratio (eg, 1:2, 1:4, 2:3, 2:5, 3:5, etc.) according to the coding setting.

The following are various examples of split information generated by a binary tree split (a set of candidates with a ratio of 1:1, 1:3, and 3:1 in this example).

For example, in addition to a flag indicating whether splitting is performed and a flag indicating the direction of splitting, a flag indicating the shape of splitting may be generated. In this case, the shape of the partition may refer to a symmetric or asymmetric partition. Among these types, when an asymmetric split is determined as a split shape, a flag indicating a split ratio may be generated, and an index may be assigned according to a predetermined candidate group. When a split ratio of 1:3 or 3:1 is supported as a candidate group, the split ratio may be selected by a 1-bit flag.

In addition to a flag indicating whether splitting is performed and a flag indicating the splitting direction, a flag indicating a splitting ratio may be generated. In this example, a group having a 1:1 symmetrical ratio may be included as a candidate group for the split ratio.

In the present invention, it is assumed that the binary tree partitioning has the same configuration as in the first example (where the flag enabling asymmetric partitioning is enabled). Unless otherwise noted, a binary tree refers to a symmetric binary tree.

As an example (2), for tree-based splitting, additional tree-based splitting may be allowed. For example, you can split using ternary tree, quadtree, octree, etc., whereby n split blocks can be obtained (3, 4 and 8 in this example, n is an integer). In the case of a ternary tree, the supported blocks (when split into multiple blocks in this example) could be blocks h–m. In the case of a quadtree, the supported blocks may be n-p blocks. In the case of an octree, the supported block may be the q block. The need to support tree-based partitioning may be implicitly determined according to the encoding/decoding setting, or the associated information may be explicitly generated. Moreover, depending on the encoding/decoding setting, partitioning may be used alone or may be used in combination with binary tree or quadtree partitioning.

For example, in the case of a binary tree, blocks such as blocks b and c may be possible. When binary tree and ternary tree are used in combination (this example assumes that the range of use of binary tree and the range of use of ternary tree overlap), blocks such as b, c, i, and l blocks may be possible. When the flag allowing additional splitting different from the existing tree is explicitly or implicitly deactivated according to the encoding/decoding setting, the resulting candidate block may be block b or c. When the flag is enabled, the resulting candidate block may be blocks b and i or blocks b, h, i and j (horizontal partitioning in this example), or may be blocks c and l or blocks c, k, l and m (vertical split in this example).

In the above example, the description is given assuming that the ratio of the lengths of the left block, middle block and right block or the top block, middle block and bottom block when split using the ternary tree method is 2:1:1 or 1:2:1, or 1: 1:2. However, the present invention is not limited to this, and other ratios may be possible depending on the encoding setting.

The following is an example of the split information generated by a ternary tree split (a 1:2:1 candidate in this example).

For example, a flag indicating a type of split may be generated in addition to a flag indicating whether a split is in progress and a flag indicating a split direction. In this case, the split type may refer to a binary tree split or a ternary tree split.

In the present invention, the adaptive encoding/decoding setting according to the partitioning scheme can be applied.

As an example, a partitioning scheme may be determined according to the block type. For example, the coding block and transform block may use quadtree partitioning, and the prediction block may use quadtree and binary tree (or ternary tree, etc.) partitioning.

As an example, a partitioning scheme may be determined according to the block size. For example, part of the range (for example, from a×b to c×d, when the latter is larger) between the maximum value and minimum value of the block can be quadtree partitioned, and part of the range (for example, from e×f to g× h) a binary tree (or ternary tree, etc.) split can be performed. In this case, the range information according to the partitioning scheme may be explicitly generated or may be implicitly defined, and these ranges may overlap.

As an example, a partitioning scheme may be determined according to the shape of the block (or the block before splitting). For example, when a block is square in shape, quadtree and binary tree (or ternary tree, etc.) splitting can be performed. Alternatively, when the block is rectangular, splitting based on a binary tree (or ternary tree, etc.) may be performed.

As an example, the split setting may be determined according to the block type. For example, in tree-based partitioning, a quadtree partitioning may be used for a coding block and a prediction block, and a binary tree partitioning may be used for a transform block. Alternatively, the allowable split depth may be set to m in the case of a coding block, the allowable split depth may be set to n in the case of a prediction block, and the allowable split depth may be set to o in the case of a transform block. Also, m, n, and o may or may not be the same.

As an example, the split setting may be determined according to the block size. For example, a portion of the range of a block (eg, a×b to c×d) may be quad-tree partitioned, a portion of the range (eg, e×f to g×h, in this example, c×d is assumed to be greater than than g×h) can be split using the binary tree method, and a part of the range (for example, from i×j to k×l, in this example it is assumed that g×h is greater than or equal to k×l) can be split using the ternary tree method . In this case, the range may contain all ranges between the maximum value and the minimum value of the block, and the ranges may be set to not overlap with each other or to overlap with each other. For example, the minimum value of the range part may be the same as the maximum value of the range part, or the minimum value of the range part may be less than the maximum value of the range part. In the case of overlapping ranges, the partition scheme having the higher maximum value may take precedence, or information may be explicitly generated as to which partition scheme to use. That is, in the partition scheme having priority, it can be determined whether or not to execute the partition scheme having lower priority according to the partition result, or the partition scheme to be used can be determined according to the partition scheme selection information.

As an example, the split setting can be determined according to the shape of the block. For example, when the block has a square shape, quadtree partitioning can be performed. Alternatively, when the block has a rectangular shape, splitting using a binary tree or ternary tree method can be performed.

As an example, a split setting may be determined according to encoding/decoding information (eg, layer type, color component, encoding mode, etc.). For example, quadtree (or binary tree or ternary tree) partitioning may be performed on a portion of a range (e.g., a×b to c×d) when the layer type is I may be performed on a portion of a range (e.g., e×f to g×h) when the layer type is P, and can be performed in a part of the range (for example, from i×j to k×l) when the layer type is B. In addition, the allowed splitting depth when splitting by the quadtree method ( or binary tree or ternary tree) may be set to m when the layer type is I, may be set to n when the layer type is P, and may be set to o when the layer type is B. In this case, m, n and o may or may not be the same. Some layer types may have the same configuration as other layers (eg P and B layers).

As another example, the allowable splitting depth in quadtree (or binary tree or ternary tree) splitting may be set to m when the color component is a luma component, and may be set to n when the color component is a chroma component, where m and n may or may not be the same. In addition, the quadtree (or binary tree or ternary tree) splitting range, in case the color component is a luminance component (e.g., a×b to c×d), may or may not be the same as the splitting range. by the quadtree (or binary tree or ternary tree) method, in case the color component is a color difference component (for example, from e×f to g×h).

As another example, the allowable split depth in quadtree (or binary tree or ternary tree) splitting may be m when the coding mode is intraframe, and may be n (in this example, n is assumed to be greater than m) when the encoding mode is interframe. Here, m and n may or may not be the same. In addition, the range of quadtree (or binary tree or ternary tree) splitting in case the coding mode is intraframe may or may not be the same as the range of quadtree (or binary tree or ternary tree) splitting in case if the coding mode is interframe.

In the case of the above example, information on whether to support the configuration of the adaptive partition candidate group according to the encoding/decoding information may be explicitly generated or may be implicitly determined.

In the above example, a case has been described in which the partitioning scheme and the partitioning setting are determined according to the encoding/decoding setting. The above example shows some of the cases according to each element and may be subject to change to obtain other cases. In addition, the partition scheme and the partition setting can be determined according to the combination of the plurality of elements. For example, the partition scheme and the partition setting may be determined according to the type, size, shape, encoding/decoding information, etc. of the block.

Also, in the above example, factors related to partitioning scheme, tuning, etc. may be implicitly defined, or information may be explicitly generated to determine whether to allow the adaptive case, as in the above example.

In the splitting setting, the splitting depth refers to the number of times the spatial splitting is relative to the original block (in this example, the splitting depth of the original block is 0), and splitting into smaller blocks can be performed with increasing splitting depth. The depth-related setting may vary according to the partitioning scheme. For example, among tree-based splitting methods, one common depth may be used for binary tree splitting depth and ternary tree splitting depth, and different depths may be used for quadtree splitting depth and binary tree splitting depth. Thus, separate depths can be used depending on the type of tree.

In the above example, when a separate split depth is used according to the tree type, the split depth can be set to 0 at the split start position (block before split in this example) of the tree. The split depth can be calculated based on the position of the start of the split, rather than based on the split range of each tree (the maximum value in this example).

In FIG. 5 shows a block splitting process according to one embodiment of the present invention. In more detail, this figure depicts an example of the size and shape of a block obtained according to one or more partitioning methods, starting from the basic coding block.

In the figure, the heavy solid line represents the base coding block, the heavy dotted line represents the quadtree split boundary, the double solid line represents the symmetric binary tree split boundary, the solid line represents the ternary tree split boundary, and the thin dashed line represents the asymmetric binary tree split boundary. . With the exception of the thick solid line, each line represents a boundary divided according to each partitioning method. The split settings (eg, split type, split information, split information configuration order, etc.) described below are not limited to the case of the corresponding example, and various changes can be made.

For convenience, the description will be given considering that separate block splitting settings are set for the top left, top right, bottom left and bottom right blocks (NxN. 64x64) with respect to the base coding block (2Nx2N. 128x128). First, four sub-blocks are obtained by one splitting operation (split depth 0 -> 1, that is, the split depth is increased by 1) in the original block, and the maximum coding block is 128x128, the minimum coding block is 8x8, and the maximum split depth is 4 given the split setting for the quadtree, which is assumed to be the setting that is normally applied to each block.

(No. 1, top left block, A1 to A6)

In this example, splitting by a single tree method (quadtree in this example) is supported, and the size and shape of a block that can be obtained by a single block splitting setting, such as a maximum coding block, a minimum coding block, and a splitting depth, can be determined. This example is a case in which there is one block that can be obtained according to splitting (splitting the width and height into two parts, respectively). The split information required for one split operation (based on the 4Mx4N block before split, split depth increased by 1) is a flag indicating whether split is performed or not (in this example, split is not performed when the flag is 0, split is executed when the flag is 1), and the resulting candidates may be 4Mx4N and 2Mx2N.

(No. 2, top right block, A7 to A11)

This example is a case in which schema partitioning of multiple trees (quadtree and binary tree in this example) is supported, and the size and shape of a block can be determined, which can be obtained through a plurality of block partitioning settings. This example assumes that in the case of a binary tree, the maximum coding block is 64x64, the minimum coding block is 4 in length, and the maximum split depth is 4.

Since this example is a case in which two or more blocks (two or four in this example) can be obtained, the split information needed for one split operation (quadtree, split depth increased by 1) can be a flag indicating whether a split is performed, a flag indicating the type of split, a flag indicating the shape of the split, and a flag indicating the direction of the split, and the resulting candidates may be 4Mx4N, 4Mx2N, 2Mx4N, 4MxN/4Mx3N, 4Mx3N/4MxN, Mx4N/3Mx4N, and 3Mx4N/ Mx4N.

When the ranges of quadtree splitting and binary tree splitting overlap (that is, in the range where both quadtree splitting and binary tree splitting can be performed in the current stage), and the current block (in the pre-split state) is the block obtained by quadtree splitting (block obtained by quadtree splitting from the parent box <when the splitting depth is 1 less than the current>), splitting information can be divided into the following cases and configured. That is, when a block maintained according to each partition setting can be obtained by a plurality of partition methods, partition information can be generated by classifying into the following processes.

(1) When quadtree splitting and binary tree splitting overlap

[Table 1]

In the above table, "a" denotes a flag indicating whether quadtree splitting is performed, and when the flag is 1, quadtree splitting (QT) is performed. When the flag is 0, "b" is checked, which is a flag indicating whether splitting is performed by the binary tree method. When b is 0, no further split is performed in the block (No Split), but when b is 1, a binary tree split is performed.

"c" stands for a flag indicating the direction of splitting. c = 0 means horizontal split (hor) and c = 1 means vertical split (ver). "d" stands for a flag indicating the shape of the split. d = 0 means symmetric partition (SBT: symmetric binary tree) and d = 1 means asymmetric partition (ABT: asymmetric binary tree). Only when d is 1 is the information about the detailed split ratio (1/4 or 3/4) in asymmetric split checked. When d is 0, in the left/right or top/bottom blocks, the ratio of the left block to the top block is 1/4, and the ratio of the right block to the bottom block is 3/4. When d is 1, the relationship is reversed.

(2) When only binary tree partitioning can be performed

In the above table, split information can be expressed using flags "b" through "e" with the exception of flag "a".

The case of block A7 in FIG. 5 represents a case in which quadtree splitting can be performed in blocks before splitting (A7 to A11) (that is, even though quadtree splitting can be performed, binary tree splitting is performed instead of quadtree splitting), and thus corresponds to the case , in which information about the split in paragraph (1) is generated.

On the other hand, the case of blocks A8-A11 corresponds to the case in which binary tree splitting is performed without first performing quadtree splitting in blocks A8-A11 before splitting (that is, quadtree splitting is no longer possible in the corresponding blocks <A8-A11>) , and thus corresponds to the case in which split information is generated in item (2).

(No. 3, bottom left block, A12 to A15)

This example is a case in which schema partitioning of multiple trees (quadtree, binary tree, and ternary tree in this example) is supported, and the size and shape of a block can be determined, which can be obtained through a variety of block partitioning settings. This example assumes that in the case of a binary tree/ternary tree, the maximum coding unit is 64x64, the minimum coding unit is 4 in length, and the maximum split depth is 4.

Since this example is a case in which two or more blocks (two, three, and four in this example) can be obtained, the split information required for one split operation may be a flag indicating whether a split is performed, a flag, indicating the split type and a flag indicating the split direction, and the resulting candidates may be 4Mx4N, 4Mx2N, 2Mx4N, 4MxN/4Mx2N/4MxN, and Mx4N/2Mx4N/Mx4N.

When the ranges of the quadtree split and the binary tree/ternary tree overlap each other, and the current block is a block obtained by quadtree split, the split information can be divided into the following cases and configured.

(1) When quadtree splitting and binary tree/ternary tree splitting overlap

[Table 2]

In the above table, "a" denotes a flag indicating whether quadtree splitting is performed, and when the flag is 1, quadtree splitting is performed. When the flag is 0, "b" is checked, which is a flag indicating whether the partitioning is performed by the binary tree method or the ternary tree method. When b is 0, no further partitioning is performed in the block, but when b is 1, binary tree or ternary tree partitioning is performed.

"c" stands for a flag indicating the direction of splitting. c = 0 means horizontal split and c = 1 means vertical split. "d" stands for a flag indicating the type of partition. d = 0 means binary tree split (BT), and d = 1 means ternary tree split (TT).

(2) When only binary tree/ternary tree partitioning can be performed

In the above table, partitioning information can be expressed using the flags "b" through "d" with the exception of the flag "a".

In FIG. 5, the case of blocks A12 and A15 is a case in which quadtree splitting can be performed in blocks A12 and A15 before splitting, and thus corresponds to the case in which split information is generated in item (1).

On the other hand, the case of blocks A13 and A14 corresponds to the case in which the ternary tree split is performed without first performing the quadtree split in blocks A13 and A14 before splitting, and thus corresponds to the case in which split information is generated at point ( 2).

(No. 4, bottom left block, A16 to A20)

This example is a case in which schema partitioning of multiple trees (quadtree, binary tree, and ternary tree in this example) is supported, and the size and shape of a block can be determined, which can be obtained through a variety of block partitioning settings. This example assumes that in the case of a binary tree/ternary tree, the maximum coding unit is 64x64, the minimum coding unit is 4 in length, and the maximum split depth is 4.

Since this example is a case in which two or more blocks (two, three and four in this example) can be obtained by splitting, the split information needed for one split operation may be a flag indicating whether splitting is performed, a flag indicating a split type, a flag indicating a split shape, and a flag indicating a split direction, and the resulting candidates may be 4Mx4N, 4Mx2N, 2Mx4N, 4MxN/4Mx3N, 4Mx3N/4MxN, Mx4N/3Mx4N, 3Mx4N/Mx4N, 4MxN/4Mx2N /4MxN and Mx4N/2Mx4N/Mx4N.

When the ranges of the quadtree split and the binary tree/ternary tree overlap each other, and the current block is a block obtained by quadtree split, the split information can be divided into the following cases and configured.

(1) When quadtree splitting and binary tree/ternary tree splitting overlap

[Table 3]

In the above table, "a" denotes a flag indicating whether quadtree splitting is performed, and when the flag is 1, quadtree splitting is performed. When the flag is 0, "b" is checked, which is a flag indicating whether splitting is performed by the binary tree method. When b is 0, no further partitioning is performed in the block, but when b is 1, binary tree or ternary tree partitioning is performed.

"c" stands for a flag indicating the direction of splitting. c = 0 means horizontal split and c = 1 means vertical split. "d" stands for a flag indicating the type of partition. d = 0 means ternary splitting, and d = 1 means binary tree splitting. When d is 1, e is checked, which is a flag for the split form. When e is 0, a symmetrical split is performed, and when e is 1, an asymmetric split is performed. When e is 1, information about the detailed split ratio in asymmetric split is checked, as in the previous example.

(2) When only binary tree/ternary tree partitioning can be performed

In the above table, split information can be expressed using flags "b" through "f" with the exception of flag "a".

In FIG. 5, the case of block A20 is a case in which quadtree splitting can be performed in blocks A16 to A19 before splitting, and thus corresponds to the case in which split information is generated in item (1).

On the other hand, the case of blocks A16 to A19 corresponds to the case in which binary tree splitting is performed without first performing quadtree splitting in blocks A16 to A19 before splitting, and thus corresponds to the case in which split information is generated at point ( 2).

Next, an inter-picture prediction method based on the fusion mode will be discussed. In merge mode, motion information can be obtained in the elements of one block. In this case, a block may refer to a coding block determined by the aforementioned splitting block. In addition, a block is a leaf node coding block and may refer to a block that is no longer split into smaller coding blocks. In the merge mode, motion information can be obtained in the elements of sub-blocks belonging to the same block. To do this, one block can be further divided into a plurality of subblocks. For example, affine motion modeling may be performed based on the fusion mode in the subblock elements. The current block can selectively use either a block-based merge mode or a sub-block-based merge mode, and a given flag can be used for this. In this case, the flag may be information indicating whether the merge mode is applied based on sub-blocks. The flag may be encoded and transmitted by the encoder based on a predetermined encoding parameter, or may be received by the decoder. The encoding parameter may mean block size/shape, component type, split type, split depth, etc. For example, when the size of the current block is greater than or equal to NxM, a flag may be passed. Otherwise, the flag may be received as a predefined value. In this case, N and M may be an integer: 8, 16 or more. N and M may be the same or different. The block-based merge mode will be described with reference to FIG. 6, and the sub-block-based merge mode will be described with reference to FIG. 7–10, respectively.

In FIG. 6 shows a method for performing inter prediction based on a block-based merge mode as an embodiment to which the present invention is applied.

As shown in FIG. 6, a merge candidate list of the current block may be compiled (S600).

The merge candidate list may comprise at least one of a spatial merge candidate or a temporal merge candidate of the current block.

The motion information of the spatial merge candidate can be obtained from the motion information of the spatial neighboring block of the current block. In this case, a spatial neighbor block is a block belonging to the same image as the image of the current block, and may mean a block adjacent to the current block. The spatial neighbor block may include at least one of the left, top, top right, bottom left, or top left neighbor of the current block. The top left adjacent block can only be used when at least one of the blocks adjacent to the left, top, top right, and bottom left blocks is not available.

The motion information of the temporary merge candidate can be obtained from the motion information of the temporary neighboring block of the current block. The temporal neighbor block is a block belonging to an image different from the image of the current block, and may be defined as a block at the same position as the current block. In this case, the block at the same position may refer to at least one of a BR block adjacent to the bottom right corner of the current block, a CTR block having the position of the center origin of the current block, or a TL block having the position of the top left origin of the current block. . Alternatively, a block at the same position may refer to a block having a position shifted by a given disparity vector from the position of the upper left origin of the current block. In this case, the disparity vector may be determined based on any one of the spatial neighbor block motion vectors described above. Alternatively, the disparity vector may be determined based on a combination of at least two motion vectors of the spatial neighboring blocks described above. The combination may refer to a calculation such as a maximum value, a minimum value, a median value, and a weighted average. For example, the disparity vector may be set to the motion vector of the left neighbor block or the top neighbor block. Alternatively, the disparity vector may be set to the median value or average between the motion vector of the left neighbor block and the motion vector of the lower left neighbor block.

The motion vector and the reference picture index of the temporary merge candidate may be obtained as the motion vector and the reference picture index of the above-described temporal neighbor block, respectively. Alternatively, the motion vector of the temporal merge candidate may be obtained as the motion vector of the temporal neighbor block, and the reference picture index of the temporal merge candidate may be set to a default value (e.g., 0) previously sent to the device for decoding regardless of the temporal neighbor block .

The merge candidate list may further comprise a combined merge candidate. A combined merge candidate can be obtained by combining n merge candidates belonging to a pre-generated list of merge candidates.

In this case, n may be an integer: 2, 3, 4 or more. The number n of merge candidates to be combined may be a fixed value previously sent to the encoder/decoder, or may be encoded and transmitted by the encoder. The transfer can be performed in at least one element of a sequence, image, layer, tile, part of a tile (package) or a given block. The number n of merge candidates to be combined may be variably determined based on the number of remaining merge candidates. In this case, the number of remaining merger candidates may mean the difference between the maximum number of merger candidates that can be included in the merger candidate list and the current number of merger candidates belonging to the merger candidate list. The maximum number may be the amount previously sent to the encoder/decoder, or it may be encoded and transmitted by the encoder. The current count can mean the number of merge candidates configured before adding the combined merge candidate. For example, when the number of remaining merge candidates is 1, two merge candidates may be used, and when the number of remaining merge candidates is greater than 1, three or more merge candidates may be used.

The positions of the n merge candidates may be positions predetermined in the merge candidate list. For example, indices (from 0 to (k-1)) may be assigned to each merge candidate belonging to the list of merge candidates. In this case, k may mean the total number of merge candidates belonging to the list of merge candidates. In this case, the positions of n merge candidates may correspond from index 0 to index (n-1) in the list of merge candidates. Alternatively, n merge candidates may be determined considering the prediction direction of each merge candidate belonging to the merge candidate list. For example, among the merge candidates belonging to the merge candidate list, only the merge candidate corresponding to bidirectional prediction may be selectively used, or only the merge candidate corresponding to unidirectional prediction may be selectively used.

The combined merge candidate may be generated using both the spatial merge candidate and the temporal merge candidate, or may be generated using only either the spatial merge candidate or the temporal merge candidate. For example, a combined merge candidate may be limited to being derived using only a spatial merge candidate. In this case, the number of merge candidates to be combined may be limited within the number of spatial merge candidates belonging to the pre-generated merge candidate list.

The combined merge candidate may be added after the spatial/temporal merge candidate in the merge candidate list. That is, the combined merge candidate index may be greater than the spatial/temporal merge candidate index. Alternatively, a combined merge candidate may be added between the spatial merge candidate and the temporal merge candidate in the merge candidate list. That is, the combined merge candidate index may be greater than the spatial merge candidate index and less than the temporal merge candidate index. Alternatively, the position of the combined merge candidate may be variably determined based on the prediction direction of the combined merge candidate. The positions of the combined merge candidates in the merge candidate list may be arranged differently depending on whether the prediction direction of the combined merge candidate is bidirectional prediction. For example, when the combined merge candidate is a bidirectional prediction, an index smaller than the index of the spatial or temporal merge candidate may be assigned and, otherwise, an index greater than the index of the spatial or temporal merge candidate may be assigned.

Hereinafter, for convenience of description, a method for obtaining a combined merge candidate based on two merge candidates will be described.

The motion information of the combined merger candidate can be obtained by a weighted average of the motion information of the first merger candidate and the motion information of the second merger candidate. Here, the weighted average is [1:1], [1:2], [1:3], [2:3], etc., and is not limited to this. The weighting factor may be predetermined in the encoder/decoder or received from the decoder. In this case, the weighting factor can be obtained by considering at least one of the distance between the current picture and the reference picture of the merge candidate or the prediction direction of the merge candidate. Alternatively, the movement information of the combined merge candidate can be obtained by obtaining the movement information in the L0 direction from the first merge candidate and the movement information in the L1 direction from the second merge candidate, and combining the movement information in the L0 direction and the movement information in the direction L1. The motion information of the combined merge candidate may be obtained based on at least one of the above-described acquisition methods, and the acquisition may be performed considering the prediction direction of the merge candidate to be combined, as described below.

In this description, the motion information may comprise at least one of a prediction direction flag, a reference picture index, or a motion vector. Motion information may be determined for L0 prediction and L1 prediction, respectively. In this case, the L0 prediction may refer to the prediction related to the L0 reference picture list, and the L1 prediction may refer to the prediction related to the L1 reference picture list.

1. When both the first fusion candidate and the second fusion candidate are unidirectional predictions

(Case 1) When the first merge candidate is the L0 prediction and the second merge candidate is the L1 prediction, the reference picture index of the combined merge candidate in the L0 direction can be obtained as the reference picture index of the first merge candidate. The prediction direction flag of the combined merge candidate in the L0 direction may be obtained as 1. The motion vector of the combined merge candidate in the L0 direction may be obtained as the motion vector of the first merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be obtained as the reference picture index of the second merge candidate. The prediction direction flag of the combined merge candidate in the L1 direction may be obtained as 1. The motion vector of the combined merge candidate in the L1 direction may be obtained as the motion vector of the second merge candidate.

(Case 2) When the first merge candidate is the L1 prediction and the second merge candidate is the L0 prediction, the reference picture index of the combined merge candidate in the L0 direction can be obtained as the reference picture index of the second merge candidate. The prediction direction flag of the combined merge candidate in the L0 direction may be obtained as 1. The motion vector of the combined merge candidate in the L0 direction may be obtained as the motion vector of the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be obtained as the reference picture index of the first merge candidate. The prediction direction flag of the combined merge candidate in the L1 direction may be obtained as 1. The motion vector of the combined merge candidate in the L1 direction may be obtained as the motion vector of the first merge candidate.

(Case 3) When the first merge candidate and the second merge candidate are L0 prediction, the reference picture index of the combined merge candidate in the L0 direction can be obtained as the reference picture index of any of the first merge candidate and the second merge candidate. For example, a reference picture index of one of the first merge candidate and the second merge candidate having the smallest index may be set as the reference picture index of the combined merge candidate in the L0 direction. The prediction direction flag of the combined merge candidate in the L0 direction may be obtained as 1. The motion vector of the combined merge candidate in the L0 direction may be obtained as a weighted average of the motion vector of the first merge candidate and the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be obtained as -1, the prediction direction flag in the L1 direction may be obtained as 0, and the motion information in the L1 direction may be obtained as 0.

(Case 4) When the first merge candidate and the second merge candidate are L1 prediction, the reference picture index of the combined merge candidate in the L0 direction can be obtained as −1, the prediction direction flag in the L0 direction can be obtained as 0, and the information about moving in the L0 direction may be obtained as 0. The reference picture index of the combined merge candidate in the L1 direction may be obtained as the reference picture index of either the first merge candidate or the second merge candidate. For example, a reference picture index of one of the first merge candidate and the second merge candidate having the smallest index may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction flag of the combined merge candidate in the L1 direction may be obtained as 1. The motion vector of the combined merge candidate in the L1 direction may be obtained as a weighted average of the motion vector of the first merge candidate and the second merge candidate.

2. When both the first fusion candidate and the second fusion candidate are bidirectional predictions

(Case 5) The reference picture index of the combined merge candidate in the L0 direction may be obtained as the reference picture index of either the first merge candidate or the second merge candidate. For example, a reference picture index of one of the first merge candidate and the second merge candidate having the smallest index may be set as the reference picture index of the combined merge candidate in the L0 direction. The prediction direction flag of the combined merge candidate in the L0 direction may be obtained as 1. The motion vector of the combined merge candidate in the L0 direction may be obtained as a weighted average of the motion vector of the first merge candidate and the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be obtained as the reference picture index of either the first merge candidate or the second merge candidate. For example, a reference picture index of one of the first merge candidate and the second merge candidate having the smallest index may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction flag of the combined merge candidate in the L1 direction may be obtained as 1. The motion vector of the combined merge candidate in the L1 direction may be obtained as a weighted average of the motion vector of the first merge candidate and the second merge candidate.

3. When the first merge candidate is a bidirectional prediction and the second merge candidate is a unidirectional prediction

(Case 6) When the second merge candidate is the L0 prediction, the reference picture index of the combined merge candidate in the L0 direction can be obtained as the reference picture index of either the first merge candidate or the second merge candidate. For example, a reference picture index of one of the first merge candidate and the second merge candidate having the smallest index may be set as the reference picture index of the combined merge candidate in the L0 direction. The prediction direction flag of the combined merge candidate in the L0 direction may be obtained as 1. The motion vector of the combined merge candidate in the L0 direction may be obtained as a weighted average of the motion vector of the first merge candidate and the second merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be obtained as the reference picture index of the first merge candidate. The prediction direction flag of the combined merge candidate in the L1 direction may be obtained as 1. The motion vector of the combined merge candidate in the L1 direction may be obtained as the motion vector of the first merge candidate.

(Case 7) When the second merge candidate is an L1 prediction, the reference picture index of the combined merge candidate in the L0 direction can be obtained as the reference picture index of the first merge candidate. The prediction direction flag of the combined merge candidate in the L0 direction may be obtained as 1. The motion vector of the combined merge candidate in the L0 direction may be obtained as the motion vector of the first merge candidate. The reference picture index of the combined merge candidate in the L1 direction may be obtained as the reference picture index of either the first merge candidate or the second merge candidate. For example, a reference picture index of one of the first merge candidate and the second merge candidate having the smallest index may be set as the reference picture index of the combined merge candidate in the L1 direction. The prediction direction flag of the combined merge candidate in the L1 direction may be obtained as 1. The motion vector of the combined merge candidate in the L1 direction may be obtained as a weighted average of the motion vector of the first merge candidate and the second merge candidate.

As shown in FIG. 6, the movement information of the current block can be obtained from the merge candidate list (S610).

In particular, the merge index of the current block may be passed. The merge index may indicate any of a plurality of merge candidates belonging to the merge candidate list. That is, a merge candidate having the same index as the merge index can be selected, and motion information of the current block can be obtained using the motion information of the selected merge candidate.

As shown in FIG. 6, inter prediction of the current block can be performed using the acquired motion information (S620).

In particular, the reference picture may be selected from the reference picture list based on the reference picture index of the current block. The reference block in the reference picture may be specified based on the motion vector of the current block. The prediction pixel of the current block may be generated using the reconstructed pixel of the specified reference block. The current block can be reconstructed by adding the prediction pixel of the current block and the residual pixel. In this case, the residual pixel may be obtained by performing at least one of inverse quantization or inverse transform on the residual coefficient transmitted by the bitstream.

In FIG. 7 shows an inter-picture prediction method based on an affine model as an embodiment to which the present invention is applied.

As shown in FIG. 7, a list of candidates for predicting motion information of the current block may be generated (S700).

The candidate list may contain one or more affine model candidates (hereinafter referred to as affine candidates). An affine candidate may refer to a candidate having a checkpoint vector. The control point vector refers to the control point motion vector for the affine model, and may be defined for the angular position of the block (eg, at least one position among top left, top right, bottom left, or bottom right).

The affine candidate may include at least one of a spatial candidate, a temporal candidate, or a customized candidate. In this case, a spatial candidate may be derived from a neighboring block vector spatially adjacent to the current block, and a temporal candidate may be derived from a neighboring block vector temporally adjacent to the current block. In this case, a neighboring block may refer to a block encoded with an affine model. The vector may refer to a motion vector or a control point vector.

A method for obtaining a space/time candidate based on a space/time neighbor block vector will be described in detail with reference to FIG. 8.

Meanwhile, a tuned candidate can be obtained based on the combination of motion vectors of the space/time neighboring blocks with respect to the current block, which will be described in detail with reference to FIG. 9.

The plurality of affine candidates described above may be placed in the candidate list based on a given priority. For example, a set of affine candidates may be placed in the candidate list in the following order: spatial candidate, temporal candidate, and tuned candidate. Alternatively, the set of affine candidates may be arranged in the candidate list in the following order: temporal candidate, spatial candidate, and tuned candidate. However, the present invention is not limited to this, and the temporary candidate may be located after the tuned candidate. Alternatively, some of the tuned candidates may be located before the spatial candidate and the rest may be located after the spatial candidate.

The list of candidates may further comprise an interim candidate based on subblocks. The movement information of the temporal candidate may be obtained as a co-located block corresponding to the current block, which will be described in detail with reference to FIG. 10.

Based on the candidate list and the candidate index, a checkpoint vector of the current block can be obtained (S710).

The candidate index may refer to an index that is encoded to obtain the checkpoint vector of the current block. The candidate index may indicate any of the plurality of affine candidates belonging to the candidate list. The checkpoint vector of the current block may be obtained using the affine candidate checkpoint vector indicated by the candidate index.

For example, the affine model type of the current block is assumed to be 4-parameter (ie, it is determined that the current block uses two checkpoint vectors). In this case, when the affine candidate indicated by the candidate index has three checkpoint vectors, only two checkpoint vectors (eg, checkpoint vectors whose indices are 0 and 1) can be selected from the three checkpoint vectors and set to as checkpoint vectors of the current block. Alternatively, the three checkpoint vectors of the indicated affine candidates may be set as the checkpoint vectors of the current block. In this case, the affine model type of the current block can be updated to 6-parameter.

Conversely, the affine model type of the current block is assumed to be 6-parameter (ie, it is determined that the current block uses three checkpoint vectors). In this case, when the affine candidate indicated by the candidate index has two checkpoint vectors, one additional checkpoint vector can be generated, and the two checkpoint vectors and the additional checkpoint vector of the affine candidate can be set as the checkpoint vectors of the current block. . An additional checkpoint vector may be derived based on at least one of the two checkpoint vectors of the affine candidate, as well as current/adjacent block size or position information.

Alternatively, the two checkpoint vectors of the specified affine candidates may be set as the checkpoint vectors of the current block. In this case, the affine model type of the current block can be updated to 4-parameter.

The motion vector of the current block may be obtained based on the checkpoint vector of the current block (S720).

The motion vector can be obtained in the elements of sub-blocks of the current block. To do this, the current block can be divided into many NxM subblocks. In this case, the NxM sub-blocks may be in the form of a rectangle (N > M or N < M) or a square (N = M). The values of N and M may be 4, 8, 16, 32 or more.

For example, the sub-block size/shape may be a fixed size/shape predetermined in the decoding device. For example, the size/shape of the sub-block may be a square, such as 4x4, 8x8, or 16x16, or a rectangle, such as 2x4, 2x8, 4x8, or 4x16. Alternatively, a subblock may be defined as a block in which the sum of the width and height is 8, 12, 16, or greater. Alternatively, a subbox may be defined as a box in which the width times the height is an integer: 16, 32, 64, or greater.

Alternatively, the size/shape of the sub-block may be obtained in a variable manner based on an attribute of the above-described block. For example, when the size of the current block is greater than or equal to a given threshold size, the current block may be split into elements of the first sub-block (for example, 8x8 and 16x16), and, otherwise, the current block may be divided into elements of the second sub-block (for example, 4x4) .

Alternatively, the size/shape information of the sub-block may be encoded and transmitted by the encoder. The information indicates at least one of the size or shape of the sub-unit that can be conveyed at least one level of sequence, image, tile group, tile, and CTU.

The breakpoint representing the current block can have a top left position and a top right position. However, the present invention is not limited to this, and the reference point may have three points: an upper left position, an upper right position, and a lower left position, or may further have a plurality of additional points.

When two breakpoints are used, a motion vector for each sub-block of the current block can be obtained using at least one of a first breakpoint vector corresponding to the first breakpoint, a second breakpoint vector corresponding to the second breakpoint, and a position (x, y) subblock or size (width or height) of the current block.

When three breakpoints are used, a motion vector for each sub-block of the current block can be obtained using at least one of a first breakpoint vector corresponding to the first breakpoint, a second breakpoint vector corresponding to the second breakpoint, a third breakpoint vector corresponding to the third breakpoint, and the position (x, y) of the subbox or the size (width or height) of the current box. In this case, a difference vector between the second checkpoint vector and the first checkpoint vector may be used, and a difference vector between the third checkpoint vector and the first checkpoint vector may be used. A difference vector may be computed for each of a horizontal direction (x direction) and a vertical direction (y direction).

The motion vector derivation process may further include a process of applying a predetermined offset to the motion vector derived from the reference point vector. The shift may be relative to the vector to improve the previously obtained motion vector. The shift may be determined based on information about at least one of the size or direction of the shift. Size (absolute) can be an integer: 1, 2, 3 or more. The direction may include at least one of a left direction, a right direction, an upward direction, or a downward direction. Information about the size and/or direction of the shift may be encoded and transmitted by the encoding device. Alternatively, the shift size may be a fixed value preset in the decoder.

Inter prediction may be performed on the current block using the obtained motion vector (S730).

In particular, the reference block may be specified using the motion vector of the current block. A reference block can be specified for each subblock of the current block. The reference block of each sub-block may refer to one reference picture. That is, sub-blocks belonging to the current block can share one reference picture. Alternatively, the reference picture index may be independently set for each sub-block of the current block.

Meanwhile, the resulting motion vector may comprise at least one of the motion vector L0 and the motion vector L1. When the motion vector contains motion vectors L0 and L1, the decoding apparatus can perform unidirectional prediction by setting any of the motion vectors L0 and L1 to 0.

The setting may be selectively performed considering at least one of the block size or the prediction mode. In this case, the block may refer to the current block or a subblock of the current block. The prediction mode may refer to skip mode, merge mode, AMVP mode, affine model prediction mode, current image reference mode, etc.

For example, when the size of the current block (or a subblock of the current block) is less than or equal to a predetermined threshold size, unidirectional prediction is performed by setting the motion vector L1 to 0, and otherwise, bidirectional prediction can be performed using the motion vectors L0 and L1. In this case, the threshold size may be defined as a block size in which at least one of the width and height is 4, 8, or 16.

Alternatively, when the current block is encoded in the first prediction mode, unidirectional prediction is performed by setting the motion vector L1 to 0, and otherwise bidirectional prediction may be performed using the motion vectors L0 and L1.

Said reference block may be set as a prediction block of the current block. The current block can be reconstructed by adding the residual block and the prediction block.

Fig. 8 relates to a method for obtaining an affine candidate from a space/temporal neighbor block checkpoint vector as an embodiment to which the present invention is applied.

The width and height of the current block 800 are cbW and cbH, respectively, and the position of the current block is (xCb, yCb). The width and height of the spatial neighbor blocks 810-850 are nbW and nbH, respectively, and the positions of the spatial neighbor blocks are (xNb, yNb). The spatial neighbor block may include at least one of the left block 810, bottom left block 840, top right block 830, top block 820, or top left block 850 of the current block. Alternatively, the spatial adjacent block may further include at least one of the block adjacent to the right side or the block adjacent to the bottom side of the upper left block 850.

A spatial candidate may have n checkpoint vectors (cpMV). In this case, the value of n may be an integer: 1, 2, 3 or more. The value of n is determined based on at least one of information about whether decoding is performed in the elements of sub-blocks, information about whether the block is encoded by an affine model, or information about the type (4-parameter or 6-parameter) of an affine model.

For example, according to the information, when the corresponding block is a block decoded in subblock elements, or encoded by an affine model, the corresponding block may have two checkpoint vectors. On the other hand, otherwise, the corresponding block may not be predicted based on the affine model.

Alternatively, according to the information, when the corresponding block is an affine model encoded block and the affine model type is 6-parameter, the corresponding block may have three checkpoint vectors. On the other hand, otherwise, the corresponding block may be subject to prediction based on the affine model.

The above-described information may be encoded and transmitted by an encoder. Alternatively, all or part of the information may be obtained from the device for decoding based on the attribute of the block. In this case, a block may refer to the current block, or may refer to a spatial/temporal neighboring block of the current block. The attribute may refer to parameters related to size, shape, position, split type, inter frame mode, residual ratio, etc. Inter frame mode is a mode predefined in the device for decoding, and may refer to merge mode, skip mode, AMVP mode, affine model, combined intra/inter frame mode, current picture reference mode, etc. Alternatively, the value of n may be obtained from the decoding device based on the block attribute described above.

In the present embodiment, the n checkpoint vectors can be expressed as the first checkpoint vector (cpMV[0]), the second checkpoint vector (cpMV[1]), the third checkpoint vector (cpMV[2]), and the nth checkpoint vector. points (cpMV[n-1]).

As an example, the first checkpoint vector (cpMV[0]), the second checkpoint vector (cpMV[1]), the third checkpoint vector (cpMV[2]), and the fourth checkpoint vector (cpMV[3]) may be vectors , corresponding to the positions of the top left origin, top right origin, bottom left origin, and bottom right origin of the block. In this case, the spatial candidate is assumed to have three control point vectors, and the three control point vectors may be arbitrary control point vectors selected from the first to n-th control point vectors. However, the present invention is not limited to this, and the spatial candidate may have two checkpoint vectors, and the two checkpoint vectors may be arbitrary checkpoint vectors selected from the first to nth checkpoint vectors.

Next, a method for obtaining a spatial candidate checkpoint vector will be described.

1. When the current block boundary does not touch the CTU boundary

The first breakpoint vector may be obtained based on at least one of the first breakpoint vector of the spatial neighbor block, the predetermined difference value, and position information (xCb, yCb) of the current block, or position information (xNb, yNb) of the spatial neighbor block.

The number of difference values may be 1, 2, 3 or more. The number of delta values may be variably determined based on the attribute of the block described above, or may be a fixed value previously sent to the device for decoding. The delta value may be defined as the delta value between one of the plurality of checkpoint vectors and another of them. For example, the difference value may include at least one of a first difference between the second checkpoint vector and the first checkpoint vector, a second difference between the third checkpoint vector and the first checkpoint vector, a third difference between the fourth checkpoint vector and the third checkpoint vector. checkpoint, or the fourth difference between the fourth checkpoint vector and the second checkpoint vector.

For example, the vector of the first control point can be obtained as in Equation 1 below.

[Equation 1]

cpMvLX[0][0] = (mvScaleHor + dHorX*(xCb - xNb) + dHorY*(yCb - yNb))

cpMvLX[0][1] = (mvScaleVer + dVerX*(xCb - xNb) + dVerY*(yCb - yNb))

In Equation 1, the variables mvScaleHor and mvScaleVer may refer to the vector of the first control point of the spatial neighbor block, or the value obtained by applying a k-offset calculation to the vector of the first control point. In this case, k may be an integer: 1, 2, 3, 4, 5, 6, 7, 8, 9 or more. The variables dHorX and dVerX correspond to the x component and y component of the first difference between the second control point vector and the first control point vector, respectively. The variables dHorY and dVerY correspond to the x component and the y component of the second difference between the third control point vector and the first control point vector, respectively. The variables described above can be obtained as in Equation 2 below.

[Equation 2]

mvScaleHor = CpMvLX[xNb][yNb][0][0] << 7

mvScaleVer = CpMvLX[xNb][yNb][0][1] << 7

dHorX = (CpMvLX[xNb + nNbW - 1][yNb][1][0] - CpMvLX[xNb][yNb][0][0]) << (7 - log2NbW)

dVerX = (CpMvLX[xNb + nNbW - 1][yNb][1][1] - CpMvLX[xNb][yNb][0][1]) << (7 - log2NbW)

dHorY = (CpMvLX[xNb][yNb + nNbH - 1][2][0] - CpMvLX[xNb][yNb][2][0]) << (7 - log2NbH)

dVerY = (CpMvLX[xNb][yNb + nNbH - 1][2][1] - CpMvLX[xNb][yNb][2][1]) << (7 - log2NbH)

The second breakpoint vector may be obtained based on at least one of the first breakpoint vector of the spatial neighbor block, the predetermined difference value, the position information (xCb, yCb) of the current block, the block size (width or height), or the position information (xNb , yNb) spatial neighboring block. In this case, the block size may refer to the size of the current block and/or the size of the spatial neighboring block. The difference value is the same as in the description of the first control point vector, so its detailed description will be omitted. However, the range and/or number of deltas used in the process of deriving the second checkpoint vector may be different from those of the first checkpoint vector.

For example, the second control point vector can be obtained as in Equation 3 below.

[Equation 3]

cpMvLX[1][0] = (mvScaleHor + dHorX*(xCb + cbWidth - xNb) + dHorY*(yCb - yNb))

cpMvLX[1][1] = (mvScaleVer + dVerX*(xCb + cbWidth - xNb) + dVerY*(yCb - yNb))

In Equation 3, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 1, so their detailed description will be omitted.

A third breakpoint vector may be obtained based on at least one of a first breakpoint vector of a spatial neighbor block, a predetermined difference value, position information (xCb, yCb) of the current block, block size (width or height), or position information (xNb , yNb) spatial neighboring block. In this case, the block size may refer to the size of the current block and/or the spatial neighboring block. The difference value is the same as in the description of the first control point vector, so its detailed description will be omitted. However, the range and/or number of deltas used in the process of deriving the third checkpoint vector may be different from those of the first checkpoint vector or the second checkpoint vector.

For example, the third control point vector can be obtained as in Equation 4 below.

[Equation 4]

cpMvLX[2][0] = (mvScaleHor + dHorX*(xCb - xNb) + dHorY*(yCb + cbHeight - yNb))

cpMvLX[2][1] = (mvScaleVer + dVerX*(xCb - xNb) + dVerY*(yCb + cbHeight - yNb))

In Equation 4, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 1, so their detailed description will be omitted. Meanwhile, through the above-described process, the n-th checkpoint vector of the spatial candidate can be obtained.

2. When the current block boundary touches the CTU boundary

The first reference point vector may be obtained based on at least one of a motion vector (MV) of a spatial neighbor block, a predetermined difference value, position information (xCb, yCb) of the current block, or position information (xNb, yNb) of the spatial neighbor block.

The motion vector may be the motion vector of the sub-block located on the lowest side of the spatial neighboring block. A subblock may be the leftmost, centered, or rightmost subblock of the plurality of subblocks located on the lowest side of the spatial adjacent block. Alternatively, the motion vector may mean the average value, the maximum value, or the minimum value of the sub-block motion vectors.

The number of difference values may be 1, 2, 3 or more. The number of delta values may be variably determined based on the attribute of the block described above, or may be a fixed value previously sent to the device for decoding. The difference value may be defined as the difference value between one of the plurality of motion vectors stored in the sub-block elements in the spatial neighbor block and another of them. For example, the difference value may refer to the difference value between the motion vector of the lower right sub-block and the motion vector of the lower left sub-block of the spatial neighboring block.

For example, the vector of the first control point can be obtained as in Equation 5 below.

[Equation 5]

cpMvLX[0][0] = (mvScaleHor + dHorX*(xCb - xNb) + dHorY*(yCb - yNb))

cpMvLX[0][1] = (mvScaleVer + dVerX*(xCb - xNb) + dVerY*(yCb - yNb))

In Equation 5, the variables mvScaleHor and mvScaleVer may refer to the motion vector (MV) of the spatial neighbor block described above, or a value obtained by applying a k-offset calculation to the motion vector. In this case, k may be an integer: 1, 2, 3, 4, 5, 6, 7, 8, 9 or more.

The variables dHorX and dVerX correspond to the x component and the y component of the given difference value, respectively. Here, the difference value refers to a difference value between the motion vector of the lower right sub-block and the motion vector of the lower left sub-block in the spatial adjacent block. The variables dHorY and dVerY can be derived from the variables dHorX and dVerX. The variables described above can be obtained as in Equation 6 below.

[Equation 6]

mvScaleHor = MvLX[xNb][yNb + nNbH - 1][0] << 7

mvScaleVer = MvLX[xNb][yNb + nNbH - 1][1] << 7

dHorX = (MvLX[xNb + nNbW - 1][yNb + nNbH - 1][0] - MvLX[xNb][yNb + nNbH - 1][0]) << (7 - log2NbW)

dVerX = (MvLX[xNb + nNbW - 1][yNb + nNbH - 1][1] - MvLX[xNb][yNb + nNbH - 1][1]) << (7 - log2NbW)

dHorY=-dVerX

dVerY = dHorX

The second reference point vector may be obtained based on at least one of a motion vector (MV) of a spatial neighbor block, a predetermined difference value, position information (xCb, yCb) of the current block, a block size (width or height), or position information ( xNb, yNb) spatial neighboring block. In this case, the block size may refer to the size of the current block and/or the spatial neighboring block. The motion vector and the difference value are the same as in the description of the first reference point vector, so their detailed description will be omitted. However, the motion vector position, range, and/or number of deltas used in the process of deriving the second breakpoint vector may be different from those of the first breakpoint vector.

For example, the second control point vector can be obtained as in Equation 7 below.

[Equation 7]

cpMvLX[1][0] = (mvScaleHor + dHorX*(xCb + cbWidth - xNb) + dHorY*(yCb - yNb))

cpMvLX[1][1] = (mvScaleVer + dVerX*(xCb + cbWidth - xNb) + dVerY*(yCb - yNb))

In Equation 7, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 5, and their detailed description will be omitted.

A third reference point vector may be obtained based on at least one of a motion vector (MV) of a spatial neighboring block, a predetermined difference value, position information (xCb, yCb) of the current block, a block size (width or height), or position information ( xNb, yNb) spatial neighboring block. In this case, the block size may refer to the size of the current block and/or the spatial neighboring block. The motion vector and the difference value are the same as in the description of the first reference point vector, so their detailed description will be omitted. However, the position of the motion vector and the range and/or number of deltas used in the process of deriving the third breakpoint vector may be different from those of the first breakpoint vector or the second breakpoint vector.

For example, the third control point vector can be obtained as in Equation 8 below.

[Equation 8]

cpMvLX[2][0] = (mvScaleHor + dHorX*(xCb - xNb) + dHorY*(yCb + cbHeight - yNb))

cpMvLX[2][1] = (mvScaleVer + dVerX*(xCb - xNb) + dVerY*(yCb + cbHeight - yNb))

In Equation 8, the variables mvScaleHor, mvScaleVer, dHorX, dVerX, dHorY, and dVerY are as described in Equation 5, and their detailed description will be omitted. Meanwhile, through the above-described process, the n-th checkpoint vector of the spatial candidate can be obtained.

The above-described affine candidate derivation process may be performed for each of the predetermined spatial neighbor blocks. The predefined spatial neighbor blocks may include at least one of the left block, bottom left block, top right block, top block, or top left block of the current block.

Alternatively, the process of obtaining an affine candidate may be performed for each group of spatial neighboring blocks. In this case, the spatial neighbor blocks can be classified into a first group containing a left block and a lower left block, and a second group containing a top right block, a top block, and an upper left block.

For example, one affine candidate may be derived from a spatial neighbor block belonging to the first group. The derivation may be performed until an available affine candidate is found based on the given priority. The priority can be the following order: left block -> bottom left block, or the reverse order. According to the priority, it is determined whether the corresponding spatial neighbor block in the first group is a block decoded by affine model prediction, and the block first decoded by affine model prediction can be selected as an affine candidate.

Similarly, one affine candidate can be obtained from a spatial neighbor block belonging to the second group. The derivation may be performed until an available affine candidate is found based on the given priority. The priority can be in the following order: top right block -> top block -> top left block, or vice versa. According to the priority, it is determined whether the corresponding spatial neighbor block in the second group is a block decoded by affine model prediction, and the block first decoded by affine model prediction can be selected as an affine candidate.

The above-described embodiment can be applied in the same/similar manner to the temporary neighbor block. In this case, the temporary neighbor block belongs to an image different from the image of the current block, and may be a block at the same position as the current block. A block at the same position may be a block having the position of the top left origin of the current block, a center position, or a position of an origin adjacent to the bottom right origin of the current block.

Alternatively, a temporal adjacent block may mean a block at a position offset by a given disparity vector from a block at the same position. In this case, the disparity vector may be determined based on the motion vector of any of the spatial neighboring blocks of the current block described above.

In FIG. 9 shows a method for deriving a tuned candidate based on a combination of motion vectors of space/time neighboring blocks in an embodiment to which the present invention is applied.

A customized candidate according to the present invention can be obtained based on a combination of at least two checkpoint vectors (hereinafter referred to as checkpoint vectors (cpMVCorner[n])) corresponding to each corner of the current block. In this case, n may be 0, 1, 2, or 3.

The checkpoint vector cPMVCorner[n] may be derived based on the motion vector of the spatial neighbor block and/or the temporal neighbor block. In this case, the spatial neighbor block may include at least one of a first neighbor block A, B, or C adjacent to the top left anchor of the current block, a second neighbor D or E adjacent to the top right anchor of the current block, or a third neighbor. block F or G adjacent to the lower left origin of the current block. The temporal adjacent block is a block belonging to an image different from the image of the current block and may refer to a block at the same position as the current block (hereinafter referred to as the fourth adjacent block Col). In this case, the fourth adjacent block may refer to a block (H, I, or J) having the position of the top left origin, top right origin, or bottom left origin of the current block, or a block adjacent to the position of the bottom right origin of the current block.

The first adjacent block may refer to the top left (A), top (B), or left (C) adjacent block of the current block. It is determined whether the motion vectors of neighboring blocks A, B, and C are available according to the given priority, and the reference point vector can be determined using the available motion vectors of neighboring blocks. The availability determination may be performed until a neighboring block is found that has an available motion vector. In this case, the priority may be in the following order: A -> B -> C. However, the present invention is not limited to this, and the order may be: A -> C -> B, C -> A -> B or B -> A -> C.

The second neighbor block may refer to the top (D) or top right (E) neighbor block of the current block. Similarly, it can be determined whether motion vectors of adjacent blocks D and E are available according to a given priority, and a checkpoint vector can be determined using the available motion vectors of neighboring blocks. The availability determination may be performed until a neighboring block is found that has an available motion vector. In this case, the priority can be in the following order: D -> E, or in the following order: E -> D.

The third adjacent block may refer to the left (F) or bottom left (G) adjacent block of the current block. Similarly, it can be determined whether a motion vector of a neighboring block is available according to a given priority, and a checkpoint vector can be determined using the available motion vector of a neighboring block. The availability determination may be performed until a neighboring block is found that has an available motion vector. In this case, the priority can be in the following order: G -> F, or in the following order: F -> G.

For example, the first breakpoint vector (cpMVCorner[0]) can be set to the first neighbor block motion vector, the second breakpoint vector (cpMVCorner[1]) can be set to the second neighbor block motion vector, and the third breakpoint vector cpMVCorner[2] can be set as the motion vector of the third neighboring block. The fourth breakpoint vector cpMVCorner[3] may be set as the motion vector of the fourth neighbor block.

Alternatively, the first checkpoint vector may be derived using a motion vector of at least one of the first neighbor block or the fourth neighbor block, where the fourth neighbor block may be an H block having an upper left datum position. The second reference point vector may be obtained using the motion vector of at least one of the second neighboring block or the fourth neighboring block. In this case, the fourth adjacent block may be an I block having an upper right reference position. The third reference point vector may be obtained using the motion vector of at least one of the third neighboring block or the fourth neighboring block. In this case, the fourth adjacent block may be a J block having a lower left reference position.

Alternatively, any of the first to fourth checkpoint vectors may be derived from the other. For example, the second control point vector can be obtained by applying a given shift vector to the first control point vector. The shift vector may be a difference vector between the third breakpoint vector and the first checkpoint vector, or it can be obtained by applying a predetermined scaling factor to the difference vector. The scaling factor may be determined based on at least one of the width or height of the current block and/or a neighboring block.

By combining at least two of the first to fourth checkpoint vectors described above, K tuned ConstK candidates according to the present invention can be determined. The value of K may be an integer: 1, 2, 3, 4, 5, 6, 7 or more. The value of K may be obtained based on information transmitted by the encoder, or may be a value previously sent to the decoder. The information may include information indicating the maximum number of customized candidates included in the list of candidates.

In particular, the first tuned candidate Const1 can be obtained by combining the vectors from the first to the third control points. For example, the first Const1 candidate configured could have a checkpoint vector as shown in Table 4 below. Meanwhile, only when the reference image information of the first neighboring block is the same as the reference image information of the second and third neighboring blocks, the checkpoint vector may be limited to the configuration shown in Table 4. In this case, the reference image information may refer to the index a reference picture indicating the position of the corresponding reference picture in the reference picture list, or may refer to a picture sequence number (POC) value indicating the output order.

[Table 4]

A second tuned Const2 candidate can be obtained by combining the vectors of the first, second, and fourth control points. For example, the second configured Const2 candidate could have a checkpoint vector as shown in Table 5 below. Meanwhile, only when the reference image information of the first neighboring block is the same as the reference image information of the second and fourth neighboring blocks, the checkpoint vector can be limited to the configuration shown in Table 5. In this case, the reference image information is as described higher.

[Table 5]

A third tuned Const3 candidate can be obtained by combining the vectors of the first, third, and fourth control points. For example, a third configured Const3 candidate could have a checkpoint vector as shown in Table 6 below. Meanwhile, only when the reference image information of the first neighboring block is the same as the reference image information of the third and fourth neighboring blocks, the checkpoint vector can be limited to the configuration shown in Table 6. In this case, the reference image information is as described higher.

[Table 6]

A fourth tuned Const4 candidate can be obtained by combining the second, third, and fourth control point vectors. For example, a fourth configured Const4 candidate could have a checkpoint vector as shown in Table 7 below. Meanwhile, only when the reference image information of the second neighboring block is the same as the reference image information of the third and fourth neighboring blocks, the configuration can be restricted as shown in Table 7. In this case, the reference image information is as described above. .

[Table 7]

A fifth tuned candidate, Const5, can be obtained by combining the vectors of the first and second control points. For example, a fifth configured Const5 candidate could have a checkpoint vector as shown in Table 8 below. Meanwhile, only when the reference image information of the first neighbor block is the same as the reference image information of the second neighbor block, the checkpoint vector can be limited to the configuration shown in Table 8. In this case, the reference image information is as described above.

[Table 8]

A sixth tuned Const6 candidate can be obtained by combining the vectors of the first and third control points. For example, the sixth configured Const6 candidate could have a checkpoint vector as shown in Table 9 below. Meanwhile, only when the reference image information of the first neighbor block is the same as the reference image information of the third neighbor block, the checkpoint vector can be limited to the configuration shown in Table 9. In this case, the reference image information is as described above.

[Table 9]

In Table 9, cpMVCorner[1] may be a second checkpoint vector derived from the first and third checkpoint vectors. The second checkpoint vector may be derived based on at least one of the first checkpoint vector, a predetermined delta value, or the current/adjacent block size. For example, the second control point vector can be obtained as in Equation 9 below.

[Equation 9]

cpMvCorner[1][0] = (cpMvCorner[0][0] << 7) + ((cpMvCorner[2][1] - cpMvCorner[0][1]) << (7 + Log2(cbHeight/cbWidth) ))

cpMvCorner[1][1] = (cpMvCorner[0][1] << 7) + ((cpMvCorner[2][0] - cpMvCorner[0][0]) << (7 + Log2(cbHeight/cbWidth) ))

In the above-described K customized candidates ConstK, the value of K does not restrict the position or priority of the customized candidates located in the list of candidates.

In addition, all of the configured first to sixth candidates may be included in the candidate list, or only some of the candidates may be included in the candidate list.

For example, when it is determined that the current block uses three checkpoint vectors, only the tuned candidate generated by the combination of three of the first to fourth checkpoint vectors may be used. When it is determined that the current block uses two checkpoint vectors, a customized candidate generated by a combination of at least two of the first to fourth checkpoint vectors may be used, or a customized candidate generated by a combination of only two of the first to fourth checkpoint vectors may be used. points.

Alternatively, only some of the tuned candidates may be included in the candidate list, given the maximum number of affine candidates included in the candidate list. In this case, the maximum number may be determined based on the maximum number information transmitted by the encoder, or may be determined variably based on the above-described attribute of the current block. In this case, the value K of the configured candidate ConstK may indicate the priority to be entered in the list of candidates.

Meanwhile, when the difference between two checkpoint vectors belonging to the tuned candidate is less than a predetermined threshold, the tuned candidate may not be entered into the candidate list. The difference value between the two reference point vectors can be divided into a difference value in the horizontal direction and a difference value in the vertical direction. Here, the difference value in the horizontal direction may refer to the difference value between the first breakpoint vector 910 and the second breakpoint vector 920, and may refer to the difference between the first breakpoint vector 910 and the third breakpoint vector 930. The threshold value may refer to 0 or a vector having a value previously sent to the device for encoding/decoding.

In FIG. 10 is a diagram showing a sub-block-based temporal candidate motion information acquisition method as an embodiment to which the present invention is applied.

The motion information of the temporal candidate according to the present invention can be obtained from the motion information of the co-located block. Motion information can be obtained in subblock elements.

In this case, the alignment block is a block belonging to an image different from the image of the current block 1010 (i.e., the alignment image), and may be a block 1040 at the same position as the current block, or a block 1030 at a position shifted by a time vector from the position of the current block. The temporal vector may be determined based on the motion vector of the spatial neighboring block of the current block. The spatial neighbor block may refer to at least one of the left, bottom left, top, top right, or top left neighbors of the current block. The time vector may be determined using only the adjacent block at the position previously sent to the encoder/decoder. For example, the pre-sent position may be the left side 1011 or the top side 1012, and may be the left side 1011 and the top side 1012. sides or in the center among many neighboring blocks. When there are a plurality of neighboring blocks on the top side, a block located on one of the leftmost side, the rightmost side, or in the center among the plurality of neighboring blocks can be used.

When obtaining a sub-block-based provisional candidate, the current block and/or the overlapping block may be split into a plurality of sub-blocks.

In this case, the subblock may have a fixed size/shape previously sent to the device for encoding/decoding. For example, a subblock is represented by an NxM block, and the values of N and M can be integers: 4, 8, or greater. A subblock can be a square (N = M) or a rectangle (N > M, N < M). It can mean size. Alternatively, the encoding device may encode and transmit sub-block size/shape information, and the decoding device may determine the sub-block size/shape based on the transmitted information.

Alternatively, the current block and/or the overlapping block may be split into a plurality of sub-blocks based on a predetermined number. In this case, the number may be a fixed number previously sent to the encoder/decoder, or may be variably determined based on the size/shape of the block. Alternatively, the amount may be determined based on the amount information transmitted by the encoder.

Next, a method for obtaining information about the movement of a temporary candidate in the elements of sub-blocks will be investigated. The motion vector of the temporal candidate may be set as the motion vector stored for each sub-block of the matching block. The temporal candidate reference picture index may be set to a value (eg, 0) predetermined in the encoding/decoding device. Alternatively, the index of the reference picture of the temporary candidate may be set to the reference picture index of the matching block or the reference picture index of the upper left sub-frame among the sub-frames. Alternatively, similar to the motion vector, the reference picture index of the temporal candidate may be set to the reference picture index stored for each sub-block.

However, when there is an unavailable sub-block among the sub-blocks belonging to the matching block, the motion vector of the unavailable sub-block may be replaced by the motion vector of the available sub-block in the matching block. In this case, an accessible subblock may refer to a block adjacent to any of the left side, right side, top side, or bottom side of the inaccessible subblock. Alternatively, the available subblock may be a block at a position previously sent to the encoder/decoder. For example, the pre-sent position may be the position of the lower right sub-block in the matching block, or the position of a sub-block including the center position of the matching block. The above-described provisional candidate based on sub-blocks can be added to the list of candidates only when there is an available sub-block. Alternatively, the above-described temporal candidate based on sub-blocks may be added to the list of candidates only when a sub-block is available in the pre-sent position.

In FIG. 11 shows an ERP projection format inter-picture prediction method as an embodiment to which the present invention is applied.

As shown in FIG. 11, in the case of the ERP projection format, a case may occur in which the reference portion indicated by the motion vector overlaps the boundary of the reference image.

In this case, the boundary of the reference picture may be extended, and the extended area may be filled with data adjacent to the boundary in the horizontal direction (see A). Alternatively, the boundary of the reference image may be extended and the extended area may be filled with data (data having continuity or correlation with the boundary) adjacent to the opposite boundary (see B). Alternatively, the data in the non-existent region can be filled in using the data adjacent to the opposite boundary (see C).

A and B require a reference picture data expansion process. However, the calculation process can be simplified. Also, C doesn't require any extra storage because of the extension. However, an additional calculation process may be required.

In the case of A and B, the size of the current picture and the size of the (expanded) reference picture may not be the same. A predetermined adjustment process for such a difference in size may be included in the prediction pixel acquisition process.

When only horizontal lengths are compared, the current picture may have a range of 0 to (pic_width - 1), and a reference picture (expanded as the right side) may have a range of -L_offset to (pic_width - 1 + R_offset) or a range of 0 to ( pic_width - 1 + L_offset + R_offset). In the first case, the prediction pixel may be obtained from an extended reference image at a position obtained by adding the coordinates of the current block and the motion vector. In the latter case, the prediction pixel may be obtained at a re-corrected position after the obtained position has been corrected. In this case, L_offest and R_offset may refer to length values extended in the left and right directions. The present invention assumes that the coordinates are calculated based on the range of the current image or an existing reference image, as before.

In case C, the position in the reference picture can be calculated by adding the coordinates of the current block and the motion vector. You can check if the calculated pixel position is outside the range of the reference image (<0.0> to <pic_width - 1, pic_height - 1>). When the calculated pixel position does not exceed the range of the reference image, a prediction pixel can be obtained from the calculated pixel position. On the other hand, when the calculated pixel position is outside the range of the reference image, the position corresponding to the calculated position can be calculated at the position opposite the inside of the border <inside image> (right side in case of left side and left side in case of right side) , and the prediction pixel can be obtained from the calculated corresponding position.

Based on at least one of the above-described embodiments, it is possible to perform inter-picture prediction at a reference picture boundary. The inter frame prediction method at the reference picture boundary may be pre-sent by the encoding/decoding apparatus, and may be determined based on selection information indicating any of the above embodiments and information associated with the selected method (data processing method, extension size information etc.), which can be applied equally to the embodiments described below.

In FIG. 12 and 15 show an inter-picture prediction method in CMP projection format as an embodiment to which the present invention is applied.

As shown in FIG. 12, in the case of the CMP projection format, continuity in 3D space may exist between the 3 x 1 top surface and the 3 x 1 bottom surface due to patch packing. However, there may be no continuity in the 3D space between the top 3 x 1 surface and the bottom 3 x 1 surface.

Similarly, there may be a case in which the reference portion indicated by the motion vector overlaps a reference image boundary or a partial surface boundary. In case A, the reference area indicates the interior of the reference image, but is located at a partial boundary of the surface (there is no mutual continuity in three-dimensional space, so the correlation is low). In case B, the reference area is located at the boundary of the reference image.

In particular, in the case of A, even though the reference portion indicates the inside of the reference image, since the reference portion contains a continuous surface boundary, inter prediction accuracy may be lowered.

In the case of A and B, the position of the reference area in the reference image can be calculated by adding the coordinates of the current block and the motion vector. It can be checked whether at least part of the reference area is outside the range of the reference image or contains a continuous border in the reference image.

When the reference area is outside the range of the reference image, a position at which continuity or correlation exists in 3D space can be determined, and a prediction pixel can be obtained from the determined position.

On the other hand, when the reference area does not exceed the range of the reference image, it can be checked whether at least one pixel in the reference area indicated by the motion vector of the current block belongs to two or more surfaces. In this case, the verification process may be performed by pixel positions such as top left, top right, bottom left, and bottom right of the current block.

When at least one pixel in the reference area does not belong to two or more surfaces, the prediction pixel can be obtained from the reference area. However, otherwise, it is possible to check if the surface has continuity with the surface to which the current block belongs. In the case of the reference section A', the upper section has continuity with the surface to which the current block belongs, and the lower section has no continuity with the surface to which the current block belongs.

When the above-described continuity exists, the prediction pixel can be obtained from the reference area. However, otherwise, the prediction pixel may be obtained from a region having continuity or correlation in three-dimensional space.

The above-described order of inter-picture prediction is given as an example, and various modifications can be made by changing the order and changing some configurations.

In the case of C and D, the reference image may be expanded and the expanded area may be filled with padding data adjacent to the boundary in the horizontal/vertical direction or using correlated area data. Case D is the same as in the ERP description, so the detailed description will be omitted. In case C, as in case A, the reference area is located within the reference image, but contains a low-correlation surface boundary. Thus, when a single extended region is not generated within a reference picture, inter-picture prediction accuracy may be reduced, which will be described in detail with reference to FIG. 13.

In the case of C and D, the size of the current picture and the size of the reference picture may not be the same due to the extension of the reference picture. In this case, the reference image may have a range from (-L_offset, -U_offset) to (pic_width - 1 + R_offset, pic_height - 1 + D_offset). In this case, L_, R_, U_ and D_ offsets may mean length values extended in the left, right, up and down directions of the reference picture. The shifts may have the same value or may have different values. The length extended according to the reference picture may be explicitly transmitted or may be implicitly determined.

For example, one of the length values that can be supported by the block split section may be supported, such as 2, 4, 8, etc. Alternatively, the extension length may be determined based on a minimum coding block, a maximum coding block, a minimum prediction block, and a maximum block. predictions. Alternatively, the length may be determined based on elements such as tiles and packs.

For example, the length may be extended by the length of the maximum prediction block, in which case prediction blocks from all blocks may be obtained within the extended range. When the area indicated by the motion vector of the current block is outside the range of the expanded reference picture, even though the reference picture is expanded, the out-of-range portion prediction pixel can be obtained by horizontal, vertical, diagonal filling, etc.

As shown in FIG. 13, cases A-D may correspond to cases in which the reference image is expanded in elements of one or more surfaces. In the case of A and B, the reference image (or fragment of the reference image) is expanded by combining surfaces having continuity. Case C and D are examples in which the reference image is separately expanded for each surface. The extended area data processing is the same/similar to the data processing of the above-described embodiment. In the above embodiment, there is one range for the reference image. In this example, since each region (called a reference picture tile) is expanded by splitting the reference picture into two or more parts, overlapping ranges can be generated.

In examples A and B, the top widened reference picture has a range of (-L_offset, -U_offset) to (pic_width - 1 + R_offset, pic_height/2 - 1 + D_offset) and the bottom widened reference picture has range from (-L_offset, pic_height/2 - U_offset) to (pic_width - 1 + R_offset, pic_height - 1 + D_offset). It is assumed that the extended left, right, top and bottom lengths of each reference picture tile are the same. However, the lengths extending in the left, right, up, and down directions may be set equal in all elements to be extended, or may be set to be different. In addition, extension lengths can be set individually according to the extension elements.

In this case, the coordinates of the range extended downward from the top reference picture tile are the same as the coordinates inside the lower reference picture tile. However, in the case of the upper reference picture, the corresponding portion has expanded correlated data, and in the case of the lower reference picture, the corresponding portion has the original data before expansion.

As a result, the reference image may have a range from (-L_offset, -U_offset) to (pic_width - 1 + R_offset, pic_height - 1 + D_offset). In general, it is correct that one pixel value is assigned to each position. However, since the reference video is expanded by dividing into two or more sections considering the characteristics of the omnidirectional video, two or more pixel values may be assigned in a portion of the range. In this case, it may be necessary to determine which pixel value is to be obtained as the predicted pixel value at the position to which two or more pixel values are assigned.

As shown in FIG. 14, in the case of E, it is possible to determine which reference picture fragment contains a larger area, and obtain a prediction value using the data of the reference picture fragment containing the larger area.

In the case F, when the reference image fragment containing the current block contains a predetermined number of pixels or more, or contains a predetermined area, data of the corresponding reference image fragment can be used. However, otherwise, it may be possible to process using reference image fragment data of a portion containing a large area, such as E.

Case G is the case in which both reference picture tiles contain equal portions. In this case, the reference picture fragment containing the current block is given priority, and data of the corresponding reference picture fragment can be used. This example can be included as detailed terms F and E.

In summary, when the area indicated by the motion vector of the current block is the boundary between the reference picture tiles, it is possible to determine the reference picture chip whose data is used according to the coverage area and whether the reference picture tiles is the same as the current block. It is possible to determine the sub-picture data of the reference picture whose data is used to perform inter-picture prediction by including all or some of the above-described conditions according to the encoding/decoding setting.

In the above example, each region is expanded by dividing the reference image into two parts. However, in this example, a case will be described in which more reference image fragments are generated in surface elements.

In FIG. 15, each surface element is assumed to be a fragment of a reference image. Similarly, it is necessary to determine which pixel value is to be obtained as a prediction pixel value at a position assigned two or more pixel values.

As shown in FIG. 15, in the case of H, the reference picture sub-picture data in the area covered more may be used, or the reference picture sub-picture data containing the current block may be used.

For example, when a reference image fragment containing the current block is included by a predetermined number of pixels or in a predetermined area or more, data of the corresponding reference image fragment can be used. Alternatively, when reference image fragment data is used in a more heavily covered area, corresponding reference image fragment data may be used. However, when the included areas are the same, it is possible to limit the use of the reference picture fragment data containing the current block.

In case I, reference image fragment data can be used in the area covered to a greater extent. In this example, reference image fragments are supported in surface elements. However, the surface tuning has a characteristic in which the 3 x 1 top portion and the 3 x 1 lower portion are discontinuous, and thus priority can be given to a reference image portion on a side having a continuous characteristic over a reference image portion on a side having a discontinuous characteristic. characteristic.

In the case of J, the reference image fragment data can be used in the area covered to a greater extent. However, the included areas are equal, and in this example, the reference picture fragment containing the current block (upper right in the 3 x 2 area) is not covered. In addition, the reference image tiles (top left and center in the 3 x 2 region) belong to the same 3 x 1 region as the reference image fragment containing the current block. In this case, priority may be given to the one closest to the reference picture fragment containing the current block.

In summary, when the area indicated by the motion vector of the current block is a boundary between reference picture tiles, it is possible to determine the reference picture slice whose data is used according to the coverage area and the priority for using the reference picture slice.

Meanwhile, a method for performing inter prediction based on affine motion simulation has been described with reference to FIG. 7-10, which can be used identically/similarly to 360-degree video encoding/decoding, and its detailed description will be omitted.

The methods of the present invention may be implemented in the form of program instructions that may be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may contain program instructions, data files, data structures, etc., alone or in combination. Program instructions recorded on a computer-readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software.

Examples of a computer-readable medium may include a hardware device specially configured to store and execute a program instruction, such as ROM, RAM, or flash memory. Examples of a program instruction may include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine codes, such as those obtained using a compiler. The above described hardware device may be configured to operate as at least one software module to perform the operations of the present invention, and vice versa.

In addition, the method or apparatus described above may be implemented by combining all or some of the configurations or functions, or may be implemented by separating configurations or functions.

Even though the description has been given above with reference to the preferred embodiments of the present invention, it should be understood that various modifications and changes can be made to the present invention by those skilled in the art without departing from the scope and without departing from the spirit and scope of the present invention set forth in the following paragraphs. the attached claims.

[Industrial applicability]

The present invention can be used in encoding/decoding video signals.

Claims (31)

1. A method for decoding a video signal, the method comprising compiling a list of candidates for the merger of the current block; obtaining motion information of the current block from the merge candidate list, the motion information comprising at least one of a prediction direction flag, a reference picture index, or a motion vector; And performing interframe prediction of the current block using motion information, wherein the merge candidate list comprises at least one of a spatial merge candidate, a temporary merge candidate, or a combined merge candidate, and the combined merge candidate is obtained by combining n merge candidates belonging to the merge candidate list, and the combined merger candidate comprises a first merger candidate and a second merger candidate, and movement information of the combined merger candidate is obtained by a weighted average of the movement information of the first merger candidate and the movement information of the second merger candidate. 2. The method of claim 1, wherein the n merge candidates are merge candidates corresponding to indices 0 to (n-1) in the merge candidate list. 3. The method of claim 1, wherein the index of the combined merge candidate is greater than the index of the temporary merge candidate. 4. The method according to claim 1, wherein the movement information of the combined merge candidate is obtained taking into account the prediction direction of the first merge candidate and the prediction direction of the second merge candidate. 5. The method of claim 1, wherein the weighted average is any of [1:1], [1:2], [1:3], or [2:3]. 6. The method according to claim 1, characterized in that the motion vector is obtained in the elements of the subblocks of the current block. 7. The method of claim 6, wherein the subblock size is a fixed size. 8. The method according to claim 6 or 7, characterized in that the subblock size is 4x4. 9. The method of claim 1, wherein performing interframe prediction of the current block using the motion information includes selecting a reference picture from the reference picture list based on the reference picture index of the current block; determining a reference block in the reference image based on a motion vector of the current block; generating a prediction pixel of the current block using the determined reference block pixel; And restoring the current block by adding the prediction pixel of the current block and the residual pixel. 10. The method of claim 9, further comprising obtaining a residual pixel by performing at least one of inverse quantization or inverse transform on the residual coefficient transmitted via the bitstream. 11. A device for decoding a video signal, and the device contains a storage device configured to store a program for performing inter-picture prediction; And a processor configured to execute at program startup the method according to any one of claims. 1-10. 12. A method for encoding a video signal, the method comprising compiling a list of candidates for the merger of the current block; obtaining motion information of the current block from the merge candidate list, the motion information comprising at least one of a prediction direction flag, a reference picture index, or a motion vector; And performing interframe prediction of the current block using motion information, wherein the merge candidate list comprises at least one of a spatial merge candidate, a temporary merge candidate, or a combined merge candidate, and the combined merge candidate is obtained by combining n merge candidates belonging to the merge candidate list, and the combined merger candidate comprises a first merger candidate and a second merger candidate, and movement information of the combined merger candidate is obtained by a weighted average of the movement information of the first merger candidate and the movement information of the second merger candidate. 13. A device for encoding a video signal, and the device contains a storage device configured to store a program for performing inter-picture prediction; And a processor capable of executing the method of claim 12 when the program is started.

Images