KR20210027838A - Video signal encoding method and apparatus and video decoding method and apparatus - Google Patents

Abstract

Provided are a video encoding method and a device using a multi-triangular prediction block encoding method are provided. An object of the present invention is to improve the coding efficiency of a video signal.

Description

Video signal processing method and apparatus TECHNICAL FIELD [Video signal encoding method and apparatus and video decoding method and apparatus]

The present invention relates to a video signal processing method and apparatus.

Video images are compression-encoded by removing spatio-temporal redundancy and inter-view redundancy, which can be stored in a format suitable for a storage medium or transmitted through a communication line.

The present invention is to improve the coding efficiency of a video signal.

In order to solve the above problems, the present invention provides a video encoding method and apparatus using a multi-triangular prediction block encoding method.

The video signal processing method and apparatus according to the present invention can improve video signal coding efficiency through a multi-triangular prediction block coding method.

One Basic coding block structure

A picture is divided into a plurality of basic coding units (Coding Tree Unit, hereinafter, CTU) as shown in Figure 1.

Picture 1

It is configured so that it does not overlap with other CTUs. For example, the CTU size may be set to 128x128 in the entire sequence, and any one of 128x128 to 256x256 may be selected and used in picture units.

A coding block/coding unit (CU) may be generated by hierarchically dividing the CTU. Prediction and transformation may be performed in units of coding units, and become a basic unit for determining a prediction coding mode. The prediction encoding mode refers to a plurality of methods of generating a predicted image, such as intra prediction (intra prediction, hereinafter, intra prediction), inter prediction (hereinafter, inter prediction), or combined prediction. It can be mentioned. In more detail, for example, a prediction block may be generated by using at least one of intra prediction, inter prediction, and complex prediction in a coding unit unit. In the inter prediction mode, when the reference picture indicates the current picture, a prediction block may be generated in an area within the current picture that has already been decoded. Since a prediction block is generated using a reference picture index and a motion vector, it can be included in inter prediction. Intra prediction is a method of generating a prediction block using information of the current picture, inter prediction is a method of generating a prediction block using information of another picture that has already been decoded, and composite prediction is a method of mixing inter prediction and intra prediction. This is how you use it.

The CTU can be divided (partitioned) into a quad tree, binary tree or triple tree, as shown in Figure 2. The divided block can be further divided into a quad tree, binary tree, or triple tree. The method of dividing the current block into four square partitions is called quad-tree partitioning, the method of dividing the current block into two non-square partitions is called binary tree partitioning, and the method of dividing the current block into three non-square partitions is called quad-tree partitioning. It is defined as binary tree partitioning.

Binary partitioning in the vertical direction ( SPLIT_BT_VER in Figure 2 ) is called vertical binary tree partitioning, and binary tree partitioning in the horizontal direction ( SPLIT_BT_HOR in Figure 2 ) is defined as horizontal binary tree partitioning.

Triple-tree partitioning in the vertical direction ( SPLIT_TT_VER in Fig. 2 ) is called vertical triple-tree partitioning, and triple-tree partitioning in the horizontal direction ( SPLIT_TT_HOR in Fig. 2 ) is defined as horizontal triple-tree partitioning.

The number of additional divisions is called partitioning depth, and the maximum value of the partitioning depth can be set differently for each sequence or picture/tile set, and different partitioning depths according to the partitioning tree type (quad tree/binary tree/triple tree) It can be set to have, and a syntax indicating this can be signaled.

Picture 2

The divided coding block may be further divided by a method such as quad partitioning, binary partitioning, or multi-partitioning to configure a coding unit, or may configure a coding unit without further partitioning.

Figure 3

As shown in Figure 3 , a coding unit can be hierarchically set for one CTU, and a coding unit can be partitioned using at least one of coding units using binary tree partitioning, quad tree partitioning/triple tree partitioning. This method is defined as multi tree partitioning.

A coding unit generated by dividing an arbitrary coding unit having a partitioning depth of k is referred to as a lower coding unit, and the partitioning depth is k+1. A coding unit having a partitioning depth k including a lower coding unit having a partitioning depth k+1 is referred to as an upper coding unit.

The partitioning type of the current coding unit may be limited according to the partitioning type of the upper coding unit and/or the partitioning type of the coding units surrounding the current coding unit.

Here, the partitioning type represents an indicator indicating which partitioning of binary tree partitioning, quad tree partitioning/triple tree partitioning is used.

2 How to generate predictive image

In video encoding, a predicted image can be generated by a plurality of methods, and a method of generating a predicted image is referred to as a predictive encoding mode.

The prediction encoding mode may include an intra prediction encoding mode, an inter prediction encoding mode, a current picture reference encoding mode, or a combined prediction mode.

The inter prediction coding mode is a prediction coding mode that generates a prediction block (prediction image) of the current block using information of a previous picture, and the intra prediction coding mode is a prediction that generates a prediction block using samples adjacent to the current block. It is called an encoding mode. A prediction block may be generated using an image that has already been reconstructed of the current picture, and this is defined as a current picture reference mode or an intra block copy mode.

A prediction block may be generated by using at least two or more of an inter prediction encoding mode, an intra prediction encoding mode, or a current picture reference encoding mode, which is referred to as a combined prediction mode.

2.1 Intra prediction coding method

In intra prediction , as shown in Fig. 4 , an already coded boundary sample around the current block is used to generate intra prediction, and this is defined as an intra reference sample.

The average value of the intra reference samples is predicted by setting the value of all samples in the prediction block (DC mode), or the vertical direction prediction sample generated by weighted prediction of the horizontal direction prediction sample and the vertical direction reference sample generated by performing horizontal reference weighted prediction. After generation, a prediction sample may be generated by weighted prediction of a horizontal direction prediction sample and a vertical direction prediction sample (Planar mode), or intra prediction may be performed using a directional intra prediction mode.

Figure 4

As shown in the left figure of Figure 5, intra prediction can be performed using 33 directions (total of 35 intra prediction modes), and 65 directions (total of 67 intra prediction modes) as shown in the right figure of Figure 5 May be. In the case of using directional intra prediction, an intra reference sample (reference reference sample) may be generated in consideration of the direction of the intra prediction mode, and intra prediction may be performed therefrom.

An intra reference sample on the left side of the coding unit is referred to as a left intra reference sample, and an intra reference sample on the upper side of the coding unit is referred to as an upper intra reference sample.

When directional intra prediction is performed, as shown in Table 1, an intra direction parameter (intraPredAng), which is a parameter indicating a prediction direction (or a prediction angle), may be set according to the intra prediction mode. Table 3 below is only an example based on a directional intra prediction mode having a value of 2 to 34 when 35 intra prediction modes are used. It goes without saying that the prediction directions (or prediction angles) of the directional intra prediction modes are further subdivided so that more than 33 directional intra prediction modes can be used.

Picture 5

Table 1

Figure 6

When intraPredAng is negative (for example, when the intra prediction mode index is between 11 and 25), as shown in Figure 6 , the left intra reference sample and the upper intra reference sample are configured in 1D according to the angle of the intra prediction mode. It can be reconstructed as a one-dimensional reference sample (Ref_1D).

Figure 7

When the intra prediction mode index is between 11 and 18, a one-dimensional reference sample can be generated in a counterclockwise direction from an intra reference sample located on the upper right side of the current block to an intra reference sample located on the lower left side, as shown in Figure 7.

In other modes, a one-dimensional reference sample may be generated using only an upper side intra reference sample or a left side intra reference sample.

Figure 8

When the intra prediction mode index is between 19 and 25, a one-dimensional reference sample can be generated in a clockwise direction from an intra reference sample located at the lower left side of the current block to an intra reference sample located at the right side of the current block as shown in Figure 8.

A weight-related parameter ifact applied to at least one reference sample determined based on the reference sample determination index iIdx and iIdx may be derived as in Equations (1) to (2) below. iIdx and i _fact are variably determined according to the slope of the directional intra prediction mode, and a reference sample specified by iIdx may correspond to an integer pel.

iIdx = (y+1) * P _ang /32 (One)

i _fact = [(y+1) * P _ang ] & 31 (2)

A prediction image may be derived by specifying at least one one-dimensional reference sample for each prediction sample. For example, a position of a one-dimensional reference sample that can be used to generate a prediction sample may be specified in consideration of a slope value of a directional intra prediction mode. Each prediction sample may have a different directional intra prediction mode. A plurality of intra prediction modes may be used for one prediction block. The plurality of intra prediction modes may be expressed as a combination of a plurality of non-directional intra prediction modes, a combination of one non-directional intra prediction mode and at least one directional intra prediction mode, or a plurality of directional intra prediction modes. It can also be expressed as a combination of modes. Different intra prediction modes may be applied for each predetermined sample group in one prediction block. A predetermined sample group may consist of at least one sample. The number of sample groups may be variably determined according to the size/number of samples of the current prediction block, or may be a fixed number pre-set in the encoder/decoder independently of the size/number of samples of the prediction block.

Specifically, for example, the position of the one-dimensional reference sample may be specified using the reference sample determination index iIdx.

When the slope of the intra prediction mode cannot be expressed with only one one-dimensional reference sample according to the slope of the intra prediction mode, a first prediction image may be generated by interpolating adjacent one-dimensional reference samples as shown in Equation (3). When the angular line according to the slope/angle of the intra prediction mode does not pass the reference sample located in the integer pel, the first prediction image may be generated by interpolating the reference sample adjacent to the left/right or above/below the corresponding angular line. . The filter coefficient of the interpolation filter used at this time may be determined based on _{i fact.} For example, the filter coefficient of the interpolation filter may be derived based on a distance between a fractional pel located on an angular line and a reference sample located on the integer pel.

P(x,y) = ((32-i _fact )/32)* Ref_1D(x+iIdx+1) + (i _fact /32)* Ref_1D(x+iIdx+2) (3)

When the slope of the intra prediction mode can be expressed with only one one-dimensional reference sample (when the i _fact value is 0), the first prediction image can be generated as shown in Equation (4) below.

P(x,y) = Ref_1D(x+iIdx+1) (4)

2.2 Wide-angle intra prediction coding method

The prediction angle of the directional intra prediction mode may be set between 45 and -135 as shown in Figure 9.

Figure 9

When the intra prediction mode is performed in the amorphous coding unit, there may be a disadvantage of predicting a current sample from an intra reference sample far from the current sample instead of an intra reference sample close to the current sample due to a predefined prediction angle.

For example, as shown in the left figure of Fig. 10 , in a coding unit whose width is larger than the height of the coding unit (hereinafter, horizontal coding unit), intra prediction is performed at a far distance L instead of a close sample T. I can. As another example, as shown in the right figure of Figure 10, in a coding unit whose height is larger than the width of the coding unit (hereinafter, the vertical coding unit), the intra-distance sample T from the distant sample T instead of the close sample L You can make predictions.

Figure 10

In the amorphous coding unit, intra prediction may be performed at a prediction angle wider than a predefined prediction angle, and this is defined as a wide angle intra prediction mode.

내지

의 예측 각도를 가질 수 있으며, 기존 인트라 예측 모드에서 사용된 각도를 벗어나는 예측 각도를 와이드 앵글 각도라고 정의 한다. Wide angle intra prediction mode

To

It can have a prediction angle of, and a prediction angle that deviates from the angle used in the existing intra prediction mode is defined as a wide angle angle.

In the left figure of Figure 10 , sample A in the horizontal direction coding unit can be predicted from the intra reference sample T using the wide-angle intra prediction mode.

In the right figure of Figure 10 , sample A in the vertical direction coding unit can be predicted from the intra reference sample L using the wide-angle intra prediction mode.

N+M intra prediction modes may be defined by adding M wide angle angles to N existing intra prediction modes. Specifically, for example, as shown in Table 2, a total of 95 intra prediction modes may be defined by adding 28 wide angle angles to 67 intra modes.

The intra prediction mode that the current block can use may be determined according to the shape of the current block. For example, 65 directional intra prediction modes may be selected from among 95 directional intra prediction modes based on at least one of a size of a current block, an aspect ratio (eg, a ratio of width and height), and a reference line index.

Table 2

The intra prediction mode angle shown in Table 2 may be adaptively determined based on at least one of a shape of a current block and a reference line index. For example, the intraPredAngle of Mode 15 may be set to have a larger value when the current block is a square than when the current block is an amorphous. Alternatively, the intraPredAngle of Mode 75 may be set to have a larger value when a non-adjacent reference line is selected than when an adjacent reference line is selected.

When using the wide-angle intra prediction mode, the length of the upper intra reference sample can be set to 2W+1, and the length of the left intra reference sample can be set to 2H+1, as shown in Figure 11.

Figure 11

In the case of using wide-angle intra prediction, when the intra prediction mode of the wide-angle intra prediction mode is encoded, the number of intra prediction modes increases, and thus encoding efficiency may be lowered. The wide-angle intra prediction mode can be encoded by replacing the existing intra prediction mode that is not used in the wide-angle intra prediction mode, and the replaced prediction mode is called a wide-angle replacement mode. The wide angle replacement mode may be an intra prediction mode in a direction opposite to the wide angle intra prediction mode.

Specifically, for example, when 35 intra predictions are used as shown in Figure 12 , wide-angle intra prediction mode 35 can be encoded with intra prediction mode 2, which is a wide-angle replacement mode, and wide-angle intra prediction mode 36 is used for wide-angle replacement. It can be encoded in intra prediction mode 3, which is a mode.

Figure 12

The replacement mode and number may be differently set according to the shape of the coding block or the ratio of the height to the width of the coding block. Specifically, for example, Table 3 shows intra prediction modes used according to the ratio of the width and height of a coding block.

Table 3

2.3 Multi-line intra prediction coding method

As shown in Figure 13 below, intra prediction may be performed on a plurality of intra reference lines, which is referred to as a multi-line intra prediction coding method.

Fig. 13

Intra prediction may be performed by selecting any one of a plurality of intra reference lines composed of an adjacent intra parallel line and a non-adjacent intra reference line, and this is referred to as a multi-line intra prediction method. Non-adjacent intra reference lines include a first non-adjacent intra reference line (non-adjacent reference line index 1), a second non-adjacent intra reference line (non-adjacent reference line index 2), and a third non-adjacent intra reference line (non-adjacent reference line). It can be composed of index 3). Only some of the non-adjacent intra reference lines may be used. For example, only a first non-adjacent intra reference line and a second non-adjacent intra reference line may be used, or only a first non-adjacent intra reference line and a third non-adjacent intra reference line may be used.

An intra reference line index (intra_luma_ref_idx), a syntax specifying a reference line used for intra prediction, may be signaled in units of coding units.

Specifically, when using an adjacent intra reference line, a first non-adjacent intra reference line, and a third non-adjacent intra reference line, intra_luma_ref_idx may be defined as shown in Table 4 below.

Table 4

When a non-adjacent intra reference line is used, it may be set not to use a non-directional intra prediction mode. That is, when a non-adjacent intra reference line is used, it may be restricted not to use the DC mode or the planar mode.

The number of samples belonging to the non-adjacent intra reference line may be set to be larger than the number of samples of the adjacent intra parallel line. Also, the number of samples of the i+1th non-adjacent intra reference line may be set to be greater than the number of samples of the ith non-adjacent intra reference line. The difference between the number of upper samples of the i-th non-adjacent intra reference line and the number of upper samples of the i-1th non-adjacent intra reference line may be represented by the reference sample number offset offsetX[i].

offsetX[1] represents a difference between the number of upper samples of the first non-adjacent intra reference line and the number of upper samples of the adjacent intra reference line. The difference between the number of left samples of the i-th non-adjacent intra reference line and the number of left samples of the i-1th non-adjacent intra reference line may be expressed as a reference sample number offset offsetY[i]. offsetY[1] represents a difference value between the number of left samples of the first non-adjacent intra reference line and the number of left samples of the adjacent intra reference line.

A non-adjacent intra reference line with an intra reference line index of i may be composed of an upper non-adjacent reference line refW + offsetX[i], a left non-adjacent reference line refH+ offsetY[i], and an upper left sample. The number of included samples may consist of refW + refH + offsetX[i] + offsetY[i] +1.

refW =2* nTbW (5)

refH =2*　*nTbH (6)

In equations (5) to (6), nTbW represents the width of the coding unit, nTbH represents the height of the coding unit, and whRatio can be defined as in the following equation (7).

whRatio = Min(log2(nTbW/nTbH),2) (7)

In the multi-line intra prediction coding method, when a non-adjacent intra reference line is used, the wide-angle intra mode may be set not to be used. Alternatively, if the MPM mode of the current coding unit is a wide-angle intra mode, the multi-line intra prediction coding method may be set not to be used.

In this case, a non-adjacent intra reference line with an intra reference line index of i may consist of an upper non-adjacent reference line W + H + offsetX[i], a left non-adjacent reference line H + W + offsetY[i], and an upper left sample. , The number of samples belonging to the non-adjacent intra reference line may be composed of 2W + 2H + offsetX[i] + offsetY[i] +1, and the values of offsetX[i] and offsetY[i] may vary according to the whRatio value. have.

For example, if the whRatio value is greater than 1, you can set the offsetX[i] value to 1 and the offsetY[i] value to 0. If the whRatio value is less than 1, you can set the offsetX[i] value to 0, and offsetY[ i] You can also set the value to 1.

2.4 Inter prediction coding and decoding method

A method of generating a prediction block (prediction image) of a block in a current picture using information of a previous picture is referred to as an inter prediction encoding mode.

A prediction block may be generated from a specific block in a previous picture of the current block.

In the encoding step, the block with the smallest reconstruction error among the blocks in the previous picture can be selected by searching around the collocated block, and the x-axis difference and y-axis difference between the upper left sample of the current block and the upper left sample of the selected block Is defined as a motion vector, and can be signaled by transmitting it to a bitstream. A block generated through interpolation from a specific block of a reference picture specified by a motion vector is called a motion compensated predictor block.

As shown in Fig. 14 , the collocated block represents the block of the picture with the same location and size of the current block and the upper left sample. The corresponding picture may be specified from the same syntax as the reference picture reference.

Fig. 14

In the inter prediction encoding mode, a prediction block may be generated in consideration of the motion of an object.

For example, if you know how and in which direction an object in the previous picture has moved in the current picture, a prediction block (prediction image) can be generated by differentiating a block considering motion in the current block, and this is defined as a motion prediction block. .

A residual block may be generated by differentiating a motion prediction block or a corresponding prediction block from the current block.

When motion occurs in an object, if the motion prediction block is used rather than the corresponding prediction block, the energy of the residual block decreases, and compression performance may be improved.

This method of using a motion prediction block is called motion compensation prediction, and most inter prediction coding uses motion compensation prediction.

A value indicating in which direction and to what extent the object in the previous picture has moved in the current picture is called a motion vector. As the motion vector, motion vectors having different pixel precisions may be used in units of sequences, groups of tiles, or blocks. For example, the pixel precision of a motion vector in a specific block may be at least one of octor-pel, quarter-pel, half-pel, integer pel, and 4 integer pel.

In the inter prediction mode, an inter prediction method using translaton motion and an affine inter prediction method using affine motion may be selectively used.

2.4.1 Afine sub-block refinement coding method

Fig. 15

As shown in Figure 15, after generating affine prediction in units of sub-blocks in affine inter prediction, affine prediction in units of sub-blocks can be refined by applying an optical flow. It is defined as (Prediction refinement with optical flow).

Afine sub-block refinement (hereinafter, PROF) may be performed using the following process.

One. Affine sub-prediction image p(i,j) is generated by performing affine motion prediction in units of sub-blocks

2. Induce horizontal gradient g _x (i,j) and vertical gradient g _y (i,j)

g _x (i,j) = P(i+1, j)-P(i-1, j)

g _y (i,j) = P(i, j+1)-P(i, j-1)

_{3. The prediction refinement P ref} (i,j) is derived using the optical flow equation as follows.

_ref (i,j) = g _x (i,j) * DiffMV _x (i,j) + g _y (i,j) * DiffMV _y (i,j)

iffMV _x (i,j) = c*i + d* j

iffMV _y (i,j) = e*i + f* j

c= f = v1x-v0x / width

e =-d = v1y-v0y/ width

4. Generate final prediction sample P'(i,j)

P'(i,j) + P(I,j) + P _ref (i,j)

2.4.2 Merge mode coding method

Motion information (motion vector, reference picture index, etc.) of the current coding unit is not encoded, and can be derived from motion information of neighboring blocks. Motion information of any one of the neighboring blocks can be set as motion information of the current coding unit, and this is defined as a merge mode.

Neighboring blocks used for the remaining mode may be a picture 16 remaining candidate indices from 0 to 4 and the block are adjacent to the coding unit as the (boundary with the abutting block of the current coding unit), as shown in Figure 16, the remaining candidate index from 5 to 26 of the It may be a non-adjacent block.

Fig 16

When the distance of the merge candidate to the current block exceeds a predefined threshold, it may be set as not available.

For example, a predefined threshold may be set to the height of the CTU (ctu_height) or ctu_height+N, and this is defined as a merge candidate usable threshold. That is, the difference between the y-axis coordinates (y _i ) of the merge candidate and the y-axis coordinates of the upper left sample of the current coding unit (hereinafter, referred to as the current coding unit reference sample) (y ₀ ) (i.e., y _i- y ₀ ) is the merge candidate availability threshold. If it is greater than the value, the merge candidate may be set as not available. Here, N is a predefined offset value. Specifically, for example, N may be set to 16 or ctu_height may be set.

Figure 17

When there are many merge candidates crossing the CTU boundary, many unusable merge candidates occur, and encoding efficiency may be lowered. Merge candidates existing above the coding unit (hereinafter, upper merge candidates) may be set as small as possible, and merge candidates existing at the left and lower portions of the coding unit (hereinafter, lower left merge candidates) may be set as many as possible.

As shown in Fig. 17, the difference between the y-axis coordinate of the current coding unit reference sample and the y-axis coordinate of the upper merge candidate may not exceed twice the height of the coding unit.

It is also possible to set so that the difference between the x-axis coordinates of the reference sample of the current coding unit and the x-axis coordinates of the lower left merge candidate as shown in Figure 17 does not exceed twice the width of the coding unit.

2.4.3 Inter-decoding area merge method

Motion information (motion vector and reference picture index) of a coding unit that is already encoded by inter prediction in the current picture can be stored in a list having a predefined size, which is defined as an inter-region motion information list.

Motion information (motion vector and reference picture index) in the inter-region motion information list is called an inter-decoding region merge candidate.

The inter decoding region merge candidate can be used as a merge candidate of the current coding unit, and this method is defined as an inter decoding region merge method.

When the tile group is initialized, the inter-region motion information list is empty, and when a partial region of a picture is encoded/decoded, it can be added to the inter-region motion information list. The initial inter decoding region merge candidate of the inter region motion information list may be signaled through the tile group header.

When the coding unit is encoded/decoded by inter prediction, motion information of the coding unit may be updated in the inter-region motion information list as shown in Fig. 18. When the number of inter-decoding region merge candidates in the inter-region motion information list is the maximum value, the value with the smallest inter-region motion information list index (firstly, the inter-decoding region merge candidate in the inter-region motion information list) is removed. , A motion vector of the most recently encoded/decoded inter-region may be added as an inter-decoding region merge candidate.

Fig. 18

The motion information mvCand of the decoded coding unit may be updated in the inter-region motion information list HmvpCandList. At this time, if the motion information of the decoded coding unit is the same as any one of the motion information in the inter-region motion information list (when both the motion vector and the reference index are the same), the inter-region motion information list is not updated, or as shown in Figure 19. The motion information mvCand of the coding unit decoded together may be stored at the end of the inter-region motion information list. At this time, if the index of HmvpCandList that has the same motion information as mvCand is hIdx, HMVPCandList [i] can be set to HVMPCandList[i-1] for all i larger than hIdx as shown in Figure 19.

Figure 19

A total of NumHmvp motion information (motion vector and reference picture index) can be stored in the inter-region motion information list, and NumHmvp is defined as the size of the inter-region motion information list.

The size of the inter-region motion information list may use a predefined value. The inter-region motion information list size may be signaled in the tile group header. For example, the size of the inter-region motion information list may be defined as 16 or 6.

In a coding unit that is inter prediction and has an affine motion vector, it may be limited not to have an inter-region motion information list.

Alternatively, in the case of inter prediction and an affine motion vector, the affine subblock vector may be added to the inter-region motion information list. In this case, the position of the sub-block may be set to the upper left or upper right, or the center sub-block.

Alternatively, the average motion vector value of each control point may be added to the inter-region merge candidate list.

If the motion vector MV0 derived by encoding/decoding a specific coding unit is the same as any one of the inter-decoding region merge candidates, MV0 may not be added to the inter-region motion information list. Alternatively, an existing inter-decoding region merge candidate having the same motion vector as MV0 may be deleted, and the index allocated to MV0 may be updated by newly including MV0 in the decoding region merge candidate.

In addition to the inter-region motion information list, an inter-region motion information long term list HmvpLTList may be configured. The size of the inter-region motion information long term list may be set equal to the size of the inter-region motion information list, or may be set to a different value.

The inter-region motion information long term list may be composed of an inter-decoding region merge candidate initially added to the tile group start position. After the inter-region motion information long term list is configured with all available values, the inter-region motion information list may be configured, or motion information in the inter-region motion information list may be set as motion information of the inter-region motion information long term list.

At this time, the inter-region motion information long-term list configured once may be updated again when no update is performed, a decoded region of the tile group is more than half of the entire tile group, or may be set to be updated every m CTU lines. The inter-region motion information list may be updated whenever the inter-region is decoded, or may be set to be updated in units of CTU lines.

The merge candidates usable in the current coding unit may be configured as follows, and may have the same search order as the configuration order.

1. Spatial merge candidate (coding block adjacent merge candidate and coding block non-adjacent merge candidate)

2. Temporal merge candidate (merge candidate derived from previous reference picture)

3. Inter-decoding domain merge candidate

4. Inter-decoding domain affine merge candidate

5. Zero motion merge candidate

First, the merge candidate list may be composed of a spatial merge candidate and a temporal merge candidate mergeCandList. The number of available spatial merge candidates and temporal merge candidates is defined as the number of available merge candidates (NumMergeCand). When the number of available merge candidates is less than the maximum allowable merge number, an inter-decoding region merge candidate may be added to the merge candidate list mergeCandList.

When adding the inter-region motion information list HmvpCandList to the merge candidate list mergeCandList, it may be checked whether motion information of the inter-decoding region merge candidate in the inter-region motion information list is the same as the motion information of the existing merge list mergeCandList. If the motion information is the same, the merge list may not be added to the mergeCandList, and if the motion information is not the same, the inter-decoding region merge candidate may be added to the merge list mergeCandList.

Among the merge candidates in the inter-decoding area, one with a large index can be added to the merge candidate list mergeCandList, or one with a small index can be added to the merge candidate list mergeCandList.

The inter-decoding region motion candidate can be used as a motion vector predictor (MVP) candidate of the current coding unit, and this method is defined as an inter-decoding region motion information prediction method.

An inter-decoding region affine motion candidate can be used as a motion vector predictor (MVP) candidate of the current coding unit, and this method is defined as an inter-decoding region motion information affine prediction method.

Motion information predictor candidates usable in the current coding unit may be configured as follows, and may have the same search order as the configuration order.

1. Spatial motion predictor candidate (same as a coding block adjacent merge candidate and a coding block non-adjacent merge candidate)

2. Temporal merge candidate (motion predictor candidate derived from previous reference picture)

3. Inter-decoding domain merge candidate

4. Inter-decoding domain affine merge candidate

5. Zero motion motion predictor candidate

2.5 Inter-Intra Combined Prediction Flag Coding Method

Among the combined prediction encoding modes, a final prediction image may be generated by weighted prediction of a prediction image (hereinafter referred to as a merge prediction image) generated through an intra prediction method on a prediction image (hereinafter referred to as a merge prediction image) generated from the merge mode. It is defined as merge-intra coupling prediction. A flag indicating whether merge-intra-combination prediction has been used A merge-intra-combination prediction flag (ciip_flag) may be signaled through a bitstream. If the ciip_flag value is 1, a merge-intra prediction image may be generated by weighted prediction of a merge prediction image acquired based on specific motion information in merge_idx and an intra prediction image acquired based on an intra prediction mode.

If the ciip_luma_mpm_flag value is 1, the intra prediction mode of the merge-intra combined prediction mode can be derived from the MPM candidate. If the ciip_luma_mpm_flag value is 0, the intra prediction mode may be derived from a predefined intra prediction mode (hereinafter, merge-intra default mode) that is not an MPM candidate.

For example, the merge-intra default mode may include at least one of a planar mode, a DC mode, a horizontal direction prediction mode, and a vertical direction prediction mode. For example, when the value of ciip_luma_mpm_flag is 0, one of the plurality of merge-intra default modes may be determined as the intra prediction mode of the current block based on the size and/or shape of the current block. Alternatively, when the value of ciip_luma_mpm_flag is 0, an index ciip_luma_default_idx for specifying any one of the multiple merge-intra depth modes may be signaled. Alternatively, when the value of ciip_luma_mpm_flag is 0, a planar mode, a DC mode, a horizontal direction prediction mode, or a vertical direction prediction mode may be determined as an intra prediction mode of the current block by default.

As another example, the planner mode may be set as the merge-intra default mode.

Table 5

In the case of using merge-intra combined prediction, as shown in Table 5, a combined intra non-directional indication flag (ciip_intra_nondir_mode_flag) indicating whether the combined intra prediction mode is a non-directional prediction mode may be signaled. The non-directional prediction mode may include at least one of a DC mode and a planar mode.

If the value of ciip_intra_nondir_mode_flag is 1, it indicates that the combined intra prediction mode is a DC mode or a planar mode. If the ciip_intra_nondir_mode_flag value is 0, it indicates that the combined intra prediction mode is not the DC mode or the planar mode.

When the ciip_intra_nondir_mode_flag value is 1, the combined intra prediction mode may signal a flag ciip_luma_nonDir_idx for specifying one of a DC mode and a planar mode. A ciip_luma_nonDir_idx value of 0 indicates planar mode, and a ciip_luma_nonDir_idx value of 1 indicates DC mode.

When the ciip_intra_nondir_mode_flag value is 0, the intra prediction mode of the current block may be derived from MPM candidates. When the ciip_intra_nondir_mode_flag value is 0, ciip_luma_mpm_idx for specifying one of the MPM candidates may be signaled.

Alternatively, when the ciip_intra_nondir_mode_flag value is 0, the intra prediction mode of the current block may be derived from the default mode(s). The default mode may include at least one of a vertical direction prediction mode, a horizontal direction prediction mode, and a diagonal direction prediction mode.

Instead of the non-directional indication flag (ciip_intra_nondir_mode_flag), a planner indication flag (ciip_intra_planar_mode_flag) or a DC indication flag (ciip_intra_DC_mode_flag) may be used. The planar message flag indicates whether the intra prediction mode of the current block is the planar mode. The DC indication flag indicates whether the intra prediction mode of the current block is a DC mode. When the planar indication flag or the DC indication flag is 0, the intra prediction mode of the current block may be determined from the default mode(s) or MPM candidates.

When the planner indication flag is used, the default mode may include at least one of a DC mode, a vertical direction prediction mode, a horizontal direction prediction mode, and a diagonal direction prediction mode.

When the DC indication flag is used, the default mode may include at least one of a planar mode, a vertical direction prediction mode, a horizontal direction prediction mode, and a diagonal direction prediction mode.

Fig. 21

When Ciip is applied, a prediction sample of the current block may be obtained by weighted prediction of a prediction image obtained by intra prediction and a prediction image obtained based on a merge mode. For example, a prediction sample can be derived as shown in Equation (21) below.

PredSamplesComb[x][y] = (w* PredSamplesIntra[x][y] + (4-w)*PredSamplesInter + 2) >> 2 (21)

In Equation (21), PredSamplesIntra represents an intra prediction sample, and PredSamplesInter represents an inter prediction sample.

In this case, a weight applied to the intra prediction sample and the inter prediction sample may be determined based on prediction information of neighboring blocks of the current block. Here, the neighboring block of the current block may include at least one of an upper neighboring block, a left neighboring block, an upper left neighboring block, an upper right neighboring block, and a lower left neighboring block.

As shown in Figure 21, the weighted prediction value w of ciip can be set based on the prediction mode information of the left block and the upper block. Specifically, the weighted prediction value w may be determined according to the number of neighboring blocks encoded in the intra prediction mode and/or the number of neighboring blocks encoded in the inter prediction mode. For example, if both the left block and the upper block are in intra prediction mode, the weighted prediction value w may be set to 3, and if both the left block and the upper block are not in intra prediction mode, the weighted prediction value w may be set to 1, otherwise The weighted predicted value w can be set to 2.

Alternatively, when the neighboring block is coded with ciip, a weight applied to the neighboring block may be determined as the weight of the current block. For example, when both the left block and the upper block are ciip-coded coding blocks, a ciip weighted prediction value of the current block may be determined based on any one of ciip weighted prediction values of the left block and the upper block. For example, a larger value among the ciip weighted predictors of the left block and/or the upper block may be determined as the ciip weighted predictive value of the current block. Alternatively, a smaller value of the ciip weighted predicted value of the left block and/or the upper block may be determined as the ciip weighted predicted value of the current block. Alternatively, the average value of the ciip weighted predicted value of the left block and the upper block may be determined as the ciip weighted predicted value of the current block. Alternatively, according to the size/shape of the current block, a ciip weighted prediction value of one of a left block and an upper block may be selected.

When any one of the left block and the upper block is coded with ciip, the ciip weighted prediction value of the neighboring block coded with ciip may be used as the weighted prediction value of the current block.

The number of merge candidates allowed under ciip mode may be limited to N or less. N may be a natural number such as 1 or 2. Among the merge candidates included in the merge candidate list under the CIIP mode, merge candidates having an index of N-1 or less may be used. Alternatively, it may be set to use only a merge candidate derived from any one of the left neighboring blocks and a merge candidate derived from any one of the upper neighboring blocks. N may be a value predefined by an encoder and a decoder. Alternatively, information indicating the number of merge candidates allowed under the ciip mode may be signaled through a bitstream. The information may indicate a difference value between the maximum number of merge candidates allowed in the merge mode and the maximum number of merge candidates allowed in the ciip mode.

When two merge candidates are used, if the ciip_flag value is 1, merge_idx may have a value of 0 or 1. In this case, the merge_idx may be determined based on the merge_idx_flag. A flag of 0 indicates that merge_idx is 0, and a flag of 1 indicates that merge_idx is 1. When one merge candidate is used, when a ciip_flag value is 1, merge_idx may have only a value of 0. In this case, the merge_idx may not be signaled as shown in Table 6 below.

Table 6

Alternatively, even when the mmvd_merge_flag value is 1, merge-intra combination prediction may be used.

For example, as shown in Table 7 below, when the mmvd_merge_flag value is 1, ciip_flag may be transmitted through a bitstream, and a predicted image derived from mmvd may be used for merge-intra combination prediction.

Table 7

If the ciip_flag value is 1, it may be set not to use PROF.

2.5.1 Triangular prediction block coding method

After dividing the currently encoded/decoded coding unit into at least one prediction block that does not have a square and/or rectangular shape, encoding/decoding may be performed. Specifically, the coding unit may be encoded/decoded into at least one prediction block using at least one or more lines such as a vertical line, a horizontal line, or a diagonal line. At this time, at least one of the information on the start point, the end point, the number of lines, the angle required for the division, the number of divided prediction blocks, and the shape of a prediction block having an arbitrary shape will be signaled through the bitstream. I can. Alternatively, depending on the intra prediction mode, the inter prediction mode, the position of the available merge candidate, etc. of the coding unit, the starting point, the end point, the number of lines for dividing the coding unit, the angle required for the division, the number of divided prediction blocks, and any shape At least one of information on the shape of a prediction block having a may be implicitly derived. The coding unit may be divided into at least one prediction block having a shape different from that of a square and/or rectangular prediction block, and intra prediction and/or inter prediction may be performed in units of the divided prediction blocks.

Fig. 21 shows an example of dividing a coding unit into two prediction blocks using a diagonal line. Dividing the coding unit into two prediction blocks using a diagonal line may be defined as symmetric diagonal partitioning. In Figure 21, it is shown that the coding unit is divided into two triangular prediction blocks of the same size.

The left figure in Figure 21 can be called left triangular partitioning, and the right figure in Figure 21 can be defined as right triangular partitioning. The prediction block to which the upper left or lower left sample of the coding unit belongs can be defined as a left triangular prediction block, the prediction block to which the upper right or lower right sample of the coding unit belongs can be defined as a right triangular prediction block, and Alternatively, the left triangular prediction block may be collectively defined as a triangular prediction block.

Fig. 21

When a neighboring block of the current coding unit is coded by diagonal partitioning, a motion vector of either a left triangular prediction block or a right triangular prediction block may be used as a merge candidate according to the position of the spatial merge candidate. For example, as shown in Figure 22, a motion vector of a triangular prediction block adjacent to the coding unit may be used as a merge candidate. If A1 is selected as the merge candidate in the left figure of Figure 22, the motion vector of the right triangular prediction block P2 adjacent to the current coding unit can be used as the merge candidate, and if B1 is selected as the merge candidate in the left figure of Figure 22, the current The motion vector of the left triangular prediction block P1 adjacent to the coding unit may be used as a merge candidate. For another example, if A1 is selected as the merge candidate in the right figure of Figure 22, the motion vector of the right triangular prediction block P2 adjacent to the current coding unit can be used, and B1 is the merge candidate in the right figure of Figure 22. If selected as, the motion vector of the right triangular prediction block P2 adjacent to the current coding unit may be used as a merge candidate.

Fig. 22

When a neighboring block or a co-located block of the current coding unit is coded by diagonal partitioning, a merge candidate may be set as not available.

Information indicating whether to apply diagonal partitioning to the coding unit may be signaled through a bitstream. For example, a triangle partition flag (triangle_partition_flag), which is a syntax indicating whether diagonal partitioning is applied to the coding unit, may be signaled in units of CU.

In addition, information indicating the direction of diagonal partitioning may be signaled through a bitstream. For example, a triangle partition type flag (triangle_partition_type_flag), which is a syntax indicating whether left triangular partitioning or right triangular partitioning is used in CU units, may be signaled in CU units. If the triangle_partion_type_flag value is 0, it indicates the left triangular partition, and if the triangle_partition_type_flag value is 1, it indicates the right triangular partition. Alternatively, information indicating the direction of diagonal partitioning may be signaled in at least one unit of a coding tree unit unit, a tile unit, a tile set (tile group), and a slice unit. In this case, among the coding units included in the level at which the information is signaled, the coding units to which the diagonal partitioning is applied may have the same partition shape.

As another example, the triangular partitioning type of the coding unit may be determined based on the triangular partitioning type of a neighboring coding unit adjacent to the coding unit. As an example, the triangular partitioning type of the coding unit may be determined to be the same as the triangular partitioning type of the neighboring coding unit. Here, the neighboring coding unit is at least one of a neighboring block located adjacent to a diagonal direction of the current coding unit, a neighboring block located adjacent to the top or left of the current coding unit, or a co-located block and a co-located block neighboring block. It can be defined as a block.

Information indicating the direction of the diagonal partitioning is signaled to the coding unit to which the first triangular partitioning is applied in the CTU, while the same diagonal partitioning direction as the first coding unit may be applied to the second and subsequent coding units to which the triangular partitioning is applied.

Each of the left triangular prediction block and the right triangular prediction block may have separate motion vectors and reference picture indexes. The motion vector of the triangular prediction block can be derived from a triangular merge candidate list consisting of a spatial merge candidate list and a temporal merge candidate list as shown in Figure 23. The maximum number of triangular merge candidates allowed for the current prediction block can be arbitrarily set.

Triangular prediction blocks may use different merge candidate lists. As an example, the merge candidate list of the right triangular prediction block may be constructed using the remaining merge candidates excluding the merge candidate indicated by the merge candidate index of the left triangular prediction block among the merge candidates of the left triangular prediction block. The maximum values of the triangular merge candidates of the left triangular prediction block and the right triangular prediction block may be set differently. As an example, the left triangular prediction block may have M triangular merge candidates, while the right triangular prediction block may have M-1 triangular merge candidates excluding merge candidates indicated by the merge candidate index of the left triangular prediction block.

As another example, triangular prediction blocks may share one merge candidate list.

The triangular merge candidates may be added to the merge candidate list in a predefined order. As an example, the following order among the merge candidates in Figure 23 may be used.

0 --> 1 --> 2 --> 3 --> 7 --> 4 --> 6

The candidate block selected from the triangular merge candidate list may be represented by a triangular merge candidate index (merge_triangle_idx).

Fig. 23

Only the merge candidate index of the left triangular prediction block or the right triangular prediction block may be signaled, and the merge candidate index of the remaining prediction blocks may be derived from the signaled merge candidate index.

As an example, only the merge candidate index of the left triangular prediction block may be signaled, and a merge candidate index adjacent to the left triangular prediction block merge candidate index may be derived as a merge candidate index of the right triangular prediction block.

Specifically, for example, if the left triangular prediction block merge candidate index is N, the right triangular prediction block merge candidate index may be derived as N+1. Only the left triangular prediction block merge candidate index may be encoded, the right triangular prediction block merge candidate index may not be encoded, and may be derived from the left triangular prediction block merge candidate index.

When the left triangular prediction block merge candidate index N is the triangular merge candidate maximum value, the right triangular prediction block merge candidate index may be derived as N-1 or 0.

Alternatively, motion information of the right triangular prediction block may be derived from a merge candidate having the same reference picture as the merge candidate indicated by the left triangular prediction block merge candidate index. Here, the same reference picture may represent a merge candidate in which at least one of the L0 reference picture and the L1 reference picture is the same. When there are a plurality of merge candidates having the same reference picture, either one may be selected based on whether bi-directional prediction or an index difference value with a merge candidate indicated by the merge candidate index of the left triangular prediction block.

A merge candidate index may be signaled for each of the left triangular prediction block and the right triangular prediction block. For example, a first merge candidate index may be signaled for a left triangular prediction block, and a second merge candidate index may be signaled for a right triangular prediction block. The motion information of the left triangular prediction block may be derived from a merge candidate signaled by the first merge candidate index, and the motion information of the right triangular prediction block may be derived from a merge candidate signaled by the second merge candidate index.

In this case, the merge candidate indicated by the first merge candidate index of the left triangular prediction block may be set not to be usable as a merge candidate of the right triangular prediction block. Accordingly, the second merge candidate index may indicate any one of the remaining merge candidates excluding the merge candidate indicated by the first merge candidate index. As an example, when the value of the second merge candidate index is smaller than the value of the first merge candidate index, the motion information of the right triangular prediction block is a merge having the same index as the second merge candidate index among merge candidates included in the merge candidate list. Can be derived from candidates. On the other hand, when the value of the second merge candidate index is the same as the value of the first merge candidate index or is greater than the first merge candidate index, the motion information of the right triangular prediction block is the second merge candidate among merge candidates included in the merge candidate list. It can be derived from a merge candidate having an index of 1 greater than the candidate index.

A range of merge candidates that can be used by the left triangular prediction block and the right triangular prediction block may be differently set. For example, motion information of the left prediction block may be derived from at least one of merge candidates adjacent to the left prediction block, while motion information of the right prediction block may be derived from at least one of merge candidates adjacent to the right prediction block.

The coding unit may be divided into two or more prediction blocks having different sizes. For example, the upper left edge of the diagonal dividing the coding unit is set to meet the left boundary or the upper boundary of the coding unit other than the upper left corner of the coding block, or the upper left edge of the diagonal line is the right boundary of the coding unit other than the lower right corner of the coding block or It can be set to cross the lower boundary. For example, as shown in Figure 24, the diagonal can be partitioned so that it passes through any one of the four corners of the coding unit (hereinafter, a diagonal corner sample) and the center of the coding unit boundary (hereinafter, a diagonal center sample). It is referred to as partitioning, and a prediction block generated by asymmetric diagonal partitioning may be defined as an asymmetric triangular prediction block.

Fig. 24

A flag isSymTriangle_flag indicating whether to generate a triangular prediction block using symmetric partitioning can be signaled. If the isSymTriangle_flag value is 1, a triangular prediction block may be generated using symmetric partitioning, and if the isSymTriangle_flag value is 0, a triangular prediction block may be generated using asymmetric partitioning.

Asym_traingle_index represents an index for specifying an asymmetric triangular prediction block, as shown in Figure 24. A triangular prediction block may be generated using the syntax table shown in Table 8 below.

Table 8

Fig. 25

As shown in Figure 25, a flag triangle_partition_type_flag indicating whether the diagonal center sample is at the upper boundary, lower boundary, right boundary, or lower boundary of the coding unit can be used.

When the triangle_partition_type_flag value is 0, it indicates that the diagonal center sample is at the upper boundary of the coding unit, and when the triangle_partition_type_flag value is 1, it indicates that the diagonal center sample is at the lower boundary of the coding unit. When the triangle_partition_type_flag value is 2, it indicates that the diagonal center sample is on the right boundary of the coding unit, and when the triangle_partition_type_flag value is 3, it indicates that the diagonal center sample is on the left boundary of the coding unit.

A flag left_diag_flag indicating whether the width of the left triangular prediction block is greater than the width of the right triangular prediction block may be signaled. When the value of left_diag_flag is 0, it indicates that the width of the left triangular prediction block is smaller than the width of the right triangular prediction block, and when the value of left_diag_flag is 1, it indicates that the width of the left triangular prediction block is greater than the width of the right triangular prediction block. Partitioning of a triangular prediction block may be derived using triangle_partition_type_flag and left_diag_flag, and a triangular prediction block may be generated using a syntax table as shown in Table 9 below.

Table 9

Different prediction methods may be used in the left triangular prediction block and the right triangular prediction block, and this is referred to as a multi-triangular prediction block coding method. For example, a merge candidate may be used in a left triangular prediction block, and intra prediction may be used in a right triangular prediction block to generate a prediction image. Conversely, a prediction image may be generated using intra prediction in the left triangular prediction block and merge candidate in the right triangular prediction block.

The intra prediction mode used in the multi-triangular prediction block coding method may be limited to the MPM mode. That is, only N MPM modes derived from neighboring blocks may be limited to be used as intra prediction modes of the multi-triangular prediction block coding method.

Alternatively, only the first MPM candidate may be limited to be used as an intra prediction mode of the multi-triangular prediction block method.

When the MPM candidate is derived, the coding unit (hereinafter, referred to as intra-triangular prediction block) in which the intra-mode is used as the multi-triangular prediction block method is set as available, and the neighboring coding unit is not the multi-triangular prediction block method and the intra mode A coding unit in which is used (hereinafter, a standard intra mode) may be set as the peripheral intra mode is not available.

Alternatively, it is possible to limit the use of at least one of the non-directional intra prediction modes (planar mode to DC mode) as the intra prediction mode of the multi-triangular prediction block method.

Using another example, both the left triangular prediction block and the right triangular prediction block may use the intra prediction mode. In this case, the intra prediction modes of the left triangular prediction block (hereinafter, the first intra triangular prediction block) and the right triangular prediction block (hereinafter, the second intra triangular prediction block) may be set differently.

Fig. 26

When performing the multi-triangular prediction coding method, four partitions (hereinafter, multi-triangular prediction partitions) may be allowed as shown in Figure 26 below. It is also possible to use the multi-triangular prediction coding method using only some of the multi-triangular prediction partitions shown in Figure 26. Specifically, for example, among the multiple triangular prediction partitions, only Part_mode 2, Part_mode 3, and Part_mode4 may be used to perform the multi-triangular prediction encoding method.

Multiple triangular prediction partitions may be derived based on prediction modes (intra mode, inter prediction mode, ciip mode) of neighboring blocks.

When all of the prediction modes of neighboring blocks are inter prediction modes, Part_mode 1, Part_mode 3, and Part_mode4 among multiple triangular prediction partitions are used, and in other cases, Part_mode 2, Part_mode 3, and Part_mode4 may be used.

Alternatively, multiple triangular prediction partitions of the current block may be derived based on prediction modes of adjacent neighboring blocks.

Specifically, for example, when the left neighboring block is in the intra mode, it may be set to use only Part_mode2 and Part_mode4 among multiple triangular prediction partitions.

For another example, when the left neighboring block is in the inter prediction mode, it may be set to use only Part_mode1 and Part_mode3 among multiple triangular prediction partitions.

For another example, when an upper neighboring block is an intra mode, it may be set to use only Part_mode2 and Part_mode3 among multiple triangular prediction partitions.

For another example, when the upper neighboring block is in the intra mode, it may be set to use only Part_mode1 and Part_mode4 among multiple triangular prediction partitions.

Image quality deterioration may occur at a boundary portion between the left triangular prediction block and the right triangular prediction block (hereinafter, a diagonal boundary region), or the continuity of image quality may deteriorate around an edge. Deterioration of image quality can be reduced by performing a process such as a smoothing filter or weighted prediction on the diagonal boundary area.

As shown in Figure 27, weighted prediction can be performed at the boundary between the left triangular prediction block and the right triangular prediction block. The sample P_Diag(x,y) in the diagonal boundary region may generate a boundary region prediction sample by weighted prediction of the left triangular prediction block P1 and the right triangular prediction block P2 as shown in Equation (2).

P_Diag(x,y) = w1 * P1(x,y) + (1-w1)* P2 (x,y) (1)

A large weight may be set for the left triangular prediction in the diagonal boundary region belonging to the left triangular prediction block, and a large weight may be set for the right triangular prediction in the diagonal boundary region belonging to the right triangular prediction block.

The size of the diagonal boundary region to which the weighted prediction is applied may be determined based on the size of a coding unit, a difference value of a motion vector of triangular prediction blocks, a POC of a reference picture, and a difference value of prediction samples at a boundary of a triangular prediction block.

Fig. 27

3 Transform and quantization

An image obtained by subtracting the predicted image from the original image is called a residual image.

The residual image may be decomposed into binary frequency components through a two-dimensional transform such as a discrete cosine transform (DCT). Even if high-frequency components are removed from an image, there is a characteristic that significant distortion does not occur visually. If the value corresponding to the high frequency is decreased or set to 0, the compression efficiency can be increased while the visual distortion is not large.

Discrete sine transform (DST) may be used according to the size of the prediction block or the prediction mode. Specifically, for example, when the prediction block/coding block is in the intra prediction mode and the size of the prediction block/coding block is smaller than NxN, the DST transform may be used, and other prediction blocks/coding blocks may be set to use the DCT.

DCT is a process that decomposes (transforms) an image into 2D frequency components using a cosine transform, and the frequency components at that time are expressed as a base image. For example, if DCT transformation is performed in an NxN block, N ² basic pattern components can be obtained. Performing DCT transformation is to obtain the size of each of the basic pattern components included in the original pixel block. The size of each basic pattern component is defined as a DCT coefficient.

In general, a discrete cosine transform (DCT) is mainly used in an image in which many non-zero components are distributed in a low frequency, and a Discrete Sine Transform (DST) may be used in an image in which a large number of high-frequency components are distributed.

DST represents a process of decomposing (converting) an image into 2D frequency components using sin transformation. A 2D image can be decomposed (transformed) into a 2D frequency component using a transformation method other than DCT or DST transformation, and this is defined as 2D image transformation.

Among the residual images, a 2D image transformation may not be performed in a specific block, and this is defined as transform skip. Quantization can be applied after the transform skip. Whether to allow the transform skip may be determined based on at least one of the size or shape of the coding unit. As an example, the transform skip may be limited to be used only in coding units of a specific size or less. For example, it is possible to set the transform skip to be used only in blocks smaller than 32x32. Alternatively, it is possible to restrict the use of transform skip to only a square block. Specifically, for example, transformation skip may be applied in units of 32x32, 16x16, 8x8, and 4x4 blocks.

DCT, DST, or 2D image transformation may be applied to an arbitrary block in the 2D image, and the transformation used in this case is defined as a first transformation. After performing the first transformation, transformation may be performed again in a partial region of the transformation block, and this is defined as a second transformation.

The first transformation may use one of a plurality of transformation cores. Specifically, for example, one of DCT2, DCT8, or DST7 may be selected and used in the transform block. Alternatively, the transform cores of the horizontal direction transformation and the vertical direction transformation of the transformation block may be individually determined. Accordingly, the horizontal direction transformation and the vertical direction transformation may be the same or different. Different transform cores may be used in the horizontal direction transformation and the vertical direction transformation, and this is defined as a multiple transformation core encoding method (Multiple Transform Selection (MTS)).

Block units for performing the first transformation and the second transformation may be set differently from each other. Specifically, for example, after performing the first transform in an 8x8 block of the residual image, the second transform may be performed for each 4x4 sub-block. For another example, after performing the first transform in each 4x4 block, the second transform may be performed in each 8x8 block.

The residual image to which the first transform is applied is defined as a first transform residual image.

DCT, DST, or 2D image transformation may be applied to the first transform residual image, and the transform used in this case is defined as a second transform. The 2D image to which the second transformation is applied is defined as a second transformation residual image.

A sample value in the block after performing the first transformation and/or the second transformation or the sample value in the block after performing the transformation skip is defined as a transform coefficient. Quantization refers to the process of dividing a transform coefficient by a predefined value to reduce the energy of a block. A value defined to apply quantization to a transform coefficient is defined as a quantization parameter.

A predefined quantization parameter may be applied in units of sequences or blocks. Typically, a quantization parameter can be defined with a value between 1 and 64.

After performing transformation and quantization, inverse quantization and inverse transformation may be performed to generate a residual reconstructed image. A first reconstructed image may be generated by adding a predicted image to the residual reconstructed image.

5 In-loop filtering

In-loop filtering is a technique that adaptively performs filtering on a decoded image in order to reduce the loss of information that occurs during quantization and encoding. A deblocking filter, a sample adaptive offset filter (SAO), and an adaptive loop filter (ALF) are examples of in-loop filtering.

A second reconstructed image may be generated by performing at least one of a deblocking filter, a sample adaptive offset (SAO), or an adaptive loop filter (ALF) on the first reconstructed image.

After applying the deblocking filter to the reconstructed image, SAO and ALF may be applied.

In the video encoding process, transformation and quantization are performed in units of blocks. A loss occurs during the quantization process, and a discontinuity occurs at the boundary of the reconstructed image. Discontinuous images appearing at the block boundary are defined as block quality deterioration (blocking artifact).

The deblocking filter is a method of mitigating block quality deterioration (blocking artifact) occurring at a block boundary of a first image and improving encoding performance.

Fig. 22

Block quality deterioration can be alleviated by performing filtering at the block boundary, and as shown in Figure 22 , whether the block is encoded in intra prediction mode, or whether the difference between the absolute values of the motion vectors of the neighboring blocks is greater than a predefined threshold. A block filter strength (BS) value may be determined based on at least one of whether or not reference pictures of neighboring blocks are identical to each other. When the BS value is 0, filtering is not performed, and when the BS value is 1 or 2, filtering may be performed at a block boundary.

Since quantization is performed on a transform coefficient, a ringing artifact occurs at the edge of an object, or a pixel value increases or decreases by a certain value compared to the original because quantization is performed in the frequency domain. The SAO can effectively reduce the ringing phenomenon by adding or subtracting a specific offset in units of blocks in consideration of the pattern of the first reconstructed image. SAO is composed of an edge offset (hereinafter referred to as EO) and a band offset (BO) according to characteristics of a reconstructed image. Edge offset is a method of adding an offset to a current sample differently according to a sample pattern of surrounding pixels. Band offset is to reduce coding errors by adding a certain value to a set of pixels with similar pixel brightness values in a region. By dividing the pixel brightness into 32 uniform bands, pixels having similar brightness values can be made into one set. For example, four adjacent bands can be grouped into one category. One category can be set to use the same offset value.

ALF (Adaptive Loop Filter) is a method of generating a second reconstructed image using one of predefined filters for a first reconstructed image or a reconstructed image that has subjected deblocking filtering to the first reconstructed image as shown in Equation (22). .

(22)

(22)

Fig. 23

In this case, the filter can be selected in units of pictures or units of CTU.

In the Luma component, you can choose one of the 5x5, 7x7 or 9x9 diamond shapes as shown in Figure 23 below. In the chroma component, it may be limited to use only the 5x5 diamond shape.

5 Video coding structure

Fig. 24

Fig. 24 is a diagram showing the video coding structure. The input video signal can be divided into a predetermined coding unit with the smallest rate distortion value. A prediction image may be generated by performing at least one of inter prediction, intra prediction, or complex prediction for each coding unit. In order to perform inter prediction in the encoding step, motion estimation may be performed. A residual image may be generated by differentiating the predicted image from the original image.

A transform coefficient may be generated by performing a first transform and/or a second transform on the residual image, or a transform skip coefficient may be generated by performing a transform skip. Quantization may be performed on a transform coefficient or a transform skip coefficient to generate a quantized transform coefficient or a quantized transform skip coefficient.

A reconstructed image may be generated by performing inverse quantization and inverse transformation on the quantized transform coefficient or the quantized transform skip coefficient. In-loop filtering may be performed on the reconstructed image to generate a filtered reconstructed image. This filtered and reconstructed image becomes the final output image in the encoding step. The filtered reconstructed image is stored as a reference picture buffer and can be used as a reference picture in another picture.

Claims (1)

Video coding and decoding method using multi-triangular prediction block coding method