KR20200132395A - Video signal encoding method and apparatus and video decoding method and apparatus - Google Patents

Abstract

An objective of the present invention is to improve a coding efficiency of a video signal. The present invention provides a method and apparatus for inter prediction encoding using merge offset vector encoding. According to the present invention, a video image is compression-encoded by removing spatio-temporal redundancy and inter-view redundancy, which can be stored in a format suitable for a storage medium or transmitted through a communication line.

Description

Video signal processing method and apparatus TECHNICAL FIELD [Video signal encoding method and apparatus and video decoding method and apparatus]

The present invention relates to a video signal processing method and apparatus.

Video images are compression-encoded by removing spatio-temporal redundancy and inter-view redundancy, which can be stored in a format suitable for a storage medium or transmitted through a communication line.

The present invention is to improve the coding efficiency of a video signal.

In order to solve the above problems, the present invention provides an inter prediction encoding method and apparatus using merge offset vector encoding.

The video signal processing method and apparatus according to the present invention can improve video signal coding efficiency through an inter prediction method using merge offset vector coding.

One Basic coding block structure

A picture is divided into a plurality of basic coding units (Coding Tree Units, hereinafter, CTU) as shown in Figure 1 .

Picture 1

It is configured not to overlap with other CTUs. For example, in the entire sequence, the CTU size can be set to 128x128, and any one of 128x128 to 256x256 can be selected and used in picture units.

A coding block/coding unit (CU) may be generated by hierarchically dividing the CTU. Prediction and transformation may be performed in units of coding units, and become a basic unit that determines a prediction coding mode. The prediction encoding mode represents a plurality of methods of generating a predicted image, such as intra prediction (intra prediction, hereinafter, intra prediction), inter prediction (hereinafter, inter prediction), or combined prediction. Can be mentioned. Specifically, for example, a prediction block may be generated by using at least one of intra prediction, inter prediction, and composite prediction in a coding unit unit. When the reference picture indicates the current picture in the inter prediction mode, a prediction block may be generated in a region within the current picture that has already been decoded. Since a prediction block is generated using a reference picture index and a motion vector, it can be included in inter prediction. Intra prediction is a method of generating a prediction block using information of the current picture, inter prediction is a method of generating a prediction block using information of another picture that has already been decoded, and composite prediction is a method of mixing inter prediction and intra prediction. This is how to use.

The CTU can be divided (partitioned) into a quad tree, binary tree or triple tree as shown in Figure 2 . The divided block can be further divided into a quad tree, binary tree, or triple tree. The method of dividing the current block into 4 square partitions is called quad tree partitioning, the method of dividing the current block into 2 irregular partitions is called binary tree partitioning, and the method of dividing the current block into 3 irregular partitions It is called binary tree partitioning.

Binary partitioning in the vertical direction (SPLIT_BT_VER in Figure 2 ) is called vertical binary tree partitioning, and binary tree partitioning in the horizontal direction (SPLIT_BT_HOR in Figure 2 ) is called horizontal binary tree partitioning.

Triple partitioning in the vertical direction (SPLIT_TT_VER in Figure 2 ) is called vertical triple-tree partitioning, and triple-tree partitioning in the horizontal direction (SPLIT_TT_HOR in Figure 2 ) is called horizontal triple tree partitioning.

The number of additional divisions is called partitioning depth, and the maximum value of the partitioning depth for each sequence or picture/tile set can be set differently, and different partitioning depths according to the partitioning tree type (quad tree/binary tree/triple tree) It can be set to have, and a syntax indicating this can be signaled.

Picture 2

The divided coding block may be further divided by a method such as quad partitioning, binary partitioning, or multi-partitioning to configure a coding unit, or may configure a coding unit without further partitioning.

Figure 3

As shown in Figure 3 , a coding unit can be hierarchically configured for one CTU, and a coding unit can be partitioned using at least one of a coding unit using binary tree partitioning, quad tree partitioning/triple tree partitioning. This method is called multi tree partitioning.

A coding unit generated by dividing an arbitrary coding unit having a partitioning depth of k is referred to as a lower coding unit, and the partitioning depth is k+1. A coding unit having a partitioning depth k including a lower coding unit having a partitioning depth k+1 is referred to as an upper coding unit.

The partitioning type of the current coding unit may be limited according to the partitioning type of the upper coding unit and/or the partitioning type of the coding units surrounding the current coding unit.

Here, the partitioning type represents an indicator indicating which partitioning is used among binary tree partitioning, quad tree partitioning/triple tree partitioning.

2 How to generate predictive image

In video encoding, a prediction image can be generated by a plurality of methods, and a method of generating a prediction image is called a prediction encoding mode.

The prediction encoding mode may include an intra prediction encoding mode, an inter prediction encoding mode, a current picture reference encoding mode, or a combined prediction mode.

The inter prediction coding mode is a prediction coding mode that generates a prediction block (prediction image) of the current block using information of a previous picture, and the intra prediction coding mode is a prediction that generates a prediction block using samples adjacent to the current block. It is called an encoding mode. A prediction block may be generated using an image that has already been reconstructed of the current picture, and this is referred to as a current picture reference mode or an intra block copy mode.

A prediction block may be generated by using at least two or more of an inter prediction encoding mode, an intra prediction encoding mode, or a current picture reference encoding mode, and this is called a combined prediction mode.

2.1 Inter prediction coding method

A method of generating a prediction block (prediction image) of a block in a current picture using information of a previous picture is called an inter prediction encoding mode.

A prediction block may be generated from a specific block in a previous picture of the current block.

In the encoding step, the block with the smallest reconstruction error among the blocks in the previous picture can be selected by searching around a collocated block, and the x-axis difference and y-axis difference between the upper left sample of the current block and the upper left sample of the selected block Is defined as a motion vector, and can be signaled by transmitting it to a bitstream. A block generated through interpolation from a specific block of a reference picture specified by a motion vector is called a motion compensated predictor block.

As shown in Figure 4, the collocated block represents the block of the picture with the same location and size of the current block and the upper left sample. The corresponding picture can be specified from the same syntax as the reference picture reference.

Figure 4

A residual block may be generated by differentiating the motion compensation prediction block from the original image of the current block.

As the motion vector, motion vectors having different pixel precisions may be used in units of sequences, slices, or blocks. For example, the pixel precision of a motion vector in a specific block may be at least one of Octor-pel, Quarter-Pel, Half-Pel, Integer pel, and 4 Integer pel.

As the inter prediction mode, an inter prediction method using translaton motion and an affine inter prediction method using affine motion may be selectively used.

2.1.1 Afine inter prediction coding and decoding method

There are many cases in which the motion of a specific object does not appear linearly in a video. For example, as shown in Figure 5, the camera zoom (Zoom-in), zoom out (Zoom-out), rotation (roation), such as affine transformation, which enables the conversion to any type of affine motion of the image using the object If only translation motion vectors are used for motion, the motion of an object cannot be effectively expressed, and coding performance may be degraded.

Picture 5

Affine motion can be expressed as the following equation (1).

(1)

(One)

Expressing affine motion using a total of six parameters is effective for images with complex motion, but encoding efficiency may be degraded due to the large number of bits used to encode affine motion parameters.

Accordingly, affine motion can be simplified and expressed with four parameters, which is called a four-parameter afine motion model. Equation (2) expresses afine motion with four parameters.

(2)

(2)

The four-parameter affine motion model may include motion vectors at two control points of the current block. The control point may include an upper left corner, an upper right corner, and a lower left corner of the current block. As an example, the motion model with a 4-parameter affine is determined by the motion vector sv0 at the upper left sample (x0,y0) of the coding unit and the motion vector sv1 at the upper right sample (x1,y1) of the coding unit as shown in the left figure of Figure 5 . Can be determined, and sv ₀ and sv ₁ are called affine seed vectors. Hereinafter, it is assumed that an affine seed vector sv ₀ located in the upper left corner is a first affine seed vector, and an affine seed vector sv ₁ located in the upper right corner is a second affine seed vector. In the 4-parameter affine motion model, it is possible to replace one of the 1/2 affine seed vectors with the affine seed vector located in the lower left corner.

Figure 6

The 6-parameter affine motion model is an affine motion model in which the motion vector sv ₂ of the residual control point (e.g., sample (x2,y2) in the lower left corner) is added to the 4-parameter affine motion model as shown in the right figure of Figure 6 . . Hereinafter, assuming that the affine seed vector sv ₀ located at the top left is the first affine seed vector, the affine seed vector sv ₁ located at the top right is the second affine seed vector, and the affine seed vector located at the bottom left Assume that sv ₂ is the third affine seed vector.

Information on the number of parameters for expressing affine motion may be encoded in the bitstream. For example, a flag indicating whether 6 parameters are used and a flag indicating whether 4 parameters are used may be encoded in a tile set, tile, coding unit, or CTU unit. Accordingly, a 4-parameter affine motion model to a 6-parameter affine motion model may be selectively used in units of a tile set, a coding unit, or a CTU.

A motion vector can be derived for each sub-block of the coding unit using the affine seed vector, as shown in Figure 7, and this is called an affine sub-block vector.

Figure 7

The affine sub-block vector can also be derived as in Equation (3) below. Here, the reference sample position (x,y) of the sub-block may be a sample located at a corner of the block (eg, an upper left sample), or a sample (eg, a center sample) in which at least one of the x-axis or y-axis is centered. .

(3)

(3)

Motion compensation may be performed in units of coding units or in units of sub-blocks within the coding unit using the afine sub-block vector, and this is called an afine inter prediction mode. In equation (3), (x ₁ -x ₀ ) may have the same value as the width of the coding unit.

2.1.2 Merge mode coding method

Motion information (motion vector, reference picture index, etc.) of the current coding unit is not encoded, and can be derived from motion information of neighboring blocks. Motion information of any one of the neighboring blocks can be set as motion information of the current coding unit, and this is called a merge mode.

Neighboring blocks used for the remaining mode may be a figure 8 remaining candidate indices from 0 to 4 and the block are adjacent to the coding unit as the (current boundary with the abutting block of the coded unit), as shown in the remaining candidate index from 5 to 26 in Figure 8 It may be a non-adjacent block.

Figure 8

If the distance of the merge candidate to the current block exceeds a predefined threshold, it may be set as not available.

For example, the predefined threshold may be set to the height of the CTU (ctu_height) or ctu_height+N, which is called a merge candidate usable threshold. That is, the y-axis coordinate (y _i ) of the merge candidate and the y-axis coordinate difference (y ₀ ) (i.e., y _i- y ₀ ) of the upper left sample of the current coding unit (hereinafter, the reference sample of the current coding unit) are the merge candidate available threshold If it is larger than the value, the merge candidate may be set as not available. Here, N is a predefined offset value. Specifically, for example, N may be set to 16 or ctu_height may be set.

Figure 9

When there are many merge candidates crossing the CTU boundary, many unusable merge candidates occur, and encoding efficiency may be lowered. Merge candidates existing above the coding unit (hereinafter, upper merge candidates) may be set as small as possible, and merge candidates existing at the left and lower sides of the coding unit (hereinafter, lower left merge candidates) may be set as many as possible.

As shown in Figure 9, the difference between the y-axis coordinate of the current coding unit reference sample and the y-axis coordinate of the upper merge candidate may be set so as not to exceed twice the height of the coding unit.

It may be set so that the difference between the x-axis coordinates of the reference sample in Figure 9 and the current coding unit reference sample and the x-axis coordinates of the lower left merge candidate does not exceed twice the width of the coding unit.

A merge candidate adjacent to the current coding unit is called an adjacent merge candidate, and a merge candidate that is not adjacent to the current coding unit is called a non-adjacent merge candidate.

A flag isAdjacentMergeflag indicating whether a merge candidate of the current coding unit is an adjacent merge candidate may be signaled.

If the isAdjacentMergeflag value is 1, it indicates that motion information of the current coding unit can be derived from an adjacent merge candidate, and if the isAdjacentMergeflag value is 0, it indicates that the motion information of the current coding unit can be derived from a non-adjacent merge candidate.

2.1.3 Merge offset vector coding (Merge with MVD, MMVD) method

The motion vector of the current coding unit may be derived by subtracting or adding an offset to the motion vector generated in the merge mode, which is defined as a merge offset vector encoding method (Merge with MVD, MMVD). A merge offset vector flag (mmvd_merge_flag), which is a flag indicating whether to use a merge offset vector, may be used. If the mmvd_merge_flag value is 1, it indicates that merge offset vector coding is used, and if the mmvd_merge_flag value is 0, it indicates that merge offset vector coding is not used.

If the merge offset vector flag value is 1, the MMVD merge candidate flag (mmvd_cand_flag), the distance index (distance_idx), and the direction index (direction_idx) may be signaled.

According to the value of the merge offset vector flag, the maximum number of merge candidates may be adaptively determined. If the merge offset vector flag has a value of 0, up to M merge candidates are used, while if the merge offset vector flag has a value of 1, N merge candidates less than M may be used.

Alternatively, whether a neighboring block can be used as a merge candidate may be adaptively determined according to the value of the merge offset vector flag. For example, when the merge offset vector flag is 1, a block adjacent to an upper right corner, a lower left corner, or an upper left corner of the current block may be set to be unavailable. Alternatively, when the merge offset vector flag is 1, the temporal merge candidate may be set to be unavailable. Alternatively, when the merge offset vector flag is 1, the pairwise merge candidate or the zero motion vector candidate may not be added to the merge candidate list.

As shown in Table 1 below, mmvd_merge_flag, mmvd_cand_flag, distance index, and direction index can be encoded.

Table 1

The merge offset vector offsetMV can be derived from the distance index and the direction index.

As shown in Fig. 10, the optimal merge offset vector value can be derived by searching the upper and lower and left and right edges of the motion derived from the merge candidate. The x-axis vector and the y-axis vector of the merge offset vector may be set to be smaller than a maximum of 32 integer samples from the merge motion vector as a reference, respectively.

Figure 10

Only when the value of the merge index is smaller than the reference value, the merge offset vector flag may be signaled. Alternatively, only when the merge candidate indicated by the merge index is bidirectional or unidirectional, the merge offset vector flag may be signaled.

The value obtained by subtracting the POC of the current picture from the POC of the reference picture L0 (abbreviation of picture order count, an indicator indicating the output order of the picture) is called the reference picture L0 difference, and the value obtained by subtracting the POC of the current picture from the POC of the reference picture L1. It is called the reference picture L1 difference.

If the sign code of the difference between the reference picture L0 and the difference between the reference picture L1 is different, offsetMV can be generated by multiplying the variable DistFromMergeMV derived from the distance index as shown in the following equations (4) to (7) by a value derived from the direction index.

offsetMVL0[0] = (DistFromMergeMV << 2) * sign[x0][y0][0] (4)

offsetMVL0[1] = (DistFromMergeMV << 2) * sign[x0][y0][1] (5)

offsetMVL1[0] = (DistFromMergeMV << 2) * sign[x0][y0][0] (6)

offsetMVL1[1] = (DistFromMergeMV << 2) * sign[x0][y0][1] (7)

If the sign code of the difference between the reference picture L0 and the difference between the reference picture L1 is the same, offsetMV can be generated by multiplying the variable DistFromMergeMV derived from the distance index as shown in Equations (8) to (11) below by a value derived from the direction index.

offsetMVL0[0] = (DistFromMergeMV << 2) * sign[x0][y0][0] (8)

offsetMVL0[1] = (DistFromMergeMV << 2) * sign[x0][y0][1] (9)

offsetMVL1[0] = -1 * (DistFromMergeMV << 2) * sign[x0][y0][0] (10)

offsetMVL1[1] = -1 * (DistFromMergeMV << 2) * sign[x0][y0][1] (11)

DistFromMergeMV can be derived as shown in Table 2 below.

Table 2

It is also possible to use fewer/more DistFromMergeMV values than those shown in Table 2.

The range of DistFromMergeMV may be determined differently according to the motion vector precision of the current block or merge candidate. For example, when the motion vector precision of the current block or merge candidate is quarter-pel, DistFromMergeMV is selected within the range of {1, 2, 4, 8, 16, 32, 64, 128}, while the current block or merge candidate When the motion vector precision of is integer-pel, DistFromMergeMV can be selected within the range of {4, 8, 16, 32, 64, 128, 256, 512}.

Alternatively, the range of DistFromMergeMV derived from the distance index may be determined differently according to the absolute value of the motion vector. As an example, when the absolute value of the motion vector is greater than a predefined value, the range of DistFromMergeMV of the first set is used, and when the absolute value of the motion vector is less than the predefined value, the DistFromMergeMV of the second set is You can use a range.

Table 3 below shows the variable sign[x][y][0] representing the x-axis offset defined according to the direction index value, and the variable sign[x][y][1] representing the y-axis offset.

Table 3

The x-axis vector and the y-axis vector of the merge offset vector may be set to be smaller than a maximum of 2 integer samples from the merge motion vector as a reference, respectively, and distance_idx may be set to have a total of 4 indices.

Alternatively, a syntax merge offset vector range flag merge_offset_range_flag indicating a search range of a merge offset vector may be signaled in a picture header or a tile set header. For example, if the merge_offset_extend_range_flag value is 1, the x-axis vector and the y-axis vector of the a merge offset vector can be set to be smaller than a maximum of 32 integer samples from the merge motion vector as a reference, respectively, and if the merge_offset_extend_range_flag value is 0, the merge offset The x-axis vector and the y-axis vector of the vector may be set to be smaller than a maximum of 2 integer samples from the merge motion vector as a reference, respectively.

Table 4

As shown in Figure 11 , the optimal merge offset vector value can be derived by searching around the motion up/down, left/right and diagonal directions derived from the merge candidate, and direction_idx can be set to have a total of 8 indices as shown in Table 6.

Figure 11

If the merge offset vector search range is as shown in the left figure of Figure 11 , use Table 6 to represent the x-axis offset variable sign[x][y][0] and the y-axis offset sign[x][y]. If [1] can be represented, and the merge offset vector search range is as shown in the right figure of Figure 11 , then using Table 6, the variable sign[x][y][0] representing the x-axis offset and the y-axis offset Can represent the variable sign[x][y][1].

Table 5

Table 6

Depending on the value of DistFromMergeMV, the range of the search direction can be configured differently. For example, when the value of DistFromMergeMV is less than the threshold value, 8 directions are searched as in the example of Table 5 or Table 6, while when the value of DistFromMergeMV is greater than the threshold value, 4 directions can be searched as in the example of Table 4. have.

Depending on the difference between MVx and MVy, the range of the search direction may be configured differently. As an example, when the difference between MVx and MVy is less than or equal to the threshold value, 8 directions are searched as in the example of Table 5 or Table 6, while the difference between MVx and MVy is greater than the threshold value, as in the example of Table 4 You can search in 4 directions.

The final motion vector can be derived by adding offsetMV to the motion vector mergeMV derived from the merge index as shown in Equations (12) to (15) below.

mvL0[0] = mergeMVL0[0]+ offsetMVL0[0] (12)

mvL0[1] = mergeMVL0[0] + offsetMVL0[0] (13)

mvL10[0] = mergeMVL1[0] + offsetMVL1[0] (14)

mvL1[1] = mergeMVL1[0] + offsetMVL1[0] (15)

Figure 12

In the case of bidirectional prediction, a motion vector may be derived by refining a bidirectional predicted image or a bidirectional motion vector in a decoder as shown in Fig. 12, which is defined as a decoder motion vector refinement (hereinafter referred to as Decoder-side motion vector refinement, DMVR). do. In DMVR, the refine motion information may not be signaled through a bitstream, and the refine motion vector may be set such that the sum of absolute difference (SAD) is the smallest value by searching around the motion vector at the decoder stage.

DMVR may be set not to be used in a coding block to which MMVD is applied.

Alternatively, DMVR may be applied only when distance_idx of MMVD is greater than or equal to a specific threshold value (dmvr application threshold). As an example, the threshold for applying dmvr may be set to 2 integer samples. Alternatively, information for determining the DMVR application threshold may be signaled through a bitstream. The information may be signaled in units of pictures, slices, or tiles.

The dmvr application threshold may be set differently according to the merge_offset_extend_range_flag value. For example, if the merge_offset_extend_range_flag value is 0, the dmvr application threshold may be set to 2 integer samples, and if the merge_offset_extend_range_flag value is 1, the dmvr application threshold may be set to 32 integer samples.

2.1.4 Inter-decoding area merge method

Motion information (motion vector and reference picture index) of a coding unit already encoded by inter prediction in the current picture can be stored in a list having a predefined size, and this is defined as an inter-region motion information list.

Motion information (motion vector and reference picture index) in the inter-region motion information list is called an inter-decoding region merge candidate.

The inter decoding region merge candidate may be used as a merge candidate of the current coding unit, and this method is defined as an inter decoding region merge method.

When the tile group is initialized, the inter-region motion information list is empty, and when a partial region of a picture is encoded/decoded, it can be added to the inter-region motion information list. The initial inter decoding region merge candidate of the inter region motion information list may be signaled through the tile group header.

If the coding unit is to inter-prediction encoding / decoding may update the motion information of the coding unit in the inter-domain motion information list, as shown in Figure 13. If the number of inter-decoding region merge candidates in the inter-region motion information list is the maximum value, the value with the smallest inter-region motion information list index (first, the inter-decoding region merge candidate in the inter-region motion information list) is removed. , The motion vector of the most recently encoded/decoded inter-region may be added as an inter-decoding region merge candidate.

Fig. 13

The motion information mvCand of the decoded coding unit may be updated in the inter-region motion information list HmvpCandList. At this time, if the motion information of the decoded coding unit is the same as any one of the motion information in the inter-region motion information list (when both the motion vector and the reference index are the same), the inter-region motion information list is not updated, or as shown in Figure 15 . The motion information mvCand of the coding unit decoded together may be stored at the end of the inter-region motion information list. At this time, if the index of HmvpCandList that has the same motion information as mvCand is hIdx, HMVPCandList [i] can be set to HVMPCandList[i-1] for all i larger than hIdx as shown in Figure 15 . When sub-block merge candidates are used in the currently decoded coding unit, motion information of the representative sub-block in the coding unit may be stored in the inter-region motion information list.

As an example, the representative sub-block in the coding unit may be set as an upper left sub-block in the coding unit or as a middle sub-block in the coding unit as shown in FIG. 14 .

Figure 14

The sub-block unit merge candidate can be derived as follows.

1. An initial shift vector (shVector) can be derived from a motion vector of a merge candidate block adjacent to the current block.

2. As shown in Equation (16), an initial shift vector may be added to the upper left sample (xSb,ySb) of the subblock in the coding unit to derive the shifted subblock having the upper left sample position (xColSb, yColSb).

(xColSb, yColSb) = (xSb + shVector[0]>> 4,ySb+shVector[1]>> 4) (16)

3. The motion vector of the collocated block corresponding to the center position of the sub-block containing (xColSb, yColSb) can be derived as the motion vector of the sub-block containing the upper left sample (xSb,ySb).

Figure 15

A total of NumHmvp motion information (motion vector and reference picture index) can be stored in the inter-region motion information list, and NumHmvp is defined as the size of the inter-region motion information list.

The size of the inter-region motion information list may use a predefined value. The inter-region motion information list size may be signaled in the tile group header. For example, the size of the inter-region motion information list may be defined as 16, may be defined as 6, or may be defined as 5.

In a coding unit that is inter prediction and has an affine motion vector, it may be limited not to have an inter-region motion information list.

Alternatively, in the case of inter prediction and having an affine motion vector, the affine sub-block vector may be added to the inter-region motion information list. In this case, the position of the sub-block may be set to the upper left or upper right, or the center sub-block.

Alternatively, the average motion vector value of each control point may be added to the inter-region merge candidate list.

When the motion vector MV0 derived by encoding/decoding a specific coding unit is the same as any one of the inter-decoding region merge candidates, MV0 may not be added to the inter-region motion information list. Alternatively, an existing inter-decoding region merge candidate having the same motion vector as MV0 may be deleted, and the index allocated to MV0 may be updated by newly including MV0 in the decoding region merge candidate.

In addition to the inter-region motion information list, an inter-region motion information long term list HmvpLTList may be configured. The inter-region motion information long term list size may be set to be the same as the inter-region motion information list size, or may be set to a different value.

The inter-region motion information long term list may be composed of an inter-decoding region merge candidate initially added to the tile group start position. After the inter-region motion information long term list is configured with all available values, the inter-region motion information list may be configured, or motion information in the inter-region motion information list may be set as motion information of the inter-region motion information long term list.

In this case, the inter-region motion information long-term list configured once may be updated again when no update is performed, a decoded region of the tile group is more than half of the entire tile group, or may be set to be updated every m CTU lines. The inter-region motion information list may be updated whenever the inter-region is decoded, or may be set to be updated in units of CTU lines.

The motion information and partition information or type of the coding unit may be stored in the inter-region motion information list. The inter-decoding region merging method may be performed using only inter-region merge candidates having a similar partition information and shape as the current coding unit.

Alternatively, an inter-region merge candidate list may be individually configured according to the block type. In this case, one of a plurality of inter-region merge candidate lists may be selected and used according to the shape of the current block.

Fig 16

As shown in Figure 16 , it can be composed of an inter-area affine motion information list and an inter-area motion information list. When the decoded coding unit is in affine inter or afine merge mode, the first affine seed vector and the second affine seed vector may be stored in the inter-region afine motion information list HmvpAfCandList. Motion information in the inter-region affine merge candidate list is called an inter-decoding region affine merge candidate.

The merge candidates usable in the current coding unit may be configured as follows, and may have the same search order as the configuration order.

1. Spatial merge candidate (coding block adjacent merge candidate and coding block non-adjacent merge candidate)

2. Temporal merge candidate (merge candidate derived from previous reference picture)

3. Inter-decoding domain merge candidate

4. Inter decoding domain affine merge candidate

5. Zero motion merge candidate

First, the merge candidate list mergeCandList may be composed of a spatial merge candidate and a temporal merge candidate. The number of available spatial merge candidates and temporal merge candidates is defined as the number of available merge candidates (NumMergeCand). When the number of available merge candidates is less than the maximum allowable merge number, an inter-decoding region merge candidate may be added to the merge candidate list mergeCandList.

When adding the inter-region motion information list HmvpCandList to the merge candidate list mergeCandList, it may be checked whether motion information of the inter-decoding region merge candidate in the inter-region motion information list is the same as the motion information of the existing merge list mergeCandList. If the motion information is the same, it is not added to the merge list mergeCandList, and if the motion information is not the same, the inter-decoding region merge candidate may be added to the merge list mergeCandList.

Among the merge candidates of the inter-decoding area, a large index may be added to the merge candidate list mergeCandList, or a small index may be added to the merge candidate list mergeCandList.

The inter decoding region motion candidate can be used as a motion vector predictor (MVP) candidate of the current coding unit, and this method is defined as an inter decoding region motion information prediction method.

An inter-decoding region affine motion candidate can be used as a motion vector predictor (MVP) candidate of the current coding unit, and this method is defined as an inter-decoding region motion information affine prediction method.

Motion information predictor candidates usable in the current coding unit may be configured as follows, and may have the same search order as the configuration order.

1. Spatial motion predictor candidate (same as the coding block adjacent merge candidate and the coding block non-adjacent merge candidate)

2. Temporal merge candidate (motion predictor candidate derived from previous reference picture)

3. Inter-decoding domain merge candidate

4. Inter decoding domain affine merge candidate

5. Zero motion motion predictor candidate

2.2 Intra prediction coding method

In intra prediction, as shown in Fig. 17 , an already coded boundary sample around the current block is used to generate intra prediction, which is called an intra reference sample.

The average value of the intra reference samples is predicted by setting the value of all samples in the prediction block (DC mode), or the vertical direction prediction sample generated by weighted prediction of the horizontal direction prediction sample and the vertical direction reference sample generated by performing horizontal reference weighted prediction After generation, a prediction sample may be generated by weighted prediction of a horizontal direction prediction sample and a vertical direction prediction sample (Planar mode), or intra prediction may be performed using a directional intra prediction mode.

Figure 17

As shown in the left figure of Figure 18 , intra prediction can be performed using 33 directions (a total of 35 intra prediction modes), and 65 directions (a total of 67 intra prediction modes) as shown in the right figure of Figure 18 . May be. When directional intra prediction is used, an intra reference sample (reference reference sample) may be generated in consideration of the direction of the intra prediction mode, and intra prediction may be performed therefrom.

An intra reference sample on the left side of the coding unit is called a left intra reference sample, and an intra reference sample on the upper side of the coding unit is called an upper intra reference sample.

When directional intra prediction is performed, an intra direction parameter (intraPredAng), which is a parameter indicating a prediction direction (or a prediction angle), may be set according to the intra prediction mode as shown in Table 7. Table 7 below is only an example based on a directional intra prediction mode having a value of 2 to 34 when 35 intra prediction modes are used. It goes without saying that the prediction directions (or prediction angles) of the directional intra prediction modes are further subdivided, so that more than 33 directional intra prediction modes may be used.

Fig. 18

Table 7

Figure 19

When intraPredAng is negative (e.g., when the intra prediction mode index is between 11 and 25), as shown in Figure 19 , the left intra reference sample and the upper intra reference sample are configured in 1D according to the angle of the intra prediction mode. It can be reconstructed as a one-dimensional reference sample (Ref_1D).

Figure 20

When the intra prediction mode index is between 11 and 18, a one-dimensional reference sample can be generated in a counterclockwise direction from an intra reference sample located on the upper right side of the current block to an intra reference sample located on the lower left side as shown in Figure 20 .

In other modes, a one-dimensional reference sample may be generated using only the upper side intra reference sample or the left side intra reference sample.

Fig. 21

When the intra prediction mode index is between 19 and 25, a one-dimensional reference sample can be generated in a clockwise direction from an intra reference sample located at the lower left side of the current block to an intra reference sample located at the right side of the current block as shown in Figure 21 .

A weight-related parameter ifact applied to at least one reference sample determined based on the reference sample determination index iIdx and iIdx may be derived as in Equations (18) to (19) below. iIdx and i _fact are variably determined according to the slope of the directional intra prediction mode, and a reference sample specified by iIdx may correspond to an integer pel.

iIdx = (y+1) * P _ang /32 (18)

i _fact = [(y+1) * P _ang ] & 31 (19)

The prediction image may be derived by specifying at least one one-dimensional reference sample for each prediction sample. For example, a position of a one-dimensional reference sample that can be used to generate a prediction sample may be specified in consideration of a slope value of a directional intra prediction mode. Each prediction sample may have a different directional intra prediction mode. A plurality of intra prediction modes may be used for one prediction block. The plurality of intra prediction modes may be represented by a combination of a plurality of non-directional intra prediction modes, a combination of one non-directional intra prediction mode and at least one directional intra prediction mode, or a plurality of directional intra prediction modes. It can also be expressed as a combination of modes. A different intra prediction mode may be applied for each predetermined sample group in one prediction block. A predetermined sample group may consist of at least one sample. The number of sample groups may be variably determined according to the size/number of samples of the current prediction block, or may be a fixed number pre-set in the encoder/decoder independently of the size/number of samples of the prediction block.

Specifically, for example, the position of the one-dimensional reference sample may be specified using the reference sample determination index iIdx.

When the slope of the intra prediction mode cannot be expressed with only one one-dimensional reference sample according to the slope of the intra prediction mode, a first prediction image may be generated by interpolating adjacent one-dimensional reference samples as shown in Equation (20). When the angular line according to the slope/angle of the intra prediction mode does not pass the reference sample located in the integer pel, the first prediction image may be generated by interpolating the reference sample adjacent to the left/right or above/below the corresponding angular line. . The filter coefficient of the interpolation filter used at this time may be determined based on i _fact . For example, the filter coefficient of the interpolation filter may be derived based on a distance between a fractional pel located on the angular line and a reference sample located on the integer pel

P(x,y) = ((32-i _fact )/32)* Ref_1D(x+iIdx+1) + (ifact/32)* Ref_1D(x+iIdx+2) (20)

When the slope of the intra prediction mode can be expressed with only one one-dimensional reference sample (when i _fact is 0), the first prediction image can be generated as shown in Equation (21) below.

P(x,y) = Ref_1D(x+iIdx+1) (21)

2.3 Wide-angle intra prediction coding method

The prediction angle of the directional intra prediction mode may be set between 45 and -135 as shown in Figure 22 .

Figure 22

When the intra prediction mode is performed in the amorphous coding unit, a disadvantage of predicting a current sample from an intra reference sample far from the current sample instead of an intra reference sample close to the current sample may occur due to a predefined prediction angle.

For example, as shown in the left figure of Fig. 23 , in a coding unit whose width is larger than the height of the coding unit (hereinafter, horizontal coding unit), intra prediction is performed at a far distance L instead of a close sample T. I can. As another example, as shown in the right figure of Fig. 2 and 3, in a coding unit whose height is larger than the width of the coding unit (hereinafter, the vertical coding unit), instead of the close sample L, from the far sample T Intra prediction can be performed.

Figure 23

In the amorphous coding unit, intra prediction may be performed at a prediction angle wider than a predefined prediction angle, which is referred to as a wide angle intra prediction mode.

내지

의 예측 각도를 가질 수 있으며, 기존 인트라 예측 모드에서 사용된 각도를 벗어나는 예측 각도를 와이드 앵글 각도라고 부른다. The wide angle intra prediction mode

To

It can have a prediction angle of, and a prediction angle that deviates from the angle used in the existing intra prediction mode is called a wide angle angle.

In the left figure of Fig. 23 , sample A in the horizontal direction coding unit can be predicted from the intra reference sample T using the wide-angle intra prediction mode.

In the right figure of Figure 23 , sample A in the vertical direction coding unit can be predicted from the intra reference sample L using the wide angle intra prediction mode.

N+M intra prediction modes may be defined by adding M wide angle angles to N existing intra prediction modes. Specifically, as shown in Table 8, a total of 95 intra prediction modes may be defined by adding 28 wide angle angles to 67 intra modes.

The intra prediction mode that the current block can use may be determined according to the shape of the current block. For example, 65 directional intra prediction modes may be selected from among 95 directional intra prediction modes based on at least one of a size of a current block, an aspect ratio (eg, a ratio of width and height), and a reference line index.

Table 8

The intra prediction mode angle shown in Table 4 may be adaptively determined based on at least one of a shape of a current block and a reference line index. For example, the intraPredAngle of Mode 15 may be set to have a larger value when the current block is a square than when the current block is a non-square. Alternatively, the intraPredAngle of Mode 75 may be set to have a larger value when a non-adjacent reference line is selected than when an adjacent reference line is selected.

When using the wide-angle intra prediction mode, the length of the upper intra reference sample can be set to 2W+1 and the length of the left intra reference sample can be set to 2H+1, as shown in Figure 24 .

Figure 24

In the case of using wide-angle intra prediction, when the intra prediction mode of the wide-angle intra prediction mode is encoded, the number of intra prediction modes increases, and thus encoding efficiency may be lowered. The wide-angle intra prediction mode can be encoded by replacing the existing intra prediction mode that is not used in the wide-angle intra prediction mode, and the replaced prediction mode is called a wide-angle replacement mode.

Specifically, for example, when 35 intra predictions are used as shown in Figure 25 , wide-angle intra prediction mode 35 can be coded with intra prediction mode 2, which is a wide-angle replacement mode, and wide-angle intra prediction mode 36 is a wide-angle replacement. It can be encoded in intra prediction mode 3, which is a mode.

Figure 25

The replacement mode and number may be differently set according to the shape of the coding block or the ratio of the height to the width of the coding block. Specifically, for example, Table 9 shows intra prediction modes used according to the ratio of the width and height of a coding block.

Table 9

3 Transform and quantization

An image obtained by subtracting the predicted image from the original image is called a residual image.

The residual image can be decomposed into binary frequency components through a two-dimensional transform such as a discrete cosine transform (DCT). Even if high-frequency components are removed from an image, there is a characteristic that significant distortion does not occur visually. If the value corresponding to the high frequency is decreased or set to 0, the compression efficiency can be increased while the visual distortion is not large.

Discrete sine transform (DST) may be used according to the size of the prediction block or the prediction mode. Specifically, for example, when the prediction block/coding block is in an intra prediction mode and the size of the prediction block/coding block is smaller than NxN, the DST transform may be used, and other prediction blocks/coding blocks may be set to use DCT.

DCT is a process that decomposes (transforms) an image into 2D frequency components using a cosine transform, and the frequency components at that time are expressed as a base image. For example, if DCT transformation is performed in an NxN block, N ² basic pattern components can be obtained. Performing DCT transformation is to obtain the size of each of the basic pattern components included in the original pixel block. The size of each basic pattern component is called a DCT coefficient.

In general, a discrete cosine transform (DCT) is mainly used for an image in which many non-zero components are distributed in a low frequency, and a Discrete Sine Transform (DST) may be used for an image in which a large number of high-frequency components are distributed.

DST represents a process of decomposing (converting) an image into 2D frequency components using sin transformation. A 2D image can be decomposed (transformed) into a 2D frequency component using a transformation method other than DCT or DST transformation, and this is called a 2D image transformation.

A 2D image transformation may not be performed in a specific block among residual images, and this is called transform skip. Quantization can be applied after the transform skip.

DCT, DST, or 2D image transformation may be applied to an arbitrary block in the 2D image, and the transformation used at this time is called a first transformation. After performing the first transformation, transformation may be performed again in a partial region of the transformation block, which is called a second transformation.

The first transformation may use one of a plurality of transformation cores. Specifically, for example, one of DCT2, DCT8, or DST7 may be selected and used in the transform block, or different transform cores may be used in the horizontal direction transformation and the vertical direction transformation of the transformation block. It is called Multiple Transform Selection (MTS). Here, DCT2, DCT8 or DST7 is called a conversion core.

The residual image to which the first transform has been applied is called a first transform residual image.

DCT, DST, or 2D image transformation may be applied to the first transform residual image, and the transform used in this case is called a second transform. The 2D image to which the second transformation is applied is called a second transformation residual image.

A sample value in a block after performing the first transform and/or the second transform is called a transform coefficient. Quantization refers to the process of dividing a transform coefficient by a predefined value to reduce the energy of a block. A value defined to apply quantization to a transform coefficient is called a quantization parameter.

A predefined quantization parameter may be applied in units of sequences or blocks. Typically, a quantization parameter can be defined with a value between 1 and 51.

After performing transformation and quantization, inverse quantization and inverse transformation may be performed to generate a residual reconstructed image. A first reconstructed image may be generated by adding a prediction image to the residual reconstructed image.

4 In-loop filtering

In-loop filtering is a technique that adaptively performs filtering on a decoded image in order to reduce the loss of information that occurs during quantization and encoding. A deblocking filter, a sample adaptive offset filter (SAO), and an adaptive loop filter (ALF) are examples of in-loop filtering.

A second reconstructed image may be generated by performing at least one of a deblocking filter, a sample adaptive offset (SAO), or an adaptive loop filter (ALF) on the first reconstructed image.

After applying the deblocking filter to the reconstructed image, SAO and ALF may be applied.

In the video encoding process, transformation and quantization are performed in units of blocks. A loss occurs during the quantization process, and a discontinuity occurs at the boundary of the reconstructed image. The discontinuous image appearing at the block boundary is called block quality deterioration (blocking artifact).

The deblocking filter is a method of mitigating block quality deterioration (blocking artifact) occurring at a block boundary of a first image and improving encoding performance.

Fig. 26

Block quality deterioration can be alleviated by performing filtering at the block boundary, and as shown in Figure 26 , whether the block is encoded in intra prediction mode, or whether the difference between the absolute values of the motion vectors of neighboring blocks is greater than a predetermined threshold. A block filter strength (BS) value may be determined based on at least one of whether or not reference pictures of neighboring blocks are identical to each other. When the BS value is 0, filtering is not performed, and when the BS value is 1 or 2, filtering may be performed at a block boundary.

Since quantization is performed on a transform coefficient, a ringing artifact occurs at the edge of an object, or a pixel value increases or decreases by a certain value compared to the original because quantization is performed in the frequency domain. The SAO can effectively reduce the ringing phenomenon by adding or subtracting a specific offset in units of blocks in consideration of the pattern of the first reconstructed image. SAO is composed of an edge offset (hereinafter referred to as EO) and a band offset (BO) according to characteristics of a reconstructed image. The edge offset is a method of adding an offset to the current sample differently according to the surrounding pixel sample pattern. Band offset is to reduce coding errors by adding a certain value to a set of pixels with similar pixel brightness values in a region. By dividing the pixel brightness into 32 uniform bands, pixels having similar brightness values can be made into one set. For example, four adjacent bands can be grouped into one category. One category can be set to use the same offset value.

ALF (Adaptive Loop Filter) is a method of generating a second reconstructed image using one of predefined filters for a first reconstructed image or a reconstructed image that has subjected deblocking filtering to the first reconstructed image as shown in Equation (22). .

(22)

(22)

Fig. 27

In this case, the filter can be selected in units of pictures or units of CTU.

In the Luma component, you can choose any of the 5x5, 7x7 or 9x9 diamond shapes as shown in Figure 27 below. In the chroma component, it may be limited to use only the 5x5 diamond shape.

5 Video coding structure

Fig. 28

Figure 28 is a diagram showing the video coding structure. The input video signal can be divided into a predetermined coding unit with the smallest rate distortion value. A prediction image may be generated by performing at least one of inter prediction, intra prediction, or complex prediction for each coding unit. In order to perform inter prediction in the encoding step, motion estimation may be performed. A residual image may be generated by differentiating the predicted image from the original image.

A transform coefficient may be generated by performing a first transform and/or a second transform on the residual image. Quantization may be performed on the transform coefficient to generate a quantized transform coefficient.

A reconstructed image may be generated by performing inverse quantization and inverse transformation on the quantized transform coefficient. In-loop filtering may be performed on the reconstructed image to generate a filtered reconstructed image. This filtered reconstructed image becomes the final output image in the encoding step. The filtered reconstructed image is stored as a reference picture buffer and can be used as a reference picture in another picture.

Claims (1)

Inter prediction coding method using merge offset vector coding