KR20220074233A - Apparatus and method for motion prediction of multi view or multi layer video - Google Patents

Abstract

The present invention relates to encoding of a multi-view or multi-layer video, from motion prediction in the conventional spatial and temporal picture in the encoding step of a multi-view or multi-layer video to motion information of a corresponding picture of another view or other component. There is an advantage in that the efficiency of motion prediction encoding can be improved by using the .

Description

Motion information prediction apparatus and method in multi-view screen or inter-layer prediction {APPARATUS AND METHOD FOR MOTION PREDICTION OF MULTI VIEW OR MULTI LAYER VIDEO}

The present invention relates to encoding/decoding of a multi-view or multi-layered image, and more particularly, to a motion information prediction technique using features of a multi-view or multi-layered image.

Recently, as the size of the display increases and the resolution increases, the amount of video data is also increasing. In addition, with the advent of new video formats pursuing immersive video, which cannot be covered by conventional 2D video, such as 3D video or multi-view video, the amount of transmitted video data is increasing tremendously.

If such a large amount of video data is transmitted or stored as it is, it becomes difficult to bear the transmission cost or storage cost. Therefore, large-capacity video compression encoding technology is essential.

There are several video compression encoding techniques. Among them, motion prediction is a method of predicting which block of the previous frame is moved from the block of the current frame by dividing the screen into several blocks. Since it is only necessary to predict the motion of a block and transmit only a motion vector, the amount of transmitted data can be reduced.

Conventional video codecs such as high efficiency video coding (HEV) or versatile video coding (VVC) use a merge mode or advanced motion vector prediction (AMVP) mode as a motion vector prediction method.

This is a candidate for motion vector prediction using neighboring blocks existing in the same frame as the current prediction block (PU: Prediction Unit, or CU: Coding Unit) and candidates derived from blocks at the same position as the current prediction block in the reference frame. Blocks are selected and movement is predicted from them.

However, in the case of 3D video or multi-view video, motion prediction can be performed more accurately by using frames of other views or other components in addition to spatial and temporal candidates as in the prior art, but there is no such attempt yet.

The inventors of the present invention have been researching and trying to solve the motion prediction problem of the multi-view or multi-layer video of the prior art. The present invention has been completed after much effort to complete an apparatus and method for motion prediction using frames of different views or different components of a multi-view or multi-layer video.

It is an object of the present invention to improve the efficiency of video encoding by using motion information of different views or other components in a multi-view or multi-layer video.

On the other hand, other objects not specified in the present invention will be additionally considered within the range that can be easily inferred from the following detailed description and effects thereof.

Motion prediction apparatus according to the present invention,

a motion vector candidate generator for generating a motion vector prediction candidate list using blocks of other components or other viewpoints for which encoding has been completed; and a motion vector prediction unit that generates motion vector prediction information of a current block by using the motion vector prediction candidate list.

The motion vector candidate generator may generate the motion vector prediction candidate list by using a block located at the same position as the current block among blocks of other components or at another time point at which the encoding is completed.

The motion vector candidate generation unit is a block located at the same position as the current block among blocks of the lower left block, the left block, the upper right block, the upper block, the upper left block, another time point at which the encoding is completed, or other component blocks of the current block. , and generating the motion vector prediction candidate list in the order of blocks located at the same position as the current block in the reference frame.

The motion vector candidate generator may determine if a block located at the same position as the current block among blocks at another time point or other component blocks at which the encoding is completed does not have motion information, an upper block and upper left block of a block located at the same position as the current block. , and generating the motion vector prediction candidate list by checking the motion information in the order of the left blocks.

The motion vector candidate generator generates one of a lower left block and a left block of the current block and one of an upper right block, an upper block, and an upper left block of the current block as the motion vector prediction candidate list, wherein the motion vector prediction is performed If motion information of blocks added to the candidate list is duplicated, the duplicated block is removed, and a block located at the same position as the current block among blocks at another time point or other components at which the encoding is completed is added to the motion vector prediction candidate list characterized in that

If the motion vector candidate generator is the same as the motion information of the block that is located at the same position as the current block, the motion vector is composed of 0 instead of the block located at the same position as the current block. A vector candidate is added to the motion vector prediction candidate list.

The motion vector candidate generation unit includes subblocks of a coding unit at a position corresponding to the current coding unit in a picture of a different view or other component corresponding to a current coding unit (CU) of the current picture to which the current block belongs. It is characterized in that the motion vector prediction candidate list is generated by combining.

The motion vector candidate generator determines the size of the subblock as a bit operation result value of shifting the size of the current coding unit to the right by two bits.

The motion vector candidate generator may fix the size of the subblock to 8x8 when the bit operation result value is less than 8x8.

The motion vector candidate generating unit converts the block to the CPMV (CPMV) of the current CU when the block located at the same position as the current block among blocks of other components or at another time point at which the encoding is completed is encoded using Affine-based motion prediction technology. It is characterized in that it is used as a Control Point Motion Vector).

The motion vector candidate generator unit uses, as a Control Point Motion Vector (CPMV), a block at the same position as the current block among blocks of another time point or other component blocks for which the encoding is completed, and performs a combination affine merge ( Affine Merge) candidates are generated.

A motion prediction method according to another embodiment of the present invention,

generating a motion vector prediction candidate list by using blocks of other views or other components from which encoding has been completed; and generating motion vector prediction information of the current block by using the motion vector prediction candidate list.

According to the present invention, there is an effect that the efficiency of motion prediction encoding can be increased by proceeding using motion information of a picture corresponding to another view or other component.

On the other hand, even if it is an effect not explicitly mentioned herein, it is added that the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as described in the specification of the present invention.

1 is a schematic structural diagram of a motion prediction apparatus according to a preferred embodiment of the present invention.
2 shows an example of neighboring blocks used for motion prediction.
3 shows an example of a subblock-based merge mode.
4 shows an example of an affine merge mode.
5 shows an example of a picture configuration of a 3D image.
6 shows an example of referring to an image from a different viewpoint in a 3D image.
7 shows an example of configuring a motion vector prediction candidate list in a 3D image.
8 shows an example of a subblock-based merge mode.
9 is a schematic flowchart of a motion prediction method according to a preferred embodiment of the present invention.
10 is a schematic structural diagram of a motion prediction apparatus of an image decoder according to another preferred embodiment of the present invention.
11 is a schematic flowchart of a motion prediction method of an image decoder according to another preferred embodiment of the present invention.
※ It is revealed that the accompanying drawings are exemplified as a reference for understanding the technical idea of the present invention, and the scope of the present invention is not limited thereby

Hereinafter, the configuration of the present invention guided by various embodiments of the present invention and effects resulting from the configuration will be described with reference to the drawings. In the description of the present invention, if it is determined that related known functions are obvious to those skilled in the art and may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. The above term may be used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a 'first component' may be termed a 'second component', and similarly, a 'second component' may also be termed a 'first component'. can Also, the singular expression includes the plural expression unless the context clearly dictates otherwise. Unless otherwise defined, terms used in the embodiments of the present invention may be interpreted as meanings commonly known to those of ordinary skill in the art.

Hereinafter, the configuration of the present invention guided by various embodiments of the present invention and effects resulting from the configuration will be described with reference to the drawings.

Hereinafter, a video codec for image encoding/decoding will be described based on ISO/IEC MPEG-I versatile video coding (VVC), but the present invention is not limited thereto and may be applied to all other video codecs.

1 is a schematic structural diagram of a motion prediction apparatus according to a preferred embodiment of the present invention.

The motion prediction apparatus 100 according to the present invention is included in the image encoder 1 and includes a motion vector candidate generator 110 and a motion vector predictor 120 . Also, the motion prediction apparatus 100 includes one or more processors and a memory (not shown). The processor is used to generate motion information by predicting motion, and program codes and data for driving the processor may be stored in the memory.

The motion vector candidate generator 110 generates a list of candidate blocks for motion prediction by using information surrounding the current block that is being encoded.

2 shows an example of neighboring blocks used for motion prediction.

In the merge mode, a candidate list for using motion information is constructed. In FIG. 2 , blocks spatially and temporally related to the current prediction unit 10 (PU) are used as candidate blocks for motion prediction.

A0, A1, B0, B1, and B2 are spatial candidates that exist in the same frame as the current prediction unit 10 and are encoded neighboring blocks.

C0 and C1 are blocks in a frame different from the current prediction unit 10, that is, a reference frame, and are temporal candidates derived from a prediction unit at the same position as the current prediction unit.

The motion vector candidate generator 110 constructs a candidate list from these spatial and temporal candidates. The merge mode motion vector prediction candidate list may be generated in the order of A0 → A1 → B0 → B1 → B2 → C0 → C1. The merge mode list may have a maximum of 5 candidates, and blocks having the same information may be excluded from the candidates. As blocks having the same information are excluded, if all five candidates are not filled, the remaining list may be filled with virtual candidates. The virtual candidates may be history-based motion vector prediction (HMVP), pairwise average motion vector prediction (PAMVP), a zero candidate, and the like.

As another method of generating motion candidates, there may be a subblock-based merging mode.

3 shows an example of a subblock-based merge mode.

In the subblock-based merge mode, a prediction signal is generated by dividing a coding unit (CU) into sub-coding units and inducing different motion vectors in units of sub-coding units. Prediction accuracy may be improved by performing prediction by inducing a motion vector in units of small sub-coding units without dividing blocks.

The motion vector candidate generator 110 first searches for a corresponding block of the current coding unit. The corresponding block of the current coding unit 22 is a block existing in the corresponding reference picture 30 (Collocated Picture) transmitted in the slice header, and is an A1 block located at the lower left of the current coding unit. When the reference picture of the A1 block is the same as the corresponding reference picture 30, the motion vector of the A1' block that is the same as the A1 position is used. find the location.

Next, a motion vector in units of sub coding units (24, 8x8 size fixed units) of the current coding unit 22 is derived from the corresponding block. To this end, the current coding unit 22 is divided into 8x8 sub coding units 24, and motion information corresponding to the center position of each sub coding unit 34 in the corresponding block is used to move each sub coding unit of the current block. The vector is derived in units of 8x8.

Finally, a motion vector derived from a corresponding block is set to a sub-block temporal MVP (SbTMVP), which is a sub-coding unit unit motion vector, through a scaling process so that it can be applied to a reference picture.

The subblock-based merge mode prediction process generates a prediction block of the current coding unit using the 8x8 block-based motion information derived through this process. In this case, only a motion compensation process is required without a motion estimation process as in the general merge mode. In addition, motion vector difference (MVD) is not transmitted to the decoder.

The motion vector candidate generator 110 may generate a motion vector prediction candidate list in AMVP mode. AVMP mode may have a maximum of two candidates.

In FIG. 2 , one candidate is selected from blocks A0 and A1 and one candidate is selected from blocks B0, B1, and B2, and duplicate motion information is removed from the list through redundancy check. If the candidate list is not completely filled by the redundant information removal, temporal candidates C0, C1 and HMVP candidates may be added.

The motion vector candidate generator 110 may also generate a motion vector prediction candidate list using the affine merge mode. The affine-based merge mode candidates are composed of inherited affine merge candidates and combination affine merge candidates.

4 shows an example of an affine merge mode.

The inherited affine merge candidate is a current coding unit derived from a Control Point Motion Vector (CPMV) of the current coding unit 40 when the upper or left neighboring coding unit of the current coding unit 40 is encoded by applying an affine-based motion vector prediction technique. (40) means CPMV.

The combinatorial affine merge candidate means the CPMV of the current coding unit 40 derived by combining neighboring motion vectors. In FIG. 4, the first available motion vector is set as CPMV1 by checking the existence of a motion vector in the order B2→→, and then CPMV2 and A1→ by checking the motion vector in the order of CPMV3. , set TMVP (C0 or C1) to CPMV4, respectively.

The set CPMVs are combined as follows and used as a combination affine merge candidate. The following is a 6-parameter affine merge candidate composed of three CPMVs.

{CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4},

{CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}

The following is a 4-parameter affine merge candidate composed of two CPMVs.

{CPMV1, CPMV2}, {CPMV1, CPMV3}

Whether the encoder uses the 6-parameter affine merge candidate or the 4-parameter information is indicated in sps_affine_type_flag in the sequence parameter set (SPS) and transmitted to the decoder.

The motion vector candidate generator 110 may extend the motion vector prediction information of the two-dimensional screen as described above to a three-dimensional image or a multi-view image.

5 shows an example of a picture configuration of a 3D image.

An image V0 of a viewpoint called a base view or an independent view is independently coded without depending on other views. Since the images V1 and V2 of different views can be encoded with reference to images of other views using a 3D encoder, they are called dependent views.

A current picture consists of several access units, and the access units are divided into a texture and a depth map.

The motion vector candidate generator 110 may generate motion vector prediction information of the current coding unit by using image information of different viewpoints in the 3D image.

6 illustrates an example of referring to an image from a different viewpoint in a 3D image.

The motion vector candidate generator 110 refers to blocks of a texture picture 60 corresponding to the depth picture 50 to generate a motion vector candidate for a depth picture 50 constituting a 3D image. can do.

When the motion vector candidate generator 110 predicts motion information from a dependent view of the 3D image, the motion information of other components may be used at another view C for which encoding is completed.

As in the previous examples, a candidate list may be constructed in a 3D image by using the normal merge mode and the AMVP mode.

7 shows an example of configuring a motion vector prediction candidate list in a 3D image.

FIG. 7B shows a coding unit of a dependent view that is currently being encoded, and FIG. 7A shows another view or other components in which encoding for referring to motion information is completed.

The motion vector candidate generator 110 is configured to block blocks ( X0, X1, X2, X3) may be additionally used to generate a list of candidates for job position information.

The motion vector candidate generator 110 may generate the candidate list in the normal merge mode. In the previous example of using spatial and temporal candidates, by additionally using candidate blocks at different viewpoints, the general merge mode candidate list is A0 → A1 → B0 → B1 → B2 → X0 → C0 (C1 when → C0 is not available) can be checked in the order of Candidates that do not overlap with each other among them are inserted into the general merge candidate list.

If X0 does not have motion information, that is, if it is encoded in the intra-picture mode, blocks that are close to the X0 block in a counterclockwise direction are checked. In other words, the motion information of the usable blocks is checked by checking the motion information in the order of X1 → X2 → X3.

The motion vector candidate generator 110 may generate the candidate list in AMVP mode. In (b) of FIG. 7 , two candidates are generated as a candidate list by selecting one candidate from A0→ and one candidate from B0→→. At this time, duplicate motion information is removed from the list through redundancy check, and if the candidate list is not fully filled by the redundancy check, in FIG. added to the list Here too, of course, duplicate information is checked and removed. Even after this process, if the number of existing candidates is less than two, zero candidates are added to the candidate set.

The motion vector candidate generator 110 may generate the candidate list in the subblock-based merge mode. 8 shows an example of a subblock-based merge mode.

In a multi-view screen including a 3D image and inter-layer prediction, information on other views or other component pictures corresponding to the current picture may be checked in addition to the corresponding reference picture of the current picture.

Therefore, the motion vector candidate generation unit 110 of the present invention may be used instead of a temporal reference picture when there is a picture of a different viewpoint or a different component (FIG. 8(b)) corresponding to the current picture ((a) of FIG. 8). A picture of a viewpoint or other component may be used.

The corresponding block of the current coding unit 91 is a block corresponding to the block 92 located at (posX, posY) in the upper left corner of the current coding unit 91 in another view corresponding to the current picture or a picture of another component. (94).

In this case, the size of the sub coding unit is not fixed to the 8x8 size, but may be variably determined according to the size of the current coding unit 91 as shown in the following equation.

size_of_sub_CU = size_of_current_CU >> 2

size_of_sub_CU is the size of the sub coding unit and size_of_current_CU is the size of the current coding unit 91 . When size_of_sub_CU becomes smaller than 8x8 by bit operation, the size may be fixed to 8x8.

The motion vector candidate generator 110 may also perform affine-based merge motion prediction by referring to pictures of other viewpoints or other components.

When blocks X0, X1, X2, and X3 in FIG. 7A are coded using the affine-based motion prediction technique, the blocks can be used as the CPMV of the current coding unit.

For example, if X0 is set to CPMV5, the combination of 6-parameter affine merge candidates can be configured as follows.

{CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV5 },

{CPMV1, CPMV3, CPMV5 }, {CPMV2, CPMV3, CPMV5 }

The motion vector prediction unit 120 predicts the motion of the current coding unit by using the motion vector prediction candidate list generated by the motion vector candidate generation unit 110 in this way to perform encoding.

9 is a schematic flowchart of a motion prediction method according to another preferred embodiment of the present invention.

The motion prediction method of the present invention may use information of other views or other components in a multi-view or multi-layer image such as a 3D image.

First, blocks that have been coded among blocks of different views or other components may be generated as a motion vector prediction candidate list (S110). A general merge mode and an AMVP mode may be used to generate the motion candidate list. It is also possible to use a subblock-based merge mode. The detailed method of each mode is as described above.

After a motion vector prediction candidate list is generated, encoding for motion prediction is performed using the candidate list (S120).

10 is a schematic structural diagram of a motion prediction apparatus in an image decoder according to another preferred embodiment of the present invention.

The motion prediction apparatus 2 of the present invention includes a motion vector candidate generating unit 210 and a motion vector applying unit 220 .

The motion vector candidate generator 210 generates motion vectors of blocks of a decoded block from another viewpoint or blocks of other components as a motion vector prediction candidate list. The general merge mode and AMVP mode may be used to generate the motion vector candidate list. It is also possible to use a subblock-based merge mode. The detailed method of each mode is as described above.

The motion vector application unit 220 applies a motion vector corresponding to a motion vector candidate index received from an encoder among the previously generated motion vector candidate list to decoding of a 3D image.

11 is a flowchart of a motion prediction method in an image decoder according to another preferred embodiment of the present invention.

First, a motion vector prediction candidate list is generated by using motion vectors of blocks of other components or blocks at another time point after decoding has been completed ( S210 ). The general merge mode and AMVP mode may be used to generate the motion vector candidate list. It is also possible to use a subblock-based merge mode.

When the motion vector prediction candidate list is completed, decoding is performed using the motion vector corresponding to the motion vector candidate index received from the encoder to apply the motion vector of the current block (S220).

In this way, by using blocks of different views or different components in a multi-view or multi-layer image for motion prediction, the present invention has the effect of increasing the efficiency of motion prediction encoding.

The protection scope of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the protection scope of the present invention cannot be limited due to obvious changes or substitutions in the technical field to which the present invention pertains.

Claims (14)

In the motion prediction apparatus for encoding a multi-view or multi-layer image:
a motion vector candidate generator for generating a motion vector prediction candidate list using blocks of other components or other views from which encoding has been completed; and
and a motion vector prediction unit generating motion vector prediction information of a current block by using the motion vector prediction candidate list.
The method of claim 1,
and the motion vector candidate generator generates the motion vector prediction candidate list by using a block located at the same position as the current block among blocks of another component or another time point at which the encoding is completed.
According to claim 1,
The motion vector candidate generation unit is a block located at the same position as the current block among blocks of the lower left block, the left block, the upper right block, the upper block, the upper left block, another time point at which the encoding is completed, or other component blocks of the current block. , characterized in that the motion vector prediction candidate list is generated in the order of blocks located at the same position as the current block in a reference frame.
4. The method of claim 3,
The motion vector candidate generator may determine if a block located at the same position as the current block among blocks at another time point or other component blocks at which the encoding is completed does not have motion information, an upper block and upper left block of a block located at the same position as the current block. , and generating the motion vector prediction candidate list by checking motion information in the order of the left blocks.
According to claim 1,
The motion vector candidate generator generates one of a lower left block and a left block of the current block and one of an upper right block, an upper block, and an upper left block of the current block as the motion vector prediction candidate list, wherein the motion vector prediction is performed If motion information of blocks added to the candidate list is duplicated, the duplicated block is removed, and a block located at the same position as the current block among blocks at another time point or other components at which the encoding is completed is added to the motion vector prediction candidate list A motion prediction device, characterized in that.
6. The method of claim 5,
If the motion vector candidate generator is the same as the motion information of the block that is located at the same position as the current block, the motion vector is composed of 0 instead of the block located at the same position as the current block. A vector candidate is added to the motion vector prediction candidate list, the motion prediction apparatus.
According to claim 1,
The motion vector candidate generation unit includes subblocks of a coding unit at a position corresponding to the current coding unit in a picture of a different view or other component corresponding to a current coding unit (CU) of the current picture to which the current block belongs. A motion prediction apparatus, characterized in that the motion vector prediction candidate list is generated by combining.
8. The method of claim 7,
The motion vector candidate generator determines the size of the subblock as a bit operation result value of shifting the size of the current coding unit to the right by two bits.
9. The method of claim 8,
Wherein the motion vector candidate generator sets the size of the subblock to 8x8 when the bit operation result value is less than 8x8.
According to claim 1,
The motion vector candidate generating unit converts the block to the CPMV (CPMV) of the current CU when the block located at the same position as the current block among blocks of other components or at another time point at which the encoding is completed is encoded using Affine-based motion prediction technology. Control Point Motion Vector), characterized in that used as a motion prediction device.
The method of claim 1,
The motion vector candidate generator unit uses, as a Control Point Motion Vector (CPMV), a block at the same position as the current block among blocks of another time point or other component blocks for which the encoding is completed, and performs a combination affine merge ( Affine Merge), a motion prediction apparatus, characterized in that the candidate is generated.
A motion prediction method for encoding a multi-view or multi-layer image performed by a controller including one or more processors and a memory, the method comprising:
generating a motion vector prediction candidate list using blocks of other views or other components from which encoding has been completed; and
and generating motion vector prediction information of a current block by using the motion vector prediction candidate list.
In the motion prediction apparatus for decoding a multi-view or multi-layer image:
a motion vector candidate generator for generating a motion vector prediction candidate list by using another time point or blocks of other components from which decoding is completed; and
and a motion vector application unit that performs decoding by applying a motion vector corresponding to an index transmitted from an encoder to a motion vector of a current block using the motion vector prediction candidate list.
A motion prediction method for decoding a multi-view or multi-layer image performed by a controller including one or more processors and a memory, the method comprising:
generating a motion vector prediction candidate list by using blocks of another time point or other components from which decoding is completed; and
and performing decoding by applying a motion vector corresponding to an index transmitted from an encoder to a motion vector of a current block using the motion vector prediction candidate list.

Images