WO2020038357A1

WO2020038357A1 - Fusion candidate list construction method, device and encoding/decoding method and device

Info

Publication number: WO2020038357A1
Application number: PCT/CN2019/101559
Authority: WO
Inventors: 杨海涛; 赵寅; 徐巍炜
Original assignee: 华为技术有限公司
Priority date: 2018-08-20
Filing date: 2019-08-20
Publication date: 2020-02-27

Abstract

The present invention provides a fusion candidate list construction method for inter-frame prediction, the method comprises: when neighbouring blocks can be obtained and the motion information of the neighbouring blocks is different from that of neighbouring blocks at specific positions, the motion information of the neighbouring blocks, used as a space fusion candidate of the current block, is added into a fusion candidate list of the current block; and when non-neighbouring blocks can be obtained, the inter prediction mode of at least one non-neighbouring block of the non-neighbouring blocks is judged as a predetermined inter-frame prediction mode, a non-neighbouring space fusion candidate of the at least one non-neighbouring block is not added into the fusion candidate list of the current block.

Description

Method and device for constructing fusion candidate list, and method and device for editing / decomposing

This application is required to be submitted to the State Intellectual Property Office of China on August 20, 2018, with the application number 201810948854.0, and the application name is "Construction Method, Device and Compilation / Solution Method and Device for Fusion Candidate List", in August 2018 The priority of a Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on 31st, application number 201811014507.7, and application name is "fusion candidate list construction method, device and method for editing / decomposing method and device", the entire contents of which are incorporated by reference in In this application.

Technical field

The embodiments of the present application relate to the field of video encoding, and more specifically, to a method for constructing a fusion candidate list in an inter prediction mode in a video encoding and decoding process, and an applied encoding and decoding method.

Background technique

Video encoding (video encoding and decoding) is widely used in digital video applications, such as broadcast digital TV, video transmission on the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems And security applications for camcorders.

With the development of the block-based hybrid video coding method in the H.261 standard in 1990, new video coding technologies and tools have been developed and formed the basis for the subsequent evolution of video coding standards. Video coding standards include MPEG-1 video, MPEG-2 video, ITU-T H.262 / MPEG-2, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10 Advanced Video Coding (Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC) ... and extensions to such standards, such as scalability and / or three-dimensional (3D) extensions. As video creation and sharing becomes more widespread, video traffic becomes the biggest burden on communication networks and data storage. Therefore, one of the goals of most video coding standards is to reduce the bit rate compared to previous standards without reducing the subjective quality of the picture. Even though the latest High Efficiency Video Coding (HEVC) can compress video about twice as much as AVC without reducing the subjective quality of the picture, new technologies are still needed to further compress the video relative to HEVC, a new generation of video coding Technology VVC (Versatile Video Coding) technology is in the process of being developed. Its goal is to further increase the compression ratio by about 50% compared to HEVC without reducing the subjective quality of the picture.

The HEVC / H.265 video coding standard or the VVC / H.266 video coding standard being developed, a frame of image will be divided into non-overlapping coding tree units (CTU), and the CTU size can be set to 64 × 64 or 128 × 128 size. Taking a 64 × 64 size CTU as an example, it contains 64 columns of pixels, each column contains 64 pixels, and each pixel contains a luminance component and / or a chrominance component. A CTU is divided into one or more coding units (Coding Units, CU). A CU contains basic coding information, including information such as prediction mode and transform coefficients. The decoding end can perform corresponding prediction processing, inverse quantization, inverse transform, reconstruction, and filtering on the CU according to the encoded information to generate a reconstructed image corresponding to the CU. A CU corresponds to a predicted image and a residual image, and the predicted image and the residual image are added to obtain a reconstructed image. The predicted image is generated by intra prediction or inter prediction, and the residual image is generated by inverse quantization and inverse transform processing of the transform coefficients.

Inter prediction is a prediction technique based on motion compensation. The main processing process is to determine the motion information of the current block, and obtain the reference image block from the reference frame of the current block according to the motion information to generate a predicted image of the current block. Among them, A current block refers to an image block undergoing encoding / decoding processing, and the current block may be a luma block or a chroma block in a coding unit. The motion information includes an inter prediction direction, a reference frame, a motion vector, and the like. The inter prediction direction indicates which prediction direction is used by the current block in forward prediction, backward prediction, or bidirectional prediction. The motion vector indicates the reference frame used for The displacement vector of the reference image block of the current block relative to the current block is predicted, so one motion vector corresponds to one reference frame. Inter-prediction of an image block can generate a predicted image using only one motion vector and using pixels in a reference frame, which is called unidirectional prediction; it can also use two motion vectors to combine using pixels in two reference frames. Generating a prediction image is called bidirectional prediction. That is, an image block can usually contain one or two motion vectors. For some multi-hypothesis inter prediction techniques, an image block may contain more than two motion vectors.

Inter-prediction indicates the reference frame (reference frame) by reference frame index (ref_idx), and indicates the reference block (reference block) of the current block (reference block) in the reference frame by the motion vector (motion vector (MV)). Position offset of the current block in the frame. A MV is a two-dimensional vector that contains horizontal and vertical displacement components. An MV corresponds to two frames, each frame has a picture order number (POC), which is used to indicate the order in which the images are displayed. On the number, so a MV also corresponds to a POC difference. The POC difference has a linear relationship with the time interval. The scaling of the motion vector usually uses a scaling method based on the POC difference value to convert a motion vector between one pair of images into a motion vector between another pair of images.

There are three commonly used inter prediction modes:

1) AMVP mode (Advanced Motion Vector Prediction): Identifies the inter prediction direction (forward, backward or bidirectional), reference frame index, motion vector prediction value index (motion vector) used by the current block in the code stream. predictor index (MVP index), motion vector residual value (MVD); reference frame queue used by inter prediction direction determination, reference frame index to determine the reference frame pointed to by the current block MV, and the motion vector prediction value The index indicates that one MVP in the MVP list is used as the predicted value of the current block MV, and one MVP and one MVD are added to obtain one MV.

2) merge / skip mode: identify the merge index in the code stream, select a merge candidate from the merge candidate list according to the merge index, and the motion vector information of the current block (including the prediction direction and reference frame , Motion vector) is determined by this merge candidate. The main difference between merge mode and skip mode is that merge mode implies that the current block has residual information, that is, the motion vector obtained from the motion candidate list is used as the motion vector prediction value of the current block, and the motion vector of the current block is determined by The predicted value of the motion vector is added to the residual value of the motion vector, and the residual of the motion vector is obtained by decoding the code stream; and the skip mode implies that the current block has no residual information (or the residual is 0), that is, from The motion vector obtained in the motion vector list is directly used as the motion vector of the current block for inter prediction; the two methods of deriving motion information are the same.

3) Affine transformation mode: obtain the motion vector of each sub-block in the current block from the two or three control point motion vectors by affine transformation.

In the HEVC standard, the fusion candidate can be the motion information of the image block adjacent to the current block, which is called spatial fusion candidate. It can also be the motion of the image block corresponding to the current block in another encoded image. Information is called temporal fusion candidate. In addition, the fusion candidate can also be a bi-predictive merge candidate composed of the forward motion information of one fusion candidate and the backward motion information of another fusion candidate, or a motion vector force A zero motion vector fusion candidate that is a zero vector (zero motion vector merge candidate).

Technical proposals received at the latest conference developed by the latest video standard VVC, such as JVET-K0286, JVET-K0198, and JVET-K0339. These proposals propose adding non-adjacent spatial fusion candidates to the fusion candidate list. (Non-adjacent spatial merge) method, which increases the number of merge candidates for merge / skip mode and improves prediction efficiency.

The construction method of the fusion candidate list in the JVET-K0286 proposal is as follows:

Step 1: Adding a spatial fusion candidate adjacent to the spatial domain of the current block to the fusion block candidate list of the current block, the method is the same as the method in HEVC. The spatial fusion candidates of adjacent spatial domains are the motion information of blocks A, B, C, D, and E in FIG. 1, and their order of joining the fusion candidate list is A, B, C, D, and E. The blocks A, B, C, ..., I in Fig. 1 are all 4x4 blocks.

Step 2: The temporal fusion candidate of the current block is added to the fusion candidate list of the current block, and the method is the same as the method in HEVC.

Step 3: Add non-adjacent spatial merge candidates that are not adjacent to the current block spatial domain to the fusion block candidate list of the current block. Non-adjacent space fusion candidates are the motion information of A1, B1, C1, D1, E1, A2, B2, C2, D2, E2, F, G, H, I blocks in Figure 1; non-adjacent space fusion candidates The order of joining the fusion candidate list is A1, B1, C1, D1, E1, F, G, H, I, A2, B2, C2, D2, E2. As a simplified solution, the JVET-K0286 proposal also proposes that non-adjacent spatial fusion candidates only include motion information of A1, B1, C1, D1, E1, A2, B2, C2, D2, and E2 blocks.

Step 4: Add other types of fusion candidates, such as bi-predictive merge candidates and zero motion vector merge candidates.

It should be noted that the length of the fusion candidate list is a preset fixed value M, such as 6 or 8 or 10. When the number of fusion candidates in the fusion candidate list reaches the preset value M, the The construction is completed, and the remaining fusion candidates are no longer added to the fusion candidate list. In addition, if a fusion candidate is the same as an existing fusion candidate in the fusion candidate list, the fusion candidate may not be added to the fusion candidate list to avoid duplicate fusion candidates appearing in the fusion candidate list, causing information redundancy.

JVET-K0339 uses more non-adjacent spatial fusion candidates, as shown in Figure 10. In FIG. 10, blocks 1 to 5 are conventional spatial fusion candidates, and blocks 6 to 48 are non-adjacent spatial fusion candidates.

When decoding, if the current block uses the skip / merge mode, the merge index is parsed from the code stream, and the fusion candidate corresponding to the merge index is selected from the fusion candidate list constructed by the above method to obtain the motion information of the current block. Perform motion compensation according to the motion information of the current block to obtain a predicted image of the current block. The prediction image of the current block is added to the residual image of the current block to obtain a reconstructed image of the current block, thereby completing decoding of the current block.

For the multiple fusions proposed in the above solution, the number of non-adjacent spatial fusion candidates is large, and the length of the fusion candidate list is small. The fusion candidate is added to the fusion candidate list during the process of adding the fusion candidate to the fusion candidate list. Filtering can improve the diversity of each fusion candidate in the fusion candidate list and their similarity with the current block motion information, thereby improving the prediction efficiency.

The method proposed in the above solution adds more non-adjacent spatial merge candidates to the fusion candidate list to determine whether to add non-believing spatial fusion candidates to the fusion candidate list. The judgment rule is based on whether a non-adjacent spatial fusion candidate overlaps with a fusion candidate list that already exists in the fusion candidate list to determine whether to add this non-adjacent spatial fusion candidate to the fusion candidate list. This judgment method needs to be more complicated Comparison logic. If the maximum number of candidates allowed by the fusion candidate list is large, this comparison will take a long time to complete, thereby affecting the performance of the codec. The motion information of non-adjacent spatial fusion candidates using skip / merge mode is easily duplicated or similar to the motion information of adjacent spatial fusion candidates, which is not conducive to improving the diversity of motion information in the fusion candidate list. In addition, if the non-adjacent spatial fusion candidate uses the affine model prediction mode and the current block uses the translation model prediction mode, these two regions have different motion types, so this non-adjacent spatial fusion candidate and the current block's The relevance of actual motion information is low and should not be added to the fusion candidate list.

Summary of the Invention

In view of this, the present invention provides an efficient method and device for establishing a fusion candidate list, a codec method applying the method, and a codec device applying the device.

A first aspect of the present invention provides a method for constructing a fusion candidate list for performing inter prediction, which includes, when the neighboring block is available and the motion information of the neighboring block is related to the neighboring block at a specific position. When the motion information is different, the motion information of the neighboring block is added as a spatial fusion candidate of the current block to the fusion candidate list of the current block; and based on the motion prediction mode of the non-adjacent block, Motion information of neighboring blocks of a specific prediction mode is excluded from the spatial fusion candidate list of the current block, that is, at least one of the non-neighboring blocks is judged to be non-phase when the non-neighboring block is available When the inter prediction mode of a neighboring block is not a predetermined inter prediction mode, and the motion information of the at least one non-adjacent block is the same as the motion information of an adjacent block or a non-adjacent block at a specific position, the at least one The motion information of a non-adjacent block is added to the fusion candidate list as a non-adjacent spatial fusion candidate for the current block. When the non-adjacent block is available, it is determined that the non-adjacent block is in the non-adjacent block. of At least one non-inter-prediction mode of adjacent blocks is predetermined inter prediction mode, not the at least one non-neighboring non-adjacent blocks added by the spatial candidate fusion integrated candidate list of the current block.

Based on this method, the present invention can remove fusion candidates that are not highly correlated with the current block's motion mode in the fusion candidate list, thereby having the opportunity to add more accurate and rich fusion candidates to the fusion candidate list, thereby improving Coding efficiency.

According to the first embodiment of the first aspect of the present invention, the method may further add a fusion candidate in a time domain to the fusion candidate list, and the fusion candidate in the time domain may preferably join a non-adjacent The spatial fusion candidate was previously added to the fusion candidate list. Specifically, in the second selection rule based on the preset, the motion information of non-adjacent blocks that are not adjacent to the current block spatial domain is used as all Before the non-adjacent spatial fusion candidate of the current block is added to the fusion candidate list of the current block, motion information of a neighboring block in a lower right corner of a co-located block in a reference frame of the current block is used as the current block. Time-domain fusion candidates are added to the fusion candidate list, wherein the position of the co-located block in the reference frame is the same as the position of the current block in the current position, or in the lower right corner of the co-located block. When the motion information of the neighboring block is not available, the motion information of the center point of the co-located block is added to the fusion candidate list as a time-domain fusion candidate of the current block.

With reference to the first aspect of the present invention or the first embodiment of the first aspect, in the second embodiment of the present invention, if the number of fusion candidates in the fusion candidate list constructed by the method does not reach a predetermined In the case of the number, in order to make full use of the coding space provided by the fusion candidate list, other fusion candidates may be added to the fusion candidate list. Preferably, in the second selection rule based on the preset, , After adding motion information of a non-adjacent block that is not adjacent to the current block spatial domain as a non-adjacent spatial fusion candidate of the current block to the fusion candidate list of the current block, and the fusion candidate The number of the fusion candidates in the list does not reach a predetermined value, and the method further includes: adding the bidirectional prediction fusion candidate to the fusion candidate list when the current block belongs to a bi-predictive slice. Or, after adding the bidirectional prediction fusion candidate to the fusion candidate list, the fusion candidate in the fusion candidate list If the number of participants does not reach the predetermined value, a zero motion vector fusion candidate is added to the fusion candidate list; or when the current block belongs to a uni-predictive slice, the motion is zero. A vector fusion candidate is added to the fusion candidate list.

With reference to the first aspect of the present invention or any implementation manner of the first aspect, in a third embodiment of the present invention, the predetermined inter prediction mode is a skip / merge mode, a simulation Affine transform prediction mode, or the inter prediction mode is a skip / merge mode and an affine (Affine) transform prediction mode. The above-mentioned specific prediction mode is not limited to the above-mentioned modes, and other inter-prediction modes, if the optical flow field mode can also be used as the above-mentioned predetermined inter-prediction mode.

In order to reduce the complexity, when there are multiple non-adjacent blocks that need to be determined, in combination with the first aspect of the present invention or any one of the embodiments of the first aspect, in a fourth embodiment of the present invention, The non-adjacent block performs an inter prediction mode determination, that is, only a non-adjacent block at a specific position is judged whether its inter prediction mode is the same as a predetermined inter prediction mode. The specific position may specifically be that at least one non-adjacent block of the non-adjacent blocks is A2, B2, C2, D2, E2 blocks, and the upper-left corner coordinate of the current block is P0 = ( x0, y0), the width and height of the current block are W and H, respectively, then the upper left corner coordinate of the A2 block is PA2 = (x0-4-2 * Sx, y0 + H-4); the B2 The upper left corner coordinate of the block is PB2 = (x0 + W-4, y0-4-2 * Sy); the upper left corner coordinate of the C2 block is PC2 = (x0 + W, y0-4-2 * Sy); The upper left corner coordinate of the D2 block is PD2 = (x0-4-2 * Sx, y0 + H); the upper left corner coordinate of the E2 block is PE2 = (x0-4-2 * Sx, y0-4-2 * Sy); or, at least one non-adjacent block among the non-adjacent blocks is A1, B1, C2, D2, and E2 blocks, and the upper-left corner coordinate of the current block is P0 = (x0, y0), where The width and height of the current block are W and H respectively, then the upper left corner coordinate of the A1 block is PA1 = (x0-4-Sx, y0 + H-4); the upper left corner coordinate of the B1 block is PB1 = ( x0 + W-4, y0-4-Sy); the upper-left corner coordinate of the C2 block is PC2 = (x0 + W, y0-4-2 * Sy); the upper-left corner coordinate of the D2 block is PD2 = ( x0-4-2 * Sx, y0 + H); the upper-left corner coordinate of the E2 block is PE2 = (x0-4-2 * Sx, y0-4-2 * Sy); or , At least one non-adjacent block among the non-adjacent blocks is A1, B1, C1, D1, E1 blocks, and the upper-left corner coordinate of the current block is P0 = (x0, y0), and the width of the current block The height and height are W and H, respectively, then the upper left corner coordinate of the A1 block is PA1 = (x0-4-Sx, y0 + H-4); the upper left corner coordinate of the B1 block is PB1 = (x0 + W- 4, y0-4-Sy); the upper-left corner coordinate of the C1 block is PC2 = (x0 + W, y0-4-Sy); the upper-left corner coordinate of the D1 block is PD1 = (x0-4-Sx, y0 + H); the coordinate of the upper left corner of the E1 block is PE1 = (x0-4-Sx, y0-4-Sy).

A second aspect of the present invention provides a decoding method, and applying the fusion candidate list constructed according to the first aspect of the present invention or the method in any of the first aspect of the present invention to the decoding process of the current block. The decoding process includes: parsing a bitstream to obtain a fusion candidate index; obtaining a corresponding fusion candidate from the fusion candidate list according to the fusion candidate index, and using the fusion candidate as the current block's Motion information; performing inter prediction on the current block according to the motion information of the current block to obtain a predicted image of the current block; obtaining a residual image of the current block; combining the predicted image of the current block and the current block And adding the residual images to obtain a reconstructed image of the current block. The decoding method provided by the second aspect of the present invention can effectively improve the decoding efficiency by using the fusion candidate list construction method provided by the first aspect of the present invention.

A third aspect of the present invention provides a coding method for constructing a fusion candidate list for inter prediction, which is constructed according to the first aspect of the present invention or the method in any embodiment of the first aspect of the present invention. The fusion candidate list is applied to the encoding process of the current block, and the encoding process includes: performing an RDO-based fusion evaluation on the current block based on each fusion candidate in the fusion candidate list (Merge estimation) and use the fusion candidate with the lowest rate-distortion cost value as the motion information of the current block; encode the current block to form encoded data based on the motion information of the current block; and use the fusion candidate with the least rate-distortion cost value A position index of the person in the fusion candidate list is appended to the encoded data.

The decoding method provided by the third aspect of the present invention can effectively improve decoding efficiency by using the fusion candidate list construction method provided by the first aspect of the present invention.

In addition, the present invention also provides an encoding device and an encoding device corresponding to the second aspect of the present invention, and a decoding device corresponding to the third aspect of the present invention to a decoding device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of an example of a video encoding system for implementing an embodiment of the present invention; FIG.

FIG. 1B is a block diagram illustrating an example of a video encoding system including any one or both of the encoder 20 of FIG. 2 and the decoder 30 of FIG. 3; FIG.

2 is a block diagram showing an example structure of a video encoder for implementing an embodiment of the present invention;

3 is a block diagram showing an example structure of a video decoder for implementing an embodiment of the present invention;

4 is a block diagram of an example of an encoding device or a decoding device;

5 is a block diagram of an example of another encoding device or decoding device;

6 is a flowchart of an example operation of a video encoder according to an embodiment to implement the method for constructing a fusion candidate list of the present invention;

7 is a flowchart of a decoding method for a video encoder to decode based on a fusion candidate list constructed in FIG. 6 according to another embodiment;

8 is a flowchart of an encoding method for a video encoder to encode based on a fusion candidate list constructed in FIG. 6 according to another embodiment;

9 is a schematic diagram of the distribution of spatially adjacent and non-adjacent blocks;

FIG. 10 is another schematic diagram of the distribution of spatially adjacent and non-adjacent blocks.

FIG. 11 is a schematic structural diagram of a device for constructing a fusion candidate athlete list according to an embodiment of the present application; FIG.

12 is a schematic structural diagram of a decoding device according to an embodiment of the present application;

13 is a schematic structural diagram of an encoding apparatus according to an embodiment of the present application;

14 is a schematic diagram of a device for implementing the methods in FIGS. 6 to 8 according to an embodiment of the present application;

If there is no specific note about the same reference symbols below, the same reference symbols refer to the same or at least functionally equivalent features.

detailed description

Hereinafter, specific embodiments of the present invention and application examples using the specific embodiments of the present invention will be described with reference to the drawings.

It should be understood that the embodiments of the present invention are not limited to the examples listed herein, they may be used in other aspects, and may include structural or logical changes not shown in the drawings.

For example, it should be understood that the disclosure in connection with the described method may be equally applicable to a corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units such as functional units to perform the described one or more method steps (e.g., one unit performs one or more steps Or multiple units, each of which performs one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific device is described based on one or more units such as functional units, the corresponding method may include a step to perform the functionality of one or more units (e.g., a step performs one or more units Functionality, or multiple steps, where each performs the functionality of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Further, it should be understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with each other, unless explicitly stated otherwise.

Video coding generally refers to processing a sequence of pictures that form a video or a video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video encoding used in this application (or this disclosure) means video encoding or video decoding. Video encoding is performed on the source side and typically involves processing (e.g., by compressing) the original video picture to reduce the amount of data required to represent the video picture (thus storing and / or transmitting more efficiently). Video decoding is performed on the destination side and usually involves inverse processing relative to the encoder to reconstruct the video picture. The video pictures (or collectively referred to as pictures, which will be explained below) referred to in the embodiments should be understood as "encoding" or "decoding" related to a video sequence. The combination of the encoding part and the decoding part is also called codec (encoding and decoding).

In the case of lossless video coding, the original video picture can be reconstructed, that is, the reconstructed video picture has the same quality as the original video picture (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed by, for example, quantization to reduce the amount of data required to represent the video picture, and the decoder side cannot completely reconstruct the video picture, that is, the quality of the reconstructed video picture is compared to the original video picture The quality is lower or worse.

Several video coding standards for H.261 belong to "lossy hybrid video codecs" (ie, combining spatial and temporal prediction in the sample domain with 2D transform coding for applying quantization in the transform domain). Each picture of a video sequence is usually partitioned into a set of non-overlapping blocks, usually encoded at the block level. In other words, the encoder side usually processes at the block (video block) level, that is, encodes the video. For example, the prediction block is generated by spatial (intra-picture) prediction and temporal (inter-picture) prediction. Processed blocks) minus the prediction block to obtain the residual block, transform the residual block in the transform domain and quantize the residual block to reduce the amount of data to be transmitted (compressed), and the decoder side will perform inverse processing relative to the encoder Partially applied to an encoded or compressed block to reconstruct the current block for representation. In addition, the encoder duplicates the decoder processing loop so that the encoder and decoder generate the same predictions (such as intra prediction and inter prediction) and / or reconstruction for processing, that is, encoding subsequent blocks.

As used herein, the term "block" may be part of a picture or frame. For the convenience of description, reference is made to VVC or the joint video coding team (Joint Collaboration Team) of the ITU-T Video Coding Experts Group (VCEG) and ISO / IEC Motion Picture Experts Group (MPEG). High-Efficiency Video Coding (HEVC) developed by Video Coding (JCT-VC) describes embodiments of the present invention. Those skilled in the art understand that the embodiments of the present invention are not limited to HEVC or VVC. Can refer to CU, PU and TU. In HEVC, a CTU is split into multiple CUs by using a quad-tree structure represented as a coding tree. A decision is made at the CU level whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode a picture region. Each CU can be further split into one, two or four PUs according to the PU split type. The same prediction process is applied within a PU, and related information is transmitted to the decoder on the basis of the PU. After obtaining a residual block by applying a prediction process based on a PU split type, a CU may be partitioned into a transform unit (TU) according to other quad-tree structures similar to a coding tree for a CU. In the latest development of video compression technology, quad-tree and binary-tree (QTBT) split frames are used to split coded blocks. In the QTBT block structure, the CU may be a square or rectangular shape. In VVC, a coding tree unit (CTU) is first divided by a quad tree structure. The quad leaf nodes are further partitioned by a binary tree structure. Binary leaf nodes are called coding units (CUs), and the segments are used for prediction and transformation processing without any other segmentation. This means that the CU, PU, and TU have the same block size in the QTBT coded block structure. At the same time, it is also proposed to use multiple partitions with QTBT block structures, such as triple-tree partitioning.

The following describes embodiments of the encoder 20, the decoder 30, and the encoding system 10 based on FIGS. 1A, 1B to 3 (before describing the embodiment of the present invention in more detail based on FIG. 6).

FIG. 1A is a conceptual or schematic block diagram of an exemplary encoding system 10, for example, a video encoding system 10 that can utilize the technology of the present application (the present disclosure). The encoder 20 (e.g., video encoder 20) and decoder 30 (e.g., video decoder 30) of video encoding system 10 represent that they can be used to perform fusion candidate list construction according to various examples described in this application, And a device example based on the technology of encoding and decoding based on the fusion selection list. As shown in FIG. 1A, the encoding system 10 includes a source device 12 for providing the encoded data 13, such as the encoded picture 13, to a destination device 14 that decodes the encoded data 13, for example.

The source device 12 includes an encoder 20, and in addition, optionally, may include a picture source 16, such as a pre-processing unit 18 of a picture pre-processing unit 18, and a communication interface or communication unit 22.

The picture source 16 may include or may be any kind of picture capture device for, for example, capturing real-world pictures, and / or any kind of pictures or comments (for screen content encoding, some text on the screen is also considered to be a picture to be encoded Or a part of an image) generating device, for example, a computer graphics processor for generating computer animated pictures, or for obtaining and / or providing real world pictures, computer animated pictures (for example, screen content, virtual reality (VR) ) Pictures) of any type of device, and / or any combination thereof (eg, augmented reality (AR) pictures).

A (digital) picture is or can be regarded as a two-dimensional array or matrix of sampling points with luminance values. The sampling points in the array may also be called pixels (short for picture element) or pixels. The number of sampling points of the array or picture in the horizontal and vertical directions (or axes) defines the size and / or resolution of the picture. In order to represent color, three color components are usually used, that is, a picture can be represented as or contain three sampling arrays. In RBG format or color space, pictures include corresponding red, green, and blue sampling arrays. However, in video coding, each pixel is usually represented in a luma / chroma format or color space, for example, YCbCr, including the luma component indicated by Y (sometimes also indicated by L) and the two chroma indicated by Cb and Cr Weight. Luma (abbreviated as luma) component Y represents luminance or gray level intensity (for example, both are the same in a grayscale picture), while two chroma (abbreviated as chroma) components Cb and Cr represent chroma or color information components . Correspondingly, a picture in the YCbCr format includes a luminance sampling array of luminance sampling values (Y), and two chrominance sampling arrays of chrominance values (Cb and Cr). Pictures in RGB format can be converted or converted to YCbCr format and vice versa. This process is also called color conversion or conversion. If the picture is black and white, the picture can include only an array of luminance samples.

The picture source 16 (e.g., the video source 16) may be, for example, a camera for capturing pictures, such as a memory of a picture memory, including or storing a previously captured or generated picture, and / or any category (internal Or external) interface. The camera may be, for example, an integrated camera that is local or integrated in the source device, and the memory may be local or, for example, an integrated memory that is integrated in the source device. The interface may be, for example, an external interface for receiving pictures from an external video source. The external video source is, for example, an external picture capture device, such as a camera, an external memory, or an external picture generation device. The external picture generation device is, for example, an external computer graphics processor, a computer. Or server. The interface may be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, an optical interface. The interface for acquiring the picture data 17 may be the same interface as the communication interface 22 or a part of the communication interface 22.

Different from the processing performed by the pre-processing unit 18 and the pre-processing unit 18, a picture or picture data 17 (for example, video data 16) may also be referred to as an original picture or original picture data 17.

The pre-processing unit 18 is configured to receive (original) picture data 17 and perform pre-processing on the picture data 17 to obtain pre-processed pictures 19 or pre-processed picture data 19. For example, the pre-processing performed by the pre-processing unit 18 may include trimming, color format conversion (for example, conversion from RGB to YCbCr), color correction, or denoising. It is understood that the pre-processing unit 18 may be an optional component.

An encoder 20 (eg, video encoder 20) is used to receive the pre-processed picture data 19 and provide the encoded picture data 21 (details will be further described below, for example, based on FIG. 2 or FIG. 4). In one example, the encoder 20 may be configured to add motion information of a neighboring block adjacent to the current block spatial domain as a spatial fusion candidate of the current block to the fusion of the current block based on a preset first selection rule. In the candidate list, the preset first selection rule includes, when the neighboring block is available and the motion information of the neighboring block is different from the motion information of the neighboring block at a specific position, then The motion information of the neighboring block is added as a spatial fusion candidate of the current block to the fusion candidate list of the current block; and based on a preset second selection rule, it will not be adjacent to the current block spatial domain The motion information of the non-adjacent block is added as a non-adjacent spatial fusion candidate of the current block to the fusion candidate list, wherein the current block has one or more of the non-adjacent spatial domains that are not adjacent Block, the preset second selection rule includes: when the non-adjacent block is available, determining that an inter prediction mode of at least one non-adjacent block in the non-adjacent block is not a predetermined inter-frame Advance When the motion information of the at least one non-adjacent block is the same as the motion information of the adjacent block or the non-adjacent block at a specific position, the motion information of the at least one non-adjacent block is used as the current block Of non-adjacent spatial fusion candidates are added to the fusion candidate list, and when the non-adjacent block is available, it is determined that the inter prediction mode of at least one non-adjacent block in the non-adjacent block is In a predetermined inter prediction mode, the non-adjacent spatial fusion candidate of the at least one non-adjacent block is not added to the fusion candidate list of the current block.

The communication interface 22 of the source device 12 can be used to receive the encoded picture data 21 and transmit it to other devices, such as the destination device 14 or any other device, for storage or direct reconstruction, or for correspondingly storing the The encoded data 13 and / or the encoded picture data 21 are processed before transmitting the encoded data 13 to other devices, such as the destination device 14 or any other device for decoding or storage.

The destination device 14 includes a decoder 30 (for example, a video decoder 30), and in addition, optionally, it may include a communication interface or communication unit 28, a post-processing unit 32, and a display device 34.

The communication interface 28 of the destination device 14 is used, for example, to receive the encoded picture data 21 or the encoded data 13 directly from the source device 12 or any other source. Any other source is, for example, a storage device, and the storage device is, for example, encoded picture data storage. device.

The communication interface 22 and the communication interface 28 can be used for direct communication through a direct communication link between the source device 12 and the destination device 14 or transmission or reception of encoded picture data 21 or encoded data 13 through any type of network The link is, for example, a direct wired or wireless connection, and any type of network is, for example, a wired or wireless network or any combination thereof, or any type of private and public network, or any combination thereof.

The communication interface 22 may be used, for example, to encapsulate the encoded picture data 21 into a suitable format, such as a packet, for transmission over a communication link or communication network.

The communication interface 28 forming a corresponding part of the communication interface 22 may be used, for example, to decapsulate the encoded data 13 to obtain the encoded picture data 21.

Both the communication interface 22 and the communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrows for the encoded picture data 13 from the source device 12 to the destination device 14 in FIG. 1A, or configured as bidirectional communication interfaces, and It can be used, for example, to send and receive messages to establish a connection, acknowledge, and exchange any other information related to a communication link and / or data transmission such as encoded picture data transmission.

The decoder 30 is configured to receive the encoded picture data 21 and provide the decoded picture data 31 or the decoded picture 31 (details will be further described below, for example, based on FIG. 3 or FIG. 5). In one example, the decoder 30 may be configured to decode the data encoded by the encoder. Specifically, the decoder 30 may parse a bitstream to obtain a fusion candidate index, and obtain a corresponding one from the fusion candidate list according to the fusion candidate index. And use the fusion candidate as the motion information of the current block; perform inter prediction on the current block according to the motion information of the current block to obtain a predicted image of the current block; and acquire the current image of the current block. A residual image; adding the predicted image of the current block and the residual image of the current block to obtain a reconstructed image of the current block.

The post-processor 32 of the destination device 14 is used to post-process decoded picture data 31 (also referred to as reconstructed picture data), for example, decoded picture 131 to obtain post-processed picture data 33, for example, post-processed Picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., conversion from YCbCr to RGB), color correction, retouching, or resampling, or any other processing, such as preparing the decoded picture data 31 to be processed by The display device 34 displays it.

The display device 34 of the destination device 14 is used to receive the post-processed picture data 33 to display a picture to, for example, a user or a viewer. The display device 34 may be or may include any kind of display for presenting a reconstructed picture, such as an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), Digital light processor (DLP) or any other display of any kind.

Although FIG. 1A illustrates the source device 12 and the destination device 14 as separate devices, the device embodiment may also include the source device 12 and the destination device 14 or both of the functionality, that is, the source device 12 or corresponding And the functionality of the destination device 14 or equivalent. In such embodiments, the same hardware and / or software, or separate hardware and / or software, or any combination thereof may be used to implement the source device 12 or corresponding functionality and the destination device 14 or corresponding functionality .

It is obvious to those skilled in the art based on the description that the existence and (exact) division of the functionality of different units or the functionality of the source device 12 and / or the destination device 14 shown in FIG. 1A may be different according to the actual device and application.

Both the encoder 20 (e.g., video encoder 20) and decoder 30 (e.g., video decoder 30) may be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (digital signal processor, DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), discrete logic, hardware, or any combination thereof. If the technology is implemented partially in software, the device may store the software's instructions in a suitable non-transitory computer-readable storage medium and may use one or more processors to execute the instructions in hardware to perform the techniques of the present disclosure . Any one of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be considered as one or more processors. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and any of the encoders or decoders may be integrated as a combined encoder / decoder in a corresponding device (Codec).

The source device 12 may be referred to as a video encoding device or a video encoding device. The destination device 14 may be referred to as a video decoding device or a video decoding device. The source device 12 and the destination device 14 may be examples of a video encoding device or a video encoding apparatus.

Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, mobile phone, smartphone, tablet or tablet computer, video camera, desktop Computer, set-top box, TV, display device, digital media player, video game console, video streaming device (such as content service server or content distribution server), broadcast receiver device, broadcast transmitter device, etc., and may not be used Or use any kind of operating system.

In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Therefore, the source device 12 and the destination device 14 may be wireless communication devices.

In some cases, the video encoding system 10 shown in FIG. 1A is merely an example, and the techniques of this application may be applicable to video encoding settings (eg, video encoding or video decoding) that do not necessarily include any data communication between encoding and decoding devices. . In other examples, data may be retrieved from local storage, streamed over a network, and the like. The video encoding device may encode the data and store the data to a memory, and / or the video decoding device may retrieve the data from the memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but only encode data to and / or retrieve data from memory and decode data.

It should be understood that for each of the examples described above with reference to video encoder 20, video decoder 30 may be used to perform the reverse process. Regarding signaling syntax elements, video decoder 30 may be used to receive and parse such syntax elements, and decode related video data accordingly. In some examples of the present invention, the video encoder 20 may add one or more syntax elements that define the specific position of the fusion candidate in the fusion candidate list and the syntax element of the inter-coding type of the spatial non-adjacent block of the current block. Entropy coded into an encoded video bitstream. In such examples, video decoder 30 may parse such syntax elements and decode related video data accordingly.

FIG. 1B is an explanatory diagram of an example of a video encoding system 40 including the encoder 20 of FIG. 2 and / or the decoder 30 of FIG. 3 according to an exemplary embodiment. The system 40 can implement the technology of the present application for constructing a fusion candidate list of the current block based on the fusion candidate construction method proposed by the present invention, and encode or decode an image based on the fusion candidate list. In the illustrated embodiment, the video encoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder implemented by the logic circuit 47 of the processing unit 46), an antenna 42, One or more processors 43, one or more memories 44, and / or a display device 45.

As shown, the imaging device 41, antenna 42, processing unit 46, logic circuit 47, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 can communicate with each other. As discussed, although video encoding system 40 is shown with video encoder 20 and video decoder 30, in different examples, video encoding system 40 may include only video encoder 20 or only video decoder 30.

In some examples, as shown, the video encoding system 40 may include an antenna 42. For example, the antenna 42 may be used to transmit or receive an encoded bit stream of video data. In addition, in some examples, the video encoding system 40 may include a display device 45. The display device 45 may be used to present video data. In some examples, as shown, the logic circuit 47 may be implemented by the processing unit 46. The processing unit 46 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. The video encoding system 40 may also include an optional processor 43, which may similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. In some examples, the logic circuit 47 may be implemented by hardware, such as dedicated hardware for video encoding, and the processor 43 may be implemented by general software, operating system, and the like. In addition, the memory 44 may be any type of memory, such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatile memory Memory (for example, flash memory, etc.). In a non-limiting example, the memory 44 may be implemented by a cache memory. In some examples, the logic circuit 47 may access the memory 44 (eg, for implementing an image buffer). In other examples, the logic circuit 47 and / or the processing unit 46 may include a memory (eg, a cache, etc.) for implementing an image buffer or the like.

In some examples, video encoder 20 implemented by logic circuits may include an image buffer (eg, implemented by processing unit 46 or memory 44) and a graphics processing unit (eg, implemented by processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include a video encoder 20 implemented by a logic circuit 47 to implement the various modules discussed with reference to FIG. 2 and / or any other encoder system or subsystem described herein. Logic circuits can be used to perform various operations discussed herein.

Video decoder 30 may be implemented in a similar manner by logic circuit 47 to implement the various modules discussed with reference to decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. In some examples, video decoder 30 implemented by a logic circuit may include an image buffer (implemented by processing unit 2820 or memory 44) and a graphics processing unit (eg, implemented by processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include a video decoder 30 implemented by a logic circuit 47 to implement various modules discussed with reference to FIG. 3 and / or any other decoder system or subsystem described herein.

In some examples, the antenna 42 of the video encoding system 40 may be used to receive an encoded bit stream of video data. As discussed, the encoded bitstream may contain data, indicators, index values, mode selection data, etc. related to encoded video frames discussed herein, such as data related to coded segmentation (e.g., transform coefficients or quantized transform coefficients) , (As discussed) optional indicators, and / or data defining code partitions). The video encoding system 40 may also include a video decoder 30 coupled to the antenna 42 and used to decode the encoded bitstream. The display device 45 is used to present video frames.

Encoder & encoding method

FIG. 2 shows a schematic / conceptual block diagram of an example of a video encoder 20 for implementing the techniques of the present invention. In the example of FIG. 2, the video encoder 20 includes a residual calculation unit 204, a transformation processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transformation processing unit 212, a reconstruction unit 214, a buffer 216, and a loop filter. A decoder unit 220, a decoded picture buffer (DPB) 230, a prediction processing unit 260, and an entropy encoding unit 270. The prediction processing unit 260 may include an inter prediction unit 244, an intra prediction unit 254, and a mode selection unit 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG. 2 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

For example, the residual calculation unit 204, the transformation processing unit 206, the quantization unit 208, the prediction processing unit 260, and the entropy encoding unit 270 form the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transformation processing unit 212, The constructing unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, and the prediction processing unit 260 form a backward signal path of the encoder, wherein the backward signal path of the encoder corresponds to To the decoder's signal path (see decoder 30 in Figure 3).

The encoder 20 receives a picture 201 or a block 203 of the picture 201 through, for example, an input 202, for example, a picture in a picture sequence forming a video or a video sequence. The picture block 203 can also be called the current picture block or the picture block to be encoded, and the picture 201 can be called the current picture or the picture to be encoded (especially when the current picture is distinguished from other pictures in video encoding, other pictures such as the same video sequence (Ie previously encoded and / or decoded pictures in the video sequence of the current picture).

segmentation

An embodiment of the encoder 20 may include a segmentation unit (not shown in FIG. 2) for segmenting the picture 201 into multiple blocks, such as the block 203, and generally into multiple non-overlapping blocks. The segmentation unit can be used to use the same block size and corresponding raster to define the block size for all pictures in the video sequence, or to change the block size between pictures or subsets or groups of pictures, and split each picture into Corresponding block. In VVC, the block structure for the next generation of video coding was introduced by J. An et al. (International Telecommunication Union, COM16-C966, September 2015, hereinafter referred to as As a quad-tree-binary-tree (QTBT) partitioning technique proposed in "VCEG Recommendation COM16-C966"), simulations have shown that the proposed QTBT structure is more than the quad-tree structure in the HEVC used Efficient. In addition, in QTBT, the CU can have a square or rectangular shape. As shown in Figure 3, the coding tree unit (CTU) is first divided by the quad tree structure. The quad leaf nodes can further pass the binary tree structure Dividing. There are two types of partitioning in binary tree partitioning: symmetrical horizontal partitioning and symmetrical vertical partitioning. In each case, the nodes are divided by dividing the nodes horizontally or vertically along the middle. Binary leaf nodes are called coding units (coding unit, CU), and the prediction and transformation processes without any further division. This means that CU, PU, and TU have the same block in the QTBT coding block structure Size. A CU is sometimes composed of coding blocks (CBs) with different color components. For example, in the case of P and B slices in 4: 2: 0 chroma format, one CU contains one luminance CB and two Chroma CB, and CUs are sometimes composed of CBs with a single component, for example, in the case of I-strips, one CU contains only one luminance CB or only two chroma CBs.

In addition, U.S. Patent Application Publication No. 20170208336 proposes a block division structure called a multi-type-tree (MTT) instead of a CU structure based on QT, BT, and / or QTBT. The MTT partition structure is still a recursive tree structure. In MTT, a plurality of different partition structures (for example, three or more) are used. For example, according to MTT technology, three or more different partition structures may be used at each depth of the tree structure for each corresponding non-leaf node of the tree structure. The depth of a node in the tree structure may refer to the length (eg, the number of divisions) of a path from the node to the root of the tree structure. A partition structure may generally refer to how many different blocks a block can be divided into. The partition structure can be a quad tree partition structure that can divide a block into four blocks, a binary tree partition structure that can divide a block into two blocks, or a triple tree partition structure that can divide a block into three blocks. In addition, the tri-tree The partition structure may not be divided by the center. The partition structure can have multiple different partition types. The division type may additionally define how to divide the block, including symmetrical or asymmetrical division, uniform or uneven division, and / or horizontal or vertical division.

In MTT, at each depth of the tree structure, the encoder 100 may be used to further divide the subtree using a particular partition type of one of the three further partition structures. For example, the encoder 100 may be used to determine specific partition types from QT, BT, triple-tree (TT), and other partition structures. In one example, the QT partition structure may include a square quadtree or a rectangular quadtree partition type. The encoder 100 may use a square quadtree partition to divide a square block by dividing the block horizontally and vertically along the center into four square blocks of equal size. Likewise, the encoder 100 may use a rectangular quadtree partition to divide a rectangular (eg, non-square) block by dividing the rectangular block horizontally and vertically along the center into four equal-sized rectangular blocks.

The BT partition structure may include at least one of a horizontally symmetric binary tree, a vertically symmetric binary tree, a horizontally asymmetric binary tree, or a vertically asymmetric binary tree partition type. For the horizontally symmetric binary tree partition type, the encoder 100 may be used to horizontally divide a block horizontally into two symmetrical blocks of the same size along the center of the block. For the vertical symmetric binary tree partition type, the encoder 100 may be used to bisect a block vertically into two symmetrical blocks of the same size along the center of the block. For the horizontal asymmetric binary tree partition type, the encoder 100 may be used to horizontally divide a block into two blocks of different sizes. For example, one block may be 1/4 of the size of the parent block, while another block may be 3/4 of the size of the parent block, similar to the PART_2N × nU or PART_2N × nD partition type. For the vertical asymmetric binary tree partition type, the encoder 100 may be used to vertically divide a block into two blocks of different sizes. For example, one block may be 1/4 of the size of the parent block, and another block may be 3/4 of the size of the parent block, similar to the PART_nL × 2N or PART_nR × 2N partition type. In other examples, the asymmetric binary tree partition type may divide the parent block into sections of different sizes. For example, one child block may be 3/8 of the parent block, and the other child block may be 5/8 of the parent block. Of course, this type of division can be vertical or horizontal.

The difference between the TT partition structure and the type of the QT or BT structure is that the TT partition structure does not partition the blocks along the center. The center areas of the blocks are kept together in the same sub-block. Different from the QT generating four blocks or the binary tree generating two blocks, three blocks are generated according to the division of the TT partition structure. Example partition types according to the TT partition structure include symmetric partition types (both horizontal and vertical) and asymmetric partition types (both horizontal and vertical). In addition, the symmetric partition type according to the TT partition structure may be uneven / uneven or equal / uniform. The asymmetric partition type according to the TT partition structure is uneven / uneven. In one example, the TT partition structure may include at least one of the following partition types: level equal / uniformly symmetrical tri-tree, vertical equal / uniformly symmetrical tri-tree, horizontally uneven / uniformly symmetrical tri-tree, vertical uneven / Uneven symmetrical tri-tree, horizontal uneven / uneven asymmetric tri-tree, or vertical uneven / uneven asymmetric tri-tree partition type

In general, the unequal / unsymmetric symmetric tri-tree partition type is a partition type that is symmetrical around the center line of the block but has at least one of the three blocks obtained with a size different from the other two. A preferred example is where the side block is 1/4 of the block size and the center block is 1/2 of the block size. The equal / uniform symmetrical tri-tree partition type is a partition type that is symmetrical around the center line of the block and the size of the resulting block is all the same. This type of division is possible if the block height or width-depending on the vertical or horizontal division-is an integer multiple of three. The uneven / uneven asymmetric tri-tree partition type is a partition type that is not symmetrical around the center line of the block and wherein at least one of the resulting blocks is not the same size as the other two.

In one example, the prediction processing unit 260 of the video encoder 20 may be used to perform any combination of the aforementioned segmentation techniques.

Like picture 201, block 203 is also or can be regarded as a two-dimensional array or matrix of sampling points with brightness values (sampling values), although its size is smaller than picture 201. In other words, block 203 may include, for example, one sampling array (e.g., a luminance array in the case of a black and white picture 201) or three sampling arrays (e.g., one luminance array and two chroma arrays in the case of a color picture) or a basis An array of any other number and / or category of color formats applied. The number of sampling points in the horizontal and vertical directions (or axes) of the block 203 defines the size of the block 203.

The encoder 20 shown in FIG. 2 is used to encode a picture 201 block by block, for example, performing encoding and prediction on each block 203.

Residual calculation

The residual calculation unit 204 is configured to calculate the residual block 205 based on the picture block 203 and the prediction block 265 (the other details of the prediction block 265 are provided below). For example, the sample value of the picture block 203 is subtracted from the prediction by sample by pixel (pixel by pixel). Sample values of block 265 to obtain residual block 205 in the sample domain.

Transform

The transform processing unit 206 is configured to apply a transform such as discrete cosine transform (DCT) or discrete sine transform (DST) on the sample values of the residual block 205 to obtain transform coefficients 207 in the transform domain. . The transform coefficient 207 may also be referred to as a transform residual coefficient, and represents a residual block 205 in a transform domain.

The transform processing unit 206 may be used to apply an integer approximation of DCT / DST, such as the transform specified for HEVC / H.265. Compared to an orthogonal DCT transform, this integer approximation is usually scaled by a factor. To maintain the norm of the residual blocks processed by the forward and inverse transforms, an additional scaling factor is applied as part of the transform process. The scaling factor is usually selected based on certain constraints, for example, the scaling factor is a power of two used for shift operations, the bit depth of the transform coefficients, the trade-off between accuracy, and implementation cost. For example, a specific scaling factor is specified on the decoder 30 side by, for example, the inverse transform processing unit 212 (and on the encoder 20 side by, for example, the inverse transform processing unit 212 as the corresponding inverse transform), and accordingly, the The 20 side specifies a corresponding scaling factor for the positive transformation through the transformation processing unit 206.

Quantify

The quantization unit 208 is used to quantize the transform coefficients 207, for example, by applying scalar quantization or vector quantization to obtain the quantized transform coefficients 209. The quantized transform coefficient 209 may also be referred to as a quantized residual coefficient 209. The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients may be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For scalar quantization, for example, different scales can be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to a finer quantization, while a larger quantization step size corresponds to a coarser quantization. An appropriate quantization step size can be indicated by a quantization parameter (QP). For example, the quantization parameter may be an index of a predefined set of suitable quantization steps. For example, smaller quantization parameters may correspond to fine quantization (smaller quantization step size), larger quantization parameters may correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include division by a quantization step size and corresponding quantization or inverse quantization performed, for example, by inverse quantization 210, or may include multiplication by a quantization step size. Embodiments according to some standards such as HEVC may use quantization parameters to determine the quantization step size. In general, the quantization step size can be calculated using a fixed-point approximation using an equation containing division based on the quantization parameter. Additional scaling factors may be introduced for quantization and inverse quantization to restore the norm of the residual block that may be modified due to the scale used in the fixed-point approximation of the equation for the quantization step size and quantization parameter. In one example embodiment, inverse transform and inverse quantization scales can be combined. Alternatively, a custom quantization table can be used and signaled from the encoder to the decoder in, for example, a bitstream. Quantization is a lossy operation, where the larger the quantization step, the greater the loss.

The inverse quantization unit 210 is used to apply the inverse quantization of the quantization unit 208 on the quantized coefficients to obtain the inverse quantized coefficients 211. For example, based on or using the same quantization step size as the quantization unit 208, the quantization scheme applied by the quantization unit 208 is applied. Inverse quantization scheme. The dequantized coefficient 211 may also be referred to as a dequantized residual coefficient 211, which corresponds to the transform coefficient 207, although the loss due to quantization is usually different from the transform coefficient.

The inverse transform processing unit 212 is used to apply an inverse transform of the transform applied by the transform processing unit 206, for example, an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), in the sample domain. Obtain an inverse transform block 213. The inverse transform block 213 may also be referred to as an inverse transform inverse quantized block 213 or an inverse transform residual block 213.

The reconstruction unit 214 (for example, the summer 214) is used to add the inverse transform block 213 (that is, the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the sample domain. For example, The sample values of the reconstructed residual block 213 are added to the sample values of the prediction block 265.

Optionally, a buffer unit 216 (or simply "buffer" 216), such as a line buffer 216, is used to buffer or store the reconstructed block 215 and corresponding sample values, for example, for intra prediction. In other embodiments, the encoder may be used to use any unfiltered reconstructed block and / or corresponding sample values stored in the buffer unit 216 for any category of estimation and / or prediction, such as intra-frame prediction.

For example, an embodiment of the encoder 20 may be configured such that the buffer unit 216 is used not only for storing the reconstructed block 215 for intra prediction 254, but also for the loop filter unit 220 (not shown in FIG. 2). Out), and / or, for example, to make the buffer unit 216 and the decoded picture buffer unit 230 form a buffer. Other embodiments may be used to use the filtered block 221 and / or blocks or samples from the decoded picture buffer 230 (neither shown in FIG. 2) as the input or basis for the intra prediction 254.

The loop filter unit 220 (or simply "loop filter" 220) is configured to filter the reconstructed block 215 to obtain the filtered block 221, so as to smoothly perform pixel conversion or improve video quality. The loop filter unit 220 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters, such as a bilateral filter, Adaptive loop filters (adaptive loop filters, ALF), or sharpening or smoothing filters, or cooperative filters. Although the loop filter unit 220 is shown as an in-loop filter in FIG. 2, in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as a filtered reconstructed block 221. The decoded picture buffer 230 may store the reconstructed encoded block after the loop filter unit 220 performs a filtering operation on the reconstructed encoded block.

An embodiment of the encoder 20 (correspondingly, the loop filter unit 220) may be used to output loop filter parameters (e.g., sample adaptive offset information), for example, directly output or by the entropy coding unit 270 or any other The entropy coding unit outputs after entropy coding, for example, so that the decoder 30 can receive and apply the same loop filter parameters for decoding.

The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for the video encoder 20 to encode video data. DPB 230 can be formed by any of a variety of memory devices, such as dynamic random access (DRAM) (including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), and resistive RAM (resistive RAM, RRAM)) or other types of memory devices. The DPB 230 and the buffer 216 may be provided by the same memory device or separate memory devices. In a certain example, a decoded picture buffer (DPB) 230 is used to store the filtered block 221. The decoded picture buffer 230 may be further used to store other previous filtered blocks of the same current picture or different pictures such as previously reconstructed pictures, such as the previously reconstructed and filtered block 221, and may provide a complete previous Reconstruction is the decoded picture (and corresponding reference blocks and samples) and / or part of the reconstructed current picture (and corresponding reference blocks and samples), for example for inter prediction. In a certain example, if the reconstructed block 215 is reconstructed without in-loop filtering, a decoded picture buffer (DPB) 230 is used to store the reconstructed block 215.

Prediction processing unit 260, also referred to as block prediction processing unit 260, is used to receive or obtain block 203 (current block 203 of current picture 201) and reconstructed picture data, such as a reference to the same (current) picture from buffer 216 Samples and / or reference picture data 231 from one or more previously decoded pictures from the decoded picture buffer 230, and used to process such data for prediction, i.e., may be provided as inter-predicted blocks 245 or intra- Prediction block 265 of prediction block 255.

The mode selection unit 262 may be used to select a prediction mode (such as an intra or inter prediction mode) and / or a corresponding prediction block 245 or 255 used as the prediction block 265 to calculate the residual block 205 and reconstruct the reconstructed block 215.

An embodiment of the mode selection unit 262 may be used to select a prediction mode (e.g., selected from those prediction modes supported by the prediction processing unit 260) that provides the best match or minimum residual (minimum residual means Better compression in transmission or storage), or provide minimal signaling overhead (minimum signaling overhead means better compression in transmission or storage), or consider or balance both. The mode selection unit 262 may be used to determine a prediction mode based on rate distortion optimization (RDO), that is, to select a prediction mode that provides the minimum code rate distortion optimization, or to select a prediction mode whose related code rate distortion meets the prediction mode selection criteria .

The prediction processing performed by an example of the encoder 20 (for example, by the prediction processing unit 260) and mode selection (for example, by the mode selection unit 262) will be explained in detail below.

As described above, the encoder 20 is used to determine or select the best or optimal prediction mode from a set of (predetermined) prediction modes. The prediction mode set may include, for example, an intra prediction mode and / or an inter prediction mode.

The set of intra prediction modes may include 35 different intra prediction modes, for example, non-directional modes such as DC (or average) mode and planar mode, or directional modes as defined in H.265, or may include 67 Different intra prediction modes, such as non-directional modes such as DC (or mean) mode and planar mode, or directional modes as defined in the developing H.266.

The set of (possible) inter prediction modes depends on the available reference pictures (i.e., at least part of the decoded pictures previously stored in DBP 230) and other inter prediction parameters, such as whether to use the entire reference picture or only the reference A part of the picture, such as a search window area surrounding the area of the current block, searches for the best matching reference block, and / or depends on, for example, whether pixel interpolation such as half-pixel and / or quarter-pixel interpolation is applied.

In addition to the above prediction modes, a skip mode and / or a direct mode can also be applied.

The prediction processing unit 260 may be further configured to divide the block 203 into smaller block partitions or sub-blocks, for example, using a quad-tree (QT) partition, a binary-tree (BT) partition, or Triple-tree (TT) segmentation, or any combination thereof, and for performing predictions, for example, for each of block partitions or sub-blocks, where the mode selection includes selecting the tree structure of the partitioned block 203 and the selection applied to the block The prediction mode for each of the partitions or sub-blocks.

The inter prediction unit 244 may include a motion estimation (ME) unit (not shown in FIG. 2) and a motion compensation (MC) unit (not shown in FIG. 2). The motion estimation unit is configured to receive or obtain picture block 203 (current picture block 203 of current picture 201) and decoded picture 231, or at least one or more previously reconstructed blocks, for example, one or more other / different previous The reconstructed block of picture 231 is decoded for motion estimation. For example, the video sequence may include the current picture and the previously decoded picture 31, or in other words, the current picture and the previously decoded picture 31 may be part of the picture sequence forming the video sequence or form the picture sequence. The construction of the fusion candidate list of the present application can be implemented by the motion estimation module.

For example, the encoder 20 may be used to select a reference block from multiple reference blocks of the same or different pictures in multiple other pictures, and provide a reference picture (or reference picture index) to a motion estimation unit (not shown in FIG. 2). ...) and / or provide an offset (spatial offset) between the position (X, Y coordinates) of the reference block and the position of the current block as an inter prediction parameter. This offset is also called a motion vector (MV).

The motion compensation unit is used for obtaining, for example, receiving inter prediction parameters, and performing inter prediction based on or using the inter prediction parameters to obtain the inter prediction block 245. Motion compensation performed by a motion compensation unit (not shown in FIG. 2) may include taking out or generating a prediction block based on a motion / block vector determined through motion estimation (possibly performing interpolation on sub-pixel accuracy). Interpolation filtering can generate additional pixel samples from known pixel samples, potentially increasing the number of candidate prediction blocks that can be used to encode picture blocks. Upon receiving the motion vector of the PU for the current picture block, the motion compensation unit 246 may locate the prediction block pointed to by the motion vector in a reference picture list. Motion compensation unit 246 may also generate syntax elements associated with blocks and video slices for use by video decoder 30 when decoding picture blocks of video slices.

The intra prediction unit 254 is configured to obtain, for example, a picture block 203 (current picture block) and one or more previously reconstructed blocks, such as reconstructed neighboring blocks, that receive the same picture for intra estimation. For example, the encoder 20 may be used to select an intra prediction mode from a plurality of (predetermined) intra prediction modes.

Embodiments of the encoder 20 may be used to select an intra prediction mode based on optimization criteria, such as based on a minimum residual (e.g., an intra prediction mode that provides a prediction block 255 that is most similar to the current picture block 203) or a minimum code rate distortion ( E.g……).

The intra prediction unit 254 is further configured to determine the intra prediction block 255 based on the intra prediction parameters of the intra prediction mode as selected. In any case, after selecting the intra prediction mode for the block, the intra prediction unit 254 is further configured to provide the intra prediction parameters to the entropy encoding unit 270, that is, to provide an indication of the selected intra prediction mode for the block. Information. In one example, the intra prediction unit 254 may be used to perform any combination of intra prediction techniques described below.

The entropy coding unit 270 is configured to apply an entropy coding algorithm or scheme (for example, a variable length coding (VLC) scheme, a context adaptive VLC (context adaptive VLC, CAVLC) scheme, an arithmetic coding scheme, and a context adaptive binary arithmetic Coding (context, adaptive binary coding, CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy Encoding method or technique) applied to one or all of the quantized residual coefficients 209, inter prediction parameters, intra prediction parameters, and / or loop filter parameters (or not applied) to obtain The encoded picture data 21 is output in the form of, for example, an encoded bit stream 21. The encoded bitstream may be transmitted to video decoder 30 or archived for later transmission or retrieval by video decoder 30. The entropy encoding unit 270 may also be used to entropy encode other syntax elements of the current video slice that is being encoded.

Other structural variations of video encoder 20 may be used to encode a video stream. For example, the non-transform-based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another embodiment, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

FIG. 3 illustrates an exemplary video decoder 30 for implementing the technique of the present application, that is, constructing a fusion candidate list of a block to be decoded (current block) and decoding a compressed image based on the constructed fusion candidate list. The video decoder 30 is configured to receive, for example, encoded picture data (eg, an encoded bit stream) 21 encoded by the encoder 20 to obtain a decoded picture 231. During the decoding process, video decoder 30 receives video data from video encoder 20, such as an encoded video bitstream and associated syntax elements representing picture blocks of encoded video slices.

In the example of FIG. 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (such as a summer 314), a buffer 316, a loop filter 320, The decoded picture buffer 330 and the prediction processing unit 360. The prediction processing unit 360 may include an inter prediction unit 344, an intra prediction unit 354, and a mode selection unit 362. In some examples, video decoder 30 may perform a decoding pass that is substantially inverse to the encoding pass described with reference to video encoder 20 of FIG. 2.

The entropy decoding unit 304 is configured to perform entropy decoding on the encoded picture data 21 to obtain, for example, quantized coefficients 309 and / or decoded encoding parameters (not shown in FIG. 3), for example, inter prediction, intra prediction parameters , (Filtered) any or all of the loop filter parameters and / or other syntax elements. The entropy decoding unit 304 is further configured to forward the inter prediction parameters, the intra prediction parameters, and / or other syntax elements to the prediction processing unit 360. Video decoder 30 may receive syntax elements at the video slice level and / or the video block level.

The inverse quantization unit 310 may be functionally the same as the inverse quantization unit 110, the inverse transform processing unit 312 may be functionally identical to the inverse transform processing unit 212, the reconstruction unit 314 may be functionally identical to the reconstruction unit 214, and the buffer 316 may be functionally Like the buffer 216, the loop filter 320 may be functionally the same as the loop filter 220, and the decoded picture buffer 330 may be functionally the same as the decoded picture buffer 230.

The prediction processing unit 360 may include an inter prediction unit 344 and an intra prediction unit 354. The inter prediction unit 344 may be functionally similar to the inter prediction unit 244 and the intra prediction unit 354 may be functionally similar to the intra prediction unit 254. . The prediction processing unit 360 is generally used to perform block prediction and / or obtain a prediction block 365 from the encoded data 21, and to receive or obtain prediction-related parameters from, for example, an entropy decoding unit 304 (explicitly or implicitly) and / or Information about the selected prediction mode.

When a video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 of the prediction processing unit 360 is used for the intra-prediction mode based on the signal representation and the previously decoded block from the current frame or picture Data to generate a prediction block 365 for a picture block of the current video slice. When a video frame is encoded as an inter-encoded (ie, B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) of the prediction processing unit 360 is based on the motion vector and received from the entropy decoding unit 304. The other syntax elements generate a prediction block 365 for a video block of the current video slice. For inter prediction, a prediction block may be generated from a reference picture in a reference picture list. The video decoder 30 may construct a reference frame list using a default construction technique based on the reference pictures stored in the DPB 330: List 0 and List 1.

The prediction processing unit 360 is configured to determine prediction information for a video block of a current video slice by analyzing a motion vector and other syntax elements, and use the prediction information to generate a prediction block for a current video block that is being decoded. For example, the prediction processing unit 360 uses some of the received syntax elements to determine a prediction mode (e.g., intra or inter prediction) of a video block used to encode a video slice, an inter prediction slice type (e.g., B slice, P slice or GPB slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each inter-coded video block for the slice, each warp for the slice The inter-prediction status and other information of the inter-coded video block to decode the video block of the current video slice.

The inverse quantization unit 310 may be used for inverse quantization (ie, inverse quantization) of the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 304. The inverse quantization process may include using the quantization parameters calculated by video encoder 20 for each video block in the video slice to determine the degree of quantization that should be applied and also to determine the degree of inverse quantization that should be applied.

The inverse transform processing unit 312 is configured to apply an inverse transform (for example, an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to generate a residual block in the pixel domain.

Reconstruction unit 314 (e.g., summer 314) is used to add inverse transform block 313 (i.e., reconstructed residual block 313) to prediction block 365 to obtain reconstructed block 315 in the sample domain, such as by The sample values of the reconstructed residual block 313 are added to the sample values of the prediction block 365.

The loop filter unit 320 (during or after the encoding cycle) is used to filter the reconstructed block 315 to obtain the filtered block 321 so as to smoothly perform pixel conversion or improve video quality. The loop filter unit 320 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters such as a bilateral filter, Adaptive loop filters (adaptive loop filters, ALF), or sharpening or smoothing filters, or cooperative filters. Although the loop filter unit 320 is shown as an in-loop filter in FIG. 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

The decoded video block 321 in a given frame or picture is then stored in a decoded picture buffer 330 that stores reference pictures for subsequent motion compensation.

The decoder 30 is used, for example, to output a decoded picture 31 through an output 332 for presentation to or review by a user.

Other variations of video decoder 30 may be used to decode the compressed bitstream. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 may directly inversely quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another embodiment, the video decoder 30 may have an inverse quantization unit 310 and an inverter conversion processing unit 312 combined into a single unit.

FIG. 4 is a schematic structural diagram of a video decoding device 400 (such as a video encoding device 400 or a video decoding device 400) according to an embodiment of the present invention. Video coding device 400 is adapted to implement the embodiments described herein. In one embodiment, the video coding device 400 may be a video decoder (such as video decoder 30 of FIG. 1A) or a video encoder (such as video encoder 20 of FIG. 1A). In another embodiment, the video decoding device 400 may be one or more of the video decoder 30 of FIG. 1A or the video encoder 20 of FIG. 1A described above.

The video decoding device 400 includes: an entry port 410 and a receiving unit (Rx) 420 for receiving data, a processor, a logic unit or a central processing unit (CPU) 430 for processing data, and a transmitter unit for transmitting data (Tx) 440 and egress port 450, and a memory 460 for storing data. The video decoding device 400 may further include a photoelectric conversion component and an electro-optic (EO) component coupled with the entrance port 410, the receiver unit 420, the transmitter unit 440, and the exit port 450 for an exit or entrance of an optical signal or an electric signal.

The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (eg, multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 is in communication with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a decoding module 470 (eg, an encoding module 470 or a decoding module 470). The encoding / decoding module 470 implements the embodiments disclosed above. For example, the encoding / decoding module 470 implements, processes, or provides various encoding operations. Therefore, the function of the video decoding device 400 is substantially improved through the encoding / decoding module 470, and the transition of the video decoding device 400 to different states is affected. Alternatively, the encoding / decoding module 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.

The memory 460 includes one or more magnetic disks, tape drives, and solid-state hard disks, which can be used as overflow data storage devices for storing programs when these programs are selectively executed, and for storing instructions and data read during program execution. The memory 460 may be volatile and / or non-volatile, and may be a read-only memory (ROM), a random access memory (RAM), a random content-addressable memory (TCAM), and / or a static state. Random access memory (SRAM).

FIG. 5 is a simplified block diagram of an apparatus 500 that can be used as either or both of the source device 12 and the destination device 14 in FIG. 1A according to an exemplary embodiment. The apparatus 500 may implement the technology of the present application for constructing a fusion candidate list and encoding or decoding an image based on the constructed fusion candidate list. The apparatus 500 may take the form of a computing system including a plurality of computing devices, or take the form of a single computing device such as a mobile phone, tablet computer, laptop computer, notebook computer, desktop computer, and the like.

The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or multiple devices capable of manipulating or processing information, existing or to be developed in the future. As shown, although a single processor such as the processor 502 can be used to practice the disclosed embodiments, speed and efficiency advantages can be achieved using more than one processor.

In one embodiment, the memory 504 in the device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device can be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 using the bus 512. The memory 504 may further include an operating system 508 and an application program 510, which contains at least one program that permits the processor 502 to perform the methods described herein. For example, applications 510 may include applications 1 through N, and applications 1 through N further include video encoding applications that perform the fusion candidate list construction described herein. The device 500 may also include additional memory in the form of a slave memory 514, which may be, for example, a memory card for use with a mobile computing device. Because a video communication session may contain a large amount of information, this information may be stored in whole or in part in the slave memory 514 and loaded into the memory 504 for processing as needed.

The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display combining a display and a touch-sensitive element operable to sense a touch input. The display 518 may be coupled to the processor 502 through a bus 512. In addition to the display 518, other output devices may be provided that allow the user to program or otherwise use the device 500, or provide other output devices as an alternative to the display 518. When the output device is a display or contains a display, the display can be implemented in different ways, including through a liquid crystal display (LCD), a cathode-ray tube (CRT) display, a plasma display, or a light emitting diode diode (LED) displays, such as organic LED (OLED) displays.

The apparatus 500 may further include or be in communication with an image sensing device 520, such as a camera or any other image sensing device 520 that can or will be developed in the future to sense an image, such as An image of a user running the device 500. The image sensing device 520 may be placed directly facing a user of the running apparatus 500. In an example, the position and optical axis of the image sensing device 520 may be configured such that its field of view includes an area immediately adjacent to the display 518 and the display 518 is visible from the area.

The device 500 may also include or be in communication with a sound sensing device 522, such as a microphone or any other sound sensing device that can or will be developed in the future to sense the sound near the device 500. The sound sensing device 522 may be placed directly facing the user of the operating device 500 and may be used to receive a sound, such as a voice or other sound, emitted by the user when the device 500 is running.

Although the processor 502 and the memory 504 of the apparatus 500 are shown in FIG. 5 as being integrated in a single unit, other configurations may be used. The operation of the processor 502 may be distributed among multiple directly-coupled machines (each machine has one or more processors), or distributed in a local area or other network. The memory 504 may be distributed among multiple machines, such as a network-based memory or a memory among multiple machines running the apparatus 500. Although only a single bus is shown here, the bus 512 of the device 500 may be formed by multiple buses. Further, the slave memory 514 may be directly coupled to other components of the device 500 or may be accessed through a network, and may include a single integrated unit, such as one memory card, or multiple units, such as multiple memory cards. Therefore, the apparatus 500 can be implemented in various configurations.

FIG. 6 is a flowchart of an example operation of a method for constructing a fusion candidate list according to an embodiment of the present invention according to the video encoder 20 and the video decoder 30 shown in FIGS. 1A and 1B. One or more functional units of the video encoder 20 or the video decoder 30 include a prediction processing unit 260/360, which can be used to execute the method of FIG. 6. In the example of FIG. 6, an improved method for constructing a fusion candidate list is proposed. In the process of adding non-adjacent spatial fusion candidates (Candidate) to the fusion candidate list (Merge and Candidate list), The inter prediction mode corresponding to the non-adjacent spatial fusion candidate determines whether to add the non-adjacent spatial fusion candidate to the fusion candidate list, thereby improving prediction efficiency.

The candidate list construction method 600 includes:

S601 adds motion information of a neighboring block adjacent to the current block spatial domain as a spatial fusion candidate of the current block to a fusion candidate list of the current block based on a preset first selection rule;

The preset first selection rule includes: when the neighboring block is available and the motion information of the neighboring block is different from the motion information of the neighboring block at a specific position, the neighboring block is changed. The motion information of the current block as a spatial fusion candidate of the current block is added to the fusion candidate list of the current block; and

S603 adds motion information of a non-adjacent block that is not adjacent to the current block spatial domain as a non-adjacent spatial fusion candidate of the current block to the fusion candidate list based on a preset second selection rule;

Wherein, the current block has one or more non-adjacent blocks that are not adjacent in the airspace, and the preset second selection rule includes: judging the non-phase when the non-adjacent blocks are available. The inter prediction mode of at least one non-adjacent block in the neighboring block is not a predetermined inter prediction mode, and the motion information of the at least one non-adjacent block is related to the motion information of the neighboring block or non-adjacent block at a specific position. When the same, the motion information of the at least one non-adjacent block is added as a non-adjacent spatial fusion candidate of the current block to the fusion candidate list. When the non-adjacent block is available, Determine that the inter prediction mode of at least one non-adjacent block among the non-adjacent blocks is a predetermined inter-prediction mode, then the non-adjacent spatial fusion candidate of the at least one non-adjacent block is not added to the current block List of fusion candidates.

Wherein, the predetermined inter prediction mode is a skip / merge mode and / or an affine transformation prediction mode.

It can be understood that at least one non-adjacent block in the non-adjacent blocks described in the above solution may be an adjacent block at a predetermined position, that is, the present invention allows a part of the non-adjacent blocks to determine an inter prediction mode, and of course Inter-prediction mode determination is also allowed for all non-adjacent blocks.

Since the number of non-adjacent spatial fusion candidates is large and the length of the fusion candidate list is small, the proper selection of fusion candidates during the process of adding fusion candidates to the fusion candidate list can improve the fusion candidate list. The diversity of each fusion candidate and their similarity to the current block motion information, thereby improving prediction efficiency. In the prior art, whether a non-adjacent space fusion candidate overlaps with a fusion candidate list in the fusion candidate list to determine whether to add the non-adjacent space fusion candidate to the fusion candidate list, but does not consider non-adjacent space. Inter prediction mode corresponding to the fusion candidate. The motion information of non-adjacent spatial fusion candidates using skip / merge mode is easily duplicated or similar to the motion information of adjacent spatial fusion candidates, which is not conducive to improving the diversity of motion information in the fusion candidate list. In addition, if the non-adjacent spatial fusion candidate uses the affine model prediction mode and the current block uses the translation model prediction mode, these two regions have different motion types, so this non-adjacent spatial fusion candidate and the current block's The relevance of actual motion information is low and should not be added to the fusion candidate list. In the present invention, in the process of obtaining non-adjacent spatial fusion candidates, the inter-prediction mode of non-adjacent blocks is used as a determination criterion, which can effectively improve the diversity of motion information in the fusion candidate list, and can also avoid excessive The fusion candidates with low correlation are added to the fusion candidate list, thereby effectively improving the performance of encoding and decoding. It should be noted that the above-mentioned preset inter prediction mode may also be a mode other than the skip mode / fusion mode or the Affine mode mentioned in the present invention, such as an optical flow field mode.

Optionally, before the step S603, the method for constructing the candidate list of the present invention may further include the following steps:

S602 adds motion information of a neighboring block in the lower right corner of a co-located block in a reference frame of the current block as a time-domain fusion candidate of the current block to the fusion candidate list, where the co-located block is in the The position in the reference frame is the same as the current position of the current block, or if the motion information of the adjacent block in the lower right corner of the co-located block is not available, the motion information of the center point of the co-located block is used as The time-domain fusion candidate of the current block is added to the fusion candidate list.

This step inherits the acquisition step of the fusion candidate in the time domain in HEVC. This step uses the characteristics of the block (co-located block) at the same position in the reference frame of the current block in the time domain to decode the co-located block in the lower right corner. Adjacent motion information is added to the fusion candidate list as a time-domain fusion candidate of the current block to enrich the diversity of the fusion candidate list. It should be noted that the fusion candidate in the time domain may also be selected from other adjacent or non-adjacent blocks adjacent to the co-located block according to requirements, which may be used as a further extension of the above steps or may be Alternatives are combined with the solution of the invention.

Optionally, after the step S603, the method for constructing the candidate list of the present invention may further include the following steps

S605: Add the bidirectional prediction fusion candidate to the fusion candidate list when the current block belongs to a bi-predictive slice, or add the bidirectional prediction fusion candidate to the fusion. After the candidate list is included, if the number of the fusion candidates in the fusion candidate list does not reach the predetermined value, a zero motion vector fusion candidate is added to the fusion candidate list; or When the current block belongs to a uni-predictive slice, a zero motion vector fusion candidate is added to the fusion candidate list.

As mentioned before, the fusion candidate list usually has a predetermined number of fusion candidates, but may occur in individual cases. After traversing the above steps S601 to S603, the fusion candidate list is still not full. That is, there is room for other fusion candidates. In order to make full use of the coding space provided by the fusion candidate list, the fusion candidate list can be filled according to the prediction type of the current block, which is included in the Adding a bidirectional prediction fusion candidate to the fusion candidate list when the current block belongs to a bi-predictive slice, or adding the bidirectional prediction fusion candidate to the fusion candidate list After that, if the number of the fusion candidates in the fusion candidate list does not reach the predetermined value, a zero motion vector fusion candidate is added to the fusion candidate list; or when the current block belongs to a single block When predicting a slice (uni-predictive slice), a zero motion vector fusion candidate is added to the fusion candidate list.

The solution of the invention determines whether to add non-adjacent spatial fusion candidates to the fusion candidate list according to the inter prediction mode of the non-adjacent spatial fusion candidates, thereby making the added non-adjacent spatial fusion candidates more accurate and diverse, which can improve Coding efficiency. The following is a specific example of constructing a fusion candidate list based on the present invention.

It is assumed that the length of the fusion candidate list is N, and N is a preset value, that is, the number of fusion candidates included in the fusion candidate list after construction is N, such as N = 5, 6, 8, 10, and the like. Specifically, in one embodiment, a method for constructing a fusion candidate list includes the following steps;

Step 1: Add the spatial fusion candidate adjacent to the spatial domain of the current block to the fusion candidate list of the current block.

This step is a conventional technique, for example, a method for obtaining a spatial fusion candidate in HEVC and adding the spatial fusion candidate to a fusion candidate list. The spatial fusion neighbors in the spatial domain are the motion information of blocks A, B, C, D, and E in FIG. 9, and the order in which they join the fusion candidate list is A, B, C, D, and E.

Assume that the coordinates of the upper left corner of the current block are P0 = (x0, y0), the width and height of the current block are W and H, and the size of the spatially adjacent A, B, C, D, and E blocks is 4x4, then the A block The upper left corner coordinate of the block is PA = (x0-4, y0 + H-4); the upper left corner coordinate of block B is PB = (x0 + W-4, y0-4); the upper left corner coordinate of block C is PC = ( x0 + W, y0-4); the coordinates of the upper left corner of the D block are PD = (x0-4, y0 + H); the coordinates of the upper left corner of the E block are PE = (x0-4, y0-4). Generally, motion information is stored in a motion vector field in units of 4x4 blocks. The motion information of a block can be found in the motion vector field by the coordinates of the upper left corner of the block. For example, the coordinate of the upper left corner of the block is (x, y), then the coordinate of its corresponding element in the motion vector field is (x >> 2, y >> 2), and ">>" indicates a right shift operation. After obtaining the coordinates of the upper left corner of the spatially adjacent blocks and their corresponding element coordinates in the motion vector length, the corresponding spatially adjacent blocks are found, such as the motion information of blocks A to E, or the motion of blocks A to E Vector information, also known as spatial fusion candidates. These motion information include, but are not limited to, the use of one or two reference picture lists and information about the reference index and motion vector of each list. The first candidate in the fusion candidate list. Is a spatially adjacent block A. In the existing HEVC standard, according to FIG. 9, up to four spatial fusion candidates can be inserted in the merge list in the stated order by sequentially checking A, B, C, D, and E in sequence. The maximum number of spatial fusion candidates allowed in VVC may not be four, it may be more, and of course it may be less, depending on the complexity of the algorithm and the gain that different numbers of spatial fusion candidates can bring. Some additional redundancy checks are performed before using all motion information of neighboring blocks as fusion candidates. These redundancy checks can be divided into two categories for two different purposes: a. Avoiding candidates with redundant motion information in the list; b. Preventing the merging of two redundant syntaxes that can be represented in other ways Partition. When M is the number of spatial fusion candidates, the complete redundancy check will be determined by

Comparison of sub-motion information. In the case of five potential spatial fusion candidates, ten motion information comparisons will be required to ensure that all candidates in the merged list have different motion information. In order to simplify the algorithm, the inspection of redundant motion information has been reduced to a subset, thereby keeping the encoding efficiency while significantly reducing the comparison logic. In the final design, no more than two comparisons are performed for each candidate, resulting in a total of five comparisons. Given the order of {A, B, C, D, E}, the motion information in the C block is only compared with the motion information in the B block, the motion information in the D block is only compared with the motion information in the A block, and The motion information in the E block is only compared with the motion information in the A and B blocks. In addition, a fusion estimation level is also introduced in HEVC, where the fusion candidate list can be obtained independently by checking whether the block containing the fusion candidate is in a merge estimation region (MER), that is, in the same MER The fusion candidate cannot be included in the fusion candidate list, thereby allowing multiple to-be-processed image blocks to perform motion estimation in the fusion mode in parallel to perform corresponding inter prediction.

Step 2: Add the time-domain fusion candidate of the current block to the fusion candidate list of the current block.

This step is an optional step, that is, this step can be selected to be performed or not performed according to requirements. In HEVC, (Sps / slice_temporal_mvp_enabled_flag) controls whether TMVP is enabled or disabled at the sequence level or the picture level. The candidate for the temporal fusion of the current block is usually obtained from the motion information of the image block in the lower right corner of the image block in the reference image of the current block at the same position as the current block. If the image block is not available, it is taken from the reference of the current block The motion information of the center position of the image block in the same position as the current block in the image. For details, please refer to Recommendation ITU-T H.265 | International / Standard ISO / IEC 23008-2 representations of the existing video coding Recommendations, 8.5.3.2 Derivation process vector components.

Step 3: According to the inter prediction mode of the non-adjacent spatial fusion candidate of the current block, add the non-adjacent spatial fusion candidate to the fusion candidate list of the current block.

There are many ways to constitute the non-adjacent spatial fusion candidates of the current block, which is not limited in the present invention. For example, the non-adjacent space fusion candidates are the motion information of A1, B1, C1, D1, E1, A2, B2, C2, D2, E2, F, G, H, I blocks in FIG. 9; The order of joining the fusion candidate list is A1, B1, C1, D1, E1, F, G, H, I, A2, B2, C2, D2, E2. For another example, in FIG. 1, the motion information of A1, B1, C1, D1, E1, A2, B2, C2, D2, and E2 blocks. The order of the non-adjacent spatial fusion candidates to join the fusion candidate list is A1, B1, C1. D1, E1, A2, B2, C2, D2, E2.

Let the coordinates of the upper left corner of the current block be P0 = (x0, y0), and the width and height of the current block are W and H, respectively. Then the upper left corner of the above Ai block (i = 1 or 2) is PAi = (x0-4 -i * Sx, y0 + H-4); the upper left corner of the Bi block is PBi = (x0 + W-4, y0-4-i * Sy); the upper left corner of the Ci block is PCi = (x0 + W , y0-4-i * Sy); the upper-left corner coordinate of the Di block is PDi = (x0-4-i * Sx, y0 + H); the upper-left corner coordinate of the Ei block is PEi = (x0-4-i * Sx , y0-4-i * Sy).

The inter prediction mode of the non-adjacent spatial fusion candidate is the inter prediction mode of the coding unit to which the non-adjacent spatial fusion candidate belongs, or the inter prediction mode of the prediction unit to which the non-adjacent spatial fusion candidate belongs. If the coding unit corresponding to the non-adjacent spatial fusion candidate is not available (for example, the coding unit corresponding to the non-adjacent spatial fusion candidate is not in the current band or the reconstruction has not been completed), or the non-adjacent spatial fusion candidate If the corresponding coding unit is available but does not use the inter prediction mode, this non-adjacent spatial fusion candidate is not available. Unavailable non-adjacent spatial fusion candidates are not added to the fusion candidate list. This is the prior art. For details, please refer to the relevant sections in the existing HEVC standard.

For an available non-adjacent spatial fusion candidate, the process of adding it to the fusion candidate list of the current block according to its inter prediction mode may include one of the following processing methods:

Method 1: If the inter-prediction mode of the non-adjacent spatial fusion candidate is skip / merge mode, this non-adjacent spatial fusion candidate is not added to the fusion candidate list; otherwise, the judgment of the non-adjacent spatial fusion candidate is determined. Whether the motion information is different from the motion information of the fusion candidate at the preset position or is different from the motion information of the fusion candidate already in the fusion candidate list. If this judgment is true, that is, if different, the non-adjacent Spatial fusion candidates are added to the fusion candidate list. To determine whether the two pieces of motion information are different, it can be determined according to whether the two prediction directions, reference frames, and motion vectors of the two pieces of motion information are the same. If any of the above parameters are different, the two pieces of motion information are considered to be different.

Method 2: If the inter-prediction mode of the non-adjacent spatial fusion candidate is an affine transformation prediction mode, the non-adjacent spatial fusion candidate is not added to the fusion candidate list; otherwise, the non-adjacent spatial fusion candidate is judged. Whether the motion information of is different from the motion information of the fusion candidate at the specific position or is different from the motion information of the fusion candidate already in the fusion candidate list, if this judgment holds, that is, if it is different, then this non-adjacent Spatial fusion candidates are added to the fusion candidate list.

Method 3: If the inter-prediction mode of the non-adjacent spatial fusion candidate is the skip / merge mode or the affine transformation prediction mode, then the non-adjacent spatial fusion candidate is not added to the fusion candidate list; otherwise, the non-adjacent spatial fusion candidate is judged Whether the motion information of the fusion candidate in the neighboring space is different from the motion information of the fusion candidate at the preset position or is different from the motion information of the fusion candidate already in the fusion candidate list. If this judgment holds, if it is different, then This non-adjacent spatial fusion candidate is added to the fusion candidate list.

Method 4: If the inter-prediction mode of the non-adjacent spatial fusion candidate is skip / merge mode and the non-adjacent spatial fusion candidate is located in the A2, B2, C2, D2, and E2 blocks, the non-adjacent spatial fusion candidate is not included. The fusion candidate is added to the fusion candidate list; otherwise, it is judged whether the motion information of the non-adjacent spatial fusion candidate is different from the motion information of the fusion candidate at a preset position or whether it is different from the existing fusion candidate in the fusion candidate list. If this judgment is true, if different, then add this non-adjacent spatial fusion candidate to the fusion candidate list.

Method 5: If the inter-prediction mode of the non-adjacent spatial fusion candidate is skip / merge mode and the non-adjacent spatial fusion candidate is located in a specific coding block, the specific block includes: A1, B1, C2 For blocks D2 and E2, this non-adjacent spatial fusion candidate is not added to the fusion candidate list; otherwise, it is determined whether the motion information of the non-adjacent spatial fusion candidate is different from the motion information of the fusion candidate at a preset position or whether Different from the motion information of existing fusion candidates in the fusion candidate list, if this judgment is true, if not, then this non-adjacent spatial fusion candidate is added to the fusion candidate list.

Method 6: If the inter-prediction mode of the non-adjacent space fusion candidate is an affine transformation mode and the non-adjacent space fusion candidate is located in the A2, B2, C2, D2, or E2 block, the non-adjacent space is not replaced. The fusion candidate is added to the fusion candidate list; otherwise, it is judged whether the motion information of the non-adjacent spatial fusion candidate is different from the motion information of the fusion candidate at a preset position or is different from at least one fusion existing in the fusion candidate list. If this judgment is true for the candidate's motion information, if it is different, then this non-adjacent spatial fusion candidate is added to the fusion candidate list.

Method 7: If the inter-prediction mode of the non-adjacent space fusion candidate is an affine transformation mode and the non-adjacent space fusion candidate is located in the A1, B1, C1, D1, and E1 blocks, the non-adjacent space is not replaced. The fusion candidate is added to the fusion candidate list; otherwise, it is judged whether the motion information of the non-adjacent spatial fusion candidate is different from the motion information of the fusion candidate at a preset position or is different from at least one fusion existing in the fusion candidate list. If this judgment is true for the candidate's motion information, if it is different, then this non-adjacent spatial fusion candidate is added to the fusion candidate list.

This step is an optional step, for example, a method for obtaining a bidirectional prediction fusion candidate and a zero motion vector fusion candidate in HEVC and adding them to the fusion candidate list.

When decoding the current block, if the current block uses the skip / merge mode, the merge index is parsed from the code stream, and the fusion candidate corresponding to the merge index is selected from the fusion candidate list constructed by the above method to obtain the motion information of the current block. Perform motion compensation according to the motion information of the current block to obtain a predicted image of the current block. The prediction image of the current block is added to the residual image of the current block to obtain a reconstructed image of the current block, thereby completing decoding of the current block.

Compared with the prior art, the inventive solution determines whether to add non-adjacent spatial fusion candidates to the fusion candidate list according to the inter prediction mode of the non-adjacent spatial fusion candidates, thereby making the added non-adjacent spatial fusion candidates more Accurate and diversified to improve coding efficiency.

FIG. 7 is a flowchart of an example operation for implementing image encoding by applying the fusion candidate list construction method constructed in FIG. 6 in an embodiment of the present invention according to the video decoder 30 shown in FIGS. 1A and 1B. One or more functional units of the video decoder 30 include a prediction processing unit 360, which may be used to perform the method of FIG. In the example of FIG. 7, a picture is decoded based on the fusion candidate list constructed by the method of FIG. 6. The decoding method 700 specifically includes:

S701 Parse the bitstream to obtain the fusion candidate index;

S703 obtains the corresponding fusion candidate from the fusion candidate list according to the fusion candidate index and uses the fusion candidate as the motion information of the current block;

For the method for constructing the fusion candidate list, please refer to FIG. 6 and the corresponding text description in this article.

S705 performs inter prediction on the current block according to the motion information of the current block to obtain a predicted image of the current block;

S707 obtain a residual image of the current block;

S709 adds the predicted image of the current block and the residual image of the current block to obtain a reconstructed image of the current block.

Before step S701 or after step S701 and before step S703, the decoding method 700 further includes: constructing a fusion candidate list of the current block according to the method in FIG. 6.

Compared with the prior art, in the above decoding method, a fusion candidate list construction method for determining whether to add a non-adjacent spatial fusion candidate to a fusion candidate list according to an inter prediction mode of a non-adjacent spatial fusion candidate is used. This makes the added non-adjacent spatial fusion candidates more accurate and diverse, and can improve decoding efficiency.

FIG. 8 is a flowchart of an example operation for implementing image encoding by applying the fusion candidate list construction method constructed in FIG. 6 in an embodiment of the present invention according to the video encoder 20 shown in FIGS. 1A and 1B. One or more functional units of the video encoder 20 include a prediction processing unit 260, which may be used to perform the method of FIG. In the example in FIG. 8, the picture is encoded based on the fusion candidate list constructed by the method in FIG. 6. The decoding method 800 specifically includes:

S801: Perform RDO-based fusion evaluation (Mergeestimation) on the current block based on each fusion candidate in the fusion candidate list, and use the fusion candidate with the lowest rate-distortion value as the motion information of the current block.

S803 encode the current block to form encoded data based on the motion information of the current block;

S805 adds a position index of the fusion candidate with the lowest rate-distortion cost value in the fusion candidate list to the encoded data.

Before step S801 or after step S801 and before step S803, the decoding method 800 further includes: constructing a fusion candidate list of the current block according to the method in FIG. 6.

Compared with the prior art, in the above encoding method, a fusion candidate list construction method for determining whether to add a non-adjacent spatial fusion candidate to a fusion candidate list according to an inter prediction mode of a non-adjacent spatial fusion candidate is used. This makes the added non-adjacent spatial fusion candidates more accurate and diverse, and can improve decoding efficiency.

The embodiment of the present application provides a device for constructing a fusion candidate sportsman list for performing inter prediction, and the device may be a video decoder, a video encoder, or a decoder. Specifically, the apparatus for constructing the fusion candidate athlete list is used to perform the steps performed by the constructing device in the method for constructing the fusion candidate athlete list. The construction apparatus provided in the embodiment of the present application may include a module corresponding to a corresponding step.

In the embodiment of the present application, functional module division may be performed on the construction device for merging candidate athlete list according to the above method example. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one process. Module. The above integrated modules may be implemented in the form of hardware or software functional modules. The division of the modules in the embodiments of the present application is schematic, and is only a logical function division. In actual implementation, there may be another division manner.

In a case where each functional module is divided according to each function, FIG. 11 illustrates a possible schematic structural diagram of a device for constructing a fusion candidate sportsman list for performing inter prediction in the foregoing embodiment. As shown in FIG. 11, a device 1100 for constructing a fusion candidate athlete list includes an airspace adjacent fusion candidate acquisition module 1101, an airspace non-adjacent fusion candidate acquisition module 1103, and an optional time domain fusion candidate acquisition module 1105 and Extended fusion candidate acquisition module 1107.

A spatial domain neighboring fusion candidate acquisition module 1101 is configured to add, based on a preset first selection rule, the motion information of a neighboring block adjacent to the current domain in the spatial domain as a spatial fusion candidate of the current block to the current block. In the fusion candidate list, wherein the preset first selection rule includes when the neighboring block is available and the motion information of the neighboring block is different from the motion information of the neighboring block at a specific position, Adding the motion information of the neighboring block as a spatial fusion candidate of the current block to the fusion candidate list of the current block;

A spatial domain non-adjacent fusion candidate acquisition module 1103 is configured to, based on a preset second selection rule, use motion information of a non-adjacent block that is not adjacent to the current domain's spatial domain as the non-adjacent space of the current block. A fusion candidate is added to the fusion candidate list, wherein the current block has one or more non-adjacent blocks that are not adjacent in the airspace, and the preset second selection rule includes: when the non-phase If neighboring blocks are available, it is determined that the inter prediction mode of at least one non-adjacent block among the non-adjacent blocks is not a predetermined inter prediction mode, and the motion information and a specific position of the at least one non-adjacent block are When the motion information of a neighboring block or a non-adjacent block is the same, adding the motion information of the at least one non-adjacent block as a non-adjacent spatial fusion candidate of the current block to the fusion candidate list, When it is determined that the inter prediction mode of at least one non-adjacent block in the non-adjacent block is a predetermined inter-prediction mode when the non-adjacent block is available, the at least one non-adjacent block is not Non-adjacent space The fusion candidate is added to the fusion candidate list of the current block.

Optionally, a time-domain fusion candidate acquisition module 1105 is configured to use the motion information of the neighboring block in the lower right corner of the co-located block at the same position in the reference frame as the time-domain fusion candidate of the current block. Or adding the motion information of the center point of the co-located block as the time domain of the current block if the motion information of an adjacent block in the lower right corner of the co-located block is not available A fusion candidate is added to the fusion candidate list.

Optionally, the fusion candidate acquisition module 1107 is extended, and the bidirectional prediction fusion candidate is added to the fusion candidate list when the current block belongs to a bi-predictive slice, or, After the bidirectional prediction fusion candidate is added to the fusion candidate list, the number of the fusion candidates in the fusion candidate list does not reach the predetermined value, then a zero motion vector fusion candidate is added to the fusion candidate list. Adding the fusion candidate list to the fusion candidate list; or adding the zero motion vector fusion candidate to the fusion candidate list when the current block belongs to a uni-predictive slice.

The invention also provides a decoding device, which may be a video decoder. Specifically, the decoding apparatus is configured to perform the decoding method described in FIG. 7. The decoding apparatus provided in the embodiment of the present application may include a module corresponding to a corresponding step.

In a case where each functional module is divided corresponding to each function, FIG. 12 shows a schematic structural diagram of an encoding device involved in the foregoing embodiment. As shown in FIG. 12, the encoding device 1200 includes: a fusion candidate athlete list constructing device 1201, a code stream analysis module 1203, a motion information acquisition module 1205, a prediction module 1207, a residual acquisition module 1209, and a reconstruction module 1211.

The fusion candidate athlete list construction device 1201 has the same function as the fusion candidate athlete list construction device 1100 in FIG. 11, and is configured to obtain a fusion candidate list of a current block;

Code stream parsing module 1203, configured to parse the code stream to obtain the fusion candidate index;

A motion information acquisition module 1205, configured to acquire a corresponding fusion candidate from the fusion candidate list according to the fusion candidate index and use the fusion candidate as the motion information of the current block;

A prediction module 1207, configured to perform inter prediction on the current block to obtain a predicted image of the current block according to the motion information of the current block;

A residual acquisition module 1209, configured to acquire a residual image of the current block;

A reconstruction module 1207 is configured to add the predicted image of the current block and the residual image of the current block to obtain a reconstructed image of the current block.

The invention also provides an encoding device, which may be a video encoder. Specifically, the decoding apparatus is configured to perform the decoding method described in FIG. 8. The encoding device provided in the embodiment of the present application may include a module corresponding to a corresponding step.

In a case where each functional module is divided corresponding to each function, FIG. 13 shows a schematic structural diagram of an encoding device involved in the foregoing embodiment. As shown in FIG. 13, the encoding device 1300 includes: a fusion candidate athlete list constructing device 1301, a motion information determination module 1303, a predictive encoding module 1305, and a fusion candidate index encoding module 1307.

The fusion candidate athlete list construction device 1301 has the same function as the fusion candidate athlete list construction device 1100 in FIG. 11 for constructing a fusion candidate list of the current block;

A motion information determination module 1303 is configured to perform an RDO-based fusion evaluation (Mergeestimation) on the current block based on each fusion candidate in the fusion candidate list, and use the fusion candidate with the smallest value of rate distortion as the fusion candidate Motion information of the current block;

A predictive encoding module 1305, configured to encode the current block to form encoded data based on the motion information of the current block; and

The fusion candidate index coding module 1307 adds a position index of the fusion candidate with the lowest rate-distortion cost value in the fusion candidate list to the encoded data.

In addition, the solution of the present invention can also be implemented in the form of a processor, that is, a memory, that is, an executable instruction set is stored in the memory, and a digital processor can execute the instruction set in the memory to implement the method shown in FIG. 6-8. Either way. Specifically, as shown in FIG. 14, the device described in FIG. 14 may be used as a decoding device 1400, an encoding device 1500, or a fusion candidate list construction device 1600 for performing inter prediction.

When the device is used as the decoding device 1400, it includes a digital processor 1401 and a memory 1403, in which an executable instruction set is stored, and the digital processor reads and stores the stored instruction set in the memory. The instruction set is used to implement the decoding method described in FIG.

When the device is used as the encoding device 1500, it includes a digital processor 1501 and a memory 1503, in which an executable instruction set is stored, and the digital processor reads and stores in the memory The instruction set is used to implement the decoding method described in FIG.

When the device is used as a fusion candidate list construction device 1600 for performing inter prediction, it includes a digital processor 1601 and a memory 1603, in which an executable instruction set is stored, and the digital processing The processor reads the instruction set stored in the memory for implementing the decoding method described in FIG. 6. In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium or a communication medium including any medium that facilitates transfer of a computer program from one place to another, according to a communication protocol, for example. . In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures used to implement the techniques described in this disclosure. The computer program product may include a computer-readable medium.

By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, flash memory, or may be used to store instructions or data structures Any other media that requires program code and is accessible by the computer. Also, any connection is properly termed a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, and microwave is used to transmit instructions from a website, server, or other remote source, then Coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. It should be understood, however, that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are actually directed to non-transitory tangible storage media. As used herein, magnetic disks and compact discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), flexible discs and Blu-ray discs, where the discs are usually magnetic The data is reproduced, while the optical disk uses a laser to reproduce the data optically. Combinations of the above should also be included within the scope of computer-readable media.

The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits , ASIC), field programmable logic array (field programmable logic arrays, FPGA) or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules for encoding and decoding, or incorporated in a composite codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a variety of devices or devices that include a wireless handset, an integrated circuit (IC), or a collection of ICs (eg, a chipset). The present disclosure describes various components, modules, or units to emphasize functional aspects of the device for performing the disclosed techniques, but does not necessarily need to be implemented by different hardware units. Specifically, as described above, the various units may be combined in a codec hardware unit in combination with suitable software and / or firmware, or provided by a collection of interoperable hardware units, which include as described above One or more processors.

Claims

A method for constructing a fusion candidate list for performing inter prediction, which is characterized by:

Based on a preset first selection rule, motion information of an adjacent block adjacent to the current block spatial domain is added as a spatial fusion candidate of the current block to the fusion candidate list of the current block, where the preset The first selection rule of includes that when the neighboring block is available and the motion information of the neighboring block is different from the motion information of the neighboring block at a specific position, using the motion information of the neighboring block as the A spatial fusion candidate of the current block is added to the fusion candidate list of the current block;

Based on a preset second selection rule, motion information of a non-adjacent block that is not adjacent to the current block spatial domain is added to the fusion candidate list as a non-adjacent spatial fusion candidate for the current block, where: The current block has one or more non-adjacent blocks that are not adjacent in the airspace, and the preset second selection rule includes: judging the non-adjacent block when the non-adjacent block is available. When the inter-prediction mode of at least one non-adjacent block in the is not a predetermined inter-prediction mode, and the motion information of the at least one non-adjacent block is the same as the motion information of an adjacent block or a non-adjacent block at a specific position , The motion information of the at least one non-adjacent block is added to the fusion candidate list as a non-adjacent spatial fusion candidate of the current block, and the judgment is made when the non-adjacent block is available When the inter prediction mode of at least one non-adjacent block among the non-adjacent blocks is a predetermined inter-prediction mode, the non-adjacent spatial fusion candidate of the at least one non-adjacent block is not added to the current block. Fusion candidate list .
The method according to claim 1, wherein, in the second selection rule based on the preset, motion information of a non-adjacent block that is not adjacent to the current block spatial domain is used as a non-phase of the current block. Before the neighboring space fusion candidate is added to the fusion candidate list of the current block, the method further includes:

Adding motion information of a neighboring block in the lower right corner of a co-located block in a reference frame of the current block as a time-domain fusion candidate of the current block to the fusion candidate list, where the co-located block is in the reference The position in the frame is the same as the current position of the current block, or if the motion information of the adjacent block in the lower right corner of the co-located block is not available, the motion information of the center point of the co-located block is used as the The time-domain fusion candidate of the current block is added to the fusion candidate list.
The method according to claim 1 or 2, wherein, in the second selection rule based on the preset, motion information of a non-adjacent block that is not adjacent to the current block spatial domain is used as the current block's After a non-adjacent spatial fusion candidate is added to the fusion candidate list of the current block, and the number of the fusion candidates in the fusion candidate list does not reach a predetermined value, the method further includes: Adding a bidirectional prediction fusion candidate to the fusion candidate list when the current block belongs to a bi-predictive slice, or adding the bidirectional prediction fusion candidate to the fusion candidate list After that, if the number of the fusion candidates in the fusion candidate list does not reach the predetermined value, a zero motion vector fusion candidate is added to the fusion candidate list; or when the current block belongs to a single block When predicting a slice (uni-predictive slice), a zero motion vector fusion candidate is added to the fusion candidate list.
The method according to claim 1 or 2, wherein the predetermined inter prediction mode is a skip / merge mode.
The method according to claim 1 or 2, wherein the predetermined inter prediction mode is an Affine transform prediction mode.
The method according to claim 1 or 2, wherein the predetermined inter prediction mode is a skip / merge mode and an affine transform prediction mode.
The method according to any one of claims 1 to 6, wherein at least one non-adjacent block among the non-adjacent blocks includes A2, B2, C2, D2, and E2 blocks, wherein the current block The upper left corner coordinate of P0 = (x0, y0), the width and height of the current block are W and H, and the upper left corner coordinate of the A2 block is PA2 = (x0-4-2 * Sx, y0 + H -4), the coordinates of the upper left corner of the B2 block are PB2 = (x0 + W-4, y0-4-2 * Sy), and the coordinates of the upper left corner of the C2 block are PC2 = (x0 + W, y0-4 -2 * Sy), the upper-left corner coordinate of the D2 block is PD2 = (x0-4-2 * Sx, y0 + H), and the upper-left corner coordinate of the E2 block is PE2 = (x0-4-2 * Sx , y0-4-2 * Sy).
The method according to any one of claims 1 to 6, wherein at least one non-adjacent block of the non-adjacent blocks includes A1, B1, C2, D2, and E2 blocks, and an upper left of the current block The angular coordinate is P0 = (x0, y0), the width and height of the current block are W and H, and the upper-left corner coordinate of the A1 block is PA1 = (x0-4-Sx, y0 + H-4), The upper left corner coordinate of the B1 block is PB1 = (x0 + W-4, y0-4-Sy), and the upper left corner coordinate of the C2 block is PC2 = (x0 + W, y0-4-2 * Sy), The upper left corner coordinate of the D2 block is PD2 = (x0-4-2 * Sx, y0 + H), and the upper left corner coordinate of the E2 block is PE2 = (x0-4-2 * Sx, y0-4-2 * Sy).
The method according to any one of claims 1 to 6, wherein at least one non-adjacent block of the non-adjacent blocks is A1, B1, C1, D1, E1 blocks, and an upper left of the current block The angular coordinate is P0 = (x0, y0), and the width and height of the current block are W and H, respectively, then the upper left corner coordinate of the A1 block is PA1 = (x0-4-Sx, y0 + H-4) The coordinates of the upper left corner of the B1 block are PB1 = (x0 + W-4, y0-4-Sy), and the coordinates of the upper left corner of the C1 block are PC2 = (x0 + W, y0-4-Sy), so The upper left coordinate of the D1 block is PD1 = (x0-4-Sx, y0 + H), and the upper left coordinate of the E1 block is PE1 = (x0-4-Sx, y0-4-Sy).
A decoding method, wherein the fusion candidate list constructed by the method according to any one of claims 1 to 9 is applied to a decoding process of the current block, and the decoding process includes:

Constructing the fusion candidate list of the current block according to the method according to any one of claims 1 to 9;

Parse the bitstream to obtain the fusion candidate index;

Obtaining a corresponding fusion candidate from the fusion candidate list according to the fusion candidate index and using the fusion candidate as the motion information of the current block;

Performing inter prediction on the current block according to the motion information of the current block to obtain a predicted image of the current block;

Acquiring a residual image of the current block;

Adding the predicted image of the current block and the residual image of the current block to obtain a reconstructed image of the current block.
A coding method, characterized in that the fusion candidate list constructed by the method according to any one of claims 1 to 9 is applied to a coding process of the current block, and the coding process includes:

Constructing the fusion candidate list of the current block according to the method according to any one of claims 1 to 9; performing RDO-based fusion on the current block based on each fusion candidate in the fusion candidate list Evaluate (Merge Estimation) and use the fusion candidate with the lowest value of rate distortion as the motion information of the current block;

Encoding the current block based on the motion information of the current block to form encoded data;

A position index of the fusion candidate with the lowest rate-distortion cost value in the fusion candidate list is appended to the encoded data.
An apparatus for constructing a fusion candidate list for performing inter prediction, which includes:

The spatial domain neighboring fusion candidate acquisition module is configured to add, based on a preset first selection rule, motion information of a neighboring block adjacent to the current domain in the spatial domain as a spatial fusion candidate of the current block to the current block. In the fusion candidate list, the preset first selection rule includes that when the neighboring block is available and the motion information of the neighboring block is different from that of the neighboring block at a specific position, then Adding the motion information of the neighboring block as a spatial fusion candidate of the current block to the fusion candidate list of the current block;

A spatial domain non-adjacent fusion candidate acquisition module is configured to, based on a preset second selection rule, use motion information of a non-adjacent block that is not adjacent to the current domain's spatial domain as the non-adjacent spatial fusion of the current block. Candidates are added to the fusion candidate list, wherein the current block has one or more non-adjacent blocks that are not adjacent in the airspace, and the preset second selection rule includes: when the non-adjacent blocks are If a block is available, it is determined that the inter prediction mode of at least one non-adjacent block in the non-adjacent block is not a predetermined inter-prediction mode, and the motion information of the at least one non-adjacent block is related to a specific position. When the motion information of an adjacent block or a non-adjacent block is the same, the motion information of the at least one non-adjacent block is added as a non-adjacent spatial fusion candidate of the current block to the fusion candidate list. If the non-adjacent block is available, determine that the inter-prediction mode of at least one non-adjacent block in the non-adjacent block is a predetermined inter-prediction mode, and then do not change the at least one non-adjacent block Non-adjacent space fusion Candidate block by fusion to join the current candidate list.
The apparatus according to claim 12, further comprising:

A time-domain fusion candidate acquisition module is configured to add motion information of a neighboring block in a lower-right corner of a co-located block at the same position in a reference frame as a time-domain fusion candidate of the current block to join the fusion. In the candidate list, or if the motion information of the adjacent block in the lower right corner of the co-located block is not available, the motion information of the center point of the co-located block is added as a time-domain fusion candidate for the current block. The fusion candidate list is described above.
The method according to claim 12 or 13, wherein the device further comprises:

Extending a fusion candidate acquisition module to add a bidirectional prediction fusion candidate to the fusion candidate list when the current block belongs to a bi-predictive slice; or, adding the bidirectional prediction fusion candidate to the list After the candidate is added to the fusion candidate list, the number of the fusion candidates in the fusion candidate list does not reach the predetermined value, then a zero motion vector fusion candidate is added to the fusion candidate list Medium; or adding a zero motion vector fusion candidate to the fusion candidate list when the current block belongs to a uni-predictive slice.
The apparatus according to claim 12 or 13, wherein the predetermined inter prediction mode is a skip / merge mode.
The apparatus according to claim 12 or 13, wherein the predetermined inter prediction mode is an Affine transform prediction mode.
The apparatus according to claim 12 or 13, wherein the predetermined inter prediction mode is a skip / merge mode or an affine transformation prediction mode.
The apparatus according to any one of claims 12 to 17, wherein at least one of the non-adjacent blocks is an A2, B2, C2, D2, or E2 block, and the current block The upper-left corner coordinate of P0 = (x0, y0), and the width and height of the current block are W and H, respectively, then the upper-left corner coordinate of the A2 block is PA2 = (x0-4-2 * Sx, y0 + H-4); the upper left corner coordinate of the B2 block is PB2 = (x0 + W-4, y0-4-2 * Sy); the upper left corner coordinate of the C2 block is PC2 = (x0 + W, y0- 4-2 * Sy); the upper-left corner coordinate of the D2 block is PD2 = (x0-4-2 * Sx, y0 + H); the upper-left corner coordinate of the E2 block is PE2 = (x0-4-2 * Sx, y0-4-2 * Sy).
The apparatus according to any one of claims 12 to 17, wherein at least one of the non-adjacent blocks is an A1, B1, C2, D2, or E2 block, and an upper left of the current block The angular coordinate is P0 = (x0, y0), and the width and height of the current block are W and H, respectively, then the upper left corner coordinate of the A1 block is PA1 = (x0-4-Sx, y0 + H-4) ; The upper left corner coordinate of the B1 block is PB1 = (x0 + W-4, y0-4-Sy); the upper left corner coordinate of the C2 block is PC2 = (x0 + W, y0-4-2 * Sy) ; The upper left corner coordinate of the D2 block is PD2 = (x0-4-2 * Sx, y0 + H); the upper left corner coordinate of the E2 block is PE2 = (x0-4-2 * Sx, y0-4- 2 * Sy).
The device according to any one of claims 12 to 17, wherein at least one non-adjacent block of the non-adjacent blocks is A1, B1, C1, D1, E1 blocks, and an upper left of the current block The angular coordinate is P0 = (x0, y0), and the width and height of the current block are W and H, respectively, then the upper left corner coordinate of the A1 block is PA1 = (x0-4-Sx, y0 + H-4) ; The upper left corner coordinate of the B1 block is PB1 = (x0 + W-4, y0-4-Sy); the upper left corner coordinate of the C1 block is PC2 = (x0 + W, y0-4-Sy); The upper left coordinate of the D1 block is PD1 = (x0-4-Sx, y0 + H); the upper left coordinate of the E1 block is PE1 = (x0-4-Sx, y0-4-Sy).
A decoding device, characterized in that the device for constructing and obtaining the fusion candidate list according to any one of claims 12 to 20 is applied to the decoding device to decode the current block, so The decoding device includes:

The fusion candidate list construction device according to any one of claims 12-20, configured to obtain a fusion candidate list of a current block;

Code stream parsing module, configured to parse the code stream to obtain the fusion candidate index;

A motion information acquisition module, configured to acquire a corresponding fusion candidate from the fusion candidate list according to the fusion candidate index and use the fusion candidate as the motion information of the current block;

A prediction module, configured to perform inter prediction on the current block according to the motion information of the current block to obtain a predicted image of the current block;

A residual acquisition module, configured to acquire a residual image of the current block;

A reconstruction module is configured to add the predicted image of the current block and the residual image of the current block to obtain a reconstructed image of the current block.
An encoding device, characterized in that the device for constructing and obtaining the fusion candidate list according to any one of claims 12 to 20 is applied to the encoding device to encode most current blocks. The decoding device includes:

The apparatus for constructing a fusion candidate list according to any one of claims 12 to 20, configured to obtain a fusion candidate list of a current block; a motion information determination module, configured to, based on the information in the fusion candidate list, Each fusion candidate performs RDO-based fusion evaluation (Merge Estimation) on the current block and uses the fusion candidate with the lowest value of rate distortion as the motion information of the current block;

A predictive encoding module, configured to encode the current block to form encoded data based on the motion information of the current block; and

The fusion candidate index encoding module adds a position index of the fusion candidate with the lowest rate-distortion value in the fusion candidate list to the encoded data.
A decoding device includes a digital processor and a memory, and an executable instruction set is stored in the memory, and the digital processor reads the instruction set stored in the memory for implementing the method as claimed in claim 10. Described decoding method.
An encoding device includes a digital processor and a memory, and an executable instruction set is stored in the memory, and the digital processor reads the instruction set stored in the memory for implementing the method as claimed in claim 11. The encoding method described.