JP2023017981A - Video processing method, encoder, non-transitory computer-readable storage medium, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image motion compensation method and an image motion compensation apparatus for improving encoding/decoding efficiency.

SOLUTION: An image processing method includes specifying a first initial motion vector and a second initial motion vector of a current image block. The first initial motion vector points to a first reference image which is a forward frame of a current image including the current image block. The second initial motion vector points to a second reference image which is a backward frame of the current image. A distance between the first reference image and the current image is equal to a distance between the second reference image and the current image. The method further includes performing prediction on the current image block according to a weighted image block obtained by performing weighted summation on the first reference image block and the second reference image block if the first and second reference images are short-term reference images. If the first and second reference images are long-term reference images, the specified operation is not executed.

SELECTED DRAWING: Figure 5

COPYRIGHT: (C)2023,JPO&INPIT

Description

COPYRIGHT STATEMENT The material disclosed by this patent document contains material that is protected by copyright. Such copyright is owned by the copyright owner. The copyright owner has no objection to the facsimile reproduction by any person of the patent document or the disclosure of the patent as it appears in the official records and archives of the Patent and Trademark Office.

The present application relates to the field of image processing, and more particularly to image motion compensation methods and apparatus.

Over the past few years, the popularity of portable, handheld, and wearable devices has continued to increase the volume of moving images. As motion picture formats become more complex, motion picture storage and transmission also become more difficult. To reduce the bandwidth occupied by video storage and transmission, video data is typically encoded and compressed at the encoding end and decoded at the decoding end.

The coding compression process includes processes such as prediction, transform, quantification, and entropy coding. Here, prediction includes two types of intra-frame prediction and inter-frame prediction, which aims to remove redundant information of the current image block to be coded using prediction block data. Intra-frame prediction acquires predicted block data using information of the current frame image. Inter-frame prediction uses information from a reference image to obtain prediction block data. The process divides the current image to be coded into multiple image blocks to be coded, and divides the image block into multiple sub-blocks. divided into image blocks, then for each sub-image block, search the image block that best matches the current sub-image block in the reference image as a predicted image block; The displacement is the motion vector. After that, the corresponding pixel values of this sub-image block and the predicted image block are subtracted to obtain a residual. The residuals corresponding to each sub-image block obtained are combined together to obtain the residual of the image block to be coded. The residual is after processing such as transformation, quantification, entropy coding, etc. to obtain an entropy coded bitstream, and an entropy coded bitstream and e.g. intra prediction mode, motion vector (or motion vector residual) , or the like, or transmitted to the decoding end.

At the decoding end of the image, after obtaining the entropy-encoded bitstream, entropy decoding is performed to obtain the corresponding residual, and the image to be decoded is determined based on information such as the motion vector and intra-frame prediction obtained by decoding. A prediction image block corresponding to the block is searched, and the value of each pixel point in the decoding target image block is obtained based on the prediction image block and the residual.

From the above explanation, when inter-frame prediction is performed, the more similar the selected reference image and the current image to be encoded are, the smaller the residual generated by inter-frame prediction becomes. I know you can raise it. Specifically, some existing techniques can use each image in the moving image to construct a high-quality specific reference image of the background content, including the scene. When inter-frame prediction is performed, the background part of the current image to be coded or the current image to be decoded is used to reduce the residual information of inter-frame prediction by referring to the high-quality specific reference image. , which increases the coding efficiency. That is, this specific reference image is a reference image for inter-frame prediction. A long-term reference image is an artificially constructed image rather than a decoded image. A long-term reference picture contains multiple image blocks, any of which are derived from a decoded picture, and different image blocks in the long-term reference picture are derived from different decoded pictures. there is a possibility.

In order to increase the coding efficiency and reduce the amount of information transmitted by the coding end, some existing techniques can directly derive motion vectors at the decoding end. The encoding end does not need to transmit motion vector information or motion vector residual information, and the decoding end can also obtain the true motion vector without decoding the motion vector information or motion vector residual information.

Some existing techniques that use motion vector derivation and techniques that use bi-directional motion prediction do not take into account the peculiarities of long-term reference pictures. Some techniques that use motion vector derivation do not consider whether the reference image to which the motion vector points is a long-term reference image, and it is possible to perform motion search in the long-term reference image when performing motion vector correction. This in turn reduces retrieval efficiency and coding efficiency. Techniques using bi-directional motion estimation operate on motion vectors based on the temporal correlation of the images, and if the reference image to which the associated motion vector is directed is the long-term reference image, the current image to be encoded, or the current Since the definition of the temporal distance between the image to be decoded and the long-term reference image is unclear, these operations may be invalid.

The present application provides an image motion compensation method and an image motion compensation device, which can enhance encoding/decoding efficiency.

In a first aspect, an image motion compensation method is provided, the method comprising:
obtaining an initial motion vector MV for the current image block;
performing motion compensation on the current image block based on the initial MV, if the reference image to which the initial MV is directed is a specific reference image;
If the reference image to which the initial MV is directed is a non-specific reference image, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV. including

In a second aspect, there is provided an image motion compensation device, the device comprising:
at least one memory for storing computer executable commands;
and at least one processor used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, the operations comprising:
obtaining an initial motion vector MV for the current image block;
performing motion compensation on the current image block based on the initial MV, if the reference image to which the initial MV is directed is a specific reference image;
If the reference image to which the initial MV is directed is a non-specific reference image, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV. That is.

In a third aspect, there is provided a computer readable storage medium on which commands are stored which, when run on the computer, cause the computer to perform the image motion compensation method according to the first aspect.

In a fourth aspect there is provided an encoding device comprising the image motion compensation apparatus according to the second aspect.

In a fifth aspect there is provided a decoding device comprising the image motion compensation apparatus according to the second aspect.

In a sixth aspect, there is provided an image motion compensation method, the method comprising:
obtaining an initial motion vector MV for the current image block;
determining a scaling ratio of the initial MV, wherein the scaling ratio of the initial MV is 1 if the initial MV is directed to a particular reference image;
scaling the initial MV based on a scaling ratio of the initial MV;
performing motion compensation on the current image block based on the scaled MV.

In a seventh aspect, there is provided an image motion compensation device, the device comprising:
at least one memory for storing computer executable commands;
and at least one processor used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, the operations comprising:
obtaining an initial motion vector MV for the current image block;
determining a scaling ratio of the initial MV, wherein the scaling ratio of the initial MV is 1 if the initial MV is directed to a particular reference image;
scaling the initial MV based on a scaling ratio of the initial MV;
and performing motion compensation on the current image block based on the scaled MV.

In an eighth aspect, there is provided a computer-readable storage medium on which commands are stored and which, when run on the computer, cause the computer to perform the image motion compensation method according to the sixth aspect.

In a ninth aspect there is provided an encoding device comprising the image motion compensation apparatus according to the seventh aspect.

In a tenth aspect there is provided a decoding device comprising the image motion compensation apparatus according to the seventh aspect.

In an eleventh aspect, there is provided an image processing method, the method comprising:
obtaining a first initial motion vector MV and a second initial MV, the first initial MV pointing to a first reference image and the second initial MV pointing to a second reference image;
when at least one of the first reference image and the second reference image is a specific reference image, predicting an image block of the current image block based on the first initial MV and the second initial MV; calculated and obtained,
When both the first reference image and the second reference image are non-specific reference images, the current image based on the gradient values of the pixel points directed by the first initial MV and the second initial MV calculating and obtaining the MV of the block, and calculating and obtaining a predicted image block of the current image block based on the MV of the current image block.

In a twelfth aspect, there is provided an image processing device, the device comprising:
at least one memory for storing computer executable commands;
and at least one processor used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, the operations comprising:
obtaining a first initial motion vector MV and a second initial MV, the first initial MV pointing to a first reference image and the second initial MV pointing to a second reference image;
when at least one of the first reference image and the second reference image is a specific reference image, predicting an image block of the current image block based on the first initial MV and the second initial MV; calculated and obtained,
When both the first reference image and the second reference image are non-specific reference images, the current image based on the gradient values of the pixel points directed by the first initial MV and the second initial MV calculating and obtaining the MV of the block, and calculating and obtaining a predicted image block of the current image block based on the MV of the current image block.

In a thirteenth aspect, there is provided a computer-readable storage medium in which commands are stored and which, when run on the computer, cause the computer to perform the image processing method according to the eleventh aspect.

A fourteenth aspect provides an encoding device comprising an image processing apparatus according to the twelfth aspect.

A fifteenth aspect provides a decoding device comprising an image processing apparatus according to the twelfth aspect.

1 is a schematic flow chart of an image motion compensation method of one embodiment of the present application; 1 is a principle schematic diagram of a two-way matching method of one embodiment of the present application; FIG. 1 is a principle schematic diagram of a template matching method of one embodiment of the present application; FIG. 1 is a schematic diagram of the DMVR technology of one embodiment of the present application; FIG. 4 is a schematic flowchart of an image processing method according to another embodiment of the present application; 1 is a schematic diagram of the principle of BIO technology in one embodiment of the present application; FIG. 1 is a schematic frame diagram of an image motion compensation device of one embodiment of the present application; FIG. 1 is a schematic frame diagram of an image processing apparatus of one embodiment of the present application; FIG. 4 is a schematic flow chart of an image motion compensation method of another embodiment of the present application; FIG. 4 is a schematic frame diagram of an image motion compensation device of another embodiment of the present application;

Technical solutions in the embodiments of the present application will be described below with reference to the drawings.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art of this application. As used herein, the technical terms used in the specification of the present application are for the purpose of describing particular embodiments only and are not intended to limit the present application.

First, related technologies and concepts related to the embodiments of the present application will be described.

A moving image consists of a plurality of images. When encoding/decoding video, different images in the video can employ different prediction methods. The prediction method adopted based on the image can divide the image into an intra-frame predicted image and an inter-frame predicted image, where the inter-frame predicted image includes a forward predicted image and a bi-directionally predicted image. . The I image is an intra-frame predicted image and is also called a key frame, and the P image is a forward predicted image. do. The B-picture is a bi-directional predictive picture, and the preceding and succeeding pictures are used as reference pictures. In one embodiment, the encoding/decoding end generates a group of pictures (GOP) step by step after encoding/decoding a plurality of pictures, and the GOP is a single I picture. , and a group of pictures consisting of at least one of a plurality of B-pictures (or bi-directionally predicted pictures) and P-pictures (or forward-predicted pictures). The decoding end reads and decodes the GOPs one by one during broadcasting, and then reads and renders the screen.

In modern video encoding/decoding standards, an image can be divided into multiple small blocks and encoded/decoded for images of different resolutions, i.e. an image can be divided into multiple image blocks. An image can be divided into any number of image blocks. For example, the image can be divided into an m×n image block matrix. An image block can have a rectangular shape, a square shape, a circular shape, or any other shape. An image block can have any size, eg, p×q pixels. Each image block can have at least one of the same size and shape. Alternatively, the two or more image blocks can have at least one of different sizes and shapes. Image blocks may or may not have any overlap. In some embodiments, this image block is called a macroblock or Largest Coding Unit (LCU) in some encoding/decoding standards. H. For the H.264 standard, this image block is called a macroblock and may be 16×16 pixels in size. For the High Efficiency Video Coding (HEVC) standard, an image block is called the largest coding unit and may be 64×64 pixels in size.

In some other embodiments, an image block may not be a macroblock or largest coding unit, but includes part of a macroblock or largest coding unit, or contains at least two complete macroblocks (or largest coding units). coding unit), or at least one complete macroblock (or maximum coding unit) and part of one macroblock (or maximum coding unit), or at least two complete macroblocks (or largest coding unit) and some macroblocks (or part of the largest coding unit). Thus, after an image is divided into a plurality of image blocks, these image blocks can be encoded/decoded respectively in the image data.

The coding process includes processes such as prediction, transform, quantification, and entropy coding. Here, prediction includes two types of intra-frame prediction and inter-frame prediction, which aims to remove redundant information of the current image block to be coded using prediction block data. Intra-frame prediction acquires predicted block data using information of the current frame image. Inter-frame prediction uses information from a reference image to obtain prediction block data. The process divides the current image to be coded into multiple image blocks to be coded, and divides the image block into multiple sub-blocks. divided into image blocks, then for each sub-image block, search the image block that best matches the current sub-image block in the reference image as a predicted image block; The displacement is the motion vector. After that, the corresponding pixel values of this sub-image block and the predicted image block are subtracted to obtain a residual. The residuals corresponding to each sub-image block obtained are combined together to obtain the residual of the image block to be coded.

In each of the embodiments of the present application, the transformation matrix allows the residual relevance of the image block to be disassociated, ie the redundant information of the image block is removed, thereby increasing the coding efficiency. The transformation of the data block in the image block usually adopts a two-dimensional transformation, that is, the encoding end multiplies the residual information of the data block with an N×M transformation matrix and its transposed matrix respectively, and after multiplication, The result is a transform coefficient. After quantifying the transform coefficients, quantified coefficients are obtained, and finally the quantified coefficients are entropy coded to obtain an entropy coded bitstream. The entropy-encoded bitstream and encoded encoding mode information, such as intra-frame prediction mode, motion vector (or motion vector residual) information, etc., are stored or transmitted to the decoding end.

At the image decoding end, entropy encoding is performed after obtaining the entropy-encoded bitstream to obtain the corresponding residual. A predicted image block corresponding to the image block is searched based on information such as the motion vector and intra-frame prediction obtained by decoding. Obtain the value of each pixel point in the current sub-image block based on the predicted image block and the residual.

In the previous sentence, it was mentioned that an encoded/decoded image is used as a reference image for current encoding/decoding. In some embodiments, further construction of the reference image can increase the similarity between the reference image and the current image to be coded/decoded.

As an example, if there is a particular encoding/decoding scene within the video content and the background in this scene is essentially unchanged, only the foreground within the video will change or move. Video monitoring, for example, belongs to this type of scene. It can be recognized that in a video monitoring scene, the monitoring camera is usually fixed and does not move, or only moves slowly, and the background remains basically unchanged. On the other hand, it can be recognized that objects such as people or vehicles captured by a video monitoring camera always move or change, and the foreground always changes. In this type of scene, a specific reference image can be constructed, which contains only high quality background information. The specific reference image may contain multiple image blocks, any image block being extracted from a decoded image, and different image blocks in the specific reference image being different decoded image blocks. It may be from a modified image. When inter-frame prediction is performed, the background portion of the current encoding/decoding target image can refer to this specific reference image, which can be used to reduce the residual information of inter-frame prediction, thereby increase decoding/decoding efficiency.

The above is a specific example of a specific reference image. In some embodiments, the specific reference image has at least one of the following properties. A composite reference, a long-term reference image, and a non-output image. For example, this particular reference image may be a synthesized long-term reference image, or may be a synthesized frame that is not output, or may be a long-term reference image that is not output, and so on. In some embodiments, synthetic frames are referred to as synthetic reference frames.

In some embodiments, a non-specific reference image may be a reference image that does not have at least one of the following properties. A synthetic frame, a long-term reference image, and a non-output reference image. For example, the specific reference image may include a reference image other than a synthesized frame, or may include a reference image other than a long-term reference image, or may include a reference image other than a reference image that is not output, or It can include reference images other than the synthesized long-term reference image, or it can include reference images other than synthesized frames that are not output, or it can include reference images other than the output long-term reference image, and so on.

In some embodiments, if an image in a video can be a reference image, it can be divided into a long-term reference image and a short-term reference image. Here, the short-term reference image is a concept corresponding to the long-term reference image. A short-term reference image exists in a certain time zone in the buffer area of the reference image, and after the decoded reference image after passing through this short-term reference image undergoes a plurality of input/output operations in the buffer area of the reference image, the short-term reference image The image is moved out of the buffer area of the reference image. The reference image buffer area can be referred to as a reference image list buffer memory, a reference image list, a reference frame list buffer memory, a reference frame list, or the like, and is unified with the reference image buffer area in the text.

A long-term reference picture (or part of the data in the long-term reference picture) can continue to exist in the buffer area of the reference picture, and this long-term reference picture (or part of the data in the long-term reference picture) is the decoded reference picture. This long-term reference picture (or part of the data in the long-term reference picture) is not affected by entry/exit operations in the reference picture buffer area of move from

Short-term and long-term reference images may be named differently in different standards, eg H.264. 264/advanced video coding (AVC), or H.264. A short-term reference picture in standards such as H.265/HEVC is called a short-term reference, and a long-term reference picture is called a long-term reference. In addition, in standards such as AVS coding standard (audio video coding standard, AVS), 1-P2, AVS2-P2, IEEE (Institute of electrical and electronics engineers, IEEE) 1857.9-P4, the long-term reference image is a background frame. (background picture). Also, in standards such as VP8 and VP9, the long-term reference picture is called a golden frame.

It should be understood that the adoption of specific terminology in the examples of this application does not imply that it must be applied to a specific scene, e.g. , H. 264/AVC or H.264/AVC. It does not indicate that it must be used for technologies that support standards such as H.265/HEVC.

The long-term reference picture described above may be obtained by constructing from image blocks extracted from multiple decoded pictures, or may be derived using multiple decoded pictures and using an existing reference frame (e.g. , reference frames stored in advance)). Of course, this constructed specific reference image may be a short-term reference image. Alternatively, the long-term reference image may not be a constructed reference image.

In the above-described embodiments, the specific reference images may include long-term reference images and the non-specific reference images may include short-term reference images.

Optionally, the type of reference frame can be identified from a special field in the stream structure.

Optionally, if the reference image is determined to be the long-term reference image, then the reference image is determined to be the specific reference image. Alternatively, if the reference image is determined to be a frame that is not output, the reference image is determined to be the specific reference image. Or, if the reference image is determined to be a synthesized frame, determine this reference image to be the specific reference image. Alternatively, if it is determined that the reference image is a frame that is not output and that this reference image is a synthesized frame, this reference image is determined to be the specific reference image.

Optionally, each type of reference image can have a corresponding mark, and then for the decoding end, the mark that the reference image has can determine whether this reference image is a specific reference image or not. .

In some embodiments, if a reference image is determined to have the mark of a long-term reference image, then this reference image is determined to be a specific reference image.

In some embodiments, a reference image is determined to be a specific reference image when it is determined that the reference image has no marks to output.

In some embodiments, if a reference image is determined to have a composite frame mark, then the reference image is determined to be a specific reference image.

In some embodiments, a reference image is determined to be a particular reference image if it has at least two of the following three marks. Long-term reference picture mark, non-output mark, synthetic frame or synthetic reference frame mark. For example, if it is determined that the reference image has a mark that is not output and it is determined that the reference image has a mark of a composite frame, then it is determined that the reference image is a specific reference image.

Specifically, an image can have a mark indicating whether it is a frame to be output or not, and if it is indicated that an image is not to be output, it indicates that this frame is a reference image; Determining whether the frame has a composite frame mark, and if so, determining the reference image as a specific reference image. When it is instructed that a certain image is to be output, it is not necessary to judge whether it is a synthesized frame or not, and it is directly determined that this frame is not a specific reference image. Or, if an image is indicated not to be output, but has a mark that is not a synthetic frame, then it can be determined that this frame is not a specific reference image.

Optionally parsing parameters from a picture header, a picture parameter set (PPS), a slice header to determine that the reference picture satisfies one of the following conditions: , the reference image is determined to be a specific reference image.

The reference image is a long-term reference image.

The reference image is a synthetic reference image.

The reference image is an image that is not output.

If the reference image is an image that is not output, it is further determined whether the reference image is a synthetic reference image.

Although the technique using motion vector derivation was mentioned in the previous sentence, performing motion search on a specific reference image when modifying the motion vector would rather reduce search efficiency and encoding/decoding efficiency. The specific reference image is artificially constructed or from a previous specific reference image in earlier temporal order, and there is no spatial relationship between image blocks in the specific reference image. However, since the image block edges have clear fluctuations, it is not very meaningful to search for motion vectors based on such a specific reference image.

Pattern Matching Motion Vector Derivation (PMMVD) technology and Decode Motion Vector Refinement (DMVR) technology are both techniques that use motion vector derivation. .

The preamble further refers to techniques that use bi-directional motion prediction, manipulating motion vectors based on the temporal correlation of images, and if the reference image to which the associated motion vector points is a particular reference image, the current encoding These operations may be invalid because the definition of the temporal distance between the target picture or the current decoding target picture and a particular reference picture is unclear. Bi-directional optical flow (BIO) prediction techniques are techniques that use bi-directional motion prediction.

Next, based on PMMVD, DMVR and BIO, the image motion compensation method of the present invention will be described with an example. It should be explained that the image motion compensation method in the present invention is not limited to these three techniques.

Before describing the image motion compensation method according to the embodiment of the present application, first, a brief description will be given of the video encoding/decoding process according to the HEVC standard.

The HEVC standard defines three inter-frame prediction modes. Inter mode, Merge mode, and Skip mode. The purpose of inter-frame prediction is to obtain a Motion Vector (MV) and then based on this motion vector determine the position of the predicted image block in the reference image. Adjacent image blocks have similar motion schemes, for example, an image block (for example, at least one of an image block to be encoded and an image block to be decoded) and an adjacent image block are the same object. , and when the cameras move, their movement distances and directions are naturally similar or the same, so there is often no need to calculate the motion vectors, and the motion vectors of neighboring image blocks are directly calculated from the current Let the motion vector of the image block of Here, in Merge mode and Skip mode, the motion vector difference (MVD) is 0, i.e. based on neighboring encoded or decoded image blocks, direct , to get the motion vector.

When at least one mode of the encoding target image block and the decoding target image block is the Merge mode, the operation principle is as follows. Construct one Motion Vector Prediction (MVP) candidate list by neighboring image blocks, select the best MVP from the MVP candidate list as the motion vector of the current image block, and then based on this motion vector , is obtained by determining the position of the predicted image block and calculating the residual after determining the predicted image block. In Merge mode, there is no MVD since the motion vector was selected from the MVP candidate list. The coding end only needs to code on the residual and the index in the MVP candidate list of the selected motion vector, and does not need to code the MVD. The decoding end can build the MVP candidate list by a similar method, and then obtain the motion vector based on the index transmitted by the encoding end. The decoding end determines the predicted image block according to the motion vector, takes the residual and decodes it to obtain the current image block.

The specific operation process of the encoding end in Merge mode is as follows.
1. Get the MVP candidate list.
2. Select the best MVP from the MVP candidate list and get the index of this MVP in the MVP candidate list.
3. Let the selected MVP be the motion vector of the current image block.
4. A predicted image block is determined from the reference image based on the motion vector.
5. Subtract the predicted image block from the current image block to get the residual.
6. Since the motion vector is selected from the MVP candidate list, there is no MVD, and the residual and the index of the selected MVP in the MVP candidate list should be sent to the decoding end.

A detailed operation process of the decoding end in Merge mode is as follows.
1. Receive the residual and the index in the MVP candidate list of the motion vector.
2. Get the MVP candidate list.
3. Based on the index, search the motion vector in the MVP candidate list to be the motion vector of the current image block.
4. Based on the motion vector, the predicted image block is determined and the residual is fetched and decoded to obtain the current image block.

The above is the process of normal Merge mode.

Skip mode is a special case of Merge mode. After obtaining the motion vector by Merge mode, if the encoder determines that the current image block and the predicted image block are basically the same based on some method, there is no need to transmit the residual data. , only the index in the MVP candidate list of the motion vector need be transmitted, and the current image block can be marked directly from the predicted image block.

In the Inter mode, the MVP is determined first, and then the MVP is modified to obtain the MVD. The encoding end not only needs to transmit the index and the residual to the decoding end, but also needs to transmit the MVD to the decoding end. Advanced Motion Vector Prediction (AMVP) is a tool that implements motion vector prediction through a competitive mechanism.

There is also an MVP candidate list in AMVP mode, and the motion vectors in this MVP candidate list are those obtained from the current image block's neighbor blocks in either the spatial domain or the temporal domain. The MVP candidate list in AMVP mode may differ from the MVP candidate list in Merge mode. The encoding end or decoding end selects the optimum MVP from the MVP candidate list. Using this MVP as a search starting point, search around it to obtain the optimum motion vector, which is the motion vector of the current image block. Based on this motion vector, the position of the predicted image block is determined, and the residual is calculated after determining the predicted image block. At the same time, this MV and MVP are subtracted from each other to obtain MVD. The encoding end encodes the residual, the MVP's index in the MVP candidate list, and the MVD and sends them to the decoding end. The decoding end can construct the MVP candidate list by a similar method, and then obtain the MVP based on the index transmitted by the encoding end. The decoding end determines the MV based on the MVP and MVD, determines the predicted image block based on the MV, takes the residual and decodes it to obtain the current image block.

The specific operation process of the encoding end in AMVP mode is as follows.
1. Get the MVP candidate list.
2. Select the best MVP from the MVP candidate list and get the index of this MVP in the MVP candidate list.
3. Determine the search starting point based on the MVP.
4. Search around the starting point to get the best motion vector.
5. A predicted image block is determined from the reference image based on the motion vector.
6. Subtract the predicted image block from the current image block to get the residual.
7. Subtract MVP from the motion vector to get MVD.
8. Send the residual, the index in the MVP candidate list of the selected MVP, and the MVD to the decoding end.

A detailed operation process of the decoding end in the AMVP mode will not be described further.

Embodiments of the present application provide an image motion compensation method 100 . FIG. 1 is a schematic flow chart of an image motion compensation method 100 of one embodiment of the present application. As shown in FIG. 1, the method 100 includes:
S110, obtaining an initial motion vector of the current image block;
S120, performing motion compensation on the current image block based on the initial motion vector if the reference image to which the initial motion vector is directed is a specific reference image;
S130, if the reference image to which this initial motion vector points is a non-specific reference image, modify this initial motion vector, obtain a modified motion vector, and perform this current image based on said modified motion vector; and performing motion compensation on the block.

In some embodiments, if the initial motion vector points to a specific reference image, motion compensation is performed directly; if the initial motion vector points to a non-specific reference image, the initial motion vector is modified; Based on the vector, motion compensation can be performed to avoid meaningless search due to obvious fluctuations of image block edges of a particular reference image, and improve coding and decoding efficiency.

In a possible embodiment, the image motion compensation method 100 of the embodiments of the present application can be applied to PMMVD techniques.

PMMVD technology is a special Merge mode based on Frame Rate Up Conversion (FRUC) technology. In such a special Merge mode, the motion information (eg MV and MVD) of the current image block is not coded in the stream but generated directly at the decoding end.

Optionally, in this possible embodiment, if S130 the reference image to which the initial motion vector is directed is a non-specific reference image, modify the initial motion vector, obtain a modified motion vector, and Performing motion compensation on the current image block based on subsequent motion vectors includes obtaining a motion vector candidate list for the current image block, and pointing any candidate motion vector in the motion vector candidate list to: determining the initial motion vector based on the motion vector candidate list; modifying the initial motion vector to obtain a modified motion vector; performing motion compensation on the current image block based on the modified motion vector.

In this embodiment, an initial motion vector is determined from the motion vector candidate list, and if the reference image to which the candidate vector in the motion vector candidate list points is a non-specific reference image, the initial motion vector is modified, and the motion after modification is determined. Obtaining the vector and performing motion compensation on the current image block based on the modified motion vector avoids making meaningless searches due to obvious swinging of image block edges of a particular reference image; Encoding/decoding efficiency can be improved.

Optionally, in a possible embodiment, obtaining a motion vector candidate list for the current image block includes determining candidate motion vectors for addition to said motion vector candidate list, and determining a reference to which said candidate motion vector is directed. If the image is a non-specific reference image, adding the candidate motion vector to the motion vector candidate list.

Specifically, in one embodiment, the method 100 determines that the reference picture to which the candidate motion vector is directed is a non-specific reference picture based on the frame mark of the reference picture to which the candidate motion vector is directed. can further include: Specifically, it is possible to determine whether the reference image to which the candidate motion vector is directed is a specific reference image based on the frame mark of the reference image to which the candidate motion vector is directed. If the reference image to which the candidate motion vector points is the specific reference image, the candidate motion vector corresponding to the specific reference image may not be added to the motion vector candidate list. Thus, when selecting the initial motion vector from the motion vector candidate list, the motion vector corresponding to the specific reference image is not selected.

It is to be understood that the motion vector in each embodiment of the present application includes three parameters: the horizontal component v _x , the vertical component v _y and the frame mark of the pointing reference image. For example, this frame mark may be a Picture Order Count (POC), or may be some other form of mark. The encoding end and the decoding end can determine the attribute of the reference image from this frame mark, and judge whether this reference image is a specific reference image or a non-specific reference image.

Optionally, in one embodiment, determining an initial motion vector based on the motion vector candidate list includes determining whether the selected initial motion vector is a motion vector pointing to a particular reference image. , if at least one of the selected initial motion vectors points to a specific reference picture, select the initial motion vector again until the pointing reference picture selects the initial motion vector of a reference picture other than the specific reference picture. can include

Specifically, if the determined initial motion vector is directed to a specific reference image based on the motion vector candidate list, the candidate motion vector is reselected from the motion vector candidate list based on a preset rule. You can choose to be the initial motion vector. This preset rule is, for example, to select the candidate motion vector with the next lowest matching cost, and if this candidate motion vector is not directed to a specific reference image, this candidate motion vector is used as the initial motion vector. you can This example is illustrative only and does not limit the application.

Taking the encoding end as an example, the motion vector derivation process in the FRUC merge mode is divided into two steps. The first step is a coding unit (CU) level motion search, and the second step is a sub-CU (Sub-CU) level motion refinement process. Similarly, the decoding end can also implement functions similar to the encoding end.

In the CU level motion search, a CU level motion vector candidate list is generated. Based on the bidirectional matching method, search the motion vector with the lowest matching cost from the CU-level motion vector candidate list, for example, MV-A. The template matching method also searches the motion vector with the lowest matching cost from the CU level motion vector candidate list, for example, MV-B. Then, based on the Rate Distortion Cost (RD-Cost) strategy used in the Merge mode strategy, determine whether the current CU uses FRUC merge mode. That is, the RD-Cost strategy is used to check the results of two matching methods (two-way matching method and template matching method). The results obtained by the matching method with the lower matching cost are further compared again with the results of the other CU modes. The matching method with the lowest matching cost among the two matching methods is the one with the lowest final matching cost, and the FRUC flag of the current CU is set to TRUE, and the decoding end is instructed to use the corresponding matching method. .

In this specific embodiment, the CU-level motion vector candidate list can correspond to the motion vector candidate list in method 100 of the examples of this application. When generating a motion vector candidate list, the motion vectors to be added to the list are scanned, and if the motion vector points to a non-specific reference picture, this motion vector can be added to the motion vector candidate list and directed to a specific reference picture. If so, discard this motion vector and do not add it to the motion vector candidate list. The motion vector candidate list in the present embodiment includes original AMVP candidate motion vectors obtained for non-specific reference pictures, at least one candidate motion vector selected from the obtained merge candidate motion vector, the motion vector obtained by interpolation from the non-specific reference image, and the upper adjacent motion vector and the left adjacent motion vector for the non-specific reference image of the current block. can contain. This motion vector candidate list may be a CU level motion vector candidate list. It should be understood that in this embodiment, the non-specific reference picture may specifically be a short-term reference picture or a short-term reference frame.

Determining the initial motion vector based on the motion vector candidate list corresponds to candidate motion vectors in the motion vector candidate list based on at least one of a bidirectional matching method and a template matching method. determining a distortion cost; and taking the motion vector with the lowest distortion cost in the candidate motion vector list as the initial motion vector.

Specifically, taking the encoding end as an example, determining the initial motion vector based on the motion vector candidate list can correspond to the CU level motion search. In the CU-level motion search, search the motion vector with the lowest matching cost from the CU-level motion vector candidate list based on the bidirectional matching method, for example, MV-A. The template matching method also searches the motion vector with the lowest matching cost from the CU level motion vector candidate list, for example, MV-B. Then, based on the RD-Cost strategy used in the Merge mode strategy, determine whether the current CU uses FRUC merge mode. That is, the RD-Cost strategy is used to check the results of two matching methods (two-way matching method and template matching method). The results obtained by the matching method with the lower matching cost are further compared again with the results of the other CU modes. In the two matching methods, the matching method with the lowest matching cost is the one with the lowest final matching cost, and the FRUC flag of the current CU is set to TRUE, and the decoding end is instructed to use the corresponding matching method. . Similarly, the decoding end can also implement functions similar to the encoding end, but the decoding end is not as complex as the encoding end. flag directly and do not need to execute the RD-Cost strategy.

Optionally, modifying an initial motion vector to obtain a modified motion vector includes generating a sub-motion vector candidate list for the current image block, wherein the sub-motion vector candidate list includes the initial motion vector. and determining a motion vector with the lowest distortion cost from the sub-motion vector candidate list as the modified motion vector.

Specifically, modifying the initial motion vector and obtaining the modified motion vector can correspond to sub-CU level motion refinement.

The motion vector with the lowest matching cost in CU-level motion search is the initial motion vector, which is taken as the starting point for CU-level motion refinement. A local search is performed around the starting point based on the matching method (two-way matching or template matching method) determined by the CU level. Specifically, in sub-CU level motion refinement, a sub-CU level motion vector candidate list can be generated. In the sub-CU level motion vector candidate list, a motion vector with a low matching cost is searched and set as the motion vector of the current CU.

A motion vector (an MV determined from a CU-level search) determined by a CU-level motion search in a sub-CU-level motion vector candidate list, and an upper adjacent motion vector, a left adjacent motion vector, and an upper left adjacent motion vector of the current image block. vector, and top, left, top-left and top-right neighboring motion vectors (MVs) and scaled versions of collocated MVs of the corresponding position of the current image block in the reference picture. ) and other time-domain derived candidate motion vectors (up to 4 ATMVP candidates and up to 4 STMVP candidates).

In the above-described specific embodiment, if the reference image to which the candidate motion vector is directed is the specific reference image, the candidate motion vector corresponding to the specific reference image is not added to the CU level motion vector candidate list, and the candidate motion vector corresponding to the specific reference image is not added. make sure that no candidate motion vector has a chance to be the initial motion vector.

In another specific embodiment, if the initial motion vector determined based on the CU-level motion vector candidate list is directed to a specific reference image, the CU-level motion vector candidates are determined according to preset rules. A candidate motion vector can be selected again from the list to be the initial motion vector. This preset rule is, for example, to select the candidate motion vector with the next lowest matching cost, and if this candidate motion vector is not directed to a specific reference image, this candidate motion vector is used as the initial motion vector. you can

FIG. 2 is a principle schematic diagram of the two-way matching method of one embodiment of the present application. As shown in FIG. 2, the bidirectional matching method searches for the closest matching between two predictive image blocks in different reference images in the motion trajectories of the current CU and derives the motion vector of the current CU. The bidirectional matching method is based on the assumption that the motion trajectory of the current image block is continuous. The motion vectors MV0 and MV1 of the two predicted image blocks shall be proportional to the time domain distance (TD0 and TD1) between the current image and the two reference images (reference image 0 and reference image 1). A motion vector candidate list can be scanned and, for example, for MV0, a motion vector pair MV0 and MV1 can be generated. Here, MV1 can be generated based on MV0, TD0 and TD1. If the distortion between the two predicted image blocks to which the motion vector pair corresponding to MV0 points is the least, this motion vector (ie MV0) is the motion vector of the current CU. When TD0=TD1, bidirectional matching changes to bidirectional matching based on a mirror image.

If one of the two reference images is the specific reference image, the definition of the time distance between the current image (the current image to be encoded or the current image to be decoded) and the specific reference image is unclear. It should be appreciated that bidirectional matching cannot be performed. Moreover, the specific reference image is artificially constructed or is from a specific reference image earlier in time sequence, and there is a spatial relationship between image blocks in the specific reference image. However, since the image block edges have obvious fluctuations, it is not very meaningful for the bidirectional matching to search the motion vector based on such a specific reference image. In a possible embodiment of the present application, the above problem is avoided by not adding candidate motion vectors corresponding to specific reference pictures to the CU-level motion vector candidate list.

FIG. 3 is a principle schematic diagram of the template matching method of one embodiment of the present application. As shown in FIG. 3, the template matching method consists of a template of the current image (at least one of the upper image block and the left image block of the current CU) and a block in the reference image (same as the size of the template). , to derive the motion vector of the current CU. After obtaining the template, if the distortion between the template and the prediction image block to which the candidate motion vector points is the least, this candidate motion vector is the motion vector of the current CU.

If the reference image is a specific reference image, the specific reference image is artificially constructed or from a specific reference image earlier in time order, and between image blocks in the specific reference image does not necessarily have spatial relevance, and image block edges have obvious fluctuations. is inaccurate and thus meaningless. In a possible embodiment of the present application, the above problem is avoided by not adding candidate motion vectors corresponding to specific reference pictures to the CU-level motion vector candidate list.

In a specific embodiment, the image motion compensation method 100 of the embodiments of the present application can be applied to DMVR technology.

The DMVR technique is a refinement technique used to make a more accurate prediction for the current image block when doing bi-prediction. Next, taking the decoding end as an example, the DMVR technology will be described in detail. The DMVR technique mainly includes two major steps, the first step is to construct a template based on the decoded image blocks corresponding to multiple initial motion vectors. The second step is to modify the initial motion vectors based on the template.

Specifically, the decoding end can generate a motion vector candidate list. For example, this motion vector candidate list may be the AMVP mode motion vector candidate list described above, or the motion vector candidate list for a Merge template. The decoding end can receive multiple indices to indicate the initial motion vectors sent from the encoding end. The decoding end obtains a plurality of initial motion vectors from the motion vector candidate list based on the index. The decoding end generates a template (e.g., a method of weighted sum of pixels) based on the decoded image blocks corresponding to the plurality of initial motion vectors, and uses the generated template to obtain the plurality of motion vectors, respectively. modify the initial motion vector of . Finally, motion compensation is performed on the current image block based on this modified motion vector.

In this specific embodiment, the initial motion vector can include a first initial motion vector and a second initial motion vector. If there is a specific reference image at the initial motion vector, the following process can be adopted. S120, if the reference image pointed to by the current initial motion vector is a specific reference image, performing motion compensation on the current image block based on the initial motion vector includes: the first initial motion vector; the current image block based on the first initial motion vector and the second initial motion vector, if the reference image to which at least one of the second initial motion vectors is directed is a specific reference image; and when the reference images directed by the first initial motion vector and the second initial motion vector are both non-specific reference images, the first initial motion vector and the second initial modify a motion vector, obtain a modified first motion vector and a modified second motion vector, and calculate the current motion vector based on the modified first motion vector and the modified second motion vector; performing motion compensation on the image blocks of . If there is no specific reference image in the initial motion vector, that is, if all the initial motion vectors are non-specific reference images, an existing DMVR processing method can be adopted.

FIG. 4 is a principle schematic diagram of the DMVR technology of one embodiment of the present application. Specifically, the initial motion vector includes a first initial motion vector (eg, may be MV0) and a second initial motion vector (eg, may be MV1), corresponding to the first initial motion vector Assuming that the decoded image block belongs to the first decoded image block of the first frame, this first frame may be the first reference image, and this first decoded image block corresponds to the first It may be a reference image block. Assuming that the decoded image block corresponding to the second motion vector belongs to the second decoded image block of the second frame, this second frame may be a second reference image, and this second decoded image block The completed image block may be a second reference image block. A weighted sum of this first reference image block and this second reference image block can be obtained to obtain a template. Here, this template can be called a bidirectional template.

Optionally, in one embodiment, the initial motion vector comprises a first initial motion vector and a second initial motion vector. Modifying the first initial motion vector and the second initial motion vector to obtain a modified first motion vector and a modified second motion vector includes: a first reference image block and a second reference image block; generating a template based on the blocks, wherein said first reference image block corresponds to said first initial motion vector and belongs to a first reference image, and said second reference image block corresponds to said second initial motion vector; corresponding to a vector and belonging to a second reference image; modifying the first initial motion vector and the second initial motion vector based on the template; obtaining a second motion vector of .

Specifically, based on the template, modifying the first initial motion vector and the second initial motion vector, and obtaining the modified first motion vector and the modified second motion vector are N using third reference image blocks to respectively match the template, wherein the N third reference image blocks correspond to N third initial motion vectors and the first reference using image belonging and M fourth reference image blocks to match the template respectively, wherein the M fourth reference image blocks correspond to M fourth initial motion vectors; and selecting one third initial motion vector from the N third initial motion vectors according to belonging to the second reference image and the matching result, and selecting the M fourth initial motion vectors; select one fourth initial motion vector from the above, and convert the one third initial motion vector and the one fourth initial motion vector to the motion vector of the current image block (i.e., the modified first motion vector, and a modified second motion vector), or used to determine the motion vector of the current image block.

Optionally, this selected third initial motion vector may be the motion vector corresponding to the lowest distortion cost. Alternatively, this selected third initial motion vector may be a motion vector corresponding to a distortion cost less than a certain value.

Optionally, this selected fourth initial motion vector may be the motion vector corresponding to the lowest distortion cost. Alternatively, this selected fourth initial motion vector may be a motion vector corresponding to a distortion cost less than a certain value.

wherein the one third initial motion vector and the one fourth initial motion vector are the motion vectors of the current image block, where the one third initial motion vector and the one fourth initial motion vector are the motion vectors of the current image block; The known image block corresponding to the initial motion vector (ie, the initial predicted image block) can be weighted summed to obtain the predicted image block.

or the one third initial motion vector and the one fourth initial motion vector can be used to determine the motion vector of the current image block, i.e. the one third initial motion vector and Each of the one fourth initial motion vectors may be an MVP. In this case, the third MVP may be used as a starting point for search optimization to obtain an optimized motion vector, and the fourth MVP may be used as a starting point for search optimization to obtain other optimized motion vectors. can. The known image blocks corresponding to the two optimized motion vectors (ie, the initial predicted image blocks) can be weighted summed to obtain the predicted image block.

Optionally, N and M may be equal.

Optionally, this third initial motion vector comprises this first initial motion vector and this fourth initial motion vector comprises this second initial motion vector, i.e. the first initial motion vector for generating the template. The reference image block corresponding to the motion vector and the reference image block corresponding to the second motion vector also need to be matched with the template respectively.

Optionally, in an embodiment of the present application, at least some initial motion vectors in the N third initial motion vectors are obtained by shifting based on the first initial motion vectors, and the M At least some of the initial motion vectors in the fourth initial motion vector of are obtained by shifting based on this second initial motion vector.

For example, the initial motion vectors other than the first initial motion vector in the N third initial motion vectors are shifted based on the first initial motion vector, for example N may be equal to 9. , eight initial motion vectors of which are obtained by shifting based on the first initial motion vector, for example, by shifting in eight directions, or by shifting different pixels vertically or horizontally. It is obtained by deviation.

Also, for example, the initial motion vectors other than the second initial motion vector in the N fourth initial motion vectors are obtained by shifting based on the second initial motion vector, for example N is equal to 9. Often, the eight initial motion vectors among them are obtained by shifting based on the second initial motion vector, such as by shifting in eight directions, or different pixels in the vertical or horizontal direction. is obtained by shifting

Optionally, in embodiments of the present application, the first reference image may be a forward frame of the current image block and the second reference image may be a backward frame of the current image block. It can be. Alternatively, the first reference image may be the forward frame of the current image block and the second reference image may be the forward frame of the current image block.

In a specific embodiment, the decoding end generates a motion vector candidate list and receives two indices to indicate the initial motion vector sent from the encoding end. The decoding end determines a DMVP condition, which is that neither of the two initial motion vectors (which may be, for example, MV0 and MV1) points to a specific reference image, and the two initial motion vectors We require the prediction directions to be opposite, ie one forward and the other backward. If the DMVR condition is satisfied, the image block corresponding to MV0 and the image block corresponding to MV1 are weighted summed to generate a bidirectional template.

In a possible embodiment of the present application, the reference picture to which the initial motion vector indicated by the two indices is directed is the specific reference picture by not adding the candidate motion vector corresponding to the specific reference picture to the motion vector candidate list. Avoid that. In another possible embodiment, if the reference picture pointed to by the initial motion vector indicated by the two indices is a specific reference picture, the decoding end initializes a motion vector that the encoding end does not point to a specific reference picture. It can be asked to repoint with the motion vector, or the decoding end returns or marks the DMVR algorithm as invalid.

In the first reference image, for the image block corresponding to MV0, the motion vectors of the neighboring eight pixel blocks retrieved by shifting one luminance pixel in at least one of the horizontal and vertical directions are: A reference list list0 can be formed with a total of 9 motion vectors of MV0. In the second reference image, for the image block corresponding to MV1, the motion vectors of the neighboring eight pixel blocks retrieved by shifting one luminance pixel in at least one of the horizontal and vertical directions are: A reference list list1 can be formed with a total of nine motion vectors of MV1.

The bidirectional template matching that the decoding end uses to perform a distortion-based search between the bidirectional template and the reconstructed blocks in the reference image finally yields the refined motion without additional motion information. get a vector. For the motion vectors in the two reference images (the motion vector in List0 and the motion vector in list1), the motion vector with the lowest matching cost replaces the original motion vector as the update motion vector. Finally, two new motion vectors (MV0' and MV1' shown in FIG. 3) replace the original MV0 and MV1. A final bidirectional prediction result is generated based on the predicted image block corresponding to MV0′ and the predicted image block corresponding to MV1′.

In DMVR technology, if the reference image is a specific reference image, the specific reference image is artificially constructed or from a specific reference image earlier in time order, and in the specific reference image There is not necessarily a spatial relationship between image blocks, and image block edges have obvious fluctuations. Therefore, there is not much significance in searching for motion vectors based on such a specific reference image, namely DMVR. It should be understood that the technique is imprecise and thus meaningless. In the embodiments of the present application, the motion vector corresponding to the specific reference image is not refined and is directly used for motion compensation, thereby avoiding the above-mentioned problems.

Embodiments of the present application provide an image processing method 200 . FIG. 5 is a schematic flow chart of an image processing method 200 of another embodiment of the present application. As shown in FIG. 5, the method 200 includes:
obtaining a first initial motion vector and a second initial motion vector, the first initial motion vector pointing to the first reference picture and the second initial M motion vector pointing to the second reference picture S210; ,
a predicted image of the current image block based on the first initial motion vector and the second initial motion vector, if at least one of the first reference image and the second reference image is a specific reference image; S220 to obtain by calculating a block;
If both the first reference image and the second reference image are non-specific reference images, the current calculating a motion vector of the image block of the current image block, and calculating S230 a prediction image block of the current image block based on the motion vector of the current image block.

The image processing method of the embodiment of the present application adopts the gradient value of the pixel point and the optimization principle, when the reference images directed by the two initial motion vectors are both non-specific reference images, A motion vector of the block is calculated and a predicted image block of the current image block is calculated. If there is a specific reference image in the reference image to which the two initial motion vectors are directed, the predicted image block of the current image block is directly calculated based on the two initial motion vectors, and the current image and the specific reference are obtained. It is possible to avoid the impossibility of prediction due to the unclear definition of the temporal distance to the image, and improve the encoding/decoding efficiency.

Optionally, S230 calculating the MV of the current image block based on the gradient values of the pixel points to which the first initial MV and the second initial MV are directed to obtain the first initial MV , calculating the MV of the current image block based on the gradient values of the pixel points to which the second initial MV is directed and an optimization principle.

In a specific embodiment, the image processing method 200 of the embodiments of the present application can be applied to improve the bi-directional motion prediction mode. Bi-directional motion prediction mode means that when coding a current image block, two initial prediction image blocks from two different reference images are simultaneously used to predict the current image block, and two initial prediction image blocks are used simultaneously. Combining predicted image blocks into one image block to be the predicted image block of the current image block. BIO prediction techniques are techniques in bi-directional motion prediction mode. In the BIO prediction technique, the motion vectors are free of more coding marks than in the normal bidirectional motion prediction mode, but the process of deriving the predicted image block is different. The BIO prediction technique is motion optimization based on block motion compensation, calculates motion vectors by optical flow mode, and is sampling point level motion optimization.

The BIO prediction technique of the embodiments of this application can include two steps. The first step is to compute and obtain the MV of the current image block based on two initial MVs (eg, the first initial MV and the second initial MV). Specifically, the MV of the current image block is calculated and obtained based on the gradient values of the pixel points directed by the first initial MV and the second initial MV and the optimization principle. a second step, based on the MV of the current image block, the first reference image block to which the first initial MV is directed, and the second reference image block to which the second initial MV is directed; is to compute and obtain a predicted image block of . Specifically, the BIO prediction technique interpolates two reference image blocks directed by two initial MVs to obtain two initial predicted image blocks of the same size as the current image block, and then uses these two initial weighted sum of the predicted image blocks and combining them as the predicted image block of the current image block.

FIG. 6 is a principle schematic diagram of the BIO technology of one embodiment of the present application. The BIO technology assumes that the motion of an object is uniform in both the horizontal and vertical directions within a short period of time, and meets the analytical conditions of the optical flow method. As shown in FIG. 6, the motion vector (v _x , v _y ) of the reference point k (k=0, 1) to which the two initial MVs are directed, and the luminance value I ^(k) are given by the following optical flow formula: It can be assumed that they are compatible.
∂I ^(k) /∂t+v _x ∂I ^(k) /∂x+v _y ∂I ^(k) /∂y=0 Formula 1
where ∂I ^(k) /∂x and ∂I ^(k) /∂y denote the horizontal and vertical components of the gradient.

The optical flow formula and Hermite are combined by interpolation to obtain the following polynomial, which is the BIO prediction value when t=0.
pred _BIO = 1/2 (I ⁽⁰⁾ + I ⁽¹⁾ + v _x /2 (τ ₁ ∂I ⁽¹⁾ / ∂x - τ ₀ ∂I ⁽⁰⁾ / ∂x) + v _y /2 ( τ ₁ ∂I ⁽¹⁾ /∂y-τ ₀ ∂I ⁽⁰⁾ /∂y)) Formula 2
where τ ₀ and τ ₁ denote the distances from the current image to reference image 0 and reference image 1, respectively, both of which can be obtained by calculating the POC of the current image and the two reference images. can.
τ ₀ = POC(current)−POC(Ref ₀ ) Formula 3
τ ₁ = POC(Ref ₁ )−POC(current) Formula 4
where POC(current) is the picture order count of the current picture, POC(Ref ₀ ) is the picture order count of reference picture 0, and POC(Ref ₁ ) is the picture order count of reference picture 1. is.

The reference images can be in different directions, one from the past and one from the future. The two reference images are also in the same direction, ie either both from the past or both from the future. If the two reference images have the same orientation, τ ₀ and τ ₁ have opposite signs. In such a situation, the two reference images cannot be the same, i.e. τ ₀ ≠ τ ₁ and the reference region has non-zero motion (MV _x0 , MV _y0 , MV _x1 , MV _y1 ≠ 0), and the motion vector is proportional to the distance in the time domain (MV _x0 /MV _x1 =MV _y0 /MV _y1 =-τ ₀ /τ ₁ ).

Assuming the same motion within a small region, the BIO motion vector can satisfy the first-order Taylor expansion formula as follows.
Δ=(I ⁽⁰⁾ −I ⁽¹⁾ +v _x (τ ₁ ∂I ⁽¹⁾ /∂x+τ ₀ ∂I ⁽⁰⁾ /∂x)+v _y (τ ₁ ∂I ⁽¹⁾ /∂y+τ ₀ ∂ I ⁽⁰⁾ /∂y)) Formula 5
where Δ is the Taylor first-order expansion of the pixel difference between two corresponding reference points (eg, both points A and B in FIG. 4) in two reference images. Gradient value and analysis by optimization method, the optimal motion vector of the current image block satisfies the square of Δ in the whole area and the minimum, so that the optimal motion vector (v _x , v _y ) can be calculated, Considering the robustness of the algorithm, v _x and v _y should be within some threshold range.

Based on the above formula, the BIO prediction technology process is as follows. For the current image block, both pixel values at the locations corresponding to the two reference images have already been obtained. I ⁽⁰⁾ and I ⁽¹⁾ in the formula denote the pixel values in the two reference images respectively, and the known ones in the above formula are I ⁽⁰⁾ , I ⁽¹⁾ and τ ₀ and τ ₁ , the gradient horizontal and vertical components are computed in the reference image, the unknowns are v _x , v _y and Δ. For every pixel point in a region, we can calculate one Δ, and use an optimization method to get Δ to get the minimum value of v _x and v _y , i.e. the required optimal motion vector is. Now, when calculating v _x and v _y , we give v _x and v _y an interval respectively, and the threshold of this interval is determined by the orientation of the two reference images with respect to the current image. After obtaining the optimal motion vector, the image block to which the optimal motion vector is directed + the residual is not directly taken as the current image block, but each pixel of the current image block is calculated by formula 2, This is also referred to as the BIO prediction and combines the predictions for each pixel together to form a predicted image block.

Optionally, in an embodiment of the present application, computing and obtaining a predicted image block of a current image block based on said first initial motion vector and said second initial motion vector is performed by said first initial A weighted sum of a first reference image block directed by a motion vector and a second reference image block directed by the second initial motion vector to obtain a predicted image block of the current image block.

Specifically, a judgment condition is added before the BIO prediction start execution, or a necessary condition is added to the BIO condition. performing BIO prediction if neither the first reference image directed by the first initial motion vector nor the second reference image directed by the second initial motion vector is a specific reference image and passes the initial BIO condition; can be done. Otherwise, directly compute the predicted image block for the current image block based on the two initial motion vectors, or return or mark the expiration of the BIO prediction algorithm.

The BIO condition may further include that prediction directions of the first initial motion vector and the second initial motion vector are different. Or alternatively, the BIO condition is that the prediction directions of the first initial motion vector and the second initial motion vector are the same, and the first initial motion vector and the second initial motion vector are both Instead of 0, the first reference image and the second reference image may be different. At the same time, the ratio of each direction component of the first initial motion vector and the second initial motion vector is the same, and both are the distance between the first reference image to which the motion vector is directed and the current image, and the distance between the second reference image and the second reference image. Equal to the ratio of the distance to the current image.

It will be appreciated that the motion vector in each embodiment of the present application includes three parameters: the horizontal component v _x , the vertical component v _y , and the frame mark of the pointing reference image. For example, this frame mark may be a POC, or other form of mark. The encoding end and the decoding end can determine the attribute of the reference image from this frame mark, and judge whether or not this reference image is a specific reference image.

Optionally, in an embodiment of the present application, determining whether the reference picture to which the initial motion vector is directed is a specific reference picture, based on a frame mark of the reference picture to which the initial motion vector is directed, It can include determining whether the reference image to which the initial motion vector is directed is a specific reference image.

Optionally, in an embodiment of the present application, S230 calculating a predicted image block of the current image block based on the motion vector of the current image block to obtain the motion vector of the current image block; calculating and obtaining a predicted image block of the current image block based on a first reference image block directed by the first initial motion vector and a second reference image block directed by the second initial motion vector; can contain. The specific calculation formulas are described in detail in the preamble and will not be further described here.

A specific embodiment of method 200 may include the following steps.

1. It is determined whether the current two initial motion vectors meet the BIO conditions, and the determination conditions are as follows.

a) The reference images directed by the two initial motion vectors are both non-specific reference images, that is, the first reference image directed by the first initial motion vector and the second reference image directed by the second initial motion vector are Both are non-specific reference images.

b) satisfy one of the following two items,
i) The two initial motion vectors are from different prediction directions (forward and backward, respectively).
ii) the two initial motion vectors are from different reference images in the same prediction direction, and neither of the two initial motion vectors is 0, and the ratio of each direction component of the two initial motion vectors is the same; is also equal to the ratio of the distances between the reference image to which the motion vector points and the current image.

An initial motion vector that satisfies a) and b) at the same time meets the BIO condition and can perform BIO prediction.

2. Based on the prediction directions of the two initial motion vectors, determine the calculation threshold, ie the threshold of v _x and v _y intervals.

3. Compute the gradient values of the pixel points pointed by the two initial motion vectors.

4. Based on the gradient value and the optimization principle, the optimal motion vector is calculated and taken as the motion vector of the current image block.

5. A BIO prediction value is obtained by the motion vector of the current image block and the reference image block to which the two initial motion vectors are directed.

b) if only i) is satisfied, perform a weighted sum of the first reference image block directed by the first initial motion vector and the second reference image block directed by the second initial motion vector to obtain the current image block; Obtain a predicted image block.

The principles, formulas and steps referred to in the examples of this application for BIO prediction can refer to the above description and will not be further described here.

In the BIO prediction technology, when the reference image is a specific reference image, the definition of the time distance between the current image (the current image to be encoded or the current image to be decoded) and the specific reference image is unclear. It should be understood that predictive techniques cannot be implemented. In an embodiment of the present application, if there is a specific reference image in the reference image pointed by the two initial motion vectors, the predicted image block of the current image block is directly calculated based on the two initial motion vectors. This avoids the problems described above.

It should be understood that the method of each embodiment of the present application can be applied to both the encoding end and the decoding end. The current image block in each embodiment of the present application may be an image block to be encoded or an image block to be decoded.

The examples of the present application merely exemplify image processing methods applied to PMMVD technology, DMVR technology, and BIO prediction technology, and the methods of the examples of the present application can also be applied to other existing or future video encoding/decoding techniques, and the embodiments of the present application are not limited thereto.

FIG. 7 is a schematic frame diagram of an image motion compensator 70 of one embodiment of the present application. As shown in FIG. 7, the image motion compensation device 70
at least one memory 701 for storing computer-executable commands;
and at least one processor 702 used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, wherein: ,
obtaining an initial motion vector MV for the current image block;
performing motion compensation on the current image block based on the initial MV, if the reference image to which the initial MV is directed is a specific reference image;
If the reference image to which the initial MV is directed is a non-specific reference image, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV. That is.

In some embodiments, the processor 702 specifically:
obtaining an MV candidate list of the current image block, wherein a reference image to which any candidate MV in the MV candidate list is directed is a non-specific reference image;
Determine the initial MV based on the MV candidate list,
Modifying the initial MV, obtaining the modified MV,
for performing motion compensation on the current image block based on the modified MV.

In some examples, the processor 702 further:
determining a candidate MV to be added to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV is directed is a non-specific reference image;

In some examples, the processor 702 further:
It is for determining that the reference image to which the candidate MV is directed is a non-specific reference image based on the frame mark of the reference image to which the candidate MV is directed.

In some embodiments, the MV candidate list includes at least one of the following candidate MVs:
an original AMVP candidate MV obtained for a non-specific reference image if said current image block is in advanced motion vector prediction AMVP mode;
merge candidate MVs obtained for non-specific reference images,
MV obtained by interpolation from a non-specific reference image;

In some embodiments, the processor 702 specifically:
determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method;
The MV having the lowest distortion cost in the MV candidate list is set as the initial MV.

In some embodiments, the processor 702 specifically:
generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the initial MV;
This is for determining the MV with the lowest distortion cost from the sub-MV candidate list as the corrected MV.

In some embodiments, the initial MVs include a first initial MV and a second initial MV,
Specifically, the processor 702 may:
When a reference image to which at least one of the first initial MV and the second initial MV is directed is a specific reference image, the current perform motion compensation on the image blocks of
When the reference image to which the first initial MV and the second initial MV are directed is a non-specific reference image, the first initial MV and the second initial MV are modified, the first MV after modification, and the modification for obtaining a later second MV and performing motion compensation on the current image block based on the modified first MV and the modified second MV;

In some embodiments, the processor 702 specifically:
generating a template based on a first reference image block and a second reference image block, wherein said first reference image block corresponds to said first initial MV and belongs to a first reference image; A second reference image block corresponds to the second initial MV and belongs to a second reference image,
Based on the template, the first initial MV and the second initial MV are modified to obtain a modified first MV and a modified second MV.

In some embodiments, the processor 702 specifically:
N third reference image blocks are used to respectively match the template, wherein the N third reference image blocks correspond to N third initial MVs and the first reference image blocks image belongs to
M fourth reference image blocks are used to respectively match the template, wherein the M fourth reference image blocks correspond to M fourth initial MVs and the second reference image belongs to
based on the matching result, selecting one third initial MV from the N third initial MVs, selecting one fourth initial MV from the M fourth initial MVs, and selecting the one fourth initial MV from the M fourth initial MVs; The three initial MVs and the one fourth initial MV are used as the MV of the current image block or used to determine the MV of the current image block.

In some embodiments, the third initial MV comprises the first initial MV, and the fourth initial MV comprises the second initial MV.

In some embodiments, at least some of the N third initial MVs are shifted based on the first initial MVs, and at least one of the M fourth initial MVs is The initial MV of the part is obtained by shifting based on the second initial MV.

In some embodiments, said N is equal to said M.

In some embodiments, the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block, or
The first reference image is the forward frame of the current image block and the second reference image is the forward frame of the current image block.

It should be understood that the image motion compensation device 70 can also be implemented by corresponding software modules and will not be further described here.

FIG. 8 is a schematic frame diagram of an image processing device 80 of another embodiment of the present application. As shown in FIG. 8, the image processing device 80
at least one memory 801 for storing computer-executable commands;
and at least one processor 802 used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, wherein: ,
obtaining a first initial motion vector MV and a second initial MV, the first initial MV pointing to a first reference image and the second initial MV pointing to a second reference image;
when at least one of the first reference image and the second reference image is a specific reference image, predicting an image block of the current image block based on the first initial MV and the second initial MV; calculated and obtained,
When both the first reference image and the second reference image are non-specific reference images, the current image based on the gradient values of the pixel points directed by the first initial MV and the second initial MV calculating and obtaining the MV of the block, and calculating and obtaining a predicted image block of the current image block based on the MV of the current image block.

In some embodiments, the processor 802 specifically:
For calculating and obtaining the MV of the current image block based on the gradient values of the pixel points directed by the first initial MV and the second initial MV and the optimization principle.

In some embodiments, the processor 802 specifically:
weighted sum of a first reference image block to which the first initial MV points and a second reference image block to which the second initial MV points to obtain a prediction image block of the current image block;

In some embodiments, the prediction directions of the first initial MV and the second initial MV are different.

In some embodiments, the prediction directions of the first initial MV and the second initial MV are the same, and the first initial MV and the second initial MV are not 0, and the first reference The image and said second reference image are different.

In some embodiments, the processor 802 specifically:
based on the MV of the current image block, a first reference image block to which the first initial MV is directed, and a second reference image block to which the second initial MV is directed, predicting a predicted image block of the current image block. It is to be calculated and obtained.

It is to be understood that the image processing device 80 can also be implemented by corresponding software modules and will not be further described here.

In some techniques that use motion vector derivation, if the motion vector points to a particular reference image, scaling the motion vector is meaningless, reducing search efficiency and encoding/decoding efficiency. . This scales the motion vectors based on the distance between the images when scaling the motion vectors, while certain reference images may be artificially constructed, other reference images and distances This is because it is based on the idea that there is no specific reference image, and there is not much significance in scaling the motion vector based on such a specific reference image.

The present application further provides an image motion compensation method. FIG. 9 is a schematic flow chart of an image motion compensation method 900 of another embodiment of the present application. As shown in FIG. 9, the method 900 includes:
obtaining S910 an initial motion vector MV for the current image block;
determining a scaling ratio for this initial MV, where the scaling ratio for this initial MV is 1 if this initial MV is directed to a particular reference image S920;
S930 scaling this initial MV based on the scaling ratio of this initial MV;
and performing motion compensation S940 on the current image block based on this scaled MV.

Here, the method of obtaining the initial MV may be the same as the method of obtaining the initial MV described above, and will not be further described here.

The image motion compensation method of the embodiment of the present application has a scaling ratio of 1 when the initial MV points to a specific reference image. If the initial MV points to a non-specific reference image, without limiting its scaling ratio, scale the motion vector based on the distance between this current image and the non-specific reference image, and the scaling strategy, and thus the motion Compensation can be performed to improve coding and decoding efficiency.

In some embodiments, the current image block is directly motion compensated based on the scaled MV.

In some embodiments, the scaled MV may be further modified to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV. Here, the method of modifying the scaled MV may be the same as the method of modifying the initial MV described above, and will not be further described here.

For example, as shown in FIG. 5, the first initial MV points to a first reference image and the second initial MV points to a second reference image. When the first reference image and the second reference image both point to the non-specific reference image, the first initial MV and the second initial MV are calculated based on the distance between the first reference image and the second reference image. to scale. Then motion compensation is performed on the current image block based on the first initial MV after scaling and the second initial MV after scaling. In one embodiment, determining a first reference image block, said first reference image block corresponding to the first initial MV after this scaling and belonging to a second reference image, determining a second reference image block. , this second reference image block corresponds to this second initial MV after scaling and belongs to the first reference image. A template is generated based on the first reference image block and the second reference image block, based on this template, the first initial MV and the second initial MV are corrected, the first MV after the correction, and the Obtain the modified second MV. Using the modified first MV and second MV, the motion vector of the current image block is calculated.

Here, the method of correcting the first initial MV and the second initial MV based on this template is the method of correcting the first initial MV and the second initial MV based on the template described above. They may be the same and will not be described further here.

The present application provides yet another image motion compensation method. In this image motion compensation method, an initial motion vector MV of a current image block is obtained, and different operations are performed when the initial MV points to a specific reference image and when the initial MV points to a non-specific reference image. and performing motion compensation on the current image block based on the initial MV.

Here, the method of obtaining the initial MV may be the same as the method of obtaining the initial MV described above, and will not be further described here.

In one embodiment, if the initial MV points to a particular reference image, this initial MV is used to perform motion compensation on the current image block. If the initial MV points to a non-specific reference image, 1, an embodiment that scales the initial MV and performs motion compensation on the current image block based on the scaled initial MV, and 2, modifies the initial MV; and 3, scaling the initial MV, modifying the scaled MV, obtaining the modified MV, the modified MV and 4, modifying the initial MV, scaling the modified MV, obtaining the scaled MV, and based on the scaled MV, the current There are four embodiments, one in which motion compensation is performed on image blocks of .

Here, the method of modifying the scaled MV or the method of modifying the initial MV may be the same as the method of modifying the initial MV described above, and will not be further described here.

In one embodiment, if the initial MV points to a particular reference image, the initial MV can be modified and motion compensation can be performed based on the modified MV. If the initial MV points to a non-specific reference image, motion compensation can be performed on the current image block after scaling the initial MV, or after scaling and modifying the initial MV. That is, in situations where the initial MV points to a particular reference image, the scaling step is skipped and the initial MV is directly modified, or the scaling ratio of the initial MV is set to 1 and then modified directly. In situations where the initial MV points to a non-specific reference picture, the initial MV can be scaled, or scaled and modified, to increase the encoding/decoding efficiency.

In other embodiments of the present application, if the initial MV points to a particular reference image, the initial MV is directly used for motion compensation, or the initial MV is modified and motion compensation is performed based on the modified MV. conduct. and if the initial MV is directed to a non-specific reference image, the initial MV is scaled and/or modified and motion compensated, various embodiments may be combined arbitrarily; The examples do not limit this.

In some embodiments, obtaining the initial motion vector MV of the current image block comprises: obtaining an MV candidate list of the current image block; and based on the MV candidate list, calculating the initial MV. determining. That is, after scaling the initial MV (including scaling with a scaling ratio of 1), select and modify the MV that the initial MV points to a non-specific reference image, and determine which type of reference image the initial MV points to Either can be modified.

Selectably, the reference image to which any candidate MV in said MV candidate list is directed is a non-specific reference image.

Optionally, obtaining the MV candidate list for the current image block determines candidate MVs for addition to the MV candidate list, if the reference image to which the candidate MV is directed is a non-specific reference image. , adding the candidate MV to the MV candidate list.

Optionally, the method may further comprise determining that the reference image to which the candidate MV is directed is a non-specific reference image based on a frame mark of the reference image to which the candidate MV is directed.

Selectably, said MV candidate list may comprise at least one of the following candidate MVs obtained for a non-specific reference image when said current image block is in Advanced Motion Vector Prediction AMVP mode: the original AMVP candidate MV obtained for the non-specific reference picture, the merge candidate MV obtained for the non-specific reference picture, the MV obtained by interpolation from the non-specific reference picture, and the upper and left neighbor MVs for the non-specific reference picture of the current block. It is a music video.

Optionally, determining the initial MVs based on the MV candidate list includes determining candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method. determining a corresponding distortion cost; and taking the MV with the lowest distortion cost in the MV candidate list as the initial MV.

Optionally, modifying the scaled MV to obtain a modified MV generates a sub-MV candidate list for the current image block, wherein the sub-MV candidate list includes the scaled MV. and determining the MV with the lowest distortion cost from the sub-MV candidate list as the modified MV.

Selectably, said initial MV comprises a first initial MV and a second initial MV, and said method is directed to at least one of said first initial MV and said second initial MV. When the reference image is a specific reference image, both the scaling ratios of the first initial MV and the second initial MV are set to 1, and based on the first initial MV and the second initial MV, the performing motion compensation on a current image block; scaling the first initial MV if the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images; and determining a scaling ratio of the second initial MV; scaling the first initial MV based on the scaling ratio of the first initial MV; and scaling the second initial MV based on the scaling ratio of the second initial MV and performing motion compensation on the current image block based on the first scaled initial MV and the second scaled initial MV.

Selectably, performing motion compensation on the current image block based on the first scaled initial MV and the second scaled initial MV comprises a first reference image block and a second reference image block. generating a template based on image blocks, wherein said first reference image block corresponds to said first initial MV after scaling and belongs to a second reference image block, said second reference image block being said scaled corresponding to a later second initial MV and belonging to a first reference image, said first reference image being a reference image to which said first initial MV is directed, said second reference image being said second initial MV and modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. ,including.

Selectably, modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV comprises N third using reference image blocks to match the template respectively, wherein the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; , using M fourth reference image blocks to respectively match the template, wherein the M fourth reference image blocks correspond to M fourth initial MVs, and the second selecting one third initial MV from the N third initial MVs and selecting one fourth initial MV from the M fourth initial MVs based on belonging to the reference image and the matching result; selecting and taking the one third initial MV and the one fourth initial MV as the MV of the current image block or using them to determine the MV of the current image block.

Selectably, the third initial MV may comprise the first initial MV and the fourth initial MV may comprise the second initial MV.

Selectably, at least some initial MVs in the N third initial MVs are obtained by shifting based on the first initial MVs, and at least some initial MVs in the M fourth initial MVs are The initial MV may be obtained by shifting based on the second initial MV.

Optionally, said N may be equal to said M.

Selectably, said first reference image is a forward frame of said current image block and said second reference image is a backward frame of said current image block, or said first reference image. is the forward frame of the current image block, and the second reference image is the forward frame of the current image block.

It is to be understood that the alternative embodiments described above can be implemented analogously to the details of method 100 and will not be further described here.

FIG. 10 is a schematic frame diagram of an image motion compensation apparatus 1000 of one embodiment of the present application. As shown in FIG. 10, the image motion compensation device 1000
at least one memory 1001 for storing computer-executable commands;
and at least one processor 1002 used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, wherein: ,
obtaining an initial motion vector MV for the current image block;
determining a scaling ratio of the initial MV, wherein the scaling ratio of the initial MV is 1 if the initial MV is directed to a particular reference image;
scaling the initial MV based on a scaling ratio of the initial MV;
and performing motion compensation on the current image block based on the scaled MV.

In some embodiments, the processor 1002 performing motion compensation on the current image block based on the scaled MV may include:
modifying the scaled MV to obtain a modified MV;
performing motion compensation on the current image block based on the modified MV.

In some embodiments, said processor 1002 obtaining an initial motion vector MV for said current image block comprises:
obtaining an MV candidate list for the current image block;
determining the initial MV based on the MV candidate list.

In some embodiments, the reference image to which any candidate MV in said MV candidate list is directed is a non-specific reference image.

In some embodiments, said processor 1002 obtaining an MV candidate list for said current image block comprises:
determining a candidate MV for adding to the MV candidate list; and adding the candidate MV to the MV candidate list if a reference image to which the candidate MV is directed is a non-specific reference image.

In some examples, the processor 1002 further:
It is for determining that the reference image to which the candidate MV is directed is a non-specific reference image based on the frame mark of the reference image to which the candidate MV is directed.

In some embodiments, the MV candidate list includes:
an original AMVP candidate MV obtained for a non-specific reference image if said current image block is in advanced motion vector prediction AMVP mode;
merge candidate MVs obtained for non-specific reference images,
an MV obtained by interpolation from a non-specific reference picture; and at least one candidate MV among an upper neighboring MV and a left neighboring MV for the non-specific reference picture of the current block.

In some embodiments, determining the initial MV based on the MV candidate list by the processor 1002 includes:
determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method;
and setting the MV with the lowest distortion cost in the MV candidate list as the initial MV.

In some embodiments, the processor 1002 modifies the scaled MV to obtain a modified MV by:
generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV;
determining the MV with the lowest distortion cost from the sub-MV candidate list as the modified MV.

In some embodiments, the initial MVs include a first initial MV and a second initial MV, and the processor 1002 specifically:
When the reference image to which at least one of the first initial MV and the second initial MV is directed is a specific reference image, the scaling ratio of the first initial MV and the second initial MV also set to 1 and performing motion compensation on the current image block based on the first initial MV and the second initial MV;
If the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, determine the scaling ratio of the first initial MV and the scaling ratio of the second initial MV; scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and scaling the first initial MV and performing motion compensation on the current image block based on the second initial MV after scaling.

In some embodiments, the processor 1002 performing motion compensation on the current image block based on the first scaled initial MV and the second scaled initial MV includes:
generating a template based on a first reference image block and a second reference image block, wherein said first reference image block corresponds to said first initial MV after scaling and belongs to a second reference image; , said second reference image block corresponds to said second initial MV after scaling and belongs to a first reference image, said first reference image being a reference image to which said first initial MV is directed, said first 2, the reference image is a reference image to which the second initial MV is directed;
modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

In some embodiments, the processor 1002 modifies the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. teeth,
N third reference image blocks are used to respectively match the template, wherein the N third reference image blocks correspond to N third initial MVs and the first reference image blocks belonging to the image;
M fourth reference image blocks are used to respectively match the template, wherein the M fourth reference image blocks correspond to M fourth initial MVs and the second reference belonging to the image;
based on the matching result, selecting one third initial MV from the N third initial MVs, selecting one fourth initial MV from the M fourth initial MVs, and selecting the one fourth initial MV from the M fourth initial MVs; 3 initial MVs and the one fourth initial MV as the MV of the current image block or using it to determine the MV of the current image block.

In some embodiments, the third initial MV comprises the first initial MV, and the fourth initial MV comprises the second initial MV.

In some embodiments, at least some of the N third initial MVs are shifted based on the first initial MVs, and at least one of the M fourth initial MVs is The initial MV of the part is obtained by shifting based on the second initial MV.

In some embodiments, said N is equal to said M.

In some embodiments, the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block, or
The first reference image is the forward frame of the current image block and the second reference image is the forward frame of the current image block.

In some embodiments, the specific reference image includes at least one of a long-term reference image, a synthesized frame, and a non-output frame.

In some examples, the processor 1002 further:
This is for determining that the reference image is the specific reference image when determining that the reference image is a frame that is not output and further determining that the reference image is a composite frame.

In some embodiments, the non-specific reference images include short-term reference images.

It should be understood that the image motion compensation device 1000 can be further implemented by corresponding software modules, and will not be further described here.

The apparatus of each embodiment of the present application can be realized by a memory and a processor, each memory is for storing commands for executing the method of the embodiments of this application, and the processor executes the above commands. It should be understood to implement and cause the apparatus to perform the method of each embodiment of the present application.

The processor referred to in the examples of the present application may be a CPU (Central Processing Unit, CPU), other general-purpose processors, Digital Signal Processors (DSP), Application Specific Integrated Circuits , ASIC), off-the-shelf Field Programmable Gate Array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or the like. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, and so on.

It should be further appreciated that the memory referred to in the examples of the present application may be volatile memory, non-volatile memory, or may include both volatile and non-volatile memory. Here, the non-volatile memory includes read-only memory (ROM), programmable read-only memory (ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), and electrically erasable programmable read. It may be only memory (Electrically EPROM, EEPROM) or flash memory. Volatile memory can be Random Access Memory (RAM), which is used as an external high speed buffer memory. By way of example but not limitation, many forms of RAM can be used, such as Static RAM (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM). , Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

If the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, memory (storage module) is integrated into the processor. There is

It should be noted that the memory discussed herein includes, but is not limited to, these and any other suitable types of memory.

Embodiments of the present application further provide a computer-readable storage medium on which commands are stored and which, when executed by the computer, cause the computer to perform the methods of each of the method embodiments described above. .

Embodiments of the present application further provide a computing device, which includes the computer-readable storage medium described above.

Embodiments of the present application further provide an encoding device, which includes at least one of an image motion compensation device 70, an image processing device 80, and an image motion compensation device 1000. FIG.

Embodiments of the present application further provide a decoding device, which includes at least one of an image motion compensator 70 , an image processor 80 and an image motion compensator 1000 .

Embodiments of the present application are applicable to aircraft, particularly in the field of unmanned aerial vehicles.

It should be understood that the division of electrical circuits, sub-circuits and sub-units in each example of the present application is only schematic. Those skilled in the art can recognize that each exemplary electrical circuit, sub-circuit, and sub-unit described by the embodiments disclosed herein can be disassembled or combined.

The embodiments described above may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer commands. When a computer loads or executes computer commands, it creates processes or functions, in whole or in part, based on the embodiments of the present application. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. Computer commands can be stored on a computer readable storage medium or transmitted from a computer readable storage medium to another computer readable storage medium, e.g., computer commands can be stored on a site, computer, server, or from the data center to other sites, computers, servers, Or it can be transmitted to the data center. A computer readable storage medium can be any available medium that can be accessed by a computer or data storage device including one or more available media integrated servers, data centers, etc. can be Usable media include magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., DVD (Digital Video Disc, DVD)), or semiconductor media (e.g., SSD (Solid State Disk, SSD)). and so on.

References to "an embodiment" or "an embodiment" throughout this specification mean that the particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of this application. Please understand. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. In addition, these specific features, structures or properties may be combined in any suitable way in one or more embodiments.

In each embodiment of the present application, the magnitude of the number of each process described above does not mean the order of execution, the order of execution of each process is determined by its function and inherent logic, and the present application It should be understood that no limitation is imposed on the implementation of the examples.

It should be understood that "B corresponding to A" in the examples of this application indicates that B is related to A and that B can be determined based on A. However, determining B based on A does not mean determining B only by A, and furthermore, B can be determined based on at least one of A and other information. Please understand.

The term "at least one" in the text only describes the relationship of related objects, there can be three relationships, for example, at least one of A and B, A exists alone, It should be understood that there are three situations where A and B exist together and B exists alone. In addition, the symbol "/" in the text usually indicates that the related objects before and after are in an "or" relationship.

Those skilled in the art can recognize that each exemplary unit and algorithmic step described by the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are ultimately implemented in a hardware or software manner is determined by the specific application of the technical solution and design constraints. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be recognized as beyond the scope of the present application.

Those skilled in the art can clearly understand that the specific working steps of the above-described systems, devices, and units can refer to the corresponding steps in the foregoing method embodiments for ease of explanation and conciseness. , will not be described further here.

It should be appreciated that in the multiple embodiments provided by this application, the disclosed systems, devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely exemplary. For example, the division of the units is only the division of logic functions, and in actual implementation, there may be other division methods, such as multiple units or components can be combined, or other systems or some features can be ignored or not implemented. Also, the shown or discussed couplings between each other or direct couplings or communication connections may be through some interfaces, and the indirect couplings or communication connections of devices or units may be , electrical, mechanical, or otherwise.

The unit described as the separate member may or may not be physically divided, and the member indicated as a unit may or may not be a physical unit. ie it can be located in one place or it can be distributed over several network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can physically exist alone, and two or more units can be integrated into one unit. may be integrated.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Modifications or replacements can be easily conceived within the technical scope of the invention, and both shall be included in the protection scope of the present application. Therefore, the protection scope of this application shall be subject to the protection scope of the appended claims.
(Item 1)
An image motion compensation method comprising:
obtaining an initial motion vector MV for the current image block;
performing motion compensation on the current image block based on the initial MV, if the reference image to which the initial MV is directed is a specific reference image;
If the reference image to which the initial MV is directed is a non-specific reference image, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV. and an image motion compensation method.
(Item 2)
If the reference image to which the initial MV is directed is a non-specific reference image, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV. The thing is
obtaining an MV candidate list of the current image block, and a reference image to which any candidate MV in the MV candidate list is directed is a non-specific reference image;
Determining the initial MV based on the MV candidate list;
Modifying the initial MV to obtain a modified MV;
performing motion compensation on the current image block based on the modified MV.
(Item 3)
Obtaining the MV candidate list for the current image block includes:
3. The method of item 2, comprising determining a candidate MV for addition to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV is directed is a non-specific reference image. .
(Item 4)
The above method is
4. The method of item 3, further comprising determining that the reference image to which the candidate MV is directed is a non-specific reference image based on the frame mark of the reference image to which the candidate MV is directed.
(Item 5)
The above MV candidate list is
Original AMVP candidate MVs obtained for a non-specific reference image if the current image block is in Advanced Motion Vector Prediction AMVP mode;
merge candidate MVs obtained for non-specific reference images,
any one of items 2 to 4, including the MV obtained by interpolation from the non-specific reference image, and at least one candidate MV among the above-neighboring MV and the left-side neighboring MV for the non-specific reference image of the current block; described method.
(Item 6)
Determining the initial MV based on the MV candidate list is
determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method;
6. The method of item 5, comprising as the initial MV the MV with the lowest distortion cost in the MV candidate list.
(Item 7)
Correcting the initial MV and obtaining the corrected MV is
generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the initial MV;
and determining the MV with the lowest distortion cost from the sub-MV candidate list as the modified MV.
(Item 8)
The initial MV includes the first initial MV and the second initial MV,
If the reference image to which the initial MV is directed is a specific reference image, performing motion compensation on the current image block based on the initial MV includes:
When the reference image to which at least one of the first initial MV and the second initial MV is directed is a specific reference image, the current performing motion compensation on the image blocks of
When the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, the first initial MV and the second initial MV are modified, and the modified first MV, and obtaining a modified second MV, and performing motion compensation on the current image block based on the modified first MV and the modified second MV.
(Item 9)
Modifying the first initial MV and the second initial MV to obtain the modified first MV and the modified second MV,
generating a template based on a first reference image block and a second reference image block, wherein said first reference image block corresponds to said first initial MV and belongs to a first reference image; the second reference image block corresponds to the second initial MV and belongs to the second reference image;
9. The method of item 8, comprising modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.
(Item 10)
Based on the template, modifying the first initial MV and the second initial MV to obtain the modified first MV and the modified second MV
N third reference image blocks are used to respectively match the template, wherein the N third reference image blocks correspond to N third initial MVs and the first reference image blocks belonging to the image;
M fourth reference image blocks are used to respectively match the template, wherein the M fourth reference image blocks correspond to M fourth initial MVs and the second reference belonging to the image;
Based on the matching result, selecting one third initial MV from the N third initial MVs, selecting one fourth initial MV from the M fourth initial MVs, and selecting the one fourth initial MV from the M fourth initial MVs. 10. The method of item 9, comprising: 3 initial MVs, and taking the one fourth initial MV as the MV of the current image block or using it to determine the MV of the current image block.
(Item 11)
11. The method of item 10, wherein the third initial MV comprises the first initial MV and the fourth initial MV comprises the second initial MV.
(Item 12)
At least some initial MVs in the N third initial MVs are obtained by shifting based on the first initial MVs, and at least some initial MVs in the M fourth initial MVs are the 12. A method according to item 10 or 11 obtained by shifting based on the second initial MV.
(Item 13)
13. The method of any one of items 10-12, wherein N is equal to M.
(Item 14)
the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block; or
14. Method according to any one of items 10 to 13, wherein said first reference image is a forward frame of said current image block and said second reference image is a forward frame of said current image block. .
(Item 15)
15. The method of any one of items 1-14, wherein the specific reference image comprises at least one of a long-term reference image, a synthetic frame, and a non-output frame.
(Item 16)
The above method is
any of items 1 to 15, further comprising determining that the reference image is the specific reference image when determining that the reference image is a frame that is not output and determining that the reference image is a composite frame. The method according to item 1.
(Item 17)
17. A method according to any one of items 1 to 16, wherein said non-specific reference images comprise short-term reference images.
(Item 18)
An image motion compensation device,
at least one memory for storing computer executable commands;
at least one processor used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, said operations comprising: ,
obtaining an initial motion vector MV for the current image block;
performing motion compensation on the current image block based on the initial MV, if the reference image to which the initial MV is directed is a specific reference image;
If the reference image to which the initial MV is directed is a non-specific reference image, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV. and an image motion compensator.
(Item 19)
Specifically, the above processor:
obtaining the MV candidate list of the current image block, wherein a reference image to which any candidate MV in the MV candidate list is directed is a non-specific reference image;
Determine the initial MV based on the MV candidate list,
Correct the above initial MV, obtain the corrected MV,
19. Image motion compensation device according to item 18, for performing motion compensation on said current image block based on said modified MV.
(Item 20)
The processor further:
19, for determining a candidate MV for adding to said MV candidate list and adding said candidate MV to said MV candidate list if the reference image to which said candidate MV is directed is a non-specific reference image; image motion compensator.
(Item 21)
The processor further:
21. The method according to any one of items 18 to 20, for determining that the reference image to which the candidate MV is directed is a non-specific reference image based on the frame mark of the reference image to which the candidate MV is directed. Image motion compensator.
(Item 22)
The above MV candidate list is
Original AMVP candidate MVs obtained for a non-specific reference image if the current image block is in Advanced Motion Vector Prediction AMVP mode;
merge candidate MVs obtained for non-specific reference images,
22. Picture motion compensation apparatus according to item 21, comprising a MV obtained by interpolation from a non-specific reference picture, and at least one candidate MV of an upper neighboring MV and a left neighboring MV for the non-specific reference picture of said current block.
(Item 23)
Specifically, the above processor:
determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method;
23. The image motion compensation device according to item 22, for making the MV with the lowest distortion cost in the MV candidate list the initial MV.
(Item 24)
Specifically, the above processor:
generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the initial MV;
21. The image motion compensation device according to any one of items 18 to 20, for determining the MV with the lowest distortion cost from the sub-MV candidate list as the modified MV.
(Item 25)
The initial MV includes the first initial MV and the second initial MV,
Specifically, the above processor:
When the reference image to which at least one of the first initial MV and the second initial MV is directed is a specific reference image, the current perform motion compensation on the image blocks of
When the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, the first initial MV and the second initial MV are modified, and the modified first MV, and a modified second MV, and performing motion compensation on the current image block based on the modified first MV and the modified second MV. Device.
(Item 26)
Specifically, the above processor:
generating a template based on a first reference image block and a second reference image block, wherein said first reference image block corresponds to said first initial MV and belongs to a first reference image; The second reference image block corresponds to the second initial MV and belongs to the second reference image,
26. Image motion according to item 25, for modifying said first initial MV and said second initial MV based on said template to obtain said modified first MV and said modified second MV. Compensator.
(Item 27)
Specifically, the above processor:
N third reference image blocks are used to respectively match the template, wherein the N third reference image blocks correspond to N third initial MVs and the first reference image blocks image belongs to
M fourth reference image blocks are used to respectively match the template, wherein the M fourth reference image blocks correspond to M fourth initial MVs and the second reference image belongs to
Based on the matching result, selecting one third initial MV from the N third initial MVs, selecting one fourth initial MV from the M fourth initial MVs, and selecting the one fourth initial MV from the M fourth initial MVs. 27. Image motion compensation apparatus according to item 26, wherein three initial MVs and said one fourth initial MV are used as the MV of said current image block or for determining the MV of said current image block. .
(Item 28)
28. The image motion compensation apparatus according to item 27, wherein said third initial MV comprises said first initial MV and said fourth initial MV comprises said second initial MV.
(Item 29)
At least some initial MVs in the N third initial MVs are obtained by shifting based on the first initial MVs, and at least some initial MVs in the M fourth initial MVs are the 29. Image motion compensator according to item 27 or 28 obtained by shifting based on the second initial MV.
(Item 30)
30. The image motion compensation device according to any one of items 27-29, wherein said N is equal to said M.
(Item 31)
the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block; or
31. An image according to any one of items 27 to 30, wherein said first reference image is a forward frame of said current image block and said second reference image is a forward frame of said current image block. motion compensator.
(Item 32)
32. The image motion compensation apparatus according to any one of items 18 to 31, wherein the specific reference image includes at least one of a long-term reference image, a synthesized frame, and a non-output frame.
(Item 33)
The processor further:
Items 18 to 32 for determining that the reference image is the specific reference image when determining that the reference image is a frame that is not output and further determining that the reference image is a composite frame. An image motion compensation device according to any one of Claims 1 to 3.
(Item 34)
Image motion compensation apparatus according to any one of items 18 to 33, wherein said non-specific reference images comprise short-term reference images.
(Item 35)
A computer readable storage medium,
A computer readable storage medium on which commands are stored and which, when run on a computer, cause a computer to perform the image motion compensation method according to any one of items 1-17.
(Item 36)
A coding device comprising an image motion compensator according to any one of items 18-34.
(Item 37)
A decoding device comprising an image motion compensator according to any one of items 18-34.
(Item 38)
An image motion compensation method comprising:
obtaining an initial motion vector MV for the current image block;
determining a scaling ratio of the initial MV, wherein the scaling ratio of the initial MV is 1 if the initial MV is directed to a particular reference image;
scaling the initial MV based on a scaling ratio of the initial MV;
performing motion compensation on the current image block based on the scaled MV.
(Item 39)
performing motion compensation on the current image block based on the scaled MV,
modifying the scaled MV to obtain a modified MV;
39. The method of item 38, comprising motion compensating the current image block based on the modified MV.
(Item 40)
Obtaining the initial motion vector MV of the current image block comprises:
obtaining a MV candidate list for the current image block;
40. The method of item 38 or 39, comprising determining the initial MV based on the MV candidate list.
(Item 41)
41. Method according to item 40, wherein the reference image to which any candidate MV in said MV candidate list is directed is a non-specific reference image.
(Item 42)
Obtaining the MV candidate list for the current image block includes:
42. The method of item 41, comprising determining a candidate MV for addition to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV is directed is a non-specific reference image. .
(Item 43)
The above method is
43. The method of item 42, further comprising determining that the reference image to which the candidate MV is directed is a non-specific reference image based on the frame mark of the reference image to which the candidate MV is directed.
(Item 44)
The above MV candidate list is
Original AMVP candidate MVs obtained for a non-specific reference image if the current image block is in Advanced Motion Vector Prediction AMVP mode;
merge candidate MVs obtained for non-specific reference images,
any one of items 40 to 43, including the MV obtained by interpolation from the non-specific reference image, and at least one candidate MV among the above-neighboring MV and the left-side neighboring MV for the non-specific reference image of the current block; described method.
(Item 45)
Determining the initial MV based on the MV candidate list is
determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method;
45. The method of item 44, comprising as the initial MV the MV with the lowest distortion cost in the MV candidate list.
(Item 46)
Correcting the MV after scaling and obtaining the corrected MV is
generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV;
46. The method of item 45, comprising determining the MV with the lowest distortion cost from the sub-MV candidate list as the modified MV.
(Item 47)
The initial MV includes a first initial MV and a second initial MV, and the method includes:
When the reference image to which at least one of the first initial MV and the second initial MV is directed is a specific reference image, the scaling ratio of the first initial MV and the second initial MV also set to 1 and performing motion compensation on the current image block based on the first initial MV and the second initial MV;
If the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, determining the scaling ratio of the first initial MV and the scaling ratio of the second initial MV; scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and scaling the first initial MV and performing motion compensation on the current image block based on the scaled second initial MV.
(Item 48)
performing motion compensation on the current image block based on the first scaled initial MV and the second scaled initial MV;
generating a template based on a first reference image block and a second reference image block, wherein said first reference image block corresponds to said first initial MV after scaling and belongs to a second reference image; , the second reference image block corresponds to the second initial MV after scaling and belongs to a first reference image, the first reference image is a reference image to which the first initial MV is directed, the first The second reference image is a reference image to which the second initial MV is directed;
48. The method of item 47, comprising modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.
(Item 49)
Based on the template, modifying the first initial MV and the second initial MV to obtain the modified first MV and the modified second MV
N third reference image blocks are used to respectively match the template, wherein the N third reference image blocks correspond to N third initial MVs and the first reference image blocks belonging to the image;
M fourth reference image blocks are used to respectively match the template, wherein the M fourth reference image blocks correspond to M fourth initial MVs and the second reference belonging to the image;
Based on the matching result, selecting one third initial MV from the N third initial MVs, selecting one fourth initial MV from the M fourth initial MVs, and selecting the one fourth initial MV from the M fourth initial MVs. 48. The method of item 47, comprising: 3 initial MVs, and the one fourth initial MV as the MV of the current image block or used to determine the MV of the current image block.
(Item 50)
50. The method of item 49, wherein the third initial MV comprises the first initial MV and the fourth initial MV comprises the second initial MV.
(Item 51)
At least some initial MVs in the N third initial MVs are obtained by shifting based on the first initial MVs, and at least some initial MVs in the M fourth initial MVs are the 51. A method according to items 49 or 50 obtained by shifting based on the second initial MV.
(Item 52)
52. The method of any one of items 49-51, wherein said N is equal to said M.
(Item 53)
the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block; or
53. Method according to any one of items 49 to 52, wherein said first reference image is a forward frame of said current image block and said second reference image is a forward frame of said current image block. .
(Item 54)
54. A method according to any one of items 38 to 53, wherein said specific reference image comprises at least one of a long term reference image, a synthetic frame and a non-output frame.
(Item 55)
The above method is
of items 38 to 54, further comprising determining that the reference image is the specific reference image when determining that the reference image is a frame that is not output and further determining that the reference image is a composite frame. A method according to any one of paragraphs.
(Item 56)
56. A method according to any one of items 38-55, wherein said non-specific reference images comprise short-term reference images.
(Item 57)
An image motion compensation device,
at least one memory for storing computer executable commands;
at least one processor used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, said operations comprising: ,
obtaining an initial motion vector MV for the current image block;
determining a scaling ratio of the initial MV, wherein the scaling ratio of the initial MV is 1 if the initial MV is directed to a particular reference image;
scaling the initial MV based on a scaling ratio of the initial MV;
performing motion compensation on the current image block based on the scaled MV.
(Item 58)
The processor performing motion compensation on the current image block based on the scaled MV,
modifying the scaled MV to obtain a modified MV;
58. The image motion compensation apparatus of item 57, comprising motion compensating the current image block based on the modified MV.
(Item 59)
Obtaining the initial motion vector MV of the current image block by the processor includes:
obtaining a MV candidate list for the current image block;
determining the initial MV based on the MV candidate list.
(Item 60)
60. Picture motion compensation apparatus according to item 59, wherein a reference picture to which any candidate MV in said MV candidate list is directed is a non-specific reference picture.
(Item 61)
The processor obtaining an MV candidate list for the current image block includes:
61. The image of item 60, comprising determining a candidate MV for addition to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV is directed is a non-specific reference image. motion compensator.
(Item 62)
The processor further:
62. Picture motion compensation device according to item 61, for determining that the reference picture to which said candidate MV is directed is a non-specific reference picture based on the frame mark of the reference picture to which said candidate MV is directed.
(Item 63)
The above MV candidate list is
Original AMVP candidate MVs obtained for a non-specific reference image if the current image block is in Advanced Motion Vector Prediction AMVP mode;
merge candidate MVs obtained for non-specific reference images,
any one of items 59 to 62, including the MV obtained by interpolation from the non-specific reference image, and at least one candidate MV of the above-neighboring MV and the left-side neighboring MV for the non-specific reference image of the current block; An image motion compensator as described.
(Item 64)
Determining the initial MV by the processor based on the MV candidate list
determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method;
64. The image motion compensation apparatus according to item 63, comprising: setting the MV with the lowest distortion cost in the MV candidate list as the initial MV.
(Item 65)
The processor correcting the scaled MV and obtaining the corrected MV is
generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV;
determining the MV with the lowest distortion cost from the sub-MV candidate list as the modified MV.
(Item 66)
The initial MV includes a first initial MV and a second initial MV, and the processor specifically:
When the reference image to which at least one of the first initial MV and the second initial MV is directed is a specific reference image, the scaling ratio of the first initial MV and the second initial MV also set to 1 and performing motion compensation on the current image block based on the first initial MV and the second initial MV;
If the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, determining the scaling ratio of the first initial MV and the scaling ratio of the second initial MV; scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and scaling the first initial MV and performing motion compensation on the current image block based on the scaled second initial MV.
(Item 67)
The processor performing motion compensation on the current image block based on the first scaled initial MV and the second scaled initial MV,
generating a template based on a first reference image block and a second reference image block, wherein said first reference image block corresponds to said first initial MV after scaling and belongs to a second reference image; , the second reference image block corresponds to the second initial MV after scaling and belongs to a first reference image, the first reference image is a reference image to which the first initial MV is directed, the first The second reference image is a reference image to which the second initial MV is directed;
Modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. Compensator.
(Item 68)
The processor correcting the first initial MV and the second initial MV based on the template to obtain the corrected first MV and the corrected second MV,
N third reference image blocks are used to respectively match the template, wherein the N third reference image blocks correspond to N third initial MVs and the first reference image blocks belonging to the image;
M fourth reference image blocks are used to respectively match the template, wherein the M fourth reference image blocks correspond to M fourth initial MVs and the second reference belonging to the image;
Based on the matching result, selecting one third initial MV from the N third initial MVs, selecting one fourth initial MV from the M fourth initial MVs, and selecting the one fourth initial MV from the M fourth initial MVs. 67. The image motion compensation apparatus of item 66, comprising: three initial MVs, and the one fourth initial MV as the MV of the current image block or used to determine the MV of the current image block. .
(Item 69)
69. Image motion compensation apparatus according to item 68, wherein said third initial MV comprises said first initial MV and said fourth initial MV comprises said second initial MV.
(Item 70)
At least some initial MVs in the N third initial MVs are obtained by shifting based on the first initial MVs, and at least some initial MVs in the M fourth initial MVs are the 70. Image motion compensator according to item 68 or 69 obtained by shifting based on the second initial MV.
(Item 71)
71. An image motion compensation apparatus according to any one of items 68-70, wherein said N is equal to said M.
(Item 72)
the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block; or
72. An image according to any one of items 68 to 71, wherein said first reference image is a forward frame of said current image block and said second reference image is a forward frame of said current image block. Device for motion compensation.
(Item 73)
73. The image motion compensation apparatus according to any one of items 57 to 72, wherein the specific reference image includes at least one of a long-term reference image, a synthesized frame and a non-output frame.
(Item 74)
The processor further:
Items 57 to 73 for determining that the reference image is the specific reference image when determining that the reference image is a frame that is not output and further determining that the reference image is a composite frame. An image motion compensation device according to any one of Claims 1 to 3.
(Item 75)
75. Image motion compensation apparatus according to any one of items 57 to 74, wherein said non-specific reference pictures comprise short-term reference pictures.
(Item 76)
A computer readable storage medium,
A computer readable storage medium on which commands are stored and which, when run on a computer, cause a computer to perform the image motion compensation method according to any one of items 38-56.
(Item 77)
A coding device comprising an image motion compensator according to any one of items 57-75.
(Item 78)
A decoding device comprising an image motion compensator according to any one of items 57-75.
(Item 79)
An image processing method comprising:
obtaining a first initial motion vector MV and a second initial MV, the first initial MV pointing to a first reference image and the second initial MV pointing to a second reference image;
when at least one of the first reference image and the second reference image is a specific reference image, based on the first initial MV and the second initial MV, a predicted image block of the current image block; calculated and obtained,
If both the first reference image and the second reference image are non-specific reference images, the current image based on the gradient values of the pixel points directed by the first initial MV and the second initial MV computing and obtaining the MV of a block, and computing and obtaining a predicted image block of the current image block based on the MV of the current image block.
(Item 80)
Calculating the MV of the current image block based on the gradient values of the pixel points directed by the first initial MV and the second initial MV, obtaining:
80. Method according to item 79, comprising computing and obtaining the MV of the current image block based on the gradient values of the pixel points to which the first initial MV and the second initial MV are directed and an optimization principle. .
(Item 81)
Calculating a predicted image block of the current image block based on the first initial MV and the second initial MV to obtain:
79 or 80 including weighted summing a first reference image block to which said first initial MV points and a second reference image block to which said second initial MV points to obtain a prediction image block of said current image block. The method described in .
(Item 82)
82. The method according to any one of items 79 to 81, wherein prediction directions of the first initial MV and the second initial MV are different.
(Item 83)
The prediction directions of the first initial MV and the second initial MV are the same, the first initial MV and the second initial MV are not 0, and the first reference image and the second reference image are 83. The method of any one of items 79-82, wherein the images are different.
(Item 84)
Calculating a predicted image block for the current image block based on the MV of the current image block to obtain:
based on the MV of the current image block, the first reference image block to which the first initial MV is directed, and the second reference image block to which the second initial MV is directed, predicting the predicted image block of the current image block. 84. A method according to any one of items 79-83, comprising obtaining by calculation.
(Item 85)
85. A method according to any one of items 79-84, wherein said specific reference image comprises at least one of a long-term reference image, a synthetic frame and a non-output frame.
(Item 86)
The above method is
determining that at least one of the first reference image and the second reference image is a non-output frame, and at least one of the first reference image and the second reference image is a synthesized frame; 86. The method according to any one of items 79 to 85, further comprising determining at least one of the first reference image and the second reference image to be the specific reference image when further determining that the Method.
(Item 87)
87. A method according to any one of items 79-86, wherein said non-specific reference images comprise short-term reference images.
(Item 88)
An image processing device,
at least one memory for storing computer executable commands;
at least one processor used alone or jointly to perform the following operations by accessing said at least one memory and executing said computer-executable commands, said operations comprising: ,
obtaining a first initial motion vector MV and a second initial MV, the first initial MV pointing to a first reference image and the second initial MV pointing to a second reference image;
when at least one of the first reference image and the second reference image is a specific reference image, based on the first initial MV and the second initial MV, a predicted image block of the current image block; calculated and obtained,
If both the first reference image and the second reference image are non-specific reference images, the current image based on the gradient values of the pixel points directed by the first initial MV and the second initial MV calculating and obtaining the MV of the block, and calculating and obtaining a predicted image block of the current image block based on the MV of the current image block.
(Item 89)
Specifically, the above processor:
89. Claimed in item 88 for calculating and obtaining the MV of said current image block based on the gradient values of the pixel points to which said first initial MV and said second initial MV are directed and an optimization principle. equipment.
(Item 90)
Specifically, the above processor:
for weighted summing a first reference image block to which said first initial MV points and a second reference image block to which said second initial MV points to obtain a prediction image block of said current image block 88. Or the device according to 89.
(Item 91)
91. The apparatus according to any one of items 88-90, wherein the prediction directions of the first initial MV and the second initial MV are different.
(Item 92)
The prediction directions of the first initial MV and the second initial MV are the same, the first initial MV and the second initial MV are not 0, and the first reference image and the second reference image are 92. Apparatus according to any one of items 88-91, wherein the images are different.
(Item 93)
Specifically, the above processor:
based on the MV of the current image block, the first reference image block to which the first initial MV is directed, and the second reference image block to which the second initial MV is directed, predicting the predicted image block of the current image block. 93. Apparatus according to any one of items 88-92 for obtaining by calculation.
(Item 94)
94. Apparatus according to any one of items 88-93, wherein said specific reference image comprises at least one of a long-term reference image, a synthetic frame and a non-output frame.
(Item 95)
The processor further:
determining that at least one of the first reference image and the second reference image is a non-output frame, and at least one of the first reference image and the second reference image is a synthesized frame; 94. The method according to any one of items 88 to 94, for determining that at least one of the first reference image and the second reference image is the specific reference image when further determining that device.
(Item 96)
96. Apparatus according to any one of items 88-95, wherein said non-specific reference images comprise short-term reference images.
(Item 97)
A computer readable storage medium,
88. A computer readable storage medium on which commands are stored which, when run on the computer, cause the computer to perform the image processing method according to any one of items 79-87.
(Item 98)
A coding device comprising an image processing apparatus according to any one of items 88-96.
(Item 99)
A decoding device comprising an image processing apparatus according to any one of items 88-96.

Claims (4)

identifying a first initial motion vector and a second initial motion vector of a current image block, wherein the first initial motion vector is a forward frame of the current image containing the current image block; pointing to a reference image, the second initial motion vector pointing to a second reference image that is a backward frame of the current image, and the distance from the first reference image to the current image being the second is the same as the distance from a reference image to the current image;
by a first reference image block directed by the first initial motion vector and the second initial motion vector, corresponding to specifying that the first reference image and the second reference image are short-term reference images; weighted summing directed second reference image blocks to obtain a weighted image block; and performing a specified operation comprising: performing a prediction for said current image block according to said weighted image block;
and C. not performing the specified action in response to identifying that the first reference image or the second reference image is a long-term reference image. a memory for storing computer-executable instructions;
By executing the said command,
identifying a first initial motion vector and a second initial motion vector of a current image block, wherein the first initial motion vector is a forward frame of the current image containing the current image block; pointing to a reference image, the second initial motion vector pointing to a second reference image that is a backward frame of the current image, and the distance from the first reference image to the current image being the second is the same as the distance from a reference image to the current image;
by a first reference image block directed by the first initial motion vector and the second initial motion vector, corresponding to specifying that the first reference image and the second reference image are short-term reference images; weighted summing directed second reference image blocks to obtain a weighted image block; and performing a specified operation comprising: performing a prediction for said current image block according to said weighted image block;
An encoder comprising: not performing the specified action in response to identifying that the first reference image or the second reference image is a long-term reference image. identifying a first initial motion vector and a second initial motion vector of a current image block, wherein the first initial motion vector is a forward frame of the current image containing the current image block; pointing to a reference image, the second initial motion vector pointing to a second reference image that is a backward frame of the current image, and the distance from the first reference image to the current image being the second is the same as the distance from a reference image to the current image;
by a first reference image block directed by the first initial motion vector and the second initial motion vector, corresponding to specifying that the first reference image and the second reference image are short-term reference images; weighted summing directed second reference image blocks to obtain a weighted image block; and performing a specified operation comprising: performing a prediction for said current image block according to said weighted image block;
and not performing the specified operation in response to identifying that the first reference image or the second reference image is a long-term reference image. temporary computer-readable storage medium; identifying a first initial motion vector and a second initial motion vector of a current image block, wherein the first initial motion vector is a forward frame of the current image containing the current image block; pointing to a reference image, the second initial motion vector pointing to a second reference image that is a backward frame of the current image, and the distance from the first reference image to the current image being the second is the same as the distance from a reference image to the current image;
by a first reference image block directed by the first initial motion vector and the second initial motion vector, corresponding to specifying that the first reference image and the second reference image are short-term reference images; weighted summing directed second reference image blocks to obtain a weighted image block; and performing a specified operation comprising: performing a prediction for said current image block according to said weighted image block;
and not executing the specified operation in response to identifying that the first reference image or the second reference image is a long-term reference image.

Images