US20210058637A1 - Efficient affine merge motion vector derivation - Google Patents

Efficient affine merge motion vector derivation Download PDF

Info

Publication number
US20210058637A1
US20210058637A1 US17/090,122 US202017090122A US2021058637A1 US 20210058637 A1 US20210058637 A1 US 20210058637A1 US 202017090122 A US202017090122 A US 202017090122A US 2021058637 A1 US2021058637 A1 US 2021058637A1
Authority
US
United States
Prior art keywords
block
blocks
sub
current
current video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US17/090,122
Other languages
English (en)
Inventor
Kai Zhang
Li Zhang
Hongbin Liu
Yue Wang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing ByteDance Network Technology Co Ltd
ByteDance Inc
Original Assignee
Beijing ByteDance Network Technology Co Ltd
ByteDance Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing ByteDance Network Technology Co Ltd, ByteDance Inc filed Critical Beijing ByteDance Network Technology Co Ltd
Assigned to BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD. reassignment BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LIU, HONGBIN, WANG, YUE
Assigned to BYTEDANCE INC. reassignment BYTEDANCE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ZHANG, KAI, ZHANG, LI
Publication of US20210058637A1 publication Critical patent/US20210058637A1/en
Priority to US17/490,220 priority Critical patent/US20220046267A1/en
Priority to US18/503,360 priority patent/US20240098295A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/527—Global motion vector estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/537—Motion estimation other than block-based
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
    • H04N19/196—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/513—Processing of motion vectors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/513—Processing of motion vectors
    • H04N19/517—Processing of motion vectors by encoding
    • H04N19/52—Processing of motion vectors by encoding by predictive encoding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/577—Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/593—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques

Definitions

  • This patent document relates to video coding and decoding techniques, devices and systems.
  • This document discloses techniques that can be used in video coding and decoding embodiments to improve performance of sub-block based coding, and in particular, when using affine motion coding mode.
  • a video processing method includes partitioning a current block into sub-blocks; deriving, for each sub-block, a motion vector, wherein the motion vector for each sub-block is associated with a position for that sub-block according to a position rule; and processing a bitstream representation of the current block using motion vectors for the sub-blocks.
  • a video processing method comprises: deriving, for a conversion between a current block and a bitstream representation of the current block using affine mode, motion vectors at control points of the current block based on a position rule; and performing the conversion between the current block and the bitstream representation using the motion vectors, and wherein the position rule specifies to exclude use of non-adjacent neighboring blocks for the deriving.
  • a method of video processing comprises: determining, for a conversion between a current block and a bitstream representation of the current block, a list of affine merge candidates for the conversion by including merge candidates from one or more neighboring block that satisfy a validity criterion based on positions of the one or more neighboring blocks; and performing the conversion between the current block and the bitstream representation using motion vectors.
  • a video encoder device that implements a video encoding method described herein is disclosed.
  • the various techniques described herein may be embodied as a computer program product stored on a non-transitory computer readable media.
  • the computer program product includes program code for carrying out the methods described herein.
  • a video decoder apparatus may implement a method as described herein.
  • FIG. 1 shows an example of sub-block based prediction.
  • FIG. 2 illustrates an example of simplified affine motion model.
  • FIG. 3 shows an example of affine motion vector field (MVF) per sub-block.
  • FIG. 4 shows an example of motion vector prediction (MVP) for AF_INTER mode.
  • FIG. 5A and FIG. 5B depict examples of Candidates for AF_MERGE.encoding mode.
  • FIG. 6 shows an example process of advanced temporal motion vector predictor (ATMVP) motion prediction for a coding unit (CU).
  • ATMVP advanced temporal motion vector predictor
  • FIG. 7 shows an example of one CU with four sub-blocks (A-D) and its neighboring blocks (a-d).
  • FIG. 8 shows an example of optical flow trajectory in video coding.
  • FIGS. 9A and 9B shows an example of bi-directional optical (BIO) coding technique without block extension.
  • FIG. 9A shows an example of access positions outside of the block and
  • FIG. 9B shows an example of padding used in order to avoid extra memory access and calculation.
  • FIG. 10 shows an example of bilateral matching.
  • FIG. 11 shows an example of template matching.
  • FIG. 12 shows an example of Unilateral motion estimation (ME) in frame rate up-conversion (FRUC).
  • ME Unilateral motion estimation
  • FRUC frame rate up-conversion
  • FIG. 13 illustrate an example implementation of interweaved prediction.
  • FIG. 14 shows an example of different positions to derive MVs for different sub-blocks, where stars represent the different positions.
  • FIG. 15 shows examples of neighboring blocks to derive v0x and v0y.
  • FIG. 16A and FIG. 16B Examples of derive MVs for the affine merge mode from left adjacent blocks coded with the affine mode (a) or from top adjacent blocks coded with the affine mode.
  • FIG. 17 shows an example of a neighboring block and a current block belonging to different coding tree unit (CTU) lines, in which an affine merge candidate from such a neighboring block is treated as invalid.
  • CTU coding tree unit
  • FIG. 18 shows an example of interweaved prediction with two dividing patterns in accordance with the disclosed technology.
  • FIG. 19A shows an example dividing pattern in which block is divided into 4 ⁇ 4 sub-blocks in accordance with the disclosed technology.
  • FIG. 19B shows an example dividing pattern in which a block is divided into 8 ⁇ 8 sub-blocks in accordance with the disclosed technology.
  • FIG. 19C shows an example dividing pattern in which a block is divided into 4 ⁇ 8 sub-blocks in accordance with the disclosed technology.
  • FIG. 19D shows an example dividing pattern in which a block is divided into 8 ⁇ 4 sub-blocks in accordance with the disclosed technology.
  • FIG. 19E shows an example dividing pattern in which a block is divided into non-uniform sub-blocks in accordance with the disclosed technology.
  • FIG. 19F shows another example dividing pattern in which a block is divided into non-uniform sub-blocks in accordance with the disclosed technology.
  • FIG. 19G shows yet another example dividing pattern in which a block is divided into non-uniform sub-blocks in accordance with the disclosed technology.
  • FIG. 20 is a block diagram of an example of a hardware platform for implementing a visual media decoding or a visual media encoding technique described in the present document.
  • FIG. 21 is a flowchart for an example method of video processing.
  • FIG. 22 is a flowchart for another example method of video processing.
  • FIG. 23 is a flowchart for another example method of video processing.
  • Section headings are used in the present document to improve readability and do not limit the techniques and embodiments described in a section to only that section.
  • This patent document is related to video/image coding technologies. Specifically, it is related to sub-block based prediction in video/image coding. It may be applied to the existing video coding standard like HEVC, or the standard (Versatile Video Coding) to be finalized. It may be also applicable to future video/image coding standards or video/image codec.
  • Sub-block based prediction is first introduced into the video coding standard by HEVC Annex I (3D-HEVC).
  • a block such as a Coding Unit (CU) or a Prediction Unit (PU)
  • PU Prediction Unit
  • Different sub-block may be assigned different motion information, such as reference index or Motion Vector (MV), and Motion Compensation (MC) is performed individually for each sub-block.
  • FIG. 1 demonstrates the concept of sub-block based prediction.
  • JVET Joint Video Exploration Team
  • sub-block based prediction is adopted in several coding tools, such as affine prediction, Alternative temporal motion vector prediction (ATMVP), spatial-temporal motion vector prediction (STMVP), Bi-directional Optical flow (BIO) and Frame-Rate Up Conversion (FRUC).
  • ATMVP Alternative temporal motion vector prediction
  • STMVP spatial-temporal motion vector prediction
  • BIO Bi-directional Optical flow
  • FRUC Frame-Rate Up Conversion
  • HEVC High Efficiency Video Coding
  • MCP motion compensation prediction
  • JEM JEM
  • affine transform motion compensation prediction is applied. As shown FIG. 2 , the affine motion field of the block is described by two control point motion vectors.
  • the motion vector field (MVF) of a block is described by the following equation:
  • sub-block based affine transform prediction is applied.
  • the sub-block size M ⁇ N is derived as in Eq. (2), where MvPre is the motion vector fraction accuracy ( 1/16 in JEM), (v 2x , v 2y ) is motion vector of the bottom-left control point, calculated according to Equation 1.
  • M and N should be adjusted downward if necessary to make it a divisor of w and h, respectively.
  • the motion vector of the center sample of each sub-block is calculated according to Eq. (1), and rounded to 1/16 fraction accuracy. Then the motion compensation interpolation filters are applied to generate the prediction of each sub-block with derived motion vector.
  • the high accuracy motion vector of each sub-block is rounded and saved as the same accuracy as the normal motion vector.
  • affine motion modes there are two affine motion modes: AF_INTER mode and AF_MERGE mode.
  • AF_INTER mode For CUs with both width and height larger than 8, AF_INTER mode can be applied.
  • An affine flag in CU level is signalled in the bitstream to indicate whether AF_INTER mode is used.
  • v 0 is selected from the motion vectors of the block A, B or C.
  • the motion vector from the neighbour block is scaled according to the reference list and the relationship among the POC of the reference for the neighbour block, the POC of the reference for the current CU and the POC of the current CU. And the approach to select v 1 from the neighbour block D and E is similar. If the number of candidate list is smaller than 2, the list is padded by the motion vector pair composed by duplicating each of the AMVP candidates. When the candidate list is larger than 2, the candidates are firstly sorted according to the consistency of the neighbouring motion vectors (similarity of the two motion vectors in a pair candidate) and only the first two candidates are kept. An RD cost check is used to determine which motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU.
  • CPMVP control point motion vector prediction
  • an index indicating the position of the CPMVP in the candidate list is signalled in the bitstream.
  • a CU When a CU is applied in AF_MERGE mode, it gets the first block coded with affine mode from the valid neighbour reconstructed blocks. And the selection order for the candidate block is from left, above, above right, left bottom to above left as shown in FIG. 5A . If the neighbour left bottom block A is coded in affine mode as shown in FIG. 5B , the motion vectors v 2 , v 3 and v 4 of the top left corner, above right corner and left bottom corner of the CU which contains the block A are derived. And the motion vector v 0 of the top left corner on the current CU is calculated according to v 2 , v 3 and v 4 . Secondly, the motion vector v 1 of the above right of the current CU is calculated.
  • the MVF of the current CU is generated.
  • an affine flag is signalled in the bitstream when there is at least one neighbour block is coded in affine mode.
  • the motion vectors temporal motion vector prediction is modified by fetching multiple sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.
  • the sub-CUs are square N ⁇ N blocks (N is set to 4 by default).
  • ATMVP predicts the motion vectors of the sub-CUs within a CU in two steps.
  • the first step is to identify the corresponding block in a reference picture with a so-called temporal vector.
  • the reference picture is called the motion source picture.
  • the second step is to split the current CU into sub-CUs and obtain the motion vectors as well as the reference indices of each sub-CU from the block corresponding to each sub-CU, as shown in FIG. 6 .
  • a reference picture and the corresponding block is determined by the motion information of the spatial neighbouring blocks of the current CU.
  • the first merge candidate in the merge candidate list of the current CU is used.
  • the first available motion vector as well as its associated reference index are set to be the temporal vector and the index to the motion source picture. This way, in ATMVP, the corresponding block may be more accurately identified, compared with TMVP, wherein the corresponding block (sometimes called collocated block) is always in a bottom-right or center position relative to the current CU.
  • a corresponding block of the sub-CU is identified by the temporal vector in the motion source picture, by adding to the coordinate of the current CU the temporal vector.
  • the motion information of its corresponding block (the smallest motion grid that covers the center sample) is used to derive the motion information for the sub-CU.
  • the motion information of a corresponding N ⁇ N block is identified, it is converted to the motion vectors and reference indices of the current sub-CU, in the same way as TMVP of HEVC, wherein motion scaling and other procedures apply.
  • the decoder checks whether the low-delay condition (i.e.
  • motion vector MV x the motion vector corresponding to reference picture list X
  • motion vector MV y the motion vector corresponding to 0 or 1 and Y being equal to 1-X
  • FIG. 7 illustrates this concept. Let us consider an 8 ⁇ 8 CU which contains four 4 ⁇ 4 sub-CUs A, B, C, and D. The neighbouring 4 ⁇ 4 blocks in the current frame are labelled as a, b, c, and d.
  • the motion derivation for sub-CU A starts by identifying its two spatial neighbours.
  • the first neighbour is the N ⁇ N block above sub-CU A (block c). If this block c is not available or is intra coded the other N ⁇ N blocks above sub-CU A are checked (from left to right, starting at block c).
  • the second neighbour is a block to the left of the sub-CU A (block b). If block b is not available or is intra coded other blocks to the left of sub-CU A are checked (from top to bottom, staring at block b).
  • the motion information obtained from the neighbouring blocks for each list is scaled to the first reference frame for a given list.
  • temporal motion vector predictor (TMVP) of sub-block A is derived by following the same procedure of TMVP derivation as specified in HEVC.
  • the motion information of the collocated block at location D is fetched and scaled accordingly.
  • all available motion vectors (up to 3) are averaged separately for each reference list. The averaged motion vector is assigned as the motion vector of the current sub-CU.
  • Bi-directional Optical flow is sample-wise motion refinement which is performed on top of block-wise motion compensation for bi-prediction.
  • the sample-level motion refinement doesn't use signalling.
  • ⁇ I (k) / ⁇ x, ⁇ I (k) / ⁇ y are horizontal and vertical components of the I (k) gradient, respectively.
  • the motion vector field (v x , v y ) is given by an equation
  • pred BIO 1/2 ⁇ ( I (0) +I (1) +v x /2 ⁇ ( ⁇ 1 ⁇ I (1) / ⁇ x ⁇ 0 ⁇ I (0) / ⁇ x )+ v y /2 ⁇ ( ⁇ 1 ⁇ I (1) / ⁇ y ⁇ 0 ⁇ I (0) / ⁇ y )).
  • ⁇ 0 and ⁇ 1 denote the distances to the reference frames as shown on a FIG. 8 .
  • the motion vector field (v 1 , v y ) is determined by minimizing the difference ⁇ between values in points A and B (intersection of motion trajectory and reference frame planes on FIGS. 9A and 9B ).
  • Model uses only first linear term of a local Taylor expansion for A:
  • the JEM uses a simplified approach making first a minimization in the vertical direction and then in the horizontal direction. This results in
  • d bit depth of the video samples.
  • I (k) , ⁇ I (k) / ⁇ x, ⁇ I (k) / ⁇ y are calculated only for positions inside the current block.
  • (2M+1) ⁇ (2M+1) square window SI centered in currently predicted point on a boundary of predicted block needs to accesses positions outside of the block (as shown in FIG. 9A ).
  • values of I (k) , ⁇ I (k) / ⁇ x, ⁇ I (k) / ⁇ y outside of the block are set to be equal to the nearest available value inside the block. For example, this can be implemented as padding, as shown in FIG. 9B .
  • BIO it's possible that the motion field can be refined for each sample.
  • a block-based design of BIO is used in the JEM.
  • the motion refinement is calculated based on 4 ⁇ 4 block.
  • the values of s n in Eq. (9) of all samples in a 4 ⁇ 4 block are aggregated, and then the aggregated values of s n in are used to derived BIO motion vectors offset for the 4 ⁇ 4 block. More specifically, the following formula is used for block-based BIO derivation:
  • MV regiment of BIO might be unreliable due to noise or irregular motion. Therefore, in BIO, the magnitude of MV regiment is clipped to a threshold value thBIO.
  • the threshold value is determined based on whether the reference pictures of the current picture are all from one direction. If all the reference pictures of the current picture are from one direction, the value of the threshold is set to 12 ⁇ 2 14-d ; otherwise, it is set to 12 ⁇ 2 13-d .
  • Gradients for BIO are calculated at the same time with motion compensation interpolation using operations consistent with HEVC motion compensation process (2D separable FIR).
  • the input for this 2D separable FIR is the same reference frame sample as for motion compensation process and fractional position (fracX, fracY) according to the fractional part of block motion vector.
  • gradient filter BIOfilterG is applied in horizontal direction corresponding to the fractional position fracX with de-scaling shift by 18-d.
  • BIOfilterG corresponding to the fractional position fracY with de-scaling shift d-8
  • signal displacement is performed using BIOfilterS in horizontal direction corresponding to the fractional position fracX with de-scaling shift by 18-d.
  • the length of interpolation filter for gradients calculation BIOfilterG and signal displacement BIOfilterF is shorter (6-tap) in order to maintain reasonable complexity.
  • Table shows the filters used for gradients calculation for different fractional positions of block motion vector in BIO. Table shows the interpolation filters used for prediction signal generation in BIO.
  • BIO is applied to all bi-predicted blocks when the two predictions are from different reference pictures.
  • BIO is disabled.
  • BIO is not applied during the OBMC process. This means that BIO is only applied in the MC process for a block when using its own MV and is not applied in the MC process when the MV of a neighboring block is used during the OBMC process.
  • a FRUC flag is signalled for a CU when its merge flag is true.
  • FRUC flag is false, a merge index is signalled and the regular merge mode is used.
  • FRUC flag is true, an additional FRUC mode flag is signalled to indicate which method (bilateral matching or template matching) is to be used to derive motion information for the block.
  • the decision on whether using FRUC merge mode for a CU is based on RD cost selection as done for normal merge candidate. That is the two matching modes (bilateral matching and template matching) are both checked for a CU by using RD cost selection. The one leading to the minimal cost is further compared to other CU modes. If a FRUC matching mode is the most efficient one, FRUC flag is set to true for the CU and the related matching mode is used.
  • Motion derivation process in FRUC merge mode has two steps.
  • a CU-level motion search is first performed, then followed by a Sub-CU level motion refinement.
  • an initial motion vector is derived for the whole CU based on bilateral matching or template matching.
  • a list of MV candidates is generated and the candidate which leads to the minimum matching cost is selected as the starting point for further CU level refinement.
  • a local search based on bilateral matching or template matching around the starting point is performed and the MV results in the minimum matching cost is taken as the MV for the whole CU.
  • the motion information is further refined at sub-CU level with the derived CU motion vectors as the starting points.
  • the following derivation process is performed for a W ⁇ H CU motion information derivation.
  • MV for the whole W ⁇ H CU is derived.
  • the CU is further split into M ⁇ M sub-CUs.
  • the value of M is calculated as in (16)
  • D is a predefined splitting depth which is set to 3 by default in the JEM. Then the MV for each sub-CU is derived.
  • the bilateral matching is used to derive motion information of the current CU by finding the closest match between two blocks along the motion trajectory of the current CU in two different reference pictures.
  • the motion vectors MV0 and MV1 pointing to the two reference blocks shall be proportional to the temporal distances, i.e., TD0 and TD1, between the current picture and the two reference pictures.
  • the bilateral matching becomes mirror based bi-directional MV.
  • template matching is used to derive motion information of the current CU by finding the closest match between a template (top and/or left neighbouring blocks of the current CU) in the current picture and a block (same size to the template) in a reference picture. Except the aforementioned FRUC merge mode, the template matching is also applied to AMVP mode.
  • AMVP has two candidates.
  • template matching method a new candidate is derived. If the newly derived candidate by template matching is different to the first existing AMVP candidate, it is inserted at the very beginning of the AMVP candidate list and then the list size is set to two (meaning remove the second existing AMVP candidate).
  • AMVP mode only CU level search is applied.
  • the MV candidate set at CU level consists of:
  • each valid MV of a merge candidate is used as an input to generate a MV pair with the assumption of bilateral matching.
  • one valid MV of a merge candidate is (MVa, refa) at reference list A.
  • the reference picture refb of its paired bilateral MV is found in the other reference list B so that refa and refb are temporally at different sides of the current picture. If such a refb is not available in reference list B, refb is determined as a reference which is different from refa and its temporal distance to the current picture is the minimal one in list B.
  • MVb is derived by scaling MVa based on the temporal distance between the current picture and refa, refb.
  • MVs from the interpolated MV field are also added to the CU level candidate list. More specifically, the interpolated MVs at the position (0, 0), (W/2, 0), (0, H/2) and (W/2, H/2) of the current CU are added.
  • the original AMVP candidates are also added to CU level MV candidate set.
  • the MV candidate set at sub-CU level consists of:
  • the scaled MVs from reference pictures are derived as follows. All the reference pictures in both lists are traversed. The MVs at a collocated position of the sub-CU in a reference picture are scaled to the reference of the starting CU-level MV.
  • ATMVP and STMVP candidates are limited to the four first ones.
  • interpolated motion field is generated for the whole picture based on unilateral ME. Then the motion field may be used later as CU level or sub-CU level MV candidates.
  • the motion field of each reference pictures in both reference lists is traversed at 4 ⁇ 4 block level.
  • the motion of the reference block is scaled to the current picture according to the temporal distance TD0 and TD1 (the same way as that of MV scaling of TMVP in HEVC) and the scaled motion is assigned to the block in the current frame. If no scaled MV is assigned to a 4 ⁇ 4 block, the block's motion is marked as unavailable in the interpolated motion field.
  • the matching cost is a bit different at different steps.
  • the matching cost is the absolute sum difference (SAD) of bilateral matching or template matching.
  • SAD absolute sum difference
  • the matching cost C of bilateral matching at sub-CU level search is calculated as follows:
  • MV and MV s indicate the current MV and the starting MV, respectively.
  • SAD is still used as the matching cost of template matching at sub-CU level search.
  • MV is derived by using luma samples only. The derived motion will be used for both luma and chroma for MC inter prediction. After MV is decided, final MC is performed using 8-taps interpolation filter for luma and 4-taps interpolation filter for chroma.
  • MV refinement is a pattern based MV search with the criterion of bilateral matching cost or template matching cost.
  • two search patterns are supported—an unrestricted center-biased diamond search (UCBDS) and an adaptive cross search for MV refinement at the CU level and sub-CU level, respectively.
  • UMBDS center-biased diamond search
  • the MV is directly searched at quarter luma sample MV accuracy, and this is followed by one-eighth luma sample MV refinement.
  • the search range of MV refinement for the CU and sub-CU step are set equal to 8 luma samples.
  • the encoder can choose among uni-prediction from list0, uni-prediction from list1 or bi-prediction for a CU. The selection is based on a template matching cost as follows:
  • cost0 is the SAD of list0 template matching
  • cost1 is the SAD of list1 template matching
  • costBi is the SAD of bi-prediction template matching.
  • the value of factor is equal to 1.25, which means that the selection process is biased toward bi-prediction.
  • the inter prediction direction selection is only applied to the CU-level template matching process.
  • a dividing pattern is defined as the way to divide a block into sub-blocks, including the size of sub-blocks and the position of sub-blocks.
  • a corresponding prediction block may be generated by deriving motion information of each sub-block based on the dividing pattern. Therefore, even for one prediction direction, multiple prediction blocks may be generated by multiple dividing patterns. Alternatively, for each prediction direction, only a dividing pattern may be applied.
  • X prediction blocks of the current block denoted as P 0 , P 1 , . . . , P x-1 are generated by sub-block based prediction with the X dividing patterns.
  • the final prediction of the current block denoted as P, can be generated as
  • FIG. 13 shows an example of interweaved prediction with two dividing patterns.
  • the coordinate of the left-top point of a CU and the size of the CU must be stored by each 4 ⁇ 4 block belonging to the CU. This information is not required to be stored in HEVC
  • the decoder must access MVs of 4 ⁇ 4 blocks not adjacent to the current CU.
  • the decoder only needs to access MVs of 4 ⁇ 4 blocks adjacent to the current CU.
  • encoding includes “transcoding” in which source video in a non-compressed format is encoded into another coded format.
  • FIG. 14 shows an example of different positions to derive MVs for different sub-blocks. Stars represent the positions. As can be seen, various different positions may be used for MV derivation.
  • a ( mvL 1 y ⁇ mvL 0 y )/ ⁇
  • b ⁇ ( mvL 1 x ⁇ mvL 0 x )/ ⁇ .
  • a ( mvT 1 x ⁇ mvT 0 x )/ ⁇
  • b ( mvT 1 y ⁇ mvT 0 y )/ ⁇ .
  • FIGS. 16A and 16B show examples of derive MVs for the affine merge mode from left adjacent blocks coded with the affine mode ( FIG. 16A ) or from top adjacent blocks coded with the affine mode ( FIG. 16B ).
  • FIG. 17 shows an example of a neighboring block and a current block belonging to different CTU lines.
  • an affine merge candidate from a neighboring block is treated as invalid (not be put into the merge candidate list) if the neighboring block belongs to a CTU line different from the current CTU line.
  • FIG. 18 shows an example of interweaved prediction with two dividing patterns in accordance with the disclosed technology.
  • a current block 1300 can be divided into multiple patterns. For example, as shown in FIG. 18 , the current block is divided into both Pattern 0 ( 1301 ) and Pattern 1 ( 1302 ). Two prediction blocks, P 0 ( 1303 ) and P 1 ( 1304 ), are generated.
  • a final prediction block P ( 1305 ) of the current block 1300 can be generated by computing a weighted sum of P 0 ( 1303 ) and P 1 ( 1304 ).
  • X prediction blocks of the current block denoted as P 0 , P 1 , . . . , P X-1
  • P The final prediction of the current block
  • (x, y) is the coordinate of a pixel in the block and w i (x, y) is the weighting value of P i .
  • the weights can be expressed as:
  • N is a non-negative value.
  • bit-shifting operation in Eq. (16) can also be expressed as:
  • the sum of the weights being a power of two allows a more efficient computation of the weighted sum P by performing a bit-shifting operation instead of a floating-point division.
  • Dividing patterns can have different shapes, or sizes, or positions of sub-blocks. In some embodiments, a dividing pattern may include irregular sub-block sizes.
  • FIGS. 19A-G show several examples of dividing patterns for a 16 ⁇ 16 block. In FIG. 19A , a block is divided into 4 ⁇ 4 sub-blocks in accordance with the disclosed technology. This pattern is also used in JEM.
  • FIG. 19B shows an example of a block being divided into 8 ⁇ 8 sub-blocks in accordance with the disclosed technology.
  • FIG. 19C shows an example of the block being divided into 8 ⁇ 4 sub-blocks in accordance with the disclosed technology.
  • FIG. 19D shows an example of the block being divided into 4 ⁇ 8 sub-blocks in accordance with the disclosed technology.
  • FIG. 19A-G show several examples of dividing patterns for a 16 ⁇ 16 block.
  • FIG. 19A a block is divided into 4 ⁇ 4 sub-blocks in accordance with the disclosed technology. This pattern is also used in JEM.
  • FIG. 19E a portion of the block is divided into 4 ⁇ 4 sub-blocks in accordance with the disclosed technology.
  • the pixels at block boundaries are divided in smaller sub-blocks with sizes like 2 ⁇ 4, 4 ⁇ 2 or 2 ⁇ 2. Some sub-blocks may be merged to form larger sub-blocks.
  • FIG. 19F shows an example of adjacent sub-blocks, such as 4 ⁇ 4 sub-blocks and 2 ⁇ 4 sub-blocks, that are merged to form larger sub-blocks with sizes like 6 ⁇ 4, 4 ⁇ 6 or 6 ⁇ 6.
  • FIG. 19G a portion of the block is divided into 8 ⁇ 8 sub-blocks. The pixels at block boundaries are divided in smaller sub-blocks with sizes like 8 ⁇ 4, 4 ⁇ 8 or 4 ⁇ 4 instead.
  • the shapes and sizes of sub-blocks in sub-block based prediction can be determined based on the shape and/or size of the coding block and/or coded block information.
  • the sub-blocks have a size of 4 ⁇ N (or 8 ⁇ N, etc.) when the current block has a size of M ⁇ N. That is, the sub-blocks have the same height as the current block.
  • the sub-blocks have a size of M ⁇ 4 (or M ⁇ 8, etc.) when the current block has a size of M ⁇ N. That is, the sub-blocks have the same width as the current block.
  • the sub-blocks have a size of A ⁇ B with A>B (e.g., 8 ⁇ 4) when the current block has a size of M ⁇ N, where M>N.
  • the sub-blocks can have the size of B ⁇ A (e.g. 4 ⁇ 8).
  • the current block has a size of M ⁇ N.
  • whether to apply interweaved prediction can be determined based on the inter-prediction direction. For example, in some embodiments, the interweaved prediction may be applied for bi-prediction but not for uni-prediction. As another example, when multiple-hypothesis is applied, the interweaved prediction may be applied for one prediction direction when there are more than one reference blocks.
  • how to apply interweaved prediction may also be determined based on the inter-prediction direction.
  • a bi-predicted block with sub-block based prediction is divided into sub-blocks with two different dividing patterns for two different reference lists. For example, a bi-predicted block is divided into 4 ⁇ 8 sub-blocks as shown in FIG. 19D when predicted from reference list 0 (L0). The same block is divided into 8 ⁇ 4 sub-blocks as shown in FIG. 19C when predicted from reference list 1 (L1). The final prediction P is calculated as
  • P 0 and P 1 are predictions from L0 and L1, respectively.
  • w 0 and w 1 are weighting values for L0 and L1, respectively.
  • XL is the number of dividing patterns for list L.
  • P i L (x, y) is the prediction generated with the i th dividing pattern and w i L (x, y) is the weighting value of P i L (x, y).
  • XL 2 dividing patterns are applied for list L. In the first dividing pattern, the block is divided into 4 ⁇ 8 sub-blocks as shown in FIG. 19D . In the second dividing pattern, the block is divided into 8 ⁇ 4 sub-blocks as shown in FIG. 19D .
  • a bi-predicted block with sub-block based prediction is considered as a combination of two uni-predicted block from L0 and L1 respectively.
  • the prediction from each list can be derived as described in the above example.
  • the final prediction P can be calculated as
  • parameters a and b are two additional weights applied to the two internal prediction blocks.
  • both a and b can be set to 1. Similar to the example above, because fewer sub-blocks are used for prediction in each direction (e.g., 4 ⁇ 8 sub-blocks as opposed to 8 ⁇ 8 sub-blocks), the bandwidth usage is better than or on par with the existing sub-block based methods. At the same time, the prediction results can be improved by using larger sub-blocks.
  • a single non-uniform pattern can be used in each uni-predicted block. For example, for each list L (e.g., L0 or L1), the block is divided into a different pattern (e.g., as shown in FIG. 19E or FIG. 19F ). The use of a smaller number of sub-blocks reduces the demand on bandwidth. The non-uniformity of the sub-blocks also increases robustness of the prediction results.
  • L e.g., L0 or L1
  • the block is divided into a different pattern (e.g., as shown in FIG. 19E or FIG. 19F ).
  • the use of a smaller number of sub-blocks reduces the demand on bandwidth.
  • the non-uniformity of the sub-blocks also increases robustness of the prediction results.
  • a multiple-hypothesis coded block there can be more than one prediction blocks generated by different dividing patterns for each prediction direction (or reference picture list). Multiple prediction blocks can be used to generate the final prediction with additional weights applied. For example, the additional weights may be set to 1/M wherein M is the total number of generated prediction blocks.
  • the encoder can determine whether and how to apply the interweaved prediction.
  • the encoder then can transmit information corresponding to the determination to the decoder at a sequence level, a picture level, a view level, a slice level, a Coding Tree Unit (CTU) (also known as a Largest Coding Unit (LCU)) level, a CU level, a PU level, a Tree Unit (TU) level, or a region level (which may include multiple CUs/PUs/Tus/LCUs).
  • CTU Coding Tree Unit
  • LCU Largest Coding Unit
  • TU Tree Unit
  • region level which may include multiple CUs/PUs/Tus/LCUs.
  • the information can be signaled in a Sequence Parameter Set (SPS), a view parameter set (VPS), a Picture Parameter Set (PPS), a Slice Header (SH), a CTU/LCU, a CU, a PU, a TU, or a first block of a region.
  • SPS Sequence Parameter Set
  • VPS view parameter set
  • PPS Picture Parameter Set
  • SH Slice Header
  • the interweaved prediction applies to existing sub-block methods like the affine prediction, ATMVP, STMVP, FRUC, or BIO. In such cases, no additional signaling cost is needed.
  • new sub-block merge candidates generated by the interweaved prediction can be inserted into a merge list, e.g., interweaved prediction+ATMVP, interweaved prediction+STMVP, interweaved prediction+FRUC etc.
  • the dividing patterns to be used by the current block can be derived based on information from spatial and/or temporal neighboring blocks. For example, instead of relying on the encoder to signal the relevant information, both encoder and decoder can adopt a set of predetermined rules to obtain dividing patterns based on temporal adjacency (e.g., previously used dividing patterns of the same block) or spatial adjacency (e.g., dividing patterns used by neighboring blocks).
  • temporal adjacency e.g., previously used dividing patterns of the same block
  • spatial adjacency e.g., dividing patterns used by neighboring blocks.
  • sub-block prediction based coding techniques e.g., affine, or ATMVP
  • other coded information e.g., skip or non-skip modes, and/or MV information
  • the encoder can determine the weighting values, and transmit the values to the decoder at sequence level, picture level, slice level, CTU/LCU level, CU level, PU level, or region level (which may include multiple CUs/PUs/Tus/LCUs).
  • the weighting values can be signaled in a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), a Slice Header (SH), a CTU/LCU, a CU, a PU, or a first block of a region.
  • the weighting values can be derived from the weighting values of a spatial and/or temporal neighboring block.
  • the interweaved prediction techniques disclosed herein can be applied in one, some, or all coding techniques of sub-block based prediction.
  • the interweaved prediction techniques can be applied to affine prediction, while other coding techniques of sub-block based prediction (e.g., ATMVP, STMVP, FRUC or BIO) do not use the interweaved prediction.
  • other coding techniques of sub-block based prediction e.g., ATMVP, STMVP, FRUC or BIO
  • all of affine, ATMVP, and STMVP apply the interweaved prediction techniques disclosed herein.
  • FIG. 20 is a block diagram of an example video bitstream processing apparatus 2000 .
  • the apparatus 2000 may be used to implement one or more of the methods described herein.
  • the apparatus 2000 may be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, and so on.
  • the apparatus 2000 may include one or more processors 2002 , one or more memories 2004 and video processing hardware 2006 .
  • the processor(s) 2002 may be configured to implement one or more methods described in the present document.
  • the memory (memories) 2004 may be used for storing data and code used for implementing the methods and techniques described herein.
  • the video processing hardware 2006 may be used to implement, in hardware circuitry, some techniques described in the present document. Note that partial or full externality of memory 2004 and circuitry 2006 from the processor 2002 electronics is optional and is an implementation choice.
  • FIG. 21 shows a flowchart for an example method 2100 for video processing.
  • the method 2100 includes partitioning ( 2102 ) a current block into sub-blocks.
  • the method 2100 further includes deriving ( 2104 ), for each sub-block, a motion vector, wherein the motion vector for each sub-block is associated with a position for that sub-block according to a position rule.
  • the method 2100 further includes processing ( 2106 ) a bitstream representation of the current block using motion vectors for the sub-blocks.
  • FIG. 22 is a flowchart for an example method 2200 for video processing.
  • the method 2200 includes deriving ( 2202 ), for a conversion between a current block and a bitstream representation of the current block using affine mode, motion vectors at control points of the current block based on a position rule.
  • the method 2200 further includes performing ( 2204 ) the conversion between the current block and the bitstream representation using the motion vectors.
  • the position rule may specify to exclude the use of non-adjacent neighboring blocks for the deriving.
  • the motion vectors may be derived without using information of a neighboring coding unit that includes at least one non-adjacent 4 ⁇ 4 block of the current block.
  • the method further includes storing and reusing at least some affine parameters of a previously converted neighboring block. In some implementations, the storing and the reusing of at least some affine parameters can be performed in two steps separately from each other.
  • FIG. 23 is a flowchart for an example method 2300 for video processing.
  • the method 2300 includes determining ( 2302 ), for a conversion between a current block and a bitstream representation of the current block, a list of affine merge candidates for the conversion by including merge candidates from one or more neighboring block that satisfy a validity criterion based on positions of the one or more neighboring blocks.
  • the method 2300 further includes performing ( 2304 ) the conversion between the current block and the bitstream representation using the motion vectors.
  • a method of video processing comprising: partitioning a current block into sub-blocks; deriving, for each sub-block, a motion vector, wherein the motion vector for each sub-block is associated with a position for that sub-block according to a position rule; and processing a bitstream representation of the current block using motion vectors for the sub-blocks.
  • a video processing method (e.g., method 2200 shown in FIG. 22 ), comprising: deriving, for a conversion between a current block and a bitstream representation of the current block using affine mode, motion vectors at control points of the current block based on a position rule; and performing the conversion between the current block and the bitstream representation using the motion vectors, and wherein the position rule specifies to exclude use of non-adjacent neighboring blocks for the deriving.
  • a method of video processing (e.g., method 2300 shown in FIG. 23 ), comprising: determining, for a conversion between a current block and a bitstream representation of the current block, a list of affine merge candidates for the conversion by including merge candidates from one or more neighboring block that satisfy a validity criterion based on positions of the one or more neighboring blocks; and performing the conversion between the current block and the bitstream representation using motion vectors.
  • a video decoding apparatus comprising a processor configured to implement a method recited in one or more of clauses 1 to 34.
  • a video encoding apparatus comprising a processor configured to implement a method recited in one or more of clauses 1 to 34.
  • a computer-readable program medium having code stored thereupon, the code comprising instructions that, when executed by a processor, causing the processor to implement a method recited in one or more of clauses 1 to 34.
  • the disclosed and other embodiments, modules and the functional operations described in this document can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in combinations of one or more of them.
  • the disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus.
  • the computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more them.
  • data processing apparatus encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.
  • a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a computer program does not necessarily correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
  • the processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
  • a processor will receive instructions and data from a read only memory or a random-access memory or both.
  • the essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
  • semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
  • magnetic disks e.g., internal hard disks or removable disks
  • magneto optical disks e.g., CD ROM and DVD-ROM disks.
  • the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
US17/090,122 2018-07-01 2020-11-05 Efficient affine merge motion vector derivation Abandoned US20210058637A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/490,220 US20220046267A1 (en) 2018-07-01 2021-09-30 Efficient affine merge motion vector derivation
US18/503,360 US20240098295A1 (en) 2018-07-01 2023-11-07 Efficient affine merge motion vector derivation

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
CN2018093943 2018-07-01
CNPCT/CN2018/093943 2018-07-01
CN2018095568 2018-07-13
CNPCT/CN2018/095568 2018-07-13
PCT/IB2019/055592 WO2020008334A1 (en) 2018-07-01 2019-07-01 Efficient affine merge motion vector derivation

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2019/055592 Continuation WO2020008334A1 (en) 2018-07-01 2019-07-01 Efficient affine merge motion vector derivation

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/490,220 Continuation US20220046267A1 (en) 2018-07-01 2021-09-30 Efficient affine merge motion vector derivation

Publications (1)

Publication Number Publication Date
US20210058637A1 true US20210058637A1 (en) 2021-02-25

Family

ID=67297216

Family Applications (3)

Application Number Title Priority Date Filing Date
US17/090,122 Abandoned US20210058637A1 (en) 2018-07-01 2020-11-05 Efficient affine merge motion vector derivation
US17/490,220 Pending US20220046267A1 (en) 2018-07-01 2021-09-30 Efficient affine merge motion vector derivation
US18/503,360 Pending US20240098295A1 (en) 2018-07-01 2023-11-07 Efficient affine merge motion vector derivation

Family Applications After (2)

Application Number Title Priority Date Filing Date
US17/490,220 Pending US20220046267A1 (en) 2018-07-01 2021-09-30 Efficient affine merge motion vector derivation
US18/503,360 Pending US20240098295A1 (en) 2018-07-01 2023-11-07 Efficient affine merge motion vector derivation

Country Status (7)

Country Link
US (3) US20210058637A1 (ko)
EP (1) EP3804327A1 (ko)
JP (2) JP2021530154A (ko)
KR (1) KR20210024487A (ko)
CN (2) CN110677675B (ko)
TW (1) TWI731363B (ko)
WO (1) WO2020008334A1 (ko)

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210400269A1 (en) * 2018-11-30 2021-12-23 Interdigital Vc Holdings, Inc. Triangle and multi-hypothesis combination for video coding and decoding
US11284068B2 (en) 2018-12-03 2022-03-22 Beijing Bytedance Network Technology Co., Ltd. Indication method of maximum number of candidates
US11431964B2 (en) 2018-11-22 2022-08-30 Beijing Bytedance Network Technology Co., Ltd. Coordination method for sub-block based inter prediction
US20220360804A1 (en) * 2018-09-21 2022-11-10 Canon Kabushiki Kaisha Video coding and decoding
US11523108B2 (en) 2019-08-10 2022-12-06 Beijing Bytedance Network Technology Co., Ltd. Position restriction for inter coding mode
US11539950B2 (en) 2019-10-02 2022-12-27 Beijing Bytedance Network Technology Co., Ltd. Slice level signaling in video bitstreams that include subpictures
US11553201B2 (en) 2019-04-02 2023-01-10 Beijing Bytedance Network Technology Co., Ltd. Decoder side motion vector derivation
US11558634B2 (en) 2018-11-20 2023-01-17 Beijing Bytedance Network Technology Co., Ltd. Prediction refinement for combined inter intra prediction mode
WO2023034640A1 (en) * 2021-09-06 2023-03-09 Beijing Dajia Internet Information Technology Co., Ltd. Candidate derivation for affine merge mode in video coding
WO2023049219A1 (en) * 2021-09-24 2023-03-30 Beijing Dajia Internet Information Technology Co., Ltd. Candidate derivation for affine merge mode in video coding
US11641467B2 (en) 2018-10-22 2023-05-02 Beijing Bytedance Network Technology Co., Ltd. Sub-block based prediction
US11695946B2 (en) 2019-09-22 2023-07-04 Beijing Bytedance Network Technology Co., Ltd Reference picture resampling in video processing
WO2023137234A1 (en) * 2022-01-17 2023-07-20 Beijing Dajia Internet Information Technology Co., Ltd. Methods and devices for candidate derivation for affine merge mode in video coding
WO2023185933A1 (en) * 2022-03-30 2023-10-05 Beijing Bytedance Network Technology Co., Ltd. Method, apparatus, and medium for video processing
US11825074B2 (en) 2018-10-08 2023-11-21 Beijing Bytedance Network Technology Co., Ltd Generation and usage of combined affine merge candidate
US11843725B2 (en) 2018-11-12 2023-12-12 Beijing Bytedance Network Technology Co., Ltd Using combined inter intra prediction in video processing
US11849108B2 (en) 2018-10-18 2023-12-19 Canon Kabushiki Kaisha Video coding and decoding
US11871025B2 (en) 2019-08-13 2024-01-09 Beijing Bytedance Network Technology Co., Ltd Motion precision in sub-block based inter prediction
US11930165B2 (en) 2019-03-06 2024-03-12 Beijing Bytedance Network Technology Co., Ltd Size dependent inter coding
US11956432B2 (en) 2019-10-18 2024-04-09 Beijing Bytedance Network Technology Co., Ltd Interplay between subpictures and in-loop filtering
US11956465B2 (en) 2018-11-20 2024-04-09 Beijing Bytedance Network Technology Co., Ltd Difference calculation based on partial position

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3878172A4 (en) * 2018-11-12 2022-08-24 HFI Innovation Inc. METHOD AND DEVICE FOR MULTIPLE HYPOTHESIS IN VIDEO CODING
JP7026064B2 (ja) * 2019-03-11 2022-02-25 Kddi株式会社 画像復号装置、画像復号方法及びプログラム
CN114071147B (zh) * 2020-07-29 2023-04-11 四川大学 一种基于双线性模型的vvc运动补偿方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9060176B2 (en) * 2009-10-01 2015-06-16 Ntt Docomo, Inc. Motion vector prediction in video coding
WO2016008157A1 (en) * 2014-07-18 2016-01-21 Mediatek Singapore Pte. Ltd. Methods for motion compensation using high order motion model
EP3205109A4 (en) * 2014-12-09 2018-03-07 MediaTek Inc. Method of motion vector predictor or merge candidate derivation in video coding
CN106303543B (zh) * 2015-05-15 2018-10-30 华为技术有限公司 视频图像编码和解码的方法、编码设备和解码设备
CN104935938B (zh) * 2015-07-15 2018-03-30 哈尔滨工业大学 一种混合视频编码标准中帧间预测方法
CN109274974B (zh) * 2015-09-29 2022-02-11 华为技术有限公司 图像预测的方法及装置
GB2561507B (en) * 2016-01-07 2021-12-22 Mediatek Inc Method and apparatus for affine merge mode prediction for video coding system
WO2017147765A1 (en) * 2016-03-01 2017-09-08 Mediatek Inc. Methods for affine motion compensation
US10602180B2 (en) * 2017-06-13 2020-03-24 Qualcomm Incorporated Motion vector prediction

Cited By (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220360804A1 (en) * 2018-09-21 2022-11-10 Canon Kabushiki Kaisha Video coding and decoding
US11825074B2 (en) 2018-10-08 2023-11-21 Beijing Bytedance Network Technology Co., Ltd Generation and usage of combined affine merge candidate
US11849108B2 (en) 2018-10-18 2023-12-19 Canon Kabushiki Kaisha Video coding and decoding
US11641467B2 (en) 2018-10-22 2023-05-02 Beijing Bytedance Network Technology Co., Ltd. Sub-block based prediction
US11956449B2 (en) 2018-11-12 2024-04-09 Beijing Bytedance Network Technology Co., Ltd. Simplification of combined inter-intra prediction
US11843725B2 (en) 2018-11-12 2023-12-12 Beijing Bytedance Network Technology Co., Ltd Using combined inter intra prediction in video processing
US11558634B2 (en) 2018-11-20 2023-01-17 Beijing Bytedance Network Technology Co., Ltd. Prediction refinement for combined inter intra prediction mode
US11956465B2 (en) 2018-11-20 2024-04-09 Beijing Bytedance Network Technology Co., Ltd Difference calculation based on partial position
US11632566B2 (en) 2018-11-20 2023-04-18 Beijing Bytedance Network Technology Co., Ltd. Inter prediction with refinement in video processing
US11632541B2 (en) 2018-11-22 2023-04-18 Beijing Bytedance Network Technology Co., Ltd. Using collocated blocks in sub-block temporal motion vector prediction mode
US11431964B2 (en) 2018-11-22 2022-08-30 Beijing Bytedance Network Technology Co., Ltd. Coordination method for sub-block based inter prediction
US11671587B2 (en) 2018-11-22 2023-06-06 Beijing Bytedance Network Technology Co., Ltd Coordination method for sub-block based inter prediction
US11647187B2 (en) * 2018-11-30 2023-05-09 Interdigital Madison Patent Holdings, Sas Triangle and multi-hypothesis combination for video coding and decoding
US20210400269A1 (en) * 2018-11-30 2021-12-23 Interdigital Vc Holdings, Inc. Triangle and multi-hypothesis combination for video coding and decoding
US11284068B2 (en) 2018-12-03 2022-03-22 Beijing Bytedance Network Technology Co., Ltd. Indication method of maximum number of candidates
US11856185B2 (en) 2018-12-03 2023-12-26 Beijing Bytedance Network Technology Co., Ltd Pruning method in different prediction mode
US11930165B2 (en) 2019-03-06 2024-03-12 Beijing Bytedance Network Technology Co., Ltd Size dependent inter coding
US11553201B2 (en) 2019-04-02 2023-01-10 Beijing Bytedance Network Technology Co., Ltd. Decoder side motion vector derivation
US11553177B2 (en) 2019-08-10 2023-01-10 Beijing Bytedance Network Technology Co., Ltd. Buffer management in subpicture decoding
US11523108B2 (en) 2019-08-10 2022-12-06 Beijing Bytedance Network Technology Co., Ltd. Position restriction for inter coding mode
US11533513B2 (en) 2019-08-10 2022-12-20 Beijing Bytedance Network Technology Co., Ltd. Subpicture size definition in video processing
US11871025B2 (en) 2019-08-13 2024-01-09 Beijing Bytedance Network Technology Co., Ltd Motion precision in sub-block based inter prediction
US11695946B2 (en) 2019-09-22 2023-07-04 Beijing Bytedance Network Technology Co., Ltd Reference picture resampling in video processing
US11546593B2 (en) 2019-10-02 2023-01-03 Beijing Bytedance Network Technology Co., Ltd. Syntax for subpicture signaling in a video bitstream
US11539950B2 (en) 2019-10-02 2022-12-27 Beijing Bytedance Network Technology Co., Ltd. Slice level signaling in video bitstreams that include subpictures
US11956432B2 (en) 2019-10-18 2024-04-09 Beijing Bytedance Network Technology Co., Ltd Interplay between subpictures and in-loop filtering
US11962771B2 (en) 2019-10-18 2024-04-16 Beijing Bytedance Network Technology Co., Ltd Syntax constraints in parameter set signaling of subpictures
WO2023034640A1 (en) * 2021-09-06 2023-03-09 Beijing Dajia Internet Information Technology Co., Ltd. Candidate derivation for affine merge mode in video coding
WO2023049219A1 (en) * 2021-09-24 2023-03-30 Beijing Dajia Internet Information Technology Co., Ltd. Candidate derivation for affine merge mode in video coding
WO2023137234A1 (en) * 2022-01-17 2023-07-20 Beijing Dajia Internet Information Technology Co., Ltd. Methods and devices for candidate derivation for affine merge mode in video coding
WO2023185933A1 (en) * 2022-03-30 2023-10-05 Beijing Bytedance Network Technology Co., Ltd. Method, apparatus, and medium for video processing

Also Published As

Publication number Publication date
US20240098295A1 (en) 2024-03-21
JP2023181441A (ja) 2023-12-21
KR20210024487A (ko) 2021-03-05
CN110677675A (zh) 2020-01-10
TW202021356A (zh) 2020-06-01
JP2021530154A (ja) 2021-11-04
CN114374850A (zh) 2022-04-19
WO2020008334A1 (en) 2020-01-09
CN110677675B (zh) 2022-02-08
US20220046267A1 (en) 2022-02-10
EP3804327A1 (en) 2021-04-14
TWI731363B (zh) 2021-06-21

Similar Documents

Publication Publication Date Title
US20210058637A1 (en) Efficient affine merge motion vector derivation
US11509929B2 (en) Multi-iteration motion vector refinement method for video processing
CN110581999B (zh) 色度解码器侧运动矢量细化
US11477458B2 (en) Mode dependent motion vector difference precision set
US11792421B2 (en) Rounding in pairwise average candidate calculations
US11871022B2 (en) Concept of interweaved prediction
US11956465B2 (en) Difference calculation based on partial position
US20210235083A1 (en) Sub-block based prediction
CN110677674B (zh) 视频处理的方法、设备和非暂时性计算机可读介质
US11930182B2 (en) Motion vector derivation between dividing patterns
US11736698B2 (en) Sub-region based determination of motion information refinement
WO2020233600A1 (en) Simplified local illumination compensation

Legal Events

Date Code Title Description
AS Assignment

Owner name: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD., CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIU, HONGBIN;WANG, YUE;REEL/FRAME:054290/0200

Effective date: 20190621

Owner name: BYTEDANCE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHANG, KAI;ZHANG, LI;SIGNING DATES FROM 20190619 TO 20190624;REEL/FRAME:054290/0104

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: ADVISORY ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION