TW201902223A - Method and apparatus of bi-directional optical flow for overlapped block motion compensation in video coding - Google Patents

Abstract

Method and apparatus of using an Inter coding tool and OBMC (Overlapped Block Motion Compensation) are disclosed. According to one method of the present invention, input data corresponding to a first block and a second block are received, where the first block is coded before the second block. For target pixels of an OBMC area located in the second block and adjacent to a target boundary between the first block and the second block, a first MV (motion vector) associated with the first block is determined and first MC (motion-compensated) pixels are derived by applying the Inter coding tool to the target pixels of the OBMC area according to the first MV and one or more first settings of the Inter coding tool associated with the first block. The first MC (motion-compensated) pixels are used for the OBMC process along with the Inter coding tool.

Description

 Method and apparatus for overlapping bi-directional optical motion compensated bidirectional optical flow in video coding   [Priority statement]

The present application claims priority to U.S. Provisional Patent Application Serial No. 62/475,975, filed on Mar. Reference is made herein to the subject matter of the application.

Embodiments of the present invention relate to motion compensation using an inter-frame coding tool that includes Bi-directional Optical Flow (BIO) to refine the motion of a bi-predictive block. In particular, the present invention relates to bandwidth reduction of an encoding system that uses BIO processing and overlapping block motion compensation.

Bidirectional optical flow (BIO)

Bidirectional optical flow (BIO) is in JCTVC-C204 (E. Alshina, et al., Bi-directional optical flow, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11,3rd Meeting: Guangzhou, CN, 7-15 October, 2010, Document: JCTVC-C204) and VCEG-AZ05 (E. Alshina, et al., Known tools performance investigation for next generation video coding , Motion estimation/compensation techniques disclosed in ITU-T SG 16 Question 6, Video Coding Experts Group (VCEG), 52nd Meeting: 19-26 June 2015, Warsaw, Poland, Document: VCEG-AZ05). The BIO derives a sample-level motion refinement based on the assumption of optical flow and stable motion, as shown in FIG. 1, where the current pixel 122 in a bi-prediction slice 120 Predicted by one pixel in reference picture 0 (130) and one pixel in reference picture 1 (110). As shown in FIG. 1, the current pixel 122 is predicted from pixel B (112) in reference picture 1 (110) and pixel A (132) in reference picture 0 (130). In Fig. 1, v _x and v _y are pixel displacement vectors in the x and y directions, which are derived using a bidirectional optical flow (BIO) model. It applies only to true bi-predictive blocks, which are predicted from two reference pictures corresponding to the previous picture and the next picture. In VCEG-AZ05, BIO uses a 5x5 window to derive motion refinement for each sample. Therefore, for N×N blocks, the motion compensation result and the corresponding gradient information of the (N+4)x(N+4) block are needed to derive the sample-based motion refinement of the NxN block. According to VCEG-AZ05, a 6-tap gradient filter and a 6-tap interpolation filter are used to generate BIO gradient information. Therefore, the computational complexity of BIO is much higher than traditional bidirectional prediction. In order to further improve the performance of BIO, the following methods have been proposed.

In conventional bidirectional prediction in HEVC, Equation (1) is used to generate a predictor, where P ⁽⁰⁾ and P ⁽¹⁾ are List 0 and List 1 predictors, respectively.

P _Conventional [ i , j ]=( P ⁽⁰⁾ [ i , j ]+ P ⁽¹⁾ [ i , j ]+1)>>1 (1)

In JCTVC-C204 and VECG-AZ05, equation (2) is used to generate a BIO predictor.

P _OpticalFlow =( P ⁽⁰⁾ [ i , j ]+ P ⁽¹⁾ [ i , j ]+ v _x [ i , j ]( I _x ⁽⁰⁾ - I _x ⁽¹⁾ [ i , j ])+ v _y [ i , j ]( I _y ⁽⁰⁾ - I _y ⁽¹⁾ [ i , j ])+1)>>1 (2)

In equation (2), I _x ⁽⁰⁾ and Ix(1) represent the x-direction gradients in the list 0 and list 1 predictors, respectively; I _y ⁽⁰⁾ and I _y ⁽¹⁾ represent list 0 and list, respectively. 1 y-direction gradient in the predictor; v _x and v _y represent offsets or displacements in the x-direction and the y-direction, respectively. The derivation processing of v _x and v _y is as follows. First, define the cost function as diffCost(x,y) to find the best values v _x and v _y . To find the best values v _x and v _y to minimize the cost function diffCost(x,y) , use a 5x5 window. The solutions of v _x and v _y can be expressed by using S ₁ , S ₂ , S ₃ , S ₅ and S ₆ .

The minimum cost function, min diffCost ( x , y ), can be derived from the following formula:

By solving equations (3) and (4), v _x and v _y can be solved according to equation (5).

among them,

In the above equation, Corresponding to the x-direction gradient of the pixel at (x, y) in the list 0 picture, Corresponds to the x-direction gradient of the pixel at (x, y) in the picture of list 1. Corresponds to the y-direction gradient of the pixel at (x, y) in the list 0 picture, and Corresponds to the y-direction gradient of the pixel at (x, y) in the picture of list 1.

In some related art, S ₂ can be ignored, and v _x and v _y can be solved according to the following equation

among them,

According to the above equation, a division operation is required in the derivation of v _x and v _y . Since there are a large number of pixel value accumulations and squares in S ₁ , S ₂ , S ₃ , S ₅ and S ₆ , the required bit-depth is large. If the bit depth of the pixel value in the video sequence is 10 bits, then a divider supporting 32 bits/28 bits is needed for v _x and the other supports 36 bits / 29 bits for frequency division. Used for v _y . If the bit depth is increased by the fractional interpolation filter, the divider becomes 40 bits/36 bits for _vx , and the frequency divider becomes 44 bits/37 bits for _vy . When the required bit depth becomes so large, it is impractical to directly use a lookup table (LUT) to replace the division. Therefore, it is desirable to develop a method of simplifying the division operation in BIO processing.

In the above equation, the parameters S1, S2, S3, S5 and S6 are related to the x-direction gradient and the y-direction gradient. For example, S1 is calculated from the sum of the x-direction gradient of the reference block in list 0 and the x-direction gradient of the reference block in list 1. The square of this sum is used as S1. S5 is calculated from the sum of the y-direction gradient of the reference block in list 0 and the y-direction gradient of the reference block in list 1. The square of this sum is used as S5. For convenience, the parameters S1, S2, S3, S5 and S6 are referred to as gradient parameters in the present disclosure. In fact, the gradient parameters S1, S2, S3, S5 and S6 are typically represented using fixed points with predefined bit depths. The derivation of v _x and v _y will require multiplication, addition and division. Among them, the implementation cost of the division operation is high.

Overlapped Block Motion Compensation (OBMC)

Overlapped block motion compensation is a motion compensation technique that estimates the intensity value of a pixel based on a motion compensated signal derived from a block motion vector (MV) near the pixel. In JCTVC-F299, OBMC is applied to symmetric motion partitioning. If the coding unit (CU) is divided into two 2NxN or Nx2N prediction units (PUs), the OBMC is applied to the horizontal boundaries of the two 2NxN prediction blocks and applied to the vertical boundaries of the two Nx2N prediction blocks. Since those partitions may have different motion vectors, the pixels at the partition boundaries may have large discontinuities, which may result in visual artifacts and may also reduce conversion/encoding efficiency. In JCTVC-F299, OBMC is introduced to smooth the boundaries of motion partitions.

FIG. 2A shows an example of an OBMC for a 2NxN partition, and FIG. 2B shows an example of an OBMC for an Nx2N partition. The dot-filled pixels represent pixels belonging to partition 0, and the blank pixels represent pixels belonging to partition 1. The overlapping regions in the luminance component are respectively defined as two or two rows of pixels on each side of the horizontal or vertical boundary. For pixel columns or rows adjacent to the partition boundary (210 or 220) (i.e., pixels labeled A in Figures 2A and 2B), the OBMC weighting factor is (3/4, 1/4). In other words, for pixel A in column 212 of partition 1, MC (motion compensated) pixel A _{1 is} generated based on MV1 of partition ₁ , and MC pixel A _{0 is} generated based on MV0 of partition ₀ . The OBMC processed pixel A is derived from (3/4 A ₁ + 1/4 A ₀ ). A similar derivation applies to the OBMC pixels in row 222. For pixels that are two or two rows away from the partition boundary (ie, pixels labeled B in 2A and 2B), the OBMC weighting factor is (7/8, 1/8). For chrominance components, the overlap regions are each defined as a column or row of pixels on each side of the horizontal or vertical boundary, and the weighting factor is (3/4, 1/4).

Currently, after normal MC processing for the current CU or PU, the MC result of the overlap region between the two CUs or PUs is generated by another MC process. Therefore, BIO is applied twice in these two MC processes to refine the two MC results separately. The above processing sequence can help skip redundant OBMC and BIO processing when two adjacent motion vectors are the same. However, the bandwidth and MC operations required for overlapping regions are increased compared to integrated OBMC processing and conventional MC processing. For example, the current PU size is 16x8, the overlap area is 16x2, and the interpolation filter in the MC is 8 taps. If OBMC is executed after normal MC, then (16+7)x(8+7)+(16+7)x(2+7)=552 reference pixels for each reference list of the current PU will be needed, and The relevant OBMC will be required. If the OBMC operation is integrated with the normal MC, the current PU and the associated OBMC need only (16 + 7) x (8 + 2 + 7) = 391 reference pixels per reference list. Therefore, when BIO and OBMC are enabled at the same time, it is desirable to reduce the computational complexity or memory bandwidth of the BIO.

In addition to the BIO encoding tools, several other inter-frame coding tools for next-generation video coding, such as pattern-based MV Derivation (PMVD), adjacent derived prediction offsets, are also disclosed. (Neighbouring-derived Prediction Offset, NPO), Local Illumination Compensation (LIC) and Affine mode motion estimation/compensation. A brief review of the inter-frame coding tools is provided below.

Graphics-based motion vector derivation (PMVD)

FIG. 3 shows an example of a FRUC bilateral matching mode in which motion information of the current block 310 is derived based on two reference pictures. The motion information of the current block is derived by finding the best match between the two blocks (320 and 330) along the motion trajectory 340 of the current block in two different reference pictures (ie, Ref0 and Ref1). Under the assumption of continuous motion trajectory, the motion vector MV0 associated with Ref0 and the motion vector MV1 associated with Ref1 pointing to the two reference blocks should be with the current picture (ie, Cur pic) and the two reference pictures Ref0 and The time distance between Ref1 (ie, TD0 and TD1) is proportional.

Figure 4 shows an example of a template matching frame rate up conversion (FRUC) mode. The adjacent regions (420a and 420b) of the current block 410 in the current picture (ie, Cur pic) are used as templates to correspond to the corresponding templates (430a and 430b) in the reference picture (ie, Ref0 in FIG. 4). match. The best match between template 420a/420b and template 430a/430b will determine the motion vector 440 derived by the decoder. Although Ref0 is only taken as an example in FIG. 4, Ref1 can also be used as a reference picture.

According to VCEG-AZ07, when merge_flag or skip_flag is true, FRUC_mrg_flag is sent. If FRUC_mrg_flag is 1, the FRUC_merge_mode is signaled to indicate whether a bilateral matching merge mode or a template matching merge mode is selected. If FRUC_mrg_flag is 0, it means that the regular merge mode is used and in this case the merge index is sent. In video coding, in order to improve coding efficiency, motion vector prediction (MVP) may be used to predict a motion vector of a block in which a candidate list is generated. The merge candidate list can be used to encode the block in merge mode. When the merge mode is used to encode a block, the motion information (eg, motion vector) of the block may be represented by one of the candidate motion vectors in the merged motion vector list. Therefore, instead of directly transmitting the motion information of the block, the merge index is transmitted to the decoder side. The decoder maintains the same merge list and uses the merge index to retrieve merge candidates sent by the merge index. In general, a merge candidate list consists of a small number of candidates, and sending a merge index is more efficient than sending motion information. When the block is encoded in the merge mode, the motion information is "merged" with the adjacent block by signaling the merge index instead of explicitly transmitting. However, the prediction residual is still transmitted. In the case where the prediction residual is zero or very small, the prediction residual is "skipped" (ie, skipped mode), and the block is encoded by the skip mode with the merge index to identify the merge in the merge list Motion vector.

Although the term FRUC refers to motion vector derivation for frame rate up-conversion, the underlying technique is intended for a decoder to derive one or more combined motion vector candidates without explicitly transmitting motion information. Therefore, FRUC is also referred to in the present disclosure as decoder derived motion information. Since the template matching method is a graphics-based motion vector derivation technique, the template matching method of FRUC is also referred to as graphics-based motion vector derivation (PMVD) in the present disclosure.

In the decoder side motion vector derivation method, a new time motion vector prediction called time derived motion vector prediction is derived by scanning all motion vectors in all reference pictures. To derive the LIST_0 time derived motion vector prediction, for each LIST_0 motion vector in the LIST_0 reference picture, the motion vector is scaled to point to the current frame. The 4x4 block pointed to by the thus scaled motion vector in the current frame is the target current block. The motion vector is further scaled to point to a reference picture, where refldx is equal to 0 in the LIST_0 for the target current block. The further scaled motion vector is stored in the LIST_0 motion vector data column of the target current block. FIGS. 5A and 5B illustrate examples of time-derived motion vector predictions that derive LIST_0 and LIST_1, respectively. In the 5A and 5B diagrams, each of the small square blocks corresponds to a 4 × 4 block. The time derived motion vector prediction process scans all motion vectors in all 4x4 blocks in all reference pictures to generate time derived LIST_0 and LIST_1 motion vector predictions for the current frame. For example, in FIG. 5A, block 510, block 512 and block 514 correspond to a 4x4 block of the current picture (ie, Cur.pic), respectively, and a LIST_0 reference picture with an index equal to 0 (ie, refidx= A 4x4 block of 0), and a 4x4 block of the LIST_0 reference picture (ie, refidx=1) with an index equal to one. Motion vectors 520 and 530 for the two blocks in the LIST_0 reference picture with index equal to 1 are known. Time derived motion vector predictions 522 and 532 can then be derived by scaling motion vectors 520 and 530, respectively. The scaled motion vector prediction is then assigned to the corresponding block. Similarly, in FIG. 5B, block 540, block 542 and block 544 correspond to a 4x4 block of the current picture (ie, Cur.pic), respectively, and a LIST_1 reference picture with an index equal to 0 (ie, refidx) A 4x4 block of =0), and a 4x4 block of the LIST_1 reference picture (ie, refidx=1) with an index equal to one. Motion vectors 550 and 560 for the two blocks in the LIST_1 reference picture with index equal to 1 are known. Time derived motion vector predictions 552 and 552 can then be derived by scaling motion vectors 550 and 560, respectively.

In the decoder motion vector derivation method, template matching is also used to generate motion vector prediction for inter-frame mode coding. When a reference picture is selected, template matching is performed to find the best template on the selected reference picture. Its corresponding motion vector is the derived motion vector prediction. The motion vector prediction is inserted into the first position of the AMVP. AMVP represents advanced motion vector prediction in which a current motion vector is predictively encoded using a candidate list. The motion vector difference between the current motion vector and the selected motion vector candidate in the candidate list is encoded.

Neighbouring-derived Prediction Offset (NPO)

The adjacent derived prediction offset (NPO) is a recently developed new coding tool to improve the motion compensated predictor with added prediction offset. With this offset, different lighting conditions between the frames can be considered. The offset is derived using a neighboring reconstructed pixel (NRP) and an extended motion compensated predictor (EMCP).

Figure 6 shows an exemplary embodiment of the derived offset. The graphics selected for NRP and EMCP are located at the N rows (612 and 622) to the left of the current PU 610 and reference block 620, respectively, and the M columns (614 and 624) above, where N and M are predetermined values. Although rectangular NRP and EMCP are shown in the examples, the graphics can have any size and shape and can be determined from any encoding parameter, such as PU or CU size, as long as NRP and EMCP use the same size and shape. The offset is calculated as the average pixel value of the NRP minus the average pixel value of the EMCP. This derived offset will be applied to the entire PU as well as the motion compensated predictor.

Figure 7 shows another exemplary embodiment of the derived offset. First, for each adjacent location (ie, the left side pixel 710 and the upper pixel 720 of the left and upper boundaries of the current block 730), a single offset is calculated as the corresponding pixel in the NRP minus the pixels in the EMCP. Exemplary offset values 6, 4, 2, and -2 for the upper adjacent position 720 and exemplary offset values 6, 6, 6, and 6 for the left adjacent position 710 are as shown in FIG.

After obtaining the offset values for the adjacent locations, the derived offset for each location in the current PU 730 will be derived as an average of the offsets from the left and upper locations, as illustrated in solution 740. For example, at the first position at the upper left corner 731 of the current PU, an offset of 6 will be generated by averaging the offsets from the left and above (ie, (6+6)/2=6). For the next position 832 on the right side, the offset is derived to 5 (ie, (6+4)/2=5). The offset values of the remaining positions can be processed and generated accordingly in raster scan order. Since the correlation between adjacent pixels and the boundary pixels is higher, the offset is also the same. According to the NPO, the offset can be adapted to the pixel location. The derived offset will be adjusted on the PU and applied separately to each PU location along with the motion compensated predictor.

Local Illumination Compensation (LIC)

Local illumination compensation (LIC) is a method of performing inter-frame prediction using adjacent samples of the current block and the reference block. It is based on a linear model using scaling factor a and offset b. The method derives a scaling factor a and an offset b by referring to neighboring samples of the current block and the reference block. The adjacent samples of the current block and the reference block correspond to an L-shape, which includes adjacent pixels above the current block and the reference block and adjacent pixels on the left side. After the scaling factor a and the offset b are derived, the LIC processed pixel l(x, y) is derived from l(x, y)=a*r(x, y)+b, where r(x, y) corresponds to Reference material for motion compensation. In addition, LIC processing can be enabled or disabled adaptively for each CU.

More details on LIC can be found in JVET-C1001 (Xu Chen, et al., "Algorithm Description of Joint Exploration Test Model 3", Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 3rd Meeting: Geneva, CH, 26 May-1 June 2016, Document: JVET-C1001).

Affine mode motion estimation and compensation

An affine model is a more general motion model than the translational motion model widely used in various coding standards. The affine mode inter-frame coding tool can describe two-dimensional block rotation and two-dimensional deformation to convert a square (or rectangle) into a parallelogram. This model can be described as follows: x'=a0+a1*x+a2*y, and y’=b0+b1*x+b2*y. (7)

In this model, a total of six parameters are used. For each pixel A(x, y) in the region of interest, the motion vector between the pixel and its corresponding reference pixel A'(x', y') is (a0+(a1-1)*x+a2 *y, b0+b1*x+(b2-1)*y). Therefore, the motion vector of each pixel also depends on the position. The six parameters of the affine model can be derived based on three known motion vectors at three different locations. The parameter derivation of the affine model is known in the art, so details are omitted here. Although only the 6-parameter affine model is shown in equation (7), a 4-parameter affine model is also used in practice.

Affinity motion compensation has been proposed for next generation video coding developed for standardization of future video coding techniques under ITU-VCEG (Video Coding Experts Group) and ITU ISO/IEC JTC1/SC29/WG11. An affine flag (ie, use_affine_flag) may be sent to indicate whether the affine mode is used for the block.

4-parameter affine prediction has also been proposed for next generation video coding. The motion vector field of an affine motion block can be described by two control point motion vectors or four parameters. Further details can be found in the document C1016 submitted to ITU-VCEG (Lin, et al., "Affine transform prediction for next generation video coding", ITU-U, Study Group 16, Question Q6/16, Contribution C1016, September 2015, Found in Geneva, CH).

According to the current practice of OBMC with BIO, motion compensation processing is performed twice (during the original MC processing and OBMC phases). This process requires not only more calculations, but also more memory access to the reference material. Therefore, it is desirable to develop techniques for reduced required computation or system bandwidth for OBMCs with BIO or similar types of coding tools.

Methods and apparatus for using inter-frame coding tools and overlapping block motion compensation (OBMC) are disclosed. In accordance with a method of the present invention, input data corresponding to a first block and a second block are received, wherein the first block is encoded prior to the second block. For a target pixel of the OBMC region located in the second block and adjacent to the target boundary between the first block and the second block, performing the following steps: determining a first motion vector associated with the first block; Deriving the first motion compensation pixel by applying an inter-frame coding tool to a target pixel of the OBMC region according to the first motion vector and one or more first settings of the inter-frame coding tool associated with the first block; Determining a second motion vector associated with the second block; applying the inter-frame coding tool according to the second motion vector and one or more second settings of the inter-frame coding tool associated with the second block Generating a second motion compensation pixel in the target pixel of the OBMC region or all pixels of the second block; or retrieving the second motion compensation pixel; generating and synchronizing based on the weighted sum of the first motion compensation pixel and the second motion compensation pixel The final motion compensated pixel corresponding to the target pixel of the OBMC region.

In one embodiment, the one or more first settings of the inter-frame encoding tool associated with the first block may be determined based on one or more first flags or patterns of the first block. The one or more second settings of the inter-frame encoding tool associated with the second block may be determined based on one or more second flags or patterns of the second block.

The first block and the second block may correspond to one prediction unit or one coding unit. In this case, the target pixel of the OBMC region may correspond to the second minimum prediction unit, and the first motion vector corresponds to the motion vector of the first minimum prediction unit, and the second motion vector corresponds to the motion vector of the second smallest prediction unit. And wherein the first smallest prediction unit is located in the first block and adjacent to the second smallest prediction unit. When the first block is located above the second block, the OBMC area may correspond to an area above the second block; when the first block is located to the left of the second block, the OBMC area may correspond to the second An area on the left side of the block.

In one embodiment, the inter-frame coding tool corresponds to BIO (bidirectional optical flow) processing. In this case, the one or more first settings of the inter-frame coding tool and the one or more second settings of the inter-frame coding tool correspond to BIO on/off settings. The one or more first settings of the inter-frame coding tool may be based on an LIC (local illumination compensation) / NPO (neighbor derived derived prediction offset) flag of the first block, or a PMVD (graphic based motion vector derivation) ) The logo, or the first PMVD mode to determine. The one or more second settings of the inter-frame coding tool may be determined based on the LIC/NPO flag of the second block, or the PMVD flag, or the PMVD mode.

In another embodiment, the inter-frame coding tool corresponds to LIC (Local Illumination Compensation). In this case, the one or more first settings and the one or more second settings of the inter-frame coding tool include values of the scaling factor a and the offset b.

In yet another embodiment, the inter-frame coding tool corresponds to affine mode motion estimation/compensation. In this case, the one or more first settings of the inter-frame coding tool and the one or more second settings comprise parameter values of a four-parameter affine model or a six-parameter affine model.

110‧‧‧Reference picture 1

120‧‧‧B-Slice

130‧‧‧Reference picture0

112‧‧‧pixel B

114, 134‧‧‧ pixel displacement vector

122‧‧‧ current pixel

132‧‧‧pixel A

210, 220‧‧‧ partition boundaries

212‧‧‧

222‧‧‧

310, 410‧‧‧ current block

320, 330‧‧‧ blocks

340‧‧‧ trajectory

420a, 420b‧‧‧ adjacent areas

430a, 430b‧‧‧ template

440, 520, 530, 550, 560 ‧ ‧ motion vectors

Blocks 510, 512, 514, 540, 542, 544‧‧

522, 532, 552, 562‧‧‧ motion vector prediction

610‧‧‧ Current forecasting unit

620‧‧‧Reference block

612, 622‧‧‧N lines

614, 624‧‧‧M column

710‧‧‧left pixel

720‧‧‧top pixels

730‧‧‧ current block

731‧‧‧Top left corner

732‧‧‧Next location

740‧‧‧Illustration

810‧‧‧ point fill area

The first block above 812‧‧

Second block below 814‧‧

822‧‧‧The first block on the left

824‧‧‧ second block on the right

880‧‧‧ adjacent pixels

910‧‧‧ Block

912, 914, 1032, 1060‧‧‧ OBMC area

1010, 1040‧‧‧ first block

1020‧‧‧ Block

1030‧‧‧Extended area

1050, 1100‧‧‧ second block

1112, 1114‧‧‧OBMC area

1210-1270‧‧ steps

Figure 1 shows an example of a bidirectional optical flow (BIO) for deriving an offset motion vector for motion refinement.

Figure 2A shows an example of an OBMC for a 2NxN partition.

Figure 2B shows an example of an OBMC for Nx2N partitioning.

Figure 3 shows an example of a FRUC bilateral matching mode in which motion information for the current block is derived based on two reference pictures.

Figure 4 shows an example of a template matching FRUC mode.

FIGS. 5A and 5B illustrate examples of time-derived motion vector predictions that derive LIST_0 and LIST_1, respectively.

Figure 6 shows an exemplary embodiment for deriving an offset of a neighboring derived prediction offset (NPO).

Figure 7 illustrates another exemplary embodiment of deriving an offset in which the upper and left pixels are averaged to form a predictor for the current pixel.

8A and 8B illustrate an extended region corresponding to an OBMC overlap region between two adjacent blocks in the vertical direction and an OBMC overlap region between two adjacent blocks in the horizontal direction, respectively. An example of an extended area.

FIGS. 9A and 9B illustrate an example of performing OBMC processing based on a prediction unit of an OBMC area located in a second block below the first block.

FIG. 10A illustrates an example of OBMC processing in accordance with an embodiment of the present invention, wherein a dotted area corresponds to an OBMC and BIO range associated with a first block.

FIG. 10B illustrates an example of OBMC processing in accordance with an embodiment of the present invention in which BIO encoding tool settings for an OBMC region of a second block are determined based on a first block located above the OBMC mix.

FIG. 10C illustrates an example of OBMC processing according to an embodiment of the present invention in which BIO encoding tool settings for an OBMC region of a second block are determined based on a first block located to the left of the OBMC hybrid.

Figure 11A shows an example of OBMC processing of an embodiment of the present invention performed according to a prediction unit based on an OBMC region located in a second block below the first block, based on the first region located above the OBMC mix The block determines the BIO encoding tool settings for the OBMC area of the second block.

FIG. 11B illustrates an example of OBMC processing of an embodiment of the present invention performed according to a prediction unit based on an OBMC region located in a second block below the first block, based on the first region located on the left side of the OBMC hybrid The block determines the BIO encoding tool settings for the OBMC area of the second block.

Fig. 12 is a diagram showing an exemplary flowchart of a video encoding system using an inter-frame coding tool and an OBMC according to an embodiment of the present invention.

The following description is the best mode for carrying out the invention. This description is made to illustrate the general principles of the invention and should not be considered as limiting. The scope of the invention is best determined by reference to the appended claims.

As mentioned earlier, the combination of bidirectional optical flow (BIO) and BIO with overlapping block motion compensation (OBMC) requires access to additional reference material, which leads to increased system bandwidth. These processes also add computational complexity. In the present invention, techniques for reducing system bandwidth and/or computational complexity associated with BIO and BIO and OBMC are disclosed.

The OBMC can be implemented by extending an area corresponding to an overlapping area between two adjacent blocks (referred to as a first block and a second block) as shown in FIGS. 8A and 8B. In Figure 8A, two adjacent blocks are shown, a first block 812 above and a second block 814 below. The dot fill area 810 corresponds to the extended area of the first block 812. In Figure 8B, two adjacent blocks are shown, a first block 822 on the left and a second block 824 on the right. The dot fill area 820 corresponds to the extended area of the first block 822.

The OBMC may also be performed based on prediction units (PUs) as shown in Figures 9A and 9B, where block 910 corresponds to the current PU to be encoded or decoded. The OBMC area 912 corresponds to the upper area of the current block affected by the OBMC processing. On the other hand, the area 914 corresponds to the left OBMC area of the current block affected by the OBMC processing. The motion vector is stored for the smallest PU (ie, 4x4 block). In order to perform the OBMC of the OBMC area 912, motion vectors (i.e., h0, h1, ..., hj) from the upper adjacent smallest PU are used for the current upper block (i.e., t0, t1, ..., tj). OBMC processing. For example, for OBMC processing, the motion vector of h1 will be used for neighboring block t1 to perform OBMC. Furthermore, according to conventional methods, the BIO inter-frame coding tool settings (eg, BIO on or off) depend on t1 or the PU covering t1 (ie, PU 910, "Second Block"). However, the OBMC of t1 needs to wait for the BIO setting information of the second block. Therefore, the OBMC of t1 cannot be executed until the BIO of the second block (ie, PU 910) is set. This will result in data buffering and additional system bandwidth. Similarly, motion vectors (i.e., v0, v1, ..., vi) from the left neighboring smallest PU are used for the OBMC block 914 on the left side of the second block (i.e., PU 910) (i.e., 10, 11, .., 1i) OBMC processing. However, when the BIO inter-frame coding tool is enabled, the BIO settings (ie, BIO on or off) are based on the current cell block (10, 11, ..., 1i) in Figure 9B. Similarly, the 11 OBMC needs to wait for the BIO setup information for the second block. Therefore, the OBMC of 11 can not be executed until the BIO setting of the second block (ie, PU 910). This will result in data buffering and additional system bandwidth.

In order to improve processing efficiency, a new method of setting BIO mode on-off of an encoding system using an OBMC mode and a BIO inter-frame coding tool is disclosed. According to an embodiment of the invention, the BIO switch parameters are determined by the first block, rather than by the second block performing BIO processing in the OBMC area. For two adjacent blocks in the horizontal direction, since the block on the left side of the boundary is often processed before the block on the right side of the boundary, the first block refers to the BIO on-off parameter of the block on the left side of the boundary, and the second A block refers to the BIO on-off parameter of the block to the right of the boundary. For two adjacent blocks in the vertical direction, the first block refers to the BIO on-off parameter of the block on the upper side of the boundary, and the second block refers to the BIO on-off parameter of the block on the lower side of the boundary. .

FIG. 10A illustrates an example of a process in accordance with an embodiment of the present invention in which a dot region 1030 corresponds to an OBMC and BIO range associated with the first tile 1010. In accordance with the present invention, the BIO encoding tool settings (e.g., on/off determination) for the extended region 1030 of the first block 1010 are based on the first block 1010. The left side boundary of the second block located in FIG. 10B is shown as the OBMC area of the point fill area 1032, using the motion vector of the first block, and also using the BIO map from the first block (ie, block 1010). The inter-frame coding tool is set for OBMC mixing of the OBMC area 1032. Therefore, the OBMC processing using the BIO encoding tool can be applied to the OBMC region 1032 at the boundary of the second block (ie, the block 1020) without waiting for the BIO setting of the second block. Similarly, as shown in FIG. 10C, the OBMC area 1060 of the second block 1050 adjacent to the boundary between the first block 1040 and the second block 1050 can perform OBMC processing with BIO without waiting The BIO setting of the second block 1050 is because the BIO inter-frame coding tool settings are based on the first block. If P corresponds to a pixel in the OBMC area (ie, P is located in the second block), the motion vector of the first block and the pixel P set by the BIO inter-frame coding tool of the first block as described above are used. The MC result can be referred to as P0. In order to execute the OBMC, the MC result of the pixel P set using the motion vector of the second block as described above and the BIO inter-frame coding tool of the second block may be referred to as P1. The OBMC of the pixel P mixed with the BIO encoding tool in the OBMC region according to an embodiment of the present invention can be derived according to the equation (8): P = C ₀ P ₀ + C ₁ P ₁ (8)

In the above equation, C0 and C1 are weighting factors. For example, if P is the first pixel from the boundary, (C ₀ , C ₁ )=(1/4, 3/4) according to an example used in next-generation video coding. If P is the second pixel from the boundary, then (C ₀ , C ₁ ) = (1/8, 7/8).

Because the BIO inter-frame coding tool is turned on or off according to some flags or mode settings of the block. The flag or mode setting of the block may include the LIC flag, the NPO flag or the FRUC (PMVD) flag, or the FRUC (PMVD) mode. For example, in existing software for Next Generation Video Coding (JEM 5.0), if the motion vector is true bidirectional and BIO is enabled, then if the LIC flag is false, the BIO will not be applied to the block. In accordance with an embodiment of the invention, a flag or mode setting (eg, LIC/NPO flag, or FRUC (PMVD) flag or FRUC (PMVD)) from the first block instead of the second block is used to determine the BIO frame. The encoding tool is arranged to generate OBMC blocks 1032 or 1060 for OBMC mixing of the second block, as shown in Figures 10B and 10C, respectively. In accordance with the present invention, the OBMC region 1032 or 1060 located in the second block uses not only the motion vector of the first block but also the flag or mode setting from the first block (eg, LIC/NPO flag, or FRUC ( PMVD) mark, or FRUC (PMVD) mode). In other words, the information of the previous block (eg, flag, motion vector, mode) is used to determine whether to turn on or off the BIO encoding tool settings of the OBMC block adjacent to the previous block, instead of using the current block. Information to determine.

Please note that the embodiment in the 10A-10C diagram according to the present invention is intended to show that the BIO of the OBMC block in the second block is determined based on the information of the first block (eg, motion vector, flag, mode). An example of an inter-frame coding tool setup. The invention is not limited to a particular parameter or configuration. For example, although the first block and the second block of the 10A-10C map are the same size, the sizes of the first block and the second block may be different.

The OBMC can also be executed based on the prediction unit. When applying the present invention, the OBMC region (i.e., 1112 or 1114) in the second block 1100 will reuse the first block (i.e., (., h0, h1, ..., hj) or (v0, v1, ..., vi)) information (for example, motion vectors, flags, patterns), as indicated by the curved arrows in Figures 11A and 11B.

Although the OBMC process enables the BIO encoding tool, other inter-frame coding tools can be enabled. The invention can also be applied to other inter-frame coding tools such as LIC and affine motion compensation.

For example, if the LIC mode is enabled for an encoding system using OBMC, the OBMC region in the second block (ie, 1112 in Figure 11A or 1114 in Figure 11B) will reuse the first block (ie, (h0) , h1,...,hj) or (v0,v1,...,vi)) (for example, LIC mode and model parameters {a,b}), as in Figures 11A and 11B The curve arrows are shown to determine the LIC encoding tool settings.

In another example, if the affine mode is enabled for an encoding system using OBMC, the OBMC region (ie, 1112 or 1114) in the second block will reuse the first block (ie, (h0, h1, .. .hj) or (v0, v1,...,vi)) (for example, control points (4-parameters or 6-parameters), CU width and height, and sub-block positions), eg 11A The graph and the curved arrows in Figure 11B are shown to determine the affine encoding tool settings.

Figure 12 shows an exemplary flow chart of a video encoding system using an inter-frame coding tool and OBMC (Overlapped Block Motion Compensation) in accordance with an embodiment of the present invention. The steps shown in the flowcharts, as well as other flowcharts in the present disclosure, may be implemented as a program executable on one or more processors (eg, one or more CPUs) on the encoder side and/or the decoder side. Code. The steps shown in the flowcharts can also be implemented based on hardware such as one or more electronic devices or processors arranged to perform the steps in the flowcharts. According to the method, input data associated with two adjacent blocks corresponding to the first block and the second block are received in step 1210, wherein the first block is encoded before the second block. The input data may include reference data, encoding parameters of the first block and the second block. In step 1220, a target pixel of the OBMC region located in the second block and adjacent to the target boundary between the first block and the second block is selected for processing by the inter-frame coding tool and the OBMC process. As shown in FIGS. 10B-10C, the target OBMC regions may correspond to regions 1032 or regions 1060 in FIGS. 10B and 10C, respectively. The target OBMC area may also correspond to the area 1112 or the area 1114 in the 11A and 11B, respectively. In step 1230, a first motion vector associated with the first block is determined. For example, the first motion vector may correspond to the motion vector of h1 in FIG. 11A or the motion vector of v1 in FIG. 11B. In step 1240, a first motion vector and one or more first settings of an inter-frame coding tool associated with the first block are derived by applying an inter-frame coding tool to the target pixel of the OBMC region. The first motion compensated pixel. The first motion compensation pixel may correspond to P ₀ associated with equation (8) above. In step 1250, a second motion vector associated with the second block is determined. For example, the second motion vector may correspond to the motion vector of t1 in FIG. 11A or the motion vector of 11 in FIG. 11B. In step 1260, the inter-frame coding tool is applied to the target pixel or the second of the OBMC region according to the second motion vector and one or more second settings of the inter-frame coding tool associated with the second block. All pixels of the block to derive the second motion compensated pixel; or retrieve the second motion compensated pixel. The second motion compensation pixel may correspond to P ₁ associated with equation (8) above. As shown in step 1270, a final motion compensated pixel corresponding to the target pixel of the OBMC region is generated based on the weighted sum of the first motion compensated pixel and the second motion compensated pixel. For example, the weighted sum can be calculated according to equation (8).

The flowchart shown above is intended to show an example of video encoding according to the present invention. Those skilled in the art can modify each step, rearrange the steps, divide the steps, or combine the steps to practice the invention without departing from the spirit of the invention. In the present disclosure, specific syntax and semantics have been used to illustrate examples for implementing embodiments of the present invention. Those skilled in the art can practice the invention by replacing the grammar and semantics with equivalent grammar and semantics without departing from the spirit of the invention.

The present invention also discloses a non-transitory computer readable medium storing program instructions that cause processing circuitry of a device to perform the video encoding method described above.

The above description is presented to enable a person of ordinary skill in the art to practice the invention. A person skilled in the art will readily appreciate that various modifications and changes can be made without departing from the spirit and scope of the invention. Therefore, the present invention is not intended to be limited to the specific embodiments shown and described. In the above detailed description, various specific details are set forth to provide a thorough understanding of the invention. Nevertheless, it will be understood by those skilled in the art that the present invention can be practiced.

Embodiments of the invention as described above may be implemented using hardware, software, or a combination thereof. For example, an embodiment of the invention may be a circuit integrated into a video compression chip or program code integrated into a video compression software to perform the described process. Embodiments of the invention may also be programmed to execute the described processing on a digital signal processor. The invention also relates to a range of functions performed by computer processors, digital signal processors, microprocessors, and field programmable gate arrays (FPGAs). In accordance with the present invention, these processors can be configured to perform specific tasks by executing computer readable software code or firmware code that defines a particular method. Software code or firmware code can be developed in different programming languages and in different formats or styles. Software code can also be compiled for different target platforms. However, the different code formats, styles, and languages of the software code, as well as other ways of configuring the code to perform the tasks, do not depart from the spirit and scope of the present invention.

The invention may be embodied in other specific forms without departing from the spirit or essential characteristics. The described embodiments are to be considered in all respects illustrative illustrative The scope of the invention is therefore intended to be limited by the appended claims Various modifications, adaptations, and combinations of various features of the described embodiments can be implemented without departing from the scope of the invention as set forth in the appended claims.

Claims (20)

 A method of video encoding, using an inter-frame coding tool and overlapping block motion compensation, the method comprising: receiving input data associated with two adjacent blocks corresponding to the first block and the second block, Wherein the first block is encoded before the second block; and for overlapping block motion compensation located in the second block and adjacent to a target boundary between the first block and the second block a target pixel of the region: determining a first motion vector associated with the first block; and according to the first motion vector and one or more first settings of the inter-frame coding tool associated with the first block Deriving a first motion compensation pixel by applying the inter-frame coding tool to the target pixel of the overlapping block motion compensation region; determining a second motion vector associated with the second block; according to the second motion a vector and one or more second settings of the inter-frame coding tool associated with the second block, by applying the inter-frame coding tool to the target pixel of the overlapping block motion compensation region or the Second block All pixels to derive a second motion compensated pixel; or to retrieve the second motion compensated pixel; and generate a weighted sum based on the first motion compensated pixel and the second motion compensated pixel to generate a motion compensation region corresponding to the overlapped block The final motion compensated pixel of the target pixel.    The method of claim 1, wherein the one of the inter-frame coding tools associated with the first block is determined based on one or more first flags or patterns of the first block Or a plurality of first settings; and determining one or more second settings of the inter-frame coding tool associated with the second block based on the one or more second flags or patterns of the second block.    The method of claim 1, wherein the first block and the second block correspond to one prediction unit or one coding unit.    The method of claim 3, wherein the target pixel of the overlapping block motion compensation region corresponds to a second minimum prediction unit, and the first motion vector corresponds to a motion vector of the first minimum prediction unit, The second motion vector corresponds to a motion vector of the second minimum prediction unit, and wherein the first minimum prediction unit is located in the first block and adjacent to the second minimum prediction unit.    The method of claim 1, wherein the overlapping block motion compensation region corresponds to an area above the second block when the first block is located above the second block; And when the first block is located on the left side of the second block, the overlapping block motion compensation area corresponds to an area on the left side of the second block.    The method of claim 1, wherein the inter-frame coding tool corresponds to bidirectional optical flow processing.    The method of claim 6, wherein the one or more first settings of the inter-frame coding tool and the one or more second settings of the inter-frame coding tool correspond to bidirectional optical flow On/Off setting.    The method of claim 6, wherein the local illumination compensation/adjacent derived prediction offset flag based on the first block, or the motion vector derivation flag based on the graphics, or the motion vector derivation mode based on the graphics Determining the one or more first settings of the inter-frame coding tool; and local illumination compensation/neighbor derived prediction offset flag based on the second block, or graphics-based motion vector derivation flag, or graphics based The motion vector derivation mode to determine the one or more second settings of the inter-frame coding tool.    The method of claim 1, wherein the inter-frame coding tool corresponds to local illumination compensation.    The method of claim 9, wherein the one or more first settings of the inter-frame coding tool and the one or more second settings comprise values of a scaling factor a and an offset b.    The method of claim 1, wherein the inter-frame coding tool corresponds to affine mode motion estimation/compensation.    The method of claim 11, wherein the one or more first settings and the one or more second settings of the inter-frame coding tool comprise a four-parameter affine model or a six-parameter imitation The parameter value of the shot model.    A video encoding apparatus that uses inter-frame coding tools and overlapping block motion compensation, the video encoding apparatus including one or more electronic circuits or processors for performing the steps of: receiving and corresponding to the first block and Input data associated with two adjacent blocks of the second block, wherein the first block is encoded before the second block; for being located in the second block and associated with the first block and the first block a target pixel of the overlapping block motion compensation region adjacent to the target boundary between the two blocks: determining a first motion vector associated with the first block; and correlating with the first block according to the first motion vector One or more first settings of the inter-frame coding tool, by applying the inter-frame coding tool to the target pixel of the overlapping block motion compensation region, deriving a first motion compensation pixel; determining a second motion vector associated with the second block; according to the second motion vector and one or more second settings of the inter-frame coding tool associated with the second block, by inter-frame coding tool application Deriving a second motion compensation pixel for the target pixel of the overlapping block motion compensation region or all pixels of the second block; or retrieving the second motion compensation pixel; and based on the first motion compensation pixel and the A weighted sum of the second motion compensated pixels, generating a final motion compensated pixel of the target pixel corresponding to the overlapped block motion compensation region.    The apparatus of claim 13, wherein the one of the inter-frame coding tools associated with the first block is determined based on one or more first flags or patterns of the first block Or a plurality of first settings; and determining one or more second settings of the inter-frame coding tool associated with the second block based on the one or more second flags or patterns of the second block.    The apparatus of claim 13, wherein the first block and the second block correspond to one prediction unit or one coding unit.    The device of claim 15, wherein the target pixel of the overlapping block motion compensation region corresponds to a second minimum prediction unit, and the first motion vector corresponds to a motion vector of the first minimum prediction unit, The second motion vector corresponds to a motion vector of the second minimum prediction unit, and wherein the first minimum prediction unit is located in the first block and adjacent to the second minimum prediction unit.    The device of claim 13, wherein the overlapping block motion compensation region corresponds to an area above the second block when the first block is located above the second block; And when the first block is located on the left side of the second block, the overlapping block motion compensation area corresponds to an area on the left side of the second block.    The apparatus of claim 13, wherein the inter-frame coding tool corresponds to bidirectional optical flow processing.    The apparatus of claim 18, wherein the one or more first settings of the inter-frame coding tool and the one or more second settings of the inter-frame coding tool correspond to bidirectional optical flow On/Off setting.    A non-transitory computer readable medium storing program instructions, the program instructions causing a processing circuit of a device to perform a video encoding method in conjunction with an inter-frame coding tool and overlapping block motion compensation, and the method includes: receiving and corresponding to the first An input data associated with two adjacent blocks of the block and the second block, wherein the first block is encoded before the second block; for being located in the second block and associated with the first block a target pixel of the overlapping block motion compensation region adjacent to the target boundary between the block and the second block: determining a first motion vector associated with the first block; and according to the first motion vector One or more first settings of the inter-frame coding tool associated with a block, the first motion compensation pixel being derived by applying the inter-frame coding tool to the target pixel of the overlapping block motion compensation region; Determining a second motion vector associated with the second block; according to the second motion vector and one or more second settings of the inter-frame coding tool associated with the second block, The inter-frame coding tool is applied to the target pixel of the overlapping block motion compensation region or all pixels of the second block to derive a second motion compensation pixel; or to retrieve the second motion compensation pixel; A weighted sum of the first motion compensation pixel and the second motion compensation pixel generates a final motion compensation pixel of the target pixel corresponding to the overlapped block motion compensation region.