WO2020198543A1 - Methods and devices for bit-depth control for bi-directional optical flow - Google Patents

Abstract

A bit-depth control method, apparatus, and a non-transitory computer-readable storage medium are provided. The method includes obtaining a first reference picture I (o) and a second reference picture I (1) associated with a video block, obtaining a first reference picture I (o) and a second reference picture I (1) associated with a video block, obtaining first prediction samples I (o) (i,j) of the video block from a reference block in the first reference picture I (o) , obtaining second prediction samples I (1) (i,j) of the video block from a reference block in the second reference picture I (1) , controlling internal bit-depths of bi-directional optical flow (BDOF) by applying right-shifting to internal BDOF parameters, obtaining motion refinements for samples in the video block based on the BDOF being applied to the video block based on the first prediction samples I (o) (i,j) and the second prediction samples I (1) (i,j), and obtaining bi-prediction samples of the video block based on the motion refinements.

Description

METHODS AND DEVICES FOR BIT-DEPTH CONTROL FOR BI-DIRECTIONAL

OPTICAL FLOW

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is based upon and claims priority to Provisional Application No. 62/823,951 filed on March 26, 2019, the entire content thereof is incorporated herein by reference in its entirety.

TECHNICAL FIELD

[0002] This application is related to video coding and compression. More specifically, this disclosure relates to methods and apparatus for bi-directional optical flow (BDOF) method for video coding.

BACKGROUND

[0003] Various video coding techniques may be used to compress video data. Video coding is performed according to one or more video coding standards. For example, video coding standards include versatile video coding (VVC), joint exploration test model (JEM), high- efficiency video coding (H.265/HEVC), advanced video coding (H.264/AVC), moving picture experts group (MPEG) coding, or the like. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, or the like) that take advantage of redundancy present in video images or sequences. An important goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradations to video quality.

SUMMARY

[0004] Examples of the present disclosure provide methods and apparatus for bit-depth control for bi-directional optical flow. According to a first aspect of the present disclosure, a bit-depth control method of BDOF for coding a video signal is provided. The method may include obtaining a first reference picture I⁽⁰⁾ and a second reference picture /⁽¹⁾ associated with a video block. The first reference picture I⁽⁰⁾ may be before a current picture and the second reference picture /⁽¹⁾ may be after the current picture in display order. The method may include obtaining first prediction samples of the video block from a reference block in the first reference picture /⁽⁰⁾. The i and j may represent a coordinate of one sample with the current picture. The method may include obtaining second prediction samples I⁽¹⁾(i, j) of the video block from a reference block in the second reference picture /⁽¹⁾. The method may include controlling internal bit-depths of the BDOF by applying right-shifting to internal BDOF parameters when a coding bit-depth is greater than 12-bit. The BDOF may use internal BDOF parameters including horizontal gradient values and vertical gradient values derived based on the first prediction samples I⁽⁰⁾(i, j) the second prediction samples I⁽¹⁾(i, j, and sample differences between the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j. The method may include obtaining motion refinements for samples in the video block based on the BDOF being applied to the video block based on the first prediction sample I⁽⁰⁾(i, j) and the second prediction sample I⁽¹⁾(i, j. The method may include obtaining bi-prediction samples of the video block based on the motion refinements.

[0005] According to a second aspect of the present disclosure, a computing device is provided. The computing device may include one or more processors, a non-transitory computer-readable memory storing instructions executable by the one or more processors. The one or more processors may be configured to obtain a first reference picture I⁽⁰⁾ and a second reference picture /⁽¹⁾ associated with a video block. The first reference picture I⁽⁰⁾ may be before a current picture and the second reference picture may be after the current picture in display order. The one or more processors may be configured to obtain first prediction samples I⁽⁰⁾(i, j) of the video block from a reference block in the first reference picture /⁽⁰⁾.

The i and j may represent a coordinate of one sample with the current picture. The one or more processors may be configured to obtain second prediction samples I⁽¹⁾(i, j) of the video block from a reference block in the second reference picture /⁽⁰⁾. The one or more processors may be configured to control internal bit-depths of the BDOF by applying right-shifting to internal BDOF parameters when a coding bit-depth is greater than 12-bit. The BDOF may use internal BDOF parameters including horizontal gradient values and vertical gradient values derived based on the first prediction samples I⁽⁰⁾(i, j) the second prediction samples I⁽¹⁾(i, j.) and sample differences between the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j . The one or more processors may be configured to obtain motion refinements for samples in the video block based on the BDOF being applied to the video block based on the first prediction sample I⁽⁰⁾(i, j) and the second prediction sample I⁽¹⁾(i, j. The one or more processors may be configured to obtain bi-prediction samples of the video block based on the motion refinements.

[0006] According to a third aspect of the present disclosure, a non-transitory computer- readable storage medium having stored therein instructions is provided. When the instructions are executed by one or more processors of the apparatus, the instructions may cause the apparatus to perform obtaining a first reference picture /⁽⁰⁾ and a second reference picture /⁽¹⁾ associated with a video block. The first reference picture /⁽⁰⁾ may be before a current picture and the second reference picture /⁽¹⁾ may be after the current picture in display order. The instructions may cause the apparatus to perform obtaining first prediction samples I⁽⁰⁾(i, j) of the video block from a reference block in the first reference picture I⁽⁰⁾ . The i and j may represent a coordinate of one sample with the current picture. The instructions may cause the apparatus to perform obtaining second prediction samples I⁽¹⁾(i, j) of the video block from a reference block in the second reference picture /⁽¹⁾. The instructions may cause the apparatus to perform controlling internal bit-depths of the BDOF by applying right-shifting to internal BDOF parameters when a coding bit-depth is greater than 12-bit. The BDOF may use internal BDOF parameters including horizontal gradient values and vertical gradient values derived based on the first prediction samples I⁽⁰⁾(i, j), the second prediction samples I⁽¹⁾(i, j. and sample differences between the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j . The instructions may cause the apparatus to perform obtaining motion refinements for samples in the video block based on the BDOF being applied to the video block based on the first prediction sample I⁽⁰⁾(i, j) and the second prediction sample I⁽¹⁾(i, j). The instructions may cause the apparatus to perform obtaining bi-prediction samples of the video block based on the motion refinements.

[0007] It is to be understood that both the foregoing general description and the following detailed description are examples only and are not restrictive of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0009] FIG. 1 is a block diagram of an encoder, according to an example of the present disclosure.

[0010] FIG. 2 is a block diagram of a decoder, according to an example of the present disclosure. [0011] FIG. 3 A is a diagram illustrating block partitions in a multi -type tree structure, according to an example of the present disclosure.

[0012] FIG. 3B is a diagram illustrating block partitions in a multi-type tree structure, according to an example of the present disclosure.

[0013] FIG. 3C is a diagram illustrating block partitions in a multi-type tree structure, according to an example of the present disclosure.

[0014] FIG. 3D is a diagram illustrating block partitions in a multi-type tree structure, according to an example of the present disclosure.

[0015] FIG. 3E is a diagram illustrating block partitions in a multi-type tree structure, according to an example of the present disclosure.

[0016] FIG. 4 is a diagram illustration of a bi-directional optical flow (BDOF) model, according to an example of the present disclosure.

[0017] FIG. 5 is a flow chart illustrating a bit-depth control method of coding a video signal, according to an example of the present disclosure.

[0018] FIG. 6 is a flow chart illustrating a method for controlling an internal bit-depth of a BDOF, according to an example of the present disclosure.

[0019] FIG. 7 is a diagram illustrating a computing environment coupled with a user interface, according to an example of the present disclosure.

DETAILED DESCRIPTION

[0020] Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise represented. The implementations set forth in the following description of example embodiments do not represent all implementations consistent with the disclosure. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the disclosure as recited in the appended claims.

[0021] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to limit the present disclosure. As used in the present disclosure and the appended claims, the singular forms“a,”“an,” and“the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It shall also be understood that the term“and/or” used herein is intended to signify and include any or all possible combinations of one or more of the associated listed items. [0022] It shall be understood that, although the terms“first,”“second,”“third,” etc. may be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one category of information from another. For example, without departing from the scope of the present disclosure, first information may be termed as second information; and similarly, second information may also be termed as first information. As used herein, the term“if’ may be understood to mean“when” or“upon” or“in response to a judgment” depending on the context.

[0023] The first version of the HEVC standard was finalized in October 2013, which offers approximately 50% bit-rate saving or equivalent perceptual quality compared to the prior generation video coding standard H.264/MPEG AVC. Although the HEVC standard provides significant coding improvements than its predecessor, there is evidence that superior coding efficiency can be achieved with additional coding tools over HEVC. Based on that, both VCEG and MPEG started the exploration work of new coding technologies for future video coding standardization one Joint Video Exploration Team (JVET) was formed in Oct. 2015 by ITU-T VECG and ISO/IEC MPEG to begin a significant study of advanced technologies that could enable substantial enhancement of coding efficiency. One reference software called the joint exploration model (JEM) was maintained by the JVET by integrating several additional coding tools on top of the HEVC test model (HM).

[0024] In Oct. 2017, the j oint call for proposals (CfP) on video compression with capability beyond HEVC was issued by ITU-T and ISO/IEC. In Apr. 2018, 23 CfP responses were received and evaluated at the 10-th JVET meeting, which demonstrated compression efficiency gain over the HEVC around 40%. Based on such evaluation results, the JVET launched a new project to develop the new generation video coding standard that is named as Versatile Video Coding (VVC). In the same month, one reference software codebase, called VVC test model (VTM), was established for demonstrating a reference implementation of the VVC standard.

[0025] Like HEVC, the VVC is built upon the block-based hybrid video coding framework.

FIG. 1 gives the block diagram of a generic block-based hybrid video encoding system. Specifically, FIG. 1 shows atypical encoder 100. The encoder 100 has video input 110, motion compensation 112, motion estimation 114, intra/inter mode decision 116, block predictor 140, adder 128, transform 130, quantization 132, prediction related info 142, intra prediction 118, picture buffer 120, inverse quantization 134, inverse transform 136, adder 126, memory 124, in-loop filter 122, entropy coding 138, and bitstream 144.

[0026] The input video signal is processed block by block (called coding units (CUs)). In VTM-1.0, a CU can be up to 128x128 pixels. However, different from the HEVC which partitions blocks only based on quad-trees, in the VVC, one coding tree unit (CTU) is split into CUs to adapt to varying local characteristics based on quad/binary/temary-tree. Additionally, the concept of multiple partition unit type in the HEVC is removed, i.e., the separation of CU, prediction unit (PU) and transform unit (TU) does not exist in the VVC anymore; instead, each CU is always used as the basic unit for both prediction and transform without further partitions. In the multi-type tree structure, one CTU is firstly partitioned by a quad-tree structure. Then, each quad-tree leaf node can be further partitioned by a binary and ternary tree structure.

[0027] As shown in FIG. 3A, 3B, 3C, 3D, and 3E, there are five splitting types, quaternary partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning.

[0028] FIG. 3 A shows a diagram illustrating block quaternary partition in a multi-type tree structure, in accordance with the present disclosure.

[0029] FIG. 3B shows a diagram illustrating block vertical binary partition in a multi-type tree structure, in accordance with the present disclosure.

[0030] FIG. 3C shows a diagram illustrating block horizontal binary partition in a multi type tree structure, in accordance with the present disclosure.

[0031] FIG. 3D shows a diagram illustrating block vertical ternary partition in a multi -type tree structure, in accordance with the present disclosure.

[0032] FIG. 3E shows a diagram illustrating block horizontal ternary partition in a multi type tree structure, in accordance with the present disclosure.

[0033] In FIG. 1, spatial prediction and/or temporal prediction may be performed. Spatial prediction (or“intra prediction”) uses pixels from the samples of already coded neighboring blocks (which are called reference samples) in the same video picture/slice to predict the current video block. Spatial prediction reduces spatial redundancy inherent in the video signal. Temporal prediction (also referred to as“inter prediction” or“motion compensated prediction”) uses reconstructed pixels from the already coded video pictures to predict the current video block. Temporal prediction reduces temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is usually signaled by one or more motion vectors (MVs), which indicate the amount and the direction of motion between the current CU and its temporal reference. Also, if multiple reference pictures are supported, one reference picture index is additionally sent, which is used to identify from which reference picture in the reference picture storage the temporal prediction signal comes from. After spatial and/or temporal prediction, the mode decision block in the encoder chooses the best prediction mode, for example, based on the rate-distortion optimization method. The prediction block is then subtracted from the current video block, and the prediction residual is de-correlated using transform and quantized. The quantized residual coefficients are inverse quantized and inverse transformed to form the reconstructed residual, which is then added back to the prediction block to form the reconstructed signal of the CU. Further, in-loop filtering, such as deblocking filter, sample adaptive offset (SAO), and adaptive in-loop filter (ALF) may be applied on the reconstructed CU before it is put in the reference picture store and used to code future video blocks. To form the output video bit-stream, coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit to be further compressed and packed to form the bit-stream.

[0034] FIG. 2 gives a general block diagram of a block-based video decoder. Specifically, FIG. 2 shows a typical decoder 200 block diagram. Decoder 200 has bitstream 210, entropy decoding 212, inverse quantization 214, inverse transform 216, adder 218, intra/inter mode selection 220, intra prediction 222, memory 230, in-loop filter 228, motion compensation 224, picture buffer 226, prediction related info 234, and video output 232.

[0035] In FIG 2, the video bit-stream is first entropy decoded at entropy decoding unit. The coding mode and prediction information are sent to either the spatial prediction unit (if intra coded) or the temporal prediction unit (if inter-coded) to form the prediction block. The residual transform coefficients are sent to inverse quantization unit and inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added together. The reconstructed block may further go through in-loop filtering before it is stored in reference picture store. The reconstructed video in reference picture store is then sent out to drive a display device, as well as used to predict future video blocks.

[0036] Bi-directional optical flow

[0037] Conventional bi-prediction in video coding is a simple combination of two temporal prediction blocks obtained from the reference pictures that are already reconstructed. However, due to the limitation of the block-based motion compensation, there could be remaining small motion that can be observed between the samples of two prediction blocks, thus reducing the efficiency of motion compensated prediction. To solve this problem, BDOF is applied in the VVC to lower the impacts of such motion for every sample inside one block.

[0038] Specifically, as shown in FIG. 4, the BDOF is a sample-wise motion refinement that is performed on top of the block-based motion-compensated predictions when bi-prediction is used. FIG. 4 shows an illustration of a BDOF model, in accordance with the present disclosure.

[0039] The motion refinement of each 4x4 sub-block is calculated by minimizing

the difference between L0 and L1 prediction samples after the BDOF is applied inside one 6x6 window W around the sub-block. Specifically, the value of is derived as

where is the floor function; clip3(min, max, x) is a function that clips a given value x inside the range of [min, max]; the symbol » represents bitwise right shift operation; the symbol « represents bitwise left shit operation; th_BDOF is the motion refinement threshold to prevent the propagated errors due to irregular local motion, which is equal to 2^13-BD, where BD is the bit- depth of input video. For example, a bit-depth represents the number of bits used to define each pixel. In (1), .

[0040] The values of S₁ S₂, S₃, S₅ and S₆ are calculated as

where

where are the sample value at coordinate of the prediction signal in list k, k =

0,1, which are generated at intermediate high precision (i.e., 16-bit) and

are the horizontal and vertical gradients of the sample that are obtained by directly calculating the difference between its two neighboring samples, i.e.,

[0041] Based on the motion refinement derived in (1), the final bi-prediction samples of the CU are calculated by interpolating the L0/L1 prediction samples along the motion trajectory based on the optical flow model, as indicated by

where shift and o_offset are the right shift value and the offset value that are applied to combine the L0 and L1 prediction signals for bi-prediction, which are equal to 15— BD and 1 « (14— BD) + 2 · (1 « 13) , respectively. Table 1 illustrates the specific bit-widths of intermediate parameters that are involved in the BDOF process. As shown in the table, the internal bit-width of the whole BDOF process does not exceed 32-bit. Additionally, the multiplication with the worst possible input happens at the product of in (1) with inputs

of 15-bit and 4-bit. Therefore, 15-bit multiplier is enough for the BDOF.

[0042] Table 1 The bit-widths of intermediate parameters of the BDOF in the VVC

[0043] Efficiency of bi-predictive prediction

[0044] Although the BDOF can enhance the efficiency of bi-predictive prediction, its design can still be further improved. Specifically, the following problems in the existing BDOF design in VVC for controlling the bit- widths of intermediate parameters are identified in this disclosure.

[0045] As shown in Table 1, the parameter (i.e., the difference between L0 and L1

prediction samples), and the parameter

and (i.e., the sum of the

horizontal/vertical L0 and L1 gradient values) are represented in the same bit- width of 11 -bit. Although such a method can facilitate the overall control of the internal bit-width for the BDOF, it is suboptimal with regards to the precision of the derived motion refinements. As shown in (4), part of the reason may be that the gradient values are calculated as the difference between neighboring prediction samples. Due to the high-pass nature of such a process, the derived gradients are less reliable in the presence of noise, e.g., the noise captured in the original video and the coding noise that are generated during the coding process. Thus, it may not always be beneficial to represent the gradient values in high bit-width.

[0046] As shown in Table 1, the maximum bit-width usage of the whole BDOF process occurs with the calculation of the vertical motion refinement

where (27-bit) is firstly left-

shifted by 3-bit then is subtracted by (30-bit). Therefore, the

maximum bit- width of the current design is equal to 31 -bit. In a practical hardware implementation, the coding process with maximal internal bit- width more than 16-bit is usually implemented by a 32-bit implementation. Therefore, the existing design does not fully utilize the valid dynamic range of the 32-bit implementation. This may lead to unnecessary precision loss of the motion refinements derived by the BDOF.

[0047] Improving the efficiency of bi-predictive prediction using BDOF

[0048] In this disclosure, one improved bit-width control method is proposed to address the two issues of the bit-width control method, as pointed out in the“problem statement” section for the existing BDOF design.

[0049] FIG. 5 shows a bit-depth control method of coding a video signal in accordance with the present disclosure.

[0050] In step 510, obtaining a first reference picture /⁽⁰⁾ and a second reference picture associated with a video block. The first reference picture /⁽⁰⁾ is before a current picture and the second reference picture /⁽¹⁾ is after the current picture in display order. For example, reference pictures may be video pictures neighboring the current picture being encoded.

[0051] In step 512, obtaining first prediction samples I⁽⁰⁾(i, j) of the video block from a reference block in the first reference picture /⁽⁰⁾, where i and j represent a coordinate of one sample with the current picture. For example, the first prediction sample I⁽⁰⁾(i, j) may be a prediction sample using a motion vector in the L0 list of the previous reference picture in display order.

[0052] In step 514, obtaining second prediction samples I I⁽¹⁾(i, j) of the video block from a reference block in the second reference picture /⁽¹⁾. For example, the second prediction sample I⁽¹⁾(i, j may be a prediction sample using a motion vector in the LI list of the next reference picture in display order.

[0053] In step 516, controlling internal bit-depths of the BDOF by applying right-shifting to internal BDOF parameters when a coding bit-depth is greater than 12-bit. The BDOF, for example, uses internal BDOF parameters including horizontal gradient values and vertical gradient values derived based on the first prediction samples I⁽⁰⁾(i, j), the second prediction samples I⁽¹⁾(i, j , and sample differences between the first prediction samples I⁽⁰⁾(i, j and the second prediction samples I⁽¹⁾(i, j) .

[0054] In step 518, obtaining motion refinements for samples in the video block based on the BDOF being applied to the video block based on the first prediction sample I⁽⁰⁾(i, j and the second prediction sample I⁽¹⁾(i, j .

[0055] In step 520, obtaining bi-prediction samples of the video block based on the motion refinements.

[0056] Firstly, to overcome the negative impacts of gradient estimation errors, additional right shift are introduced in the proposed method when calculating the gradient values and in (4), i.e., to lower the internal bit-width of gradient values.

Specifically, the horizontal and vertical gradients at each sample position are calculated as

[0057] Moreover, additional bit-shift

is introduced to the calculation of variables

and q(ί,j) in order to control the entire BDOF process so that it is operated at appropriate internal bit- widths, as depicted as:

[0058] As will be seen in Table 2, due to the modification to the number of right-shifted bits that are applied in (6) and (7), the dynamic ranges of the parameters and

q(ί,j) will be different, compared to the existing BDOF design in Table 1 where the three parameters are represented in the same dynamic range (i.e., 21-bit). Such change can increase the bit-widths of the internal parameters and , which could potentially increase

the maximal bit-width of internal BDOF process to be beyond 32-bit. Thus, to ensure 32-bit implementation, two additional clipping operations are introduced in calculating the values of S₂ and S₆. Specifically, in the proposed method, the values of two parameters are calculated as

where B₂ and B₆ are the parameters to control the output dynamic ranges of S₂ and S₆ , respectively. It should be noticed that different from the gradient calculation the clipping operations in (8) are only applied once to calculate the motion refinement of each 4x4 sub- block inside one BDOF CU, i.e., being invoked based on the 4x4 unit. Therefore, the corresponding complexity increase due to the clipping operations introduced in the proposed method is very negligible.

[0059] In practice, different values of , B₂ and B₆ may be applied to achieve

different trade-offs between the intermediate bit-width and the precision of internal BDOF derivations. As one embodiment of the disclosure, it is proposed to set

and

to 2, B₂ to 25 and B₆ to 27. As another embodiment of the disclosure, it is proposed to set to 1,

to 4, B₂ to 26 and B₆ to 28.

[0060] Table 2 illustrates the corresponding bit-width of each intermediate parameter when the proposed bit-width control method is applied to the BDOF. In Table 2, gray colors highlight the changes that are applied in the proposed bit-width control method compared to the existing BDOF design in VVC. As can be seen in Table 2, with the proposed bit-width control method, the internal bit-width of the whole BDOF process does not exceed 32-bit. Additionally, by the proposed design, the maximal bit- width is just 32-bit which can fully utilize the available dynamic range of 32-bit hardware implementation. On the other hand, as shown in the table, the multiplication with the worst possible input happens at the product of

where the input is 14-bit and the input

is 6-bit. Therefore, like the existing BDOF design, one 16- bit multiplier is also large enough when the proposed method is applied.

[0061] Table 2 The bit-widths of intermediate parameters of the proposed method

[0062] FIG. 6 shows an example method for controlling an internal bit-depth of a BDOF in accordance with the present disclosure.

[0063] In step 610, obtaining a first horizontal gradient value of a first prediction sample I⁽⁰⁾(i, j) based on a difference between a first prediction sample I⁽⁰⁾(i+ 1, j and a first prediction sample I⁽⁰⁾(i- 1, j . [0064] In step 612, obtaining a second horizontal gradient value of a second prediction sample I⁽¹⁾(i, j based on a difference between a second prediction sample I⁽¹⁾(i+ 1, j and a second prediction sample I⁽¹⁾(i- 1, j .

[0065] In step 614, obtaining a first vertical gradient value of a first prediction sample I⁽⁰⁾(i, j) based on a difference between a first prediction sample I⁽⁰⁾(i, j+ 1 and a first prediction sample I⁽⁰⁾(i, j- 1 .

[0066] In step 616, obtaining a second vertical gradient value of a second prediction sample I⁽¹⁾(i, j) based on a difference between a second prediction sample I⁽¹⁾(i, j+ 1 and a second prediction sample I⁽¹⁾(i, j- 1 .

[0067] In step 618, right shifting the first and second horizontal gradient values by a first shift value.

[0068] In step 620, right shifting the first and second vertical gradient values by the first shift value.

[0069] In the above method, the clipping operations, as in equation (8), are added to avoid the overflow of the intermediate parameter when deriving

and

. However, such clippings are only needed when the correlation parameters are accumulated in the large local window. When one small window is applied, the overflow may not be possible. Therefore, in another embodiment of the disclosure, the following bit-depth control method is proposed for the BDOF method without clipping, as described as follows.

[0070] First, the gradient values and in (4) at each sample position are

calculated as

[0071] Second, the correlation parameters and that are used for the

BDOF process are calculated as:

[0072] Third, the values of S₁, S₂, S₃, S₅ and S₆ are calculated as

[0073] Fourth, the motion refinement of each 4x4 sub-block is derived as

[0074] Fifth, the final bi-prediction samples of the CU are calculated by interpolating the L0/L1 prediction samples along the motion trajectory based on the optical flow model, as indicated by

[0075] The above BDOF bit-width control method is built upon one assumption that the internal bit-depth used for coding the video cannot exceed 12-bit such that the precision of the output signal from the motion compensation (MC) is 14-bit. In other words, the BDOF bit- width control method as specified in (9) to (13) cannot guarantee that all the bit-depths of internal BDOF operations are within 32-bit, when the internal bit-depth is greater than 12-bit. To address such overflow problem for high internal bit-depth, an improved BDOF bit-depth control method is disclosed in the following by introducing additional bit-wise right shifts which are dependent on the applied internal bit-depth after the MC stage. In this method, when the internal bit-depth is greater than 12-bit, the MC output signal is always shifted to 14-bit such that the existing BDOF bit-depth control method that is designed for the 8 to 12-bit internal bit-depth can be reused for the BDOF process of high bit-depth video. Specifically, assuming bit-depth is the internal bit-depth, the proposed method may be implemented with the following steps:

[0076] First, the gradient values and in (4) at each sample position are

calculated as

[0077] Second, the correlation parameters that are used for the

BDOF process are calculated as:

[0078] Third, the values of 5₁ S₂, S₃, S₅ and S₆ are calculated as

[0079] Fourth, the motion refinement of each 4x4 sub-block is is derived as

where th_BDOF is the motion refinement threshold, which is calculated based on the internal bit- depth as 1«max(5, bit-depth-7). In another example, th_BDOF may be calculated based on the internal bit-depth as 1<< bit-depth-7. In other words, to control the dynamic range of BDOF motion refinement, the motion refinement threshold is determined as two to the power of the coding bit-depth minus 7.

[0080] FIG. 7 shows a computing environment 710 coupled with a user interface 760. The computing environment 710 can be part of data processing server. The computing environment 710 includes processor 720, memory 740, and I/O interface 750.

[0081] The processor 720 typically controls overall operations of the computing environment 710, such as the operations associated with the display, data acquisition, data communications, and image processing. The processor 720 may include one or more processors to execute instructions to perform all or some of the steps in the above-described methods. Moreover, the processor 720 may include one or more modules that facilitate the interaction between the processor 720 and other components. The processor may be a Central Processing Unit (CPU), a microprocessor, a single chip machine, a GPU, or the like.

[0082] The memory 740 is configured to store various types of data to support the operation of the computing environment 710. Memory 740 may include predetermine software 742. Examples of such data comprise instructions for any applications or methods operated on the computing environment 710, video datasets, image data, etc. The memory 740 may be implemented by using any type of volatile or non-volatile memory devices, or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic or optical disk.

[0083] The I/O interface 750 provides an interface between the processor 720 and peripheral interface modules, such as a keyboard, a click wheel, buttons, and the like. The buttons may include but are not limited to, a home button, a start scan button, and a stop scan button. The I/O interface 750 can be coupled with an encoder and decoder.

[0084] In an embodiment, there is also provided a non-transitory computer-readable storage medium comprising a plurality of programs, such as comprised in the memory 740, executable by the processor 720 in the computing environment 710, for performing the above- described methods. For example, the non-transitory computer-readable storage medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disc, an optical data storage device or the like.

[0085] The non-transitory computer-readable storage medium has stored therein a plurality of programs for execution by a computing device having one or more processors, where the plurality of programs when executed by the one or more processors, cause the computing device to perform the above-described method for motion prediction.

[0086] In an embodiment, the computing environment 710 may be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field- programmable gate arrays (FPGAs), graphical processing units (GPUs), controllers, micro- controllers, microprocessors, or other electronic components, for performing the above methods.

[0087] The description of the present disclosure has been presented for purposes of illustration, and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative implementations will be apparent to those of ordinary skill in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.

[0088] The examples were chosen and described in order to explain the principles of the disclosure, and to enable others skilled in the art to understand the disclosure for various implementations and to best utilize the underlying principles and various implementations with various modifications as are suited to the particular use contemplated. Therefore, it is to be understood that the scope of the disclosure is not to be limited to the specific examples of the implementations disclosed and that modifications and other implementations are intended to be included within the scope of the present disclosure.

Claims

CLAIMS What is claimed is:

1. A bit-depth control method of bi-directional optical flow (BDOF) for coding a video signal, comprising:

obtaining a first reference picture /⁽⁰⁾ and a second reference picture /⁽¹⁾ associated with a video block, wherein the first reference picture /⁽⁰⁾ is before a current picture and the second reference picture /⁽¹⁾ is after the current picture in display order;

obtaining first prediction samples I⁽⁰⁾(i, j) of the video block from a reference block in the first reference picture /⁽⁰⁾, wherein i and j represent a coordinate of one sample with the current picture;

obtaining second prediction samples I⁽¹⁾(i, j) of the video block from a reference block in the second reference picture /⁽¹⁾;

controlling internal bit-depths of the BDOF by applying right-shifting to internal

BDOF parameters when a coding bit-depth is greater than 12-bit, wherein the internal BDOF parameters comprising horizontal gradient values and vertical gradient values derived based on the first prediction samples I⁽⁰⁾(i, j), the second prediction samples I⁽¹⁾(i, j) and sample differences between the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽⁰⁾(i, j); obtaining motion refinements for samples in the video block based on the BDOF

being applied to the video block based on the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j); and

obtaining bi-prediction samples of the video block based on the motion refinements.

2. The method of claim 1 , wherein controlling the internal bit-depths of the BDOF by applying right-shifting to the internal BDOF parameters when the coding bit-depth is greater than 12-bit comprises:

obtaining a first horizontal gradient value of a first prediction sample I⁽⁰⁾(i, j) based on a difference between a first prediction sample I⁽⁰⁾(i+ 1, j and a first prediction sample

I⁽⁰⁾(i- 1, j ;

obtaining a second horizontal gradient value of a second prediction sample

based on a difference between a second prediction sample I⁽¹⁾(i+ 1, j and a second prediction sample I⁽¹⁾(i- 1, j ;

obtaining a first vertical gradient value of a first prediction sample I⁽⁰⁾(i, j) based on a difference between a first prediction sample I⁽⁰⁾(i, j+ 1 and a first prediction sample I⁽⁰⁾(i, j- 1 ;

obtaining a second vertical gradient value of a second prediction sample

based on a difference between a second prediction sample I⁽¹⁾(i, j+ 1 and a second prediction sample I⁽¹⁾(i, j- 1 ;

right shifting the first and second horizontal gradient values by a first shift value; and right shifting the first and second vertical gradient values by the first shift value.

3. The method of claim 2, wherein the first shift value is equal to the coding bit-depth minus 6.

4. The method of claim 1, further comprising:

obtaining first correlation values, wherein the first correlation values are a sum of the horizontal gradient values based on the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j);

obtaining second correlation values, wherein the second correlation values are a sum of the vertical gradient values based on the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j) ;

modifying the first correlation values by right shifting the first correlation values using a second shift value; and

modifying the second correlation values by right shifting the second correlation values using the second shift value.

5. The method of claim 4, wherein the second shift value is equal to the coding bit- depth minus 11.

6. The method of claim 4, further comprising:

obtaining first modified prediction samples by right shifting the first prediction samples I⁽⁰⁾(i, j) using a third shift value; obtaining second modified prediction samples by right shifting the second prediction samples I⁽¹⁾(i, j) using the third shift value; and

obtaining third correlation values, wherein the third correlation values are a difference between the first modified prediction samples and the second modified prediction samples.

7. The method of claim 6, wherein the third shift value is equal to the coding bit-depth minus 8.

8. The method of claim 6, further comprising:

obtaining a first internal summation value based on a sum of the first correlation values squared within each 4x4 sub-block of the video block;

obtaining a second internal summation value based on a sum of multiplications of the first correlation values and the second correlation values within each 4x4 sub-block of the video block;

obtaining a third internal summation value based on a sum of multiplications of the first correlation values and the third correlation values within each 4x4 sub-block of the video block;

obtaining a fourth internal summation value based on a sum of the second correlation values squared within each 4x4 sub-block of the video block;

obtaining a fifth internal summation value based on a sum of multiplications of the second correlation values and the third correlation values within each 4x4 sub-block of the video block;

obtaining a horizontal motion refinement value based on the quotient of the third internal summation value and the first internal summation value, wherein motion refinement values comprise the horizontal motion refinement value;

obtaining a vertical motion refinement value based on the second internal summation value, the fourth internal summation value, the fifth internal summation value, and the horizontal motion refinement value, wherein motion refinement values comprise the vertical motion refinement value; and

clipping the horizontal motion refinement value and the vertical motion refinement value based on a motion refinement threshold.

9. The method of claim 8, wherein the motion refinement threshold is determined as two to the power of the coding bit-depth minus 7.

10. A computing device comprising:

one or more processors;

a non-transitory computer-readable storage medium storing instructions executable by the one or more processors, wherein the one or more processors are configured to:

obtain a first reference picture /⁽⁰⁾ and a second reference picture /⁽¹⁾ associated with a video block, wherein the first reference picture /⁽⁰⁾ is before a current picture and the second reference picture /⁽¹⁾ is after the current picture in display order;

obtain first prediction samples I⁽⁰⁾(i, j) of the video block from a reference block in the first reference picture / ⁽⁰⁾ . wherein i and j represent a coordinate of one sample with the current picture;

obtain second prediction samples I⁽¹⁾(i, j) of the video block from a reference block in the second reference picture /⁽¹⁾;

control internal bit-depths of a bi-directional optical flow (BDOF) by applying right-shifting to internal BDOF parameters when a coding bit-depth is greater than 12- bit, wherein the internal BDOF parameters comprising horizontal gradient values and vertical gradient values derived based on the first prediction samples I⁽⁰⁾(i, j) the second prediction samples I⁽¹⁾(i, j), and sample differences between the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j) :

obtain motion refinements for samples in the video block based on the BDOF being applied to the video block based on the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j); and

obtain bi-prediction samples of the video block based on the motion refinements.

11. The computing device of claim 10, wherein the one or more processors configured to control the internal bit-depths of the BDOF by applying right-shifting to the internal BDOF parameters when the coding bit-depth is greater than 12-bit are further configured to:

obtain a first horizontal gradient value of a first prediction sample I⁽⁰⁾(i, j) based on a difference between a first prediction sample I⁽⁰⁾(i+ 1, j and a first prediction sample I⁽⁰⁾(i- 1, j ;

obtain a second horizontal gradient value of a second prediction sample I⁽¹⁾(i, j) based on a difference between a second prediction sample I⁽¹⁾(i+ 1, j and a second prediction sample I⁽¹⁾(i- 1, j ;

obtain a first vertical gradient value of a first prediction sample I⁽⁰⁾(i, j) based on a difference between a first prediction sample I⁽⁰⁾(i, j+ 1 and a first prediction sample I⁽⁰⁾(i, j- 1 ;

obtain a second vertical gradient value of a second prediction sample I⁽¹⁾(i, j) based on a difference between a second prediction sample I⁽¹⁾(i, j+ 1 and a second prediction sample I⁽¹⁾(i, j- 1 ;

right shift the first and second horizontal gradient values by a first shift value; and right shift the first and second vertical gradient values by the first shift value.

12. The computing device of claim 11, wherein the first shift value is equal to the coding bit-depth minus 6.

13. The computing device of claim 10, wherein the one or more processors are further configured to:

obtain first correlation values, wherein the first correlation values are a sum of the horizontal gradient values based on the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j);

obtain second correlation values, wherein the second correlation values are a sum of the vertical gradient values based on the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j) ;

modify the first correlation values by right shifting the first correlation values using a second shift value; and

modify the second correlation values by right shifting the second correlation values using the second shift value.

14. The computing device of claim 13, wherein the second shift value is equal to the coding bit-depth minus 11.

15. The computing device of claim 13, wherein the one or more processors are further configured to:

obtain first modified prediction samples by right shifting the first prediction samples I⁽⁰⁾(i, j) using a third shift value;

obtain second modified prediction samples by right shifting the second prediction samples I⁽¹⁾(i, j) using the third shift value; and

obtain third correlation values, wherein the third correlation values are a difference between the first modified prediction samples and the second modified prediction samples.

16. The computing device of claim 15, wherein the third shift value is equal to the coding bit-depth minus 8.

17. The computing device of claim 15, wherein the one or more processors are further configured to:

obtain a first internal summation value based on a sum of the first correlation values squared within each 4x4 sub-block of the video block;

obtain a second internal summation value based on a sum of multiplications of the first correlation values and the second correlation values within each 4x4 sub-block of the video block;

obtain a third internal summation value based on a sum of multiplications of the first correlation values and the third correlation values within each 4x4 sub-block of the video block;

obtain a fourth internal summation value based on a sum of the second correlation values squared within each 4x4 sub-block of the video block;

obtain a fifth internal summation value based on a sum of multiplications of the second correlation values and the third correlation values within each 4x4 sub-block of the video block;

obtain a horizontal motion refinement value based on the quotient of the third internal summation value and the first internal summation value, wherein motion refinement values comprise the horizontal motion refinement value;

obtain a vertical motion refinement value based on the second internal summation value, the fourth internal summation value, the fifth internal summation value, and the horizontal motion refinement value, wherein motion refinement values comprise the vertical motion refinement value; and

clip the horizontal motion refinement value and the vertical motion refinement value based on a motion refinement threshold.

18. The computing device of claim 17, wherein the motion refinement threshold is determined as two to the power of the coding bit-depth minus 7.

19. A non-transitory computer-readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform:

obtaining a first reference picture /⁽⁰⁾ and a second reference picture /⁽¹⁾ associated with a video block, wherein the first reference picture /⁽⁰⁾ is before a current picture and the second reference picture /⁽¹⁾ is after the current picture in display order;

obtaining first prediction samples I⁽⁰⁾(i, j) of the video block from a reference block in the first reference picture I⁽⁰⁾, wherein i and j represent a coordinate of one sample with the current picture;

obtaining second prediction samples I⁽¹⁾(i, j) of the video block from a reference block in the second reference picture I⁽¹⁾;

controlling internal bit-depths of a bi-directional optical flow (BDOF) by applying right-shifting to internal BDOF parameters when a coding bit-depth is greater than 12-bit, wherein the internal BDOF parameters comprising horizontal gradient values and vertical gradient values derived based on the first prediction samples I⁽⁰⁾(i, j, the second prediction samples I⁽¹⁾(i, j , and sample differences between the first prediction samples I⁽⁰⁾(i, j and the second prediction samples I⁽¹⁾(i, j;

obtaining motion refinements for samples in the video block based on the BDOF being applied to the video block based on the first prediction samples I⁽⁰⁾(i, j and the second prediction samples I⁽¹⁾(i, j ; and obtaining bi-prediction samples of the video block based on the motion refinements.

20. The non-transitory computer-readable storage medium of claim 19, wherein the plurality of programs caused the computing device to perform controlling the internal bit- depths of the BDOF by applying right-shifting to the internal BDOF parameters when the coding bit-depth is greater than 12-bit further cause the computing device to perform:

obtaining a first horizontal gradient value of a first prediction sample I⁽⁰⁾(i, j based on a difference between a first prediction sample /⁽⁰⁾(i + 1 ,j) and a first prediction sample I⁽⁰⁾(i- 1, j ;

obtaining a second horizontal gradient value of a second prediction sample based on a difference between a second prediction sample I⁽¹⁾(i+ 1, j and a second prediction sample I⁽⁰⁾(i- 1, j ;

obtaining a first vertical gradient value of a first prediction sample I⁽⁰⁾(i, j) based on a difference between a first prediction sample I⁽⁰⁾(i, j+ 1 and a first prediction sample I⁽⁰⁾(i, j- 1 ;

obtaining a second vertical gradient value of a second prediction sample

based on a difference between a second prediction sample I⁽¹⁾(i, j+ 1 and a second prediction sample I⁽⁰⁾(i, j- 1 ;

right shifting the first and second horizontal gradient values by a first shift value; and right shifting the first and second vertical gradient values by the first shift value.

21. The non-transitory computer-readable storage medium of claim 20, wherein the first shift value is equal to the coding bit-depth minus 6.

22. The non-transitory computer-readable storage medium of claim 19, wherein the plurality of programs further cause the computing device to perform:

obtaining first correlation values, wherein the first correlation values are a sum of the horizontal gradient values based on the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j) ;

obtaining second correlation values, wherein the second correlation values are a sum of the vertical gradient values based on the first prediction samples I⁽⁰⁾(i, j) and the second prediction samples I⁽¹⁾(i, j); modifying the first correlation values by right shifting the first correlation values using a second shift value; and

modifying the second correlation values by right shifting the second correlation values using the second shift value.

23. The non-transitory computer-readable storage medium of claim 22, wherein the second shift value is equal to the coding bit-depth minus 11.

24. The non-transitory computer-readable storage medium of claim 22, wherein the plurality of programs further cause the computing device to perform:

obtaining first modified prediction samples by right shifting the first prediction samples I⁽⁰⁾(i, j) using a third shift value;

obtaining second modified prediction samples by right shifting the second prediction samples I⁽¹⁾(i, j) using the third shift value; and

obtaining third correlation values, wherein the third correlation values are a difference between the first modified prediction samples and the second modified prediction samples.

25. The non-transitory computer-readable storage medium of claim 24, wherein the third shift value is equal to the coding bit-depth minus 8.

26. The non-transitory computer-readable storage medium of claim 24, wherein the plurality of programs further cause the computing device to perform:

obtaining a first internal summation value based on a sum of the first correlation values squared within each 4x4 sub-block of the video block;

obtaining a second internal summation value based on a sum of multiplications of the first correlation values and the second correlation values within each 4x4 sub-block of the video block;

obtaining a third internal summation value based on a sum of multiplications of the first correlation values and the third correlation values within each 4x4 sub-block of the video block;

obtaining a fourth internal summation value based on a sum of the second correlation values squared within each 4x4 sub-block of the video block; obtaining a fifth internal summation value based on a sum of multiplications of the second correlation values and the third correlation values within each 4x4 sub-block of the video block;

obtaining a horizontal motion refinement value based on the quotient of the third internal summation value and the first internal summation value, wherein motion refinement values comprise the horizontal motion refinement value;

obtaining a vertical motion refinement value based on the second internal summation value, the fourth internal summation value, the fifth internal summation value, and the horizontal motion refinement value, wherein motion refinement values comprise the vertical motion refinement value; and

clipping the horizontal motion refinement value and the vertical motion refinement value based on a motion refinement threshold.

27. The non-transitory computer-readable storage medium of claim 26, wherein the motion refinement threshold is determined as two to the power of the coding bit-depth minus 7.