KR20230123951A - Bidirectional optical flow in video coding - Google Patents

Abstract

A method of decoding video data includes determining that bi-directional optical flow (BDOF) is enabled for a block of video data; dividing the block into a plurality of subblocks based on a determination that BDOF is enabled for the block; determining individual distortion values for each sub-block of one or more sub-blocks of the plurality of sub-blocks; determining that one of the BDOFs per pixel is performed or the BDOF is bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values; determining prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed; and reconstructing the block based on the predicted samples.

Description

Bidirectional optical flow in video coding

This application claims priority to U.S. Application Serial No. 17/645,233, filed on December 20, 2021, and U.S. Provisional Application No. 63/129,190, filed on December 22, 2020, the entire contents of each of which are incorporated herein by reference. incorporated by reference. US Application Serial No. 17/645,233 claims the benefit of US Provisional Application Serial No. 63/129,190, filed on December 22, 2020.

technology field

This disclosure relates to video encoding and video decoding.

Digital video capabilities include digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, A wide range of devices, including digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite wireless telephones, so-called "smart phones", video teleconferencing devices, video streaming devices, and the like. can be integrated into Digital video devices are MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265/High Efficiency Video It implements video coding techniques, such as those described in the standards defined by HEVC, and extensions to those standards. Video devices may transmit, receive, encode, decode, and/or store digital video information more efficiently by implementing such video coding techniques.

Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be divided into coding tree units (CTUs), coding units (CUs) ) and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks of the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks of the same picture, or temporal prediction with respect to reference samples in other reference pictures. there is. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames.

In general, this disclosure provides methods for decoder-side motion vector derivation (e.g., template matching, bilateral matching, decoder-side motion vector (MV) refinement, and/or bidirectional optical flow (BDOF)). Describe the techniques. The techniques of this disclosure can be applied to any of the existing video codecs, such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), Essential Video Coding (EVC), or an efficient coding tool in any future video coding standards. may be

In one or more examples, for BDOF, a video encoder and a video decoder (eg, a video coder) determine whether per-pixel BDOF is performed on subblocks of a block, or whether BDOF is bypassed. It may also be configured to selectively determine. That is, the video coder may choose either per-pixel BDOF or per-pixel BDOF (or BDOF in general) is bypassed. In this way, the example techniques, when combined together, may facilitate selection between coding modes that may provide better coding performance (e.g., where a video coder has a per-pixel BDOF of one is performed for the subblock or BDOF is bypassed for the subblock).

Moreover, in some examples, determining whether to perform per-pixel BDOF or bypass BDOF for a subblock may be based on determining a distortion value and comparing the distortion value to a threshold value. In some examples, the video coder may be configured to determine the distortion value in such a way that the calculations used to determine the distortion value can be reused by the video coder when performing per-pixel BDOF. For example, when a video coder performs per-pixel BDOF, the video coder may reuse results from calculations performed to determine distortion values for performing per-pixel BDOF.

In one example, this disclosure describes a method of decoding video data, the method comprising: determining that bi-directional optical flow (BDOF) is enabled for a block of video data; dividing the block into a plurality of subblocks based on a determination that BDOF is enabled for the block; for each sub-block of one or more sub-blocks of the plurality of sub-blocks, determining individual distortion values; determining that one of the per-pixel BDOFs is performed or the BDOFs are bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values; determining prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed; and reconstructing the block based on the prediction samples.

In one example, this disclosure describes a device for decoding video data, the device including a memory configured to store the video data; and processing circuitry coupled to the memory, wherein the processing circuitry determines that bi-directional optical flow (BDOF) is enabled for the block of video data; divide the block into a plurality of subblocks based on determining that BDOF is enabled for the block; for each subblock of one or more subblocks of the plurality of subblocks, determine respective distortion values; determine that one of the BDOFs per pixel is performed or the BDOF is bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values; determine prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed; and reconstruct the block based on the predicted samples.

In one example, this disclosure describes a computer readable storage medium storing instructions that, when executed, cause one or more processors to determine that bidirectional optical flow (BDOF) is enabled for a block of video data. let it; divide the block into a plurality of subblocks based on a determination that BDOF is enabled for the block; for each sub-block of one or more sub-blocks of the plurality of sub-blocks, determine respective distortion values; determine that one of the per-pixel BDOFs are performed or the BDOFs are bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values; determine prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed; and to reconstruct a block based on the prediction samples.

In one example, this disclosure describes a device for decoding video data, the device comprising: means for determining that bi-directional optical flow (BDOF) is enabled for a block of video data; means for dividing the block into a plurality of subblocks based on determining that BDOF is enabled for the block; means for determining respective distortion values for each sub-block of one or more of the plurality of sub-blocks; means for determining that one of per-pixel BDOF is performed or BDOF is bypassed for each sub-block of one or more sub-blocks of the plurality of sub-blocks based on the respective distortion values; means for determining prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed; and means for reconstructing the block based on the prediction samples.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

1 is a block diagram illustrating an example video encoding and decoding system that may perform the techniques of this disclosure.
2A and 2B are conceptual diagrams illustrating an exemplary QuadTree Binary Tree (QTBT) structure, and a corresponding Coding Tree Unit (CTU).
3 is a block diagram illustrating an example video encoder that may perform the techniques of this disclosure.
4 is a block diagram illustrating an example video decoder that may perform the techniques of this disclosure.
5A and 5B are conceptual diagrams illustrating examples of spatial neighboring motion vector candidates for merge mode and advanced motion vector predictor (AMVP) mode, respectively.
6A and 6B are conceptual diagrams illustrating examples of temporal motion vector predictor (TMVP) candidates and motion vector scaling, respectively.
7 is a conceptual diagram illustrating template matching performed on a search area around an initial motion vector (MV).
8 is a conceptual diagram illustrating examples of proportional motion vector differences based on temporal distances.
9 is a conceptual diagram illustrating examples of motion vector differences that are mirrored regardless of temporal distances.
10 is a conceptual diagram illustrating an example of a 3x3 square search pattern in the search range of [-8,8].
11 is a conceptual diagram illustrating an example of decoding-side motion vector refinement.
12 is a conceptual diagram illustrating an extended coding unit (CU) used in bi-directional optical flow (BDOF).
13 is a flowchart illustrating an exemplary process of per-pixel BDOF with subblock bypass.
14 is a conceptual diagram illustrating an example of BDOF per pixel of an 8x8 subblock.
15 is a flowchart illustrating an example method for decoding a current block, in accordance with the techniques of this disclosure.
16 is a flowchart illustrating an example method for encoding a current block, in accordance with the techniques of this disclosure.

A video encoder may be configured to generate a predictive block from one or more reference blocks in one or more reference pictures having one or more motion vectors for the block. The video encoder determines prediction blocks and residuals between the blocks, and signal information representing the residuals and information used to determine motion vectors. A video decoder receives information representing residuals and information used to determine motion vectors. A video decoder determines motion vector(s), determines reference block(s) from the motion vector(s), and generates a predictive block. A video decoder adds the predictive block to the residual to reconstruct the block.

In some cases, the reference block and predictive block are the same block. However, it is not required in all examples that the reference block and the prediction block are the same. In some examples, eg, in bi-prediction, a video encoder and a video decoder may determine a first reference block based on a first motion vector and a second reference block based on a second motion vector. A video encoder and a video decoder may blend the first and second reference blocks to generate a predictive block.

Moreover, in some examples, the video encoder and video decoder may generate the predictive block based on adjustments to sample values of the first and second reference blocks. One example method for adjusting sample values to generate samples of a predictive block is referred to as bi-directional optical flow (BDOF). For example, assume that I ⁽⁰⁾ (x,y) refers to the first reference block and I ⁽¹⁾ (x,y) refers to the second reference block. In BDOF, a predictive block may be considered as I ⁽⁰⁾ (x,y) plus I ⁽¹⁾ (x,y). As described below, the video encoder and video decoder may determine adjustment factors (ie, b(x,y)) and add the adjustment factors to the predictive block as part of the process of determining predictive samples. (i.e. I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) + b(x,y)). There may be additional scaling and offsetting of the result of I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) + b(x,y) to determine the prediction samples.

In BDOF, a video encoder and a video decoder utilize a motion vector to determine adjustment factors (eg, multiplied or added factors) for adjusting sample values of a predictive block to generate predictive samples. As an example, a video encoder and a video decoder may generate predictive samples by adding corresponding values of the first reference block, the corresponding samples of the second reference block, and the motion refinement.

Various types of BDOF techniques may exist. One example of BDOF is subblock BDOF, and another example of BDOF techniques is per-pixel BDOF. In a subblock BDOF, a video encoder and a video decoder determine a motion refinement (also referred to as refined motion) for a subblock. For subblock BDOF, a video encoder and a video decoder use the same motion refinement to adjust samples from a predictive block, where the predictive block is a first reference block and a second reference block (e.g., the first reference block block and the second reference block, or a weighted average of the first reference block and the second reference block). In per-pixel BDOF, a video encoder and a video decoder may determine motion refinement factors, which may be different for two or more samples in the current block. For per-pixel BDOF, the video encoder and video decoder determine the motion refinements determined for the sample-per-pixel (also the refined referred to as motions) may be used.

Whether BDOF or other refinement techniques may be selectively enabled at the block level, but whether BDOF is applied at the sub-block level may be inferred based on the distortion values. For example, a video encoder may enable BDOF for a block and may signal information indicating that BDOF is enabled for the block.

In response, the video decoder may divide the block into a plurality of subblocks based on determining that BDOF is enabled for the block. Although BDOF is enabled for a block, a video decoder may determine whether BDOF is actually performed or bypassed on a subblock-by-subblock basis. For example, a video decoder determines, for each subblock of one or more subblocks of a plurality of subblocks, respective distortion values.

According to one or more examples described in this disclosure, the video decoder performs one of per-pixel BDOF for each subblock of one or more subblocks of a plurality of subblocks based on respective distortion values, or BDOF is bypassed. may decide to become For example, a video decoder may determine a first distortion value for a first subblock and determine that per-pixel BDOF is performed for the first subblock based on the first distortion value. The video decoder may determine a second distortion value for the second subblock, determine that BDOF is bypassed for the second subblock based on the second distortion value, and so on.

In one or more examples, if the video decoder determines that BDOF is performed, the video decoder may perform per-pixel BDOF, and other BDOF techniques may not be available to the video decoder. That is, the video decoder may determine, on a subblock-by-subblock basis, that one of the BDOFs per pixel is performed or the BDOF is bypassed for each subblock. If BDOF is performed, the BDOF technique available to the video decoder may be per-pixel BDOF, and other BDOF techniques may not be available.

In one or more examples, as described above, a video decoder may determine distortion values for determining whether per-pixel BDOF is performed or whether BDOF is bypassed on a subblock-by-subblock basis. In some examples, as will be described in more detail below, a video decoder may reuse calculations used to determine distortion values for determining per-pixel motion refinement for per-pixel BDOF. For example, for a first subblock, a video decoder may determine a first distortion value. Assume that for the first subblock, the video decoder has determined that per-pixel BDOF is enabled. In some examples, rather than recalculating all of the values necessary to determine the per-pixel motion refinement, the video decoder may determine from the calculations performed by the video decoder to determine that the per-pixel BDOF is performed to determine the per-pixel motion refinement. It may also be configured to reuse the results of

A video decoder may be configured to determine predictive samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed. For example, it is assumed that BDOF is performed per pixel for subblocks. In this example, a video decoder may generate predictive samples for a subblock by refining samples of a predictive block (eg, a block generated from combining two reference blocks) based on per-pixel motion refinement. there is. As another example, assume that BDOF is bypassed for a subblock. In this example, the video decoder may not perform refinement of the samples of the predictive block to generate the predictive samples. Rather, the samples of the predictive block may be the same as the predictive samples (or possibly with some adjustment not based on BDOF). For example, when BDOF is bypassed, the video encoder and video decoder may generate predictive samples by determining a weighted average of corresponding samples in the first reference block and the second reference block.

A video decoder may reconstruct a block based on the predictive samples. For example, a video decoder may receive residual values representing differences between prediction samples and samples of a block, and add the residual values to the prediction samples to reconstruct the block. The above examples are described from the point of view of a video decoder. A video encoder may be configured to perform similar techniques. For example, the prediction samples generated by the video decoder should be the same as the prediction samples generated by the video encoder. Thus, a video encoder may perform techniques similar to those described above to determine predictive samples in the same way as a video decoder.

1 is a block diagram illustrating an example video encoding and decoding system 100 that may perform the techniques of this disclosure. The techniques of this disclosure relate generally to coding (encoding and/or decoding) video data. Generally, video data includes any data for processing video. Accordingly, video data may include video metadata such as raw, unencoded video, encoded video, decoded (eg, reconstructed) video, and signaling data.

As shown in FIG. 1 , system 100 , in this example, includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116 . In particular, source device 102 provides video data to destination device 116 via computer readable medium 110 . Source device 102 and destination device 116 may be desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, It may include any of a wide variety of devices, including digital media players, video gaming consoles, video streaming devices, broadcast receiver devices, and the like. In some cases, source device 102 and destination device 116 may be equipped for wireless communication and, therefore, may be referred to as wireless communication devices.

In the example of FIG. 1 , source device 102 includes a video source 104 , a memory 106 , a video encoder 200 , and an output interface 108 . Destination device 116 includes input interface 122 , video decoder 300 , memory 120 , and display device 118 . According to this disclosure, video encoder 200 of source device 102 and video decoder 300 of destination device 116 perform template matching, bilateral matching, decoder-side motion vector (MV) refinement, and bi-directional optical flow and It may be configured to apply the techniques for the same decoder-side motion vector derivation techniques. Thus, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, a source device and a destination device may include other components or arrangements. For example, source device 102 may receive video data from an external video source, such as an external camera. Likewise, destination device 116 may interface with an external display device, rather than including an integrated display device.

System 100 as shown in FIG. 1 is only one example. In general, any digital video encoding and/or decoding device performs techniques for decoder-side motion vector derivation techniques such as template matching, bilateral matching, decoder-side motion vector (MV) refinement, and bidirectional optical flow (BDOF). You may. Source device 102 and destination device 116 are merely examples of such coding devices for which source device 102 generates coded video data for transmission to destination device 116 . This disclosure refers to a “coding” device as a device that performs coding (encoding and/or decoding) of data. Accordingly, video encoder 200 and video decoder 300 represent examples of coding devices, particularly a video encoder and video decoder, respectively. In some examples, source device 102 and destination device 116 may operate in a substantially symmetric manner such that each of source device 102 and destination device 116 includes video encoding and decoding components. . Thus, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116, eg, for video streaming, video playback, video broadcasting, or video telephony.

In general, video source 104 represents a source of video data (ie, raw, unencoded video data), a sequential series of pictures of video data to video encoder 200 that encodes the data for the pictures. s (also referred to as "frames"). Video source 104 of source device 102 may include a video capture device, such as a video camera, a video archive containing previously captured raw video, and/or a video feed interface for receiving video from a video content provider. may be As a further alternative, video source 104 may generate computer graphics-based data as source video, or as a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange pictures from received order (sometimes referred to as “display order”) into coding order for coding. Video encoder 200 may generate a bitstream that includes encoded video data. Source device 102 then converts the encoded video data to computer-readable medium 110 via output interface 108 for reception and/or retrieval by, e.g., input interface 122 of destination device 116. ) can also be output.

Memory 106 of source device 102 and memory 120 of destination device 116 represent general purpose memories. In some examples, memories 106 and 120 may store raw video data, eg, raw video from video source 104 and raw, decoded video data from video decoder 300 . Additionally or alternatively, memories 106 and 120 may store software instructions executable by, eg, video encoder 200 and video decoder 300 , respectively. Although memory 106 and memory 120 are shown separately from video encoder 200 and video decoder 300 in this example, video encoder 200 and video decoder 300 also serve functionally similar or equivalent purposes. It should be understood that it may include internal memories for Moreover, memories 106 and 120 may store encoded video data, eg, output from video encoder 200 and input to video decoder 300 . In some examples, portions of memories 106 and 120 may be allocated as one or more video buffers, eg, to store raw, decoded, and/or encoded video data.

Computer readable medium 110 may represent any type of medium or device capable of transferring encoded video data from source device 102 to destination device 116 . In one example, computer readable medium 110 enables source device 102 to transmit encoded video data directly to destination device 116 in real time, e.g., over a radio frequency network or a computer-based network. represents a medium of communication for In accordance with a communication standard, such as a wireless communication protocol, output interface 108 may modulate a transmission signal containing encoded video data, and input interface 122 may demodulate a received transmission signal. Communication media may include any wireless or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from source device 102 to destination device 116.

In some examples, source device 102 may output encoded data from output interface 108 to storage device 112 . Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122 . Storage device 112 may be a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. may include any of a variety of distributed or locally accessed data storage media, such as

In some examples, source device 102 may output the encoded video data to a file server 114 or other intermediate storage device that may store the encoded video data generated by source device 102. Destination device 116 may access stored video data from file server 114 via streaming or download.

File server 114 may be any type of server device capable of storing encoded video data and transmitting the encoded video data to destination device 116 . File server 114 may be a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), and a content delivery network. (CDN) device, Hypertext Transfer Protocol (HTTP) server, Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and/or Network Attached Storage (NAS) device. File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, and the like. may be

Destination device 116 may access the encoded video data from file server 114 over any standard data connection, including an Internet connection. This may be a wireless channel (eg, Wi-Fi connection), a wired connection (eg, digital subscriber line (DSL), cable modem, etc.), or both, suitable for accessing encoded video data stored on file server 114. A combination of all may be included. Input interface 122 is configured to operate according to any one or more of the various protocols discussed above for retrieving or receiving media data from file server 114, or other such protocols for retrieving media data. It could be.

Output interface 108 and input interface 122 are wireless transmitters/receivers, modems, wired networking components (eg, Ethernet cards), wireless communication components that operate in accordance with any of the various IEEE 802.11 standards. , or other physical components. For examples in which output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 may be configured for 4G, 4G-LTE (long term evolution), LTE Advanced, 5G, and the like. In accordance with cellular communication standards, it may be configured to transmit data such as encoded video data. In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 conform to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (eg, ZigBee™), the Bluetooth™ standard, and the like. s, may be configured to transmit data such as encoded video data. In some examples, source device 102 and/or destination device 116 may include separate system-on-a-chip (SoC) devices. For example, source device 102 may include video encoder 200 and/or an SoC device to perform functionality due to output interface 108 , and destination device 116 may include video decoder 300 and/or a SoC device for performing functionality due to input interface 122 .

The techniques of this disclosure may be used for over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions such as dynamic adaptive streaming over HTTP (DASH), data storage It may be applied to video coding in support of any of a variety of multimedia applications, such as digital video encoded onto a medium, decoding of digital video stored on a data storage medium, or other applications.

Input interface 122 of destination device 116 receives the encoded video bitstream from computer readable medium 110 (eg, communication medium, storage device 112, file server 114, etc.). An encoded video bitstream includes syntax elements with values describing processing and/or characteristics of video blocks or other coded units (eg, slices, pictures, groups of pictures, sequences, etc.) The same may include signaling information defined by video encoder 200 that is also used by video decoder 300 . Display device 118 displays decoded pictures of the decoded video data to a user. Display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

Although not shown in FIG. 1 , in some examples, video encoder 200 and video decoder 300 may be integrated with an audio encoder and/or an audio decoder, respectively, and include both audio and video in a common data stream. may include suitable MUX-DEMUX units, or other hardware and/or software, to handle multiplexed streams that If applicable, MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP).

Video encoder 200 and video decoder 300 may each include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete It may be implemented as any of a variety of suitable encoder and/or decoder circuitry, such as logic, software, hardware, firmware or any combinations thereof. If the techniques are implemented partly in software, a device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. . That is, there may be a computer readable storage medium storing instructions that, when executed, cause one or more processors to perform the example techniques described in this disclosure. Each of video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in a separate device. . A device that includes video encoder 200 and/or video decoder 300 may include an integrated circuit, a microprocessor, and/or a wireless communication device such as a cellular telephone.

The following describes video coding standards. Video coding standards, including its scalable video coding (SVC) and multi-view video coding (MVC) extensions, include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO /IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC). Additionally, High Efficiency Video Coding (HEVC) or ITU-T H.265, including Range Extension, Multiview Extension (MV-HEVC) and Scalable Extension (SHVC), ISO/IEC Motion Picture Experts Group (MPEG) and the Joint Collaboration Team on 3D Video Coding Extension Development (JCT-3V) of the ITU-T Video Coding Experts Group (VCEG) as well as the Joint Collaboration Team on Video Coding (JCT-VC). The HEVC specification is available from ITU-T H.265, "Series H: Audiovisual and Multimedia Systems, Infrastructure of Audiovisual Services-Coding of Moving Video, High efficiency Video Coding," The International Telecommunication Union. December 2016, 664 Pages.

ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) implement the current HEVC standard (its current extensions and short-term extensions for screen content coding and high-dynamic-range coding). We are working on standardization of future video coding technologies with compression capabilities significantly exceeding those of The groups are working together on this exploratory activity in a collaborative collaborative effort known as the Joint Video Exploration Team (JVET) to evaluate compression technology designs proposed by their experts in this area. The latest version of the reference software, namely VVC Test Model 10 (VTM 10.0), can be downloaded from https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM.

Video encoder 200 and video decoder 300 implement a video coding standard such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC) or extensions thereof, such as multi-view and/or scalable video It may operate according to coding extensions. Alternatively, video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards, such as ITU-T H.266, also referred to as Versatile Video Coding (VVC). A draft of the VVC standard is from Bross et al., "Versatile Video Coding (Draft 10)," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 18 ^th Meeting: by teleconference, 22 June - 1 July 2020, JVET-S2001-vA (hereinafter "VVC Draft 10"). Editorial refinements of VVC Draft 10 can be found in Bross et al., "Versatile Video Coding Editorial Refinements on Draft 10," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11; 20 ^th Meeting: by teleconference, 7-16 Oct.2020, described in JVET-T2001-v2. An algorithm description of the Versatile Video Coding and Test Model 10 (VTM 10.0) can be found in J.Chen, Y.Ye and S.Kim, "Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11)," JVET-T2002, Dec. 2020 (hereinafter referred to as JVET-T2002). However, the techniques of this disclosure are not limited to any particular coding standard.

In general, video encoder 200 and video decoder 300 may perform block-based coding of pictures. The term “block” generally refers to a structure containing data to be processed (eg, to be encoded, decoded, or otherwise used in an encoding and/or decoding process). For example, a block may include a two-dimensional matrix of samples of luminance and/or chrominance data. In general, video encoder 200 and video decoder 300 may code video data represented in YUV (eg, Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for samples of a picture, video encoder 200 and video decoder 300 may code luminance and chrominance components, where: Chrominance components may include both red and blue hue chrominance components. In some examples, video encoder 200 converts received RGB formatted data to a YUV representation prior to encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, pre- and post-processing units (not shown) may perform these transformations.

This disclosure may generally refer to coding (eg, encoding and decoding) of pictures to include the process of encoding or decoding data of a picture. Similarly, this disclosure may refer to coding of blocks of a picture to include the process of encoding or decoding data for the blocks, eg, prediction and/or residual coding. An encoded video bitstream typically includes a series of values for syntax elements that indicate coding decisions (eg, coding modes) and partitioning of pictures into blocks. Accordingly, references to coding a picture or block should generally be understood as coding values for syntax elements forming the picture or block.

HEVC defines various blocks including coding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video coder (such as video encoder 200) partitions a coding tree unit (CTU) into CUs according to a quadtree structure. That is, the video coder partitions the CTUs and CUs into four equal non-overlapping squares, and each node in the quadtree has either zero or four child nodes. Nodes with no child nodes may be referred to as “leaf nodes,” and the CUs of such leaf nodes may include one or more PUs and/or one or more TUs. A video coder may further partition PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents partitioning of TUs. In HEVC, PUs represent inter-prediction data, while TUs represent residual data. Intra-predicted CUs contain intra-prediction information such as an intra-mode indication.

As another example, video encoder 200 and video decoder 300 may be configured to operate according to VVC. According to VVC, a video coder (such as video encoder 200) partitions a picture into a plurality of coding tree units (CTUs). Video encoder 200 may partition the CTU according to a tree structure, such as a quadtree binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concept of multiple partition types, such as HEVC's separation between CUs, PUs, and TUs. The QTBT structure includes two levels: a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary tree partitioning. A root node of the QTBT structure corresponds to a CTU. Leaf nodes of binary trees correspond to coding units (CUs).

In the MTT partitioning structure, blocks are divided using quadtree (QT) partitions, binary tree (BT) partitions, and one or more types of triple tree (TT) (also referred to as ternary tree (TT)) partitions. It can also be partitioned. A triple or ternary tree partition is a partition in which a block is divided into three subblocks. In some examples, a triple or ternary tree partition splits a block into three subblocks without splitting the original block through the center. Partitioning types in MTT (eg, QT, BT, and TT) may be symmetric or asymmetric.

In some examples, video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, while in other examples, video encoder 200 and video decoder 300 has one QTBT/MTT structure for the luminance component and another QTBT/MTT structure for both chrominance components (or two QTBT/MTT structures for individual chrominance components). ) may use two or more QTBT or MTT structures such as

Video encoder 200 and video decoder 300 may be configured to use quadtree partitioning per HEVC, QTBT partitioning, MTT partitioning, or other partitioning structures. For purposes of explanation, the description of the techniques of this disclosure is presented in terms of QTBT partitioning. However, it should be understood that the techniques of this disclosure may also be applied to video coders configured to use quadtree partitioning, or other types of partitioning as well.

In some examples, a CTU is a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples of a picture having three sample arrays, or three separate color planes used to code the samples and Contains the CTB of samples of a picture or a monochrome picture coded using syntax structures. A CTB may be an NxN block of samples for some value of N such that the division of a component into CTBs is partitioning. A component is a single sample or array from one of the three arrays (luma and two chroma) that make up a picture in 4:2:0, 4:2:2, or 4:4:4 color format, or monochrome A single sample or array of arrays that make up a picture in a format. In some examples, a coding block is an MxN block of samples for some values of M and N such that the division of a CTB into coding blocks is partitioning.

Blocks (eg, CTUs or CUs) may be grouped in a variety of ways in a picture. As an example, a brick may refer to a rectangular area of CTU rows within a particular tile in a picture. A tile may be a rectangular area of CTUs within a particular tile column and a particular tile row in a picture. A tile column refers to a rectangular region of CTUs with a height equal to the height of the picture and a width specified by syntax elements (eg, as in a picture parameter set). A tile row refers to a rectangular region of CTUs with a height specified by syntax elements (eg, as in a picture parameter set) and a width equal to the width of the picture.

In some examples, a tile may be partitioned into multiple bricks, each of which may include one or more CTU rows within the tile. A tile that is not partitioned into multiple bricks may also be referred to as a brick. However, bricks that are a true subset of tiles may not be referred to as tiles.

Bricks in a picture may also be arranged into slices. A slice may be an integer number of bricks of a picture that may be exclusively included in a single network abstraction layer (NAL) unit. In some examples, a slice contains only a contiguous sequence of multiple complete tiles or complete bricks of one tile.

This disclosure may use “N×N” and “N by N” interchangeably to refer to sample dimensions of a block (such as a CU or other video block) in terms of vertical and horizontal dimensions, e.g., 16 ×16 samples or 16 by 16 samples. In general, a 16×16 CU will have 16 samples in the vertical direction (y = 16) and 16 samples in the horizontal direction (x = 16). Similarly, an N×N CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N denotes a non-negative integer value. Samples in a CU may be arranged in rows and columns. Moreover, CUs do not necessarily have the same number of samples in the horizontal direction as in the vertical direction. For example, CUs may contain N×M samples, where M is not necessarily equal to N.

Video encoder 200 encodes video data for CUs representing prediction and/or residual information, and other information. Prediction information indicates how a CU is to be predicted in order to form a predictive block for the CU. Residual information generally indicates sample-by-sample differences between samples of a CU before encoding and a predictive block.

To predict a CU, video encoder 200 may generally form a predictive block for the CU via inter-prediction or intra-prediction. Inter-prediction generally refers to predicting a CU from data of a previously coded picture, whereas intra-prediction generally refers to predicting a CU from previously coded data of the same picture. To perform inter-prediction, video encoder 200 may use one or more motion vectors to generate a predictive block. Video encoder 200 may generally perform a motion search to identify a reference block that closely matches a CU, eg, in terms of differences between the CU and the reference block. Video encoder 200 determines the sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared differences (MSD), or whether the reference block closely matches the current CU. Other such difference calculations may be used to determine the difference metric. In some examples, video encoder 200 may predict the current CU using uni-prediction or bi-prediction.

Some examples of VVC also provide an affine motion compensation mode, which may be considered an inter-prediction mode. In affine motion compensation mode, video encoder 200 may determine two or more motion vectors representing non-translational motion, such as zooming in or out, rotation, perspective motion, or other irregular motion types.

To perform intra-prediction, video encoder 200 may select an intra-prediction mode to generate a predictive block. Some examples of VVC provide 67 intra-prediction modes including planar mode and DC mode as well as various directional modes. In general, video encoder 200 selects an intra-prediction mode that describes neighboring samples for a current block (eg, a block of a CU) from which to predict samples of the current block. Such samples are generally above, above and to the left of, or to the left of, the current block in the same picture as the current block, assuming that video encoder 200 codes the CTUs and CUs in raster scan order (left to right, top to bottom). may be in

Video encoder 200 encodes data representing a prediction mode for a current block. For example, for inter-prediction modes, video encoder 200 may encode data indicating which of the various available inter-prediction modes is being used, as well as motion information for the corresponding mode. For uni- or bi-directional inter-prediction, for example, video encoder 200 may encode motion vectors using Advanced Motion Vector Prediction (AMVP) or merge mode. Video encoder 200 may encode motion vectors for an affine motion compensation mode using similar modes.

Following prediction, such as intra-prediction or inter-prediction, of a block, video encoder 200 may calculate residual data for the block. Residual data, such as a residual block, represents sample-by-sample differences between a block and a prediction block for the block, formed using a corresponding prediction mode. Video encoder 200 may apply one or more transforms to the residual block to produce transformed data in the transform domain instead of the sample domain. For example, video encoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, video encoder 200 may apply a secondary transform subsequent to the first transform, such as a mode dependent non-separable secondary transform (MDNSST), a signal dependent transform, a Karhunen-Loeve transform (KLT), or the like. Video encoder 200 produces transform coefficients following application of one or more transforms.

As mentioned above, following any transforms to generate transform coefficients, video encoder 200 may perform quantization of the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to provide additional compression, possibly reducing the amount of data used to represent the transform coefficients. By performing the quantization process, video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, video encoder 200 may round down an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, video encoder 200 may perform a bitwise right-shift of the value to be quantized.

Following quantization, video encoder 200 may scan the transform coefficients to generate a one-dimensional vector from the two-dimensional matrix containing the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) transform coefficients in front of the vector and lower energy (and therefore higher frequency) transform coefficients in the back of the vector. In some examples, video encoder 200 may utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other examples, video encoder 200 may perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 200 may entropy encode the one-dimensional vector, eg, according to context-adaptive binary arithmetic coding (CABAC). Video encoder 200 may also entropy encode values for syntax elements that describe metadata associated with encoded video data for use by video decoder 300 in decoding the video data.

To perform CABAC, video encoder 200 may assign a context within a context model to a symbol to be transmitted. Context may relate to whether neighboring values of a symbol are zero values, for example. The probability determination may be based on the context assigned to the symbol.

Video encoder 200 sends syntax data, e.g., block-based syntax data, picture-based syntax data, and sequence-based syntax data, to video decoder 300, e.g., picture header, block header, slice header, or other syntax data, For example, it may additionally generate in a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS). Likewise, video decoder 300 may decode such syntax data to determine how to decode the corresponding video data.

In this way, video encoder 200 converts encoded video data, e.g., syntax elements that describe the partitioning of a picture into blocks (e.g., CUs) and bits that contain prediction and/or residual information for the blocks. You can also create streams. Ultimately, video decoder 300 may receive the bitstream and decode the encoded video data.

In general, video decoder 300 performs a process that is reversible to that performed by video encoder 200 to decode encoded video data of a bitstream. For example, video decoder 300 may decode values for syntax elements of a bitstream using CABAC in a manner that is substantially similar to, but reversible to, the CABAC encoding process of video encoder 200 . Syntax elements may define partitioning information for partitioning of a picture into CTUs and partitioning of each CTU according to a corresponding partition structure, such as a QTBT structure, to define the CUs of a CTU. Syntax elements may further define prediction and residual information for blocks of video data (eg, CUs).

Residual information may be represented by quantized transform coefficients, for example. Video decoder 300 may inverse quantize and inverse transform the quantized transform coefficients of the block to reconstruct a residual block for the block. Video decoder 300 uses the signaled prediction mode (intra- or inter-prediction) and related prediction information (eg, motion information for inter-prediction) to form a predictive block for the block. Video decoder 300 may then combine the predictive block and the residual block (on a sample-by-sample basis) to reproduce the original block. Video decoder 300 may perform additional processing, such as performing a deblocking process to reduce visual artifacts along the boundaries of a block.

In accordance with the techniques of this disclosure, video encoder 200 and video decoder 300 may be configured to perform bi-directional optical flow (BDOF). For example, video encoder 200 may be configured to perform BDOF as part of encoding the current block, and video decoder 300 may be configured to perform BDOF as part of decoding the current block.

As described in more detail, in some examples, a video coder (e.g., video encoder 200 and/or video decoder 300) divides an input block into a plurality of subblocks, such that a size of the input block divides the input block into a plurality of subblocks, which is less than or equal to the size of a coding unit, determines that a bidirectional optical flow (BDOF) is to be applied to a subblock of the plurality of subblocks based on a condition being satisfied, and Divide into a plurality of sub-subblocks, and determine a refined motion vector for one or more of the sub-subblocks, wherein the refined motion vector for a sub-subblock of the one or more sub-subblocks is determine the refined motion vector, which is the same for a plurality of samples in a sub-subblock, and perform BDOF on a subblock based on the refined motion vector for one or more sub-subblocks. may be

As another example, a video coder divides an input block into a plurality of subblocks, wherein a size of the input block is less than or equal to a size of a coding unit, and based on a condition being satisfied determine that a bidirectional optical flow (BDOF) is to be applied to a subblock of the plurality of subblocks, divide the subblock into a plurality of sub-subblocks, and refine motion for each of one or more samples in the subblock It may be configured to determine a vector and perform BDOF for the subblock based on the refined motion vector for each of one or more samples in the subblock.

For example, as described above, video encoder 200 or video decoder 300 determines a refined motion vector for each of one or more samples in a subblock, and determines the number of the one or more samples in a subblock. BDOF may be performed based on the refined motion vectors for each. In this disclosure, performing BDOF based on a refined motion vector for each of one or more samples in a subblock is referred to as “per-pixel BDOF”. For example, for per-pixel BDOF, rather than having the same one refined motion vector for all samples in a subblock, the refined motion vector for each sample in a subblock is determined separately.

A refined motion vector may not necessarily mean that a motion vector for a subblock is changed. Rather, the refined motion vector for the sample may be used to determine the amount by which the samples in the predictive block are adjusted to generate the predictive sample. For example, for a first sample of a first subblock, a first refined motion vector may indicate how much to adjust the first sample in the predictive block to generate the first prediction sample, and the first subblock For a second sample of , the second refined motion vector may indicate how much to adjust the second sample in the prediction to generate the second prediction sample, and so on.

According to one or more examples described in this disclosure, video encoder 200 and video decoder 300 generate a per-pixel for each subblock of one or more subblocks of a block (eg, an input block) based on respective distortion values. It may be determined that either BDOF is performed or BDOF is bypassed. For example, as described above, video encoder 200 and video decoder 300 may perform per-pixel BDOF based on a condition being satisfied. The condition may be satisfied when the distortion value of the subblock is greater than the threshold value.

Accordingly, in some examples, options for video encoder 200 and video decoder 300 include performing per-pixel BDOF for a subblock based on whether the distortion value for the subblock is greater than or less than a threshold. It may be configured to bypass BDOF or BDOF. For example, in some techniques, it may be possible for video encoder 200 and video decoder 300 to perform per-pixel BDOF on a subblock-by-subblock basis, but not determine whether BDOF is bypassed. In some techniques where BDOF can be bypassed on a subblock-by-subblock basis, per-pixel BDOF may not have been available. With the example techniques described in this disclosure, video encoder 200 and video decoder 300 may be configured to optionally perform per-pixel BDOF or bypass BDOF, which reduces decoding overhead as appropriate. Balancing may result in better video compression.

In one or more examples, to encode or decode video data, respectively, video encoder 200 and video decoder 300 determine that BDOF is enabled for the block of video data and determine that BDOF is enabled for the block based on, or more generally, may be configured to divide a block into a plurality of subblocks when BDOF is enabled for the block. Video encoder 200 and video decoder 300 may determine, for each subblock of one or more of the plurality of subblocks, separate distortion values. Exemplary ways of determining individual distortion values are described in more detail below. Video encoder 200 and video decoder 300 determine whether one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of one or more subblocks of a plurality of subblocks based on the respective distortion values. and determine predictive samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed.

Video encoder 200 may determine, and may signal residual values, that indicate a difference between the prediction samples and the samples of the block. Video decoder 300 may receive residual values indicative of a difference between the prediction samples and the samples of the block, and may add the residual values to the prediction samples to reconstruct the block. In some examples, to receive the residual values, video decoder 300 may be configured to receive information indicative of the residual values, from which video decoder 300 determines the residual values.

This disclosure may generally refer to “signaling” certain information, such as syntax elements. The term “signaling” may generally refer to the communication of values and/or other data for syntax elements used to decode encoded video data. That is, video encoder 200 may signal values for syntax elements in the bitstream. In general, signaling refers to producing a value in a bitstream. As noted above, source device 102 provides a bitstream in non-real-time or substantially real-time, as may occur when storing syntax elements in storage device 112 for later retrieval by destination device 116. to the destination device 116.

2A and 2B are conceptual diagrams illustrating an example quadtree binary tree (QTBT) structure 130 , and a corresponding coding tree unit (CTU) 132 . Solid lines indicate quadtree splitting, and dotted lines indicate binary tree splitting. At each split (i.e. non-leaf) node of the binary tree, a flag is signaled to indicate which split type (i.e. horizontal or vertical) is being used, in this example 0 indicates horizontal split. and 1 indicates vertical division. For quadtree splitting, there is no need to indicate the splitting type because quadtree nodes split a block horizontally and vertically into four subblocks with the same size. Accordingly, syntax elements (such as splitting information) for the domain tree level (ie, solid lines) of the QTBT structure 130 and (splitting information) for the prediction tree level (ie, dotted lines) of the QTBT structure 130 ) syntax elements, which video encoder 200 may encode and video decoder 300 may decode. Video encoder 200 may encode and video decoder 300 may decode video data, such as prediction and transform data, for the CUs represented by the terminal leaf nodes of QTBT structure 130 .

In general, CTU 132 of FIG. 2B may be associated with parameters defining the sizes of blocks corresponding to nodes of QTBT structure 130 at first and second levels. These parameters are CTU size (representing the size of CTU 132 in samples), minimum quadtree size (MinQTSize, representing minimum allowed quadtree leaf node size), maximum binary tree size (MaxBTSize, maximum allowed binary tree size). root node size), maximum binary tree depth (MaxBTDepth, indicating maximum allowed binary tree depth), and minimum binary tree size (MinBTSize, indicating minimum allowed binary tree leaf node size).

A root node of the QTBT structure corresponding to a CTU may have four child nodes at the first level of the QTBT structure, each of which may be partitioned according to quadtree partitioning. That is, nodes at the first level are either leaf nodes (with no child nodes) or have 4 child nodes. The example of QTBT structure 130 shows such nodes as including a parent node and child nodes with solid lines for branches. If the nodes of the first level are not larger than the maximum allowed binary tree root node size (MaxBTSize), the nodes may be further partitioned by individual binary trees. Binary tree splitting of one node can be repeated until the nodes resulting from the split reach either the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). The example of QTBT structure 130 shows such nodes as having dotted lines for branches. A binary tree leaf node is referred to as a coding unit (CU) used for prediction (eg, intra-picture or inter-picture prediction) and transformation, without any further partitioning. As discussed above, CUs may also be referred to as “video blocks” or “blocks”.

In one example of the QTBT partitioning structure, the CTU size is set as 128x128 (luma samples and two corresponding 64x64 chroma samples), MinQTSize is set as 16x16, MaxBTSize is set as 64x64, (both width and height for all) MinBTSize is set as 4, and MaxBTDepth is set as 4. Quadtree partitioning is first applied to the CTU to create quad-tree leaf nodes. Quadtree leaf nodes may have a size from 16x16 (ie, MinQTSize) to 128x128 (ie, CTU size). If the quadtree leaf node is 128x128, then the leaf quadtree node will not be further split by the binary tree because the size exceeds MaxBTSize (ie 64x64 in this example). Otherwise, the quadtree leaf nodes will be further partitioned by the binary tree. Thus, a quadtree leaf node is also the root node for a binary tree, and has a binary tree depth of zero. When the binary tree depth reaches MaxBTDepth (4 in this example) no further splits are allowed. A binary tree node with a width equal to MinBTSize (in this example, 4) implies that no further vertical splits (ie, splits of width) are allowed for that binary tree node. Similarly, a binary tree node with a height equal to MinBTSize implies that no further horizontal splits (ie, splits of height) are allowed for that binary tree node. As mentioned above, the leaf nodes of the binary tree are referred to as CUs and are further processed according to prediction and transformation without further partitioning.

3 is a block diagram illustrating an example video encoder 200 that may perform the techniques of this disclosure. 3 is provided for purposes of explanation and should not be considered limiting of the techniques as generally exemplified and described in this disclosure. For purposes of explanation, this disclosure describes a video encoder 200 in accordance with the techniques of VVC (ITU-T H.266, under development), and HEVC (ITU-T H.265). However, the techniques of this disclosure may be performed by video encoding devices configured for other video coding standards.

In the example of FIG. 3 , video encoder 200 includes video data memory 230 , mode select unit 202 , residual generation unit 204 , transform processing unit 206 , quantization unit 208 , inverse quantization unit 210 , inverse transform processing unit 212 , reconstruction unit 214 , filter unit 216 , decoded picture buffer (DPB) 218 , and entropy encoding unit 220 . Video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction Any or all of unit 214 , filter unit 216 , DPB 218 , and entropy encoding unit 220 may be implemented in one or more processors or in processing circuitry. For example, the units of video encoder 200 may be implemented as one or more circuits or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Moreover, video encoder 200 may include additional or alternative processors or processing circuitry to perform these and other functions.

Video data memory 230 may store video data to be encoded by the components of video encoder 200 . Video encoder 200 may receive video data stored in video data memory 230 , eg, from video source 104 ( FIG. 1 ). DPB 218 may act as a reference picture memory that stores reference video data for use in prediction of subsequent video data by video encoder 200 . Video data memory 230 and DPB 218 may be memory devices such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. It may be formed by any of the memory devices. Video data memory 230 and DPB 218 may be provided by the same memory device or separate memory devices. In various examples, video data memory 230 may be on-chip with other components of video encoder 200 as illustrated, or off-chip with respect to those components.

In this disclosure, references to video data memory 230 are limited to memory internal to video encoder 200 unless specifically described as such, or memory external to video encoder 200 unless specifically described as such. should not be construed as being Rather, reference to video data memory 230 should be understood as a reference memory that stores video data that video encoder 200 receives for encoding (eg, video data for a current block to be encoded). Memory 106 of FIG. 1 may also provide temporary storage of outputs from various units of video encoder 200 .

The various units of FIG. 3 are illustrated to aid understanding of the operations performed by video encoder 200 . The units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Fixed function circuits refer to circuits that provide specific functionality and are preset for operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, programmable circuits may execute software or firmware that causes the programmable circuits to operate in a manner defined by instructions in the software or firmware. Fixed function circuits may execute software instructions (eg, to receive parameters or output parameters), but the types of operations they perform are generally immutable. In some examples, one or more of the units may be discrete circuit blocks (fixed function or programmable), and in some examples one or more of the units may be integrated circuits.

Video encoder 200 may include arithmetic logic units (ALUs), basic function units (EFUs), digital circuits, analog circuits, and/or programmable cores formed from programmable circuits. . In examples where the operations of video encoder 200 are performed using software executed by programmable circuits, memory 106 (FIG. 1) contains instructions of software that video encoder 200 receives and executes ( object code), or other memory (not shown) within video encoder 200 may store such instructions.

Video data memory 230 is configured to store received video data. Video encoder 200 may retrieve a picture of video data from video data memory 230 and provide the video data to residual generation unit 204 and mode select unit 202 . Video data in video data memory 230 may be raw video data to be encoded.

Mode select unit 202 includes motion estimation unit 222 , motion compensation unit 224 , and intra-prediction unit 226 . Mode select unit 202 may include additional functional units to perform video prediction according to other prediction modes. As examples, mode select unit 202 may include a palette unit, an intra-block copy unit (which may be part of motion estimation unit 222 and/or motion compensation unit 224), an affine unit, a linear model (LM) units and the like.

Mode select unit 202 typically coordinates multiple encoding passes to test combinations of encoding parameters and resulting rate-distortion values for those combinations. Encoding parameters may include partitioning of CTUs into CUs, prediction modes for CUs, transform types for residual data of CUs, quantization parameters for residual data of CUs, and the like. Mode select unit 202 may ultimately select a combination of encoding parameters with better rate-distortion values than other tested combinations.

Video encoder 200 may partition a picture retrieved from video data memory 230 into a series of CTUs and encapsulate one or more CTUs within a slice. Mode select unit 202 may partition the CTUs of a picture according to a tree structure, such as the quadtree structure or QTBT structure of HEVC described above. As described above, video encoder 200 may form one or more CUs from partitioning a CTU according to a tree structure. Such a CU may also be generically referred to as a “video block” or “block”.

In general, mode select unit 202 also controls components thereof (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to obtain a current block (e.g., current CU , or, in HEVC, an overlapping portion of a PU and a TU). For inter-prediction of a current block, motion estimation unit 222 identifies one or more closely matching reference blocks in one or more reference pictures (eg, one or more previously coded pictures stored in DPB 218). To do so, motion search may be performed. In particular, motion estimation unit 222 determines whether a potential reference block is currently based on, e.g., sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared differences (MSD), and the like. You can also calculate a value indicating how similar it is to a block. Motion estimation unit 222 may generally perform these calculations using sample-by-sample differences between the considered reference block and the current block. Motion estimation unit 222 may identify the reference block with the lowest value resulting from these calculations that indicates the reference block that most closely matches the current block.

Motion estimation unit 222 may form one or more motion vectors (MVs) that define positions of reference blocks in reference pictures relative to the position of the current block in the current picture. Motion estimation unit 222 may then provide the motion vectors to motion compensation unit 224 . For example, for unidirectional inter-prediction, motion estimation unit 222 may provide a single motion vector, whereas for bidirectional inter-prediction, motion estimation unit 222 may provide two motion vectors. there is. Motion compensation unit 224 may then use the motion vectors to generate a predictive block. For example, motion compensation unit 224 may use the motion vector to retrieve the data of the reference block. As another example, if the motion vector has fractional-sample precision, motion compensation unit 224 may interpolate values for the predictive block according to one or more interpolation filters. Moreover, for bi-directional inter-prediction, motion compensation unit 224 retrieves the data for two reference blocks identified by the respective motion vectors, and retrieves the data, e.g., through sample-by-sample averaging or weighted averaging. can also be combined.

As another example, for intra-prediction or intra-predictive coding, intra-prediction unit 226 may generate a predictive block from samples neighboring the current block. For example, for directional modes, intra-prediction unit 226 generally mathematically combines the values of neighboring samples and populates these computed values in a defined direction over the current block to obtain a predictive block. can also create As another example, for the DC mode, intra-prediction unit 226 may calculate an average of neighboring samples for the current block, and generate a predictive block to include this resulting average for each sample of the predictive block. there is.

Mode select unit 202 provides the predictive block to residual generation unit 204. Residual generation unit 204 receives a raw, unencoded version of the current block from video data memory 230 and a predictive block from mode select unit 202 . Residual generation unit 204 calculates sample-by-sample differences between the current block and the predictive block. The resulting sample-by-sample differences define the residual block for the current block. In some examples, residual generation unit 204 may also determine differences between sample values in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

In examples in which mode select unit 202 partitions CUs into PUs, each PU may be associated with a luma prediction unit and corresponding chroma prediction units. Video encoder 200 and video decoder 300 may support PUs having various sizes. As indicated above, the size of a CU may refer to the size of the luma coding block of the CU and the size of a PU may refer to the size of the luma prediction unit of the PU. Assuming that the size of a particular CU is 2Nx2N, video encoder 200 supports PU sizes of 2Nx2N or NxN for intra-prediction, and symmetric PU sizes of 2Nx2N, 2NxN, Nx2N, NxN, etc. for inter-prediction sizes may be supported. Video encoder 200 and video decoder 300 may also support asymmetric partitioning on PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter prediction.

In examples in which mode select unit 202 does not further partition a CU into PUs, each CU may be associated with a luma coding block and corresponding chroma coding blocks. As above, the size of a CU may refer to the size of a luma coding block of the CU. Video encoder 200 and video decoder 300 may support CU sizes of 2Nx2N, 2NxN, or Nx2N.

For other video coding techniques, such as intra-block copy mode coding, affine mode coding and linear model (LM) mode coding, as some examples, mode select unit 202, via individual units associated with the coding techniques, Generates a prediction block for the current block being encoded. For some examples, such as palette mode coding, mode select unit 202 may not generate a predictive block, but instead generate syntax elements indicating a way to reconstruct the block based on the selected palette. In such modes, mode select unit 202 may provide these syntax elements to entropy encoding unit 220 to be encoded.

As described above, residual generation unit 204 receives video data for a current block and a corresponding predictive block. Residual generation unit 204 then generates a residual block for the current block. To generate a residual block, residual generation unit 204 calculates sample-by-sample differences between the current block and the predictive block.

Transform processing unit 206 applies one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a “transform coefficient block”). Transform processing unit 206 may apply various transforms to the residual block to form a transform coefficient block. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, transform processing unit 206 may perform multiple transforms on the residual block, eg, a primary transform and a secondary transform, eg, a rotation transform. In some examples, transform processing unit 206 does not apply transforms to the residual block.

Quantization unit 208 may quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 may quantize the transform coefficients of a transform coefficient block according to a quantization parameter (QP) value associated with the current block. Video encoder 200 may adjust the degree of quantization applied to transform coefficient blocks associated with the current block by adjusting the QP value associated with the CU (eg, via mode select unit 202 ). Quantization may introduce loss of information, and thus, quantized transform coefficients may have lower precision than the original transform coefficients generated by transform processing unit 206.

Inverse quantization unit 210 and inverse transform processing unit 212 may apply inverse quantization and inverse transforms to the quantized transform coefficient block, respectively, to reconstruct a residual block from the transform coefficient block. Reconstruction unit 214 will generate a reconstructed block corresponding to the current block (potentially with some degree of distortion) based on the predicted block produced by mode select unit 202 and the reconstructed residual block. may be For example, reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples from the predictive block produced by mode select unit 202 to produce a reconstructed block.

Filter unit 216 may perform one or more filter operations on the reconstructed blocks. For example, filter unit 216 may perform deblocking operations to reduce blocking artifacts along the edges of CUs. In some examples, operations of filter unit 216 may be skipped.

Video encoder 200 stores the reconstructed blocks in DPB 218 . For example, in examples in which the operations of filter unit 216 are not performed, reconstruction unit 214 may store the reconstructed blocks to DPB 218 . In examples in which the operations of filter unit 216 are performed, filter unit 216 may store the filtered reconstructed blocks to DPB 218 . Motion estimation unit 222 and motion compensation unit 224 retrieve reference pictures from DPB 218 formed from the reconstructed (and potentially filtered) blocks to inter-block blocks of subsequently encoded pictures. can also be predicted. Additionally, intra-prediction unit 226 may use reconstructed blocks within DPB 218 of the current picture to intra-predict other blocks in the current picture.

In general, entropy encoding unit 220 may entropy encode syntax elements received from other functional components of video encoder 200 . For example, entropy encoding unit 220 may entropy encode the quantized transform coefficient blocks from quantization unit 208 . As another example, entropy encoding unit 220 may entropy encode prediction syntax elements from mode select unit 202 (eg, motion information for inter-prediction or intra-mode information for intra-prediction). . Entropy encoding unit 220 may perform one or more entropy encoding operations on syntax elements, another example of video data, to generate entropy-encoded data. For example, entropy encoding unit 220 may perform a context-adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable-to-variable (V2V) length coding operation, a syntax-based context-adaptive binary arithmetic coding (SBAC) operation, A probability interval partitioning entropy (PIPE) coding operation, an exponential-Golomb encoding operation, or another type of entropy encoding operation may be performed on the data. In some examples, entropy encoding unit 220 may operate in a bypass mode in which syntax elements are not entropy encoded.

Video encoder 200 may output a bitstream that includes entropy encoded syntax elements necessary to reconstruct blocks of a picture or slice. In particular, entropy encoding unit 220 may output a bitstream.

The operations described above are described in terms of blocks. Such description should be understood as operations on luma coding block and/or chroma coding blocks. As described above, in some examples, the luma coding block and chroma coding blocks are luma and chroma components of a CU. In some examples, the luma coding block and chroma coding blocks are luma and chroma components of a PU.

In some examples, operations performed on a luma coding block need not be repeated for chroma coding blocks. As an example, operations to identify motion vectors (MVs) and reference pictures for luma coding blocks need not be repeated to identify MVs and reference pictures for chroma blocks. Rather, the MV for the luma coding block may be scaled to determine the MV for chroma blocks, and the reference picture may be the same. As another example, the intra-prediction process may be the same for luma coding block and chroma coding blocks.

Video encoder 200 represents an example of a device configured to encode video data, the device including a memory configured to store video data, and one or more processing units implemented in circuitry, the one or more processing units comprising input Dividing a block into a plurality of subblocks, wherein a size of the input block is less than or equal to a size of a coding unit, and bidirectional optical flow (BDOF) based on a condition being satisfied Determining that it will be applied to a subblock among a plurality of subblocks, dividing the subblock into a plurality of sub-subblocks, and determining a refined motion vector for one or more of the sub-subblocks, - determine the refined motion vector for a sub-subblock of the sub-blocks, which is the same for a plurality of samples in the sub-subblock, and the refinement for one or more sub-subblocks It is configured to perform BDOF on the subblock based on the resulting motion vector.

As another example, one or more processing units implemented in circuitry divides an input block into a plurality of subblocks, wherein a size of the input block is less than or equal to a size of a coding unit, and divides the input block into a plurality of subblocks; , determine that bidirectional optical flow (BDOF) is to be applied to a subblock of the plurality of subblocks based on a condition being satisfied, divide the subblock into a plurality of sub-subblocks, and one or more samples in the subblock determine a refined motion vector for each of the s, and perform BDOF for the subblock based on the refined motion vector for each of one or more samples in the subblock.

As another example, processing circuitry of video encoder 200 determines that bi-directional optical flow (BDOF) is enabled for a block of video data, and based on the determination that BDOF is enabled for the block, determines the block as a plurality of divide into subblocks, for each subblock of one or more subblocks of the plurality of subblocks, determine respective distortion values, and determine respective distortion values of each of the one or more subblocks of the plurality of subblocks based on the respective distortion values. Determine that one of the BDOFs is performed or the BDOF is bypassed per pixel for a subblock of , and predict for each subblock of the one or more subblocks based on the determination that the BDOF is performed or the BDOF is bypassed per pixel. It may be configured to determine samples, determine residual values representing a difference between the predicted samples and the block, and signal information representing the residual values.

4 is a block diagram illustrating an example video decoder 300 that may perform the techniques of this disclosure. 4 is provided for purposes of explanation and is not limiting on the techniques as broadly exemplified and described in this disclosure. For purposes of explanation, this disclosure describes a video decoder 300 in accordance with the techniques of VVC (ITU-T H.266, under development), and HEVC (ITU-T H.265). However, the techniques of this disclosure may be performed by video coding devices configured for other video coding standards.

In the example of FIG. 4 , video decoder 300 includes coded picture buffer (CPB) memory 320 , entropy decoding unit 302 , prediction processing unit 304 , inverse quantization unit 306 , inverse transform processing unit 308 , reconstruction unit 310 , filter unit 312 , and decoded picture buffer (DPB) 314 . CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB ( Any or all of 314) may be implemented in one or more processors or in processing circuitry. For example, the units of video decoder 300 may be implemented as one or more circuits or logic elements as part of hardware circuitry, or as part of a processor, ASIC, or FPGA. Moreover, video decoder 300 may include additional or alternative processors or processing circuitry to perform these and other functions.

Prediction processing unit 304 includes motion compensation unit 316 and intra-prediction unit 318 . Prediction processing unit 304 may include additional units to perform prediction according to other prediction modes. As examples, prediction processing unit 304 may include a palette unit, an intra-block copy unit (which may form part of motion compensation unit 316 ), an affine unit, a linear model (LM) unit, etc. . In other examples, video decoder 300 may include more, fewer, or different functional components.

CPB memory 320 may store video data, such as an encoded video bitstream, to be decoded by the components of video decoder 300 . Video data stored in CPB memory 320 may be obtained, for example, from computer readable medium 110 (FIG. 1). CPB memory 320 may include a CPB that stores encoded video data (eg, syntax elements) from an encoded video bitstream. CPB memory 320 may also store video data other than syntax elements of a coded picture, such as temporary data representing outputs from various units of video decoder 300 . DPB 314 generally stores decoded pictures that video decoder 300 may output and/or use as reference video data when decoding pictures or subsequent data of an encoded video bitstream. CPB memory 320 and DPB 314 may be formed by any of a variety of memory devices, such as DRAM including SDRAM, MRAM, RRAM, or other types of memory devices. CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, CPB memory 320 may be on-chip with other components of video decoder 300 or off-chip with respect to those components.

Additionally or alternatively, in some examples, video decoder 300 may retrieve coded video data from memory 120 (FIG. 1). That is, memory 120 may store data as discussed above with CPB memory 320 . Likewise, memory 120 may store instructions to be executed by video decoder 300 when some or all of the functionality of video decoder 300 is implemented in software to be executed by processing circuitry of video decoder 300 . there is.

Various units shown in FIG. 4 are illustrated to aid understanding of the operations performed by video decoder 300 . The units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Similar to FIG. 3, fixed function circuits refer to circuits that provide specific functionality and are preset for operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, programmable circuits may execute software or firmware that causes the programmable circuits to operate in a manner defined by instructions in the software or firmware. Fixed function circuits may execute software instructions (eg, to receive parameters or output parameters), but the types of operations they perform are generally immutable. In some examples, one or more of the units may be discrete circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be integrated circuits.

Video decoder 300 may include ALUs, EFUs, digital circuits, analog circuits, and/or programmable cores formed from programmable circuits. In examples where the operations of video decoder 300 are performed by software executing on programmable circuits, on-chip or off-chip memory may be used to store instructions of software that video decoder 300 receives and executes (e.g., , object code) may be stored.

Entropy decoding unit 302 may receive encoded video data from the CPB and entropy decode the video data to reproduce syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 decoded video data based on syntax elements extracted from the bitstream. can also create

In general, video decoder 300 reconstructs a picture on a block-by-block basis. Video decoder 300 may perform a reconstruction operation on each block individually (where the block currently being reconstructed, ie, being decoded, may be referred to as a “current block”).

Entropy decoding unit 302 may entropy decode syntax elements that define quantized transform coefficients of a quantized transform coefficient block, as well as transform information, such as a quantization parameter (QP) and/or transform mode indication(s). . Inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine a degree of quantization and, similarly, a degree of inverse quantization for inverse quantization unit 306 to apply. Inverse quantization unit 306 may, for example, perform a bitwise left-shift operation to inverse quantize the quantized transform coefficients. Thereby, inverse quantization unit 306 may form a transform coefficient block that includes transform coefficients.

After inverse quantization unit 306 forms a transform coefficient block, inverse transform processing unit 308 may apply one or more inverse transforms to the transform coefficient block to produce a residual block associated with the current block. For example, inverse transform processing unit 308 may apply an inverse DCT, inverse integer transform, inverse Karhunen-Loeve transform (KLT), inverse rotation transform, inverse directional transform, or other inverse transform to the transform coefficient block.

Moreover, prediction processing unit 304 generates a predictive block according to the predictive information syntax elements entropy decoded by entropy decoding unit 302 . For example, if the prediction information syntax elements indicate that the current block is inter-predicted, motion compensation unit 316 may generate the predictive block. In this case, the predictive information syntax elements may indicate a reference picture in DPB 314 from which to derive the reference block, as well as a motion vector that identifies the position of the reference block in the reference picture relative to the position of the current block in the current picture. there is. Motion compensation unit 316 may generally perform the inter-prediction process in a manner substantially similar to that described with respect to motion compensation unit 224 (FIG. 3).

As another example, if the prediction information syntax elements indicate that the current block is intra-predicted, intra-prediction unit 318 may generate the predictive block according to the intra-prediction mode indicated by the prediction information syntax elements. Again, intra-prediction unit 318 may generally perform the intra-prediction process in a manner substantially similar to that described with respect to intra-prediction unit 226 (FIG. 3). Intra-prediction unit 318 may retrieve data of neighboring samples for the current block from DPB 314 .

Reconstruction unit 310 may reconstruct a current block using the predictive block and the residual block. For example, reconstruction unit 310 may add samples of the residual block to corresponding samples of the predictive block to reconstruct the current block.

Filter unit 312 may perform one or more filter operations on the reconstructed blocks. For example, filter unit 312 may perform deblocking operations to reduce blockiness artifacts along the edges of reconstructed blocks. Operations of filter unit 312 are not necessarily performed in all examples.

Video decoder 300 may store the reconstructed blocks in DPB 314 . For example, in examples where the operations of filter unit 312 are not performed, reconstruction unit 310 may store the reconstructed blocks to DPB 314 . In examples in which the operations of filter unit 312 are performed, filter unit 312 may store the filtered reconstructed blocks to DPB 314 . As discussed above, DPB 314 may provide reference information to prediction processing unit 304, such as previously decoded pictures for subsequent motion compensation and samples of the current picture for intra-prediction. . Moreover, video decoder 300 may output decoded pictures (eg, decoded video) from DPB 314 for subsequent presentation on a display device, such as display device 118 of FIG. 1 .

In this way, the video decoder 300 represents an example of a video decoding device, the video decoding device including a memory configured to store video data, and one or more processing units implemented in circuitry, the one or more processing units comprising: Dividing the input block into a plurality of subblocks, wherein the size of the input block is less than or equal to the size of a coding unit, bidirectional optical flow (BDOF) based on a condition being satisfied Determining that is to be applied to a subblock of the plurality of subblocks, dividing the subblock into a plurality of sub-subblocks, and determining a refined motion vector for one or more of the sub-subblocks, wherein one or more Determine the refined motion vector for a sub-subblock of the sub-subblocks, which is the same for a plurality of samples in the sub-subblock, and for one or more sub-subblocks. It is configured to perform BDOF on a subblock based on the refined motion vector.

As another example, one or more processing units implemented in circuitry divides an input block into a plurality of subblocks, wherein a size of the input block is less than or equal to a size of a coding unit, and divides the input block into a plurality of subblocks; , determine that bidirectional optical flow (BDOF) is to be applied to a subblock of the plurality of subblocks based on a condition being satisfied, divide the subblock into a plurality of sub-subblocks, and one or more samples in the subblock determine a refined motion vector for each of the s, and perform BDOF for the subblock based on the refined motion vector for each of one or more samples in the subblock.

As another example, processing circuitry (e.g., motion compensation unit 316) of video decoder 300 determines that bi-directional optical flow (BDOF) is enabled for a block of video data, and BDOF is enabled for the block. divides the block into a plurality of sub-blocks based on the determination that it is, determines individual distortion values for each sub-block of one or more sub-blocks among the plurality of sub-blocks, and determines a plurality of sub-blocks based on the individual distortion values. For each subblock of one or more subblocks of the blocks, determine that one of the per-pixel BDOFs are performed or the BDOF is bypassed, and based on the determination that the per-pixel BDOFs are performed or the BDOFs are bypassed, one or more subblocks of the blocks are performed. It may be configured to determine predictive samples for each subblock of the blocks, and to reconstruct the block based on the predictive samples. For example, processing circuitry may receive residual values representing differences between the predicted samples and samples of the block, and add the residual values to the predicted samples to reconstruct the block.

The following describes the CU structure and motion vector prediction in HEVC. The following may provide additional context to the above description of CU and motion vector prediction, and may include some repetitions of the above description to aid understanding.

In HEVC, the largest coding unit in a slice is referred to as a coding tree block (CTB) or coding tree unit (CTU). CTB contains a quad-tree, and its nodes are coding units. The size of the CTB can range from 16x16 to 64x64 (although technically 8x8 CTB sizes can be supported) in the HEVC main profile. A coding unit (CU) can be the same size of a CTB to be as small as 8x8. Each coding unit is coded in one mode, ie inter or intra. When a CU is inter-coded, the CU may be further partitioned into 2 or 4 prediction units (PUs), or may be just one PU if no further partition is applied. When two PUs exist in one CU, the PUs can be half-size rectangles or two rectangles that are ¼ or ¾ the size of the CU. When CUs are inter-coded, each PU has one set of motion information derived with a unique inter-prediction mode.

Next, motion vector prediction is explained. In the HEVC standard, there are two inter-prediction modes: merge (skip is considered as a special case of merge) and advanced motion vector prediction (AMVP) modes, respectively for a prediction unit (PU).

For either AMVP mode or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. Motion vector(s) as well as reference indices in the merge mode of the current PU are generated by taking one candidate from the MV candidate list.

The MV candidate list includes up to 5 candidates for merge mode and only 2 candidates for AMVP mode. A merge candidate may contain a set of motion information, eg, motion vectors corresponding to both reference picture lists (List 0 and List 1) and reference indices. Once a merge candidate is identified by a merge index, reference pictures used for prediction of current blocks as well as associated motion vectors are determined. On the other hand, under the AMVP mode for each potential prediction direction from either List 0 or List 1, since the AMVP candidate contains only motion vectors, along with the MV predictor (MVP) index for the MV candidate list, the reference The index must be explicitly signaled. For AMVP mode, the predicted motion vectors can be further refined. Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks.

The spatial neighbor candidates are described next. For example, FIGS. 5A and 5B are conceptual diagrams illustrating examples of spatial neighboring motion vector candidates for merge mode and advanced motion vector predictor (AMVP) mode, respectively.

Although spatial MV candidates are derived from neighboring blocks shown in FIGS. 5A and 5B for a particular PU (PU ₀ ) 500 , the methods of generating candidates from blocks are different for merge and AMVP modes. In the merge mode, up to four spatial MV candidates can be derived in the order shown in Fig. 5A with numerical values, and the order is as follows: i.e., as shown in Fig. 5A, left (0, A1), top (1, B1), top right (2, B0), bottom left (3, A0), and top left (4, B2).

In AVMP mode, the neighboring blocks are divided into two groups: the left group consisting of blocks 0 and 1, and the upper group consisting of blocks 2, 3, and 4, as shown in PU0 502 of FIG. 5B. divided into groups For each group, potential candidates in neighboring blocks that refer to the same reference picture as indicated by the signaled reference index have the highest priority to be selected to form the group's final candidates. It is possible that all neighboring blocks do not contain a motion vector pointing to the same reference picture. Thus, if such a candidate cannot be found, the first available candidate may be scaled to form a final candidate, thus temporal distance differences may be compensated for.

The following describes temporal motion vector prediction in HEVC. A temporal motion vector predictor (TMVP) candidate, if enabled and available, is added to the MV candidate list after the spatial motion vector candidates. The process of motion vector derivation for a TMVP candidate is the same for both merge and AMVP modes, but the target reference index for a TMVP candidate in merge mode is always set to 0.

The primary block location for TMVP candidate derivation is block “T” illustrated as block 602, as shown in FIG. It is the lower right block outside the collocated PU. However, if the block is located outside the current CTB row or motion information is not available, then the block is replaced with the central block of the PU, illustrated as block 604 .

The motion vector for the TMVP candidate is derived from the collocated PU of the collocated picture, indicated at the slice level. A motion vector for a collocated PU is referred to as a collocated MV. Similar to the temporal direct mode in AVC, to derive the TMVP candidate motion vector, the collocated MV must be scaled to compensate for temporal distance differences, as shown in FIG. 6B.

The following describes additional aspects of motion prediction in HEVC. Several aspects of merge and AMVP modes are worth mentioning as follows. Motion vector scaling: It is assumed that the value of motion vectors is proportional to the distance of pictures at their presentation time. A motion vector associates two pictures: a reference picture and the picture containing the motion vector (ie, the containing picture). When a motion vector is used to predict another motion vector, the distance between the containing picture and the reference picture is calculated based on picture order count (POC) values.

For a motion vector to be predicted, both its associated containing picture and reference picture may be different. Thus, a new distance is calculated (based on the POC). And, the motion vector is scaled based on these two POC distances. For a spatial neighbor candidate, the containing pictures for the two motion vectors are the same, while the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporal neighbor candidates.

Artificial motion vector candidate generation: If the motion vector candidate list is not complete, artificial motion vector candidates are generated and inserted at the end of the list until it has all candidates.

For merge mode, there are two types of artificial MV candidates: a combined candidate derived only for B-slices, and zero candidates used only for AMVP if the first type does not provide enough artificial candidates. do. For each pair of candidates that are already in the candidate list and have the necessary motion information, the bi-directionally combined motion vector candidates are the first candidate's motion vector referencing a picture in list 0 and the second candidate referencing a picture in list 1. It is derived by combining candidate motion vectors.

Pruning Process for Candidate Insertion: Candidates from different blocks may occur to be identical, which reduces the efficiency of the merge/AMVP candidate list. A pruning process is applied to solve these problems. This pruning process compares one candidate to other candidates in the current candidate list to avoid inserting the same candidate in a particular range. To reduce complexity, instead of comparing each potential candidate to all other existing candidates, only a limited number of pruning processes are applied.

The following describes template matching prediction. Template matching (TM) prediction is a special merging mode based on frame rate up conversion (FRUC) techniques. In this mode, the block's motion information is not signaled but derived at the decoder side (eg, by video decoder 300). TM prediction applies to both AMVP mode and regular merge mode. In the AMVP mode, MVP candidate selection is determined based on template matching to pick up the one that reaches the minimum difference between the current block template and the reference block template. For regular merge mode, the TM mode flag is signaled to indicate the use of TM, then the TM is applied to the merge candidate indicated by the merge index for MV refinement.

As shown in FIG. 7 , template matching is the closest match between a template in the current frame 700 (the top and/or left neighboring blocks of the current CU) and a block in the reference frame 702 (same size as the template). It is used to derive the motion information of the current CU by finding a close match. With the AMVP candidate selected based on the initial matching error, the MVP of the AMVP candidate is refined by template matching. For a merge candidate indicated by the signaled merge index, the merged MVs of the merge candidates corresponding to L0 and L1 are independently refined by template matching, and then the less accurate are further refined back to the better as before .

For the cost function, motion compensated interpolation may be utilized if the motion vector points to fractional sample positions. To reduce complexity, bi-linear interpolation instead of the regular 8-tap DCT-IF interpolation is used for both template matching to generate templates for reference pictures. The matching cost (C) of template matching is calculated as follows:

In the above equation, w is a weighting factor empirically set to 4, and MV and MV ^s are the currently testing MV and the initial MV (i.e. MVP candidate in AMVP mode or merged motion in merge mode), respectively. indicate The sum of absolute differences (SAD) is used as the matching cost of template matching.

When TM is used, motion is refined by using only luma samples. The derived motion may be used for both luma and chroma for MC (motion compensated) inter prediction. After the MV is determined, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

Regarding the search method , MV refinement is a pattern-based MV search with a criterion of template matching cost. Two search patterns are supported: diamond search and cross search for MV refinement. MV is directly searched with 1/4 luma sample MVD accuracy into diamond pattern, followed by 1/4 luma sample MVD accuracy into cross pattern, which is then followed by 1/8 luma sample MVD refinement into cross pattern do. The search range of MV refinement is set equal to (-8, +8) luma samples around the initial MV.

The following describes bilateral matching prediction. Bilateral matching (also referred to as bilateral merging) (BM) prediction is another merging mode based on frame rate up conversion (FRUC) techniques. When a block is determined to apply the BM mode, two starting motion vectors (MV0 and MV1) are derived by using the signaled merge candidate index to select a merge candidate from the constructed merge list. A bilateral matching search may be around MV0 and MV1. The final MV0' and MV1' are derived based on the minimum bilateral matching cost.

The motion vector difference MVD0 800 (MV0' - denoted by MV0) and MVD1 802 (MV1' - denoted by MV1) pointing to two reference blocks is the difference between the current picture and the two reference pictures. It may be proportional to temporal distances (TD), eg, TD0 and TD1. 8 illustrates an example of MVD0 and MVD1, where TD1 is four times TD0.

However, there is an optional design in which MVD0 and MVD1 are mirrored regardless of temporal distances (TD0 and TD1). 9 illustrates an example of mirrored MVD0 900 and MVD1 902, where TD1 is four times TD0.

Bilateral matching performs a local search around the initial MV0 and MV1 to derive the final MV0' and MV1'. The local search applies a 3x3 square search pattern to loop over the search range [-8, 8]. At each search iteration, the bilateral matching cost of the 8 surrounding MVs in the search pattern is calculated and compared to the bilateral matching cost of the central MV. The MV with the minimum bilateral matching cost becomes the new central MV in the next search iteration. A local search ends when the current center MV has a minimum cost within a 3x3 square search pattern or when the local search reaches a predefined maximum search iteration. 10 illustrates an example of a 3x3 square search pattern 1000 in the search range [−8, 8].

The following describes decoder-side motion vector refinement. To increase the accuracy of MVs in merge mode, decoder-side motion vector refinement (DMVR) is applied in VVC. In the bidirectional prediction operation, a refined MV is searched around initial MVs in the reference picture list L0 and the reference picture list L1. The DMVR method calculates the distortion between two candidate blocks in reference picture list L0 and list L1. As illustrated in FIG. 11 , SAD between blocks 1102 and 1100 based on each MV candidate around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate a bi-directional prediction signal.

The refined MV derived by the DMVR process is used to generate inter prediction samples and is also used in temporal motion vector prediction for future picture coding. The original MV is used in the deblocking process and also used in spatial motion vector prediction for future CU coding.

DMVR is a subblock-based merge mode with a predefined maximum processing unit of 16x16 luma samples. If the width and/or height of the CU is greater than 16 luma samples, the CU may be further divided into subblocks having a width and/or height equal to 16 luma samples.

The search method is described below. In DVMR, the search points surrounding the initial MV and MV offset may follow the MV difference mirroring rule. For example, any points checked by the DMVR, denoted by a candidate MV pair (MV0, MV1), may conform to the following two equations:

In the above equation, MV_offset represents a refinement offset between an initial MV and a refined MV in one of the reference pictures. The refinement search range is two integer luma samples from the initial MV. The search includes an integer-sample offset search stage and a fractional-sample refinement stage.

A 25-point full search is applied to the integer-sample offset search. The SAD of the initial MV pair is calculated first. If the SAD of the initial MV pair is less than the threshold, the integer-sample stage of the DMVR ends. Otherwise, the SADs of the remaining 24 points are calculated and checked in raster scanning order. The point with the smallest SAD is selected as the output of the integer-sample offset search stage. To reduce the penalty of uncertainty in DMVR refinement, the original MV may be preferred during the DMVR process. SAD between reference blocks referred to by initial MV candidates is reduced by 1/4 of the SAD value.

Integer sample search is followed by fractional sample refinement. To save computational complexity, the fractional sample refinement is derived by using the parametric error surface equation instead of an additional search into the SAD comparison. Fractional sample refinement is invoked conditionally based on the output of the integer sample search stage. If the integer-sample search stage ends with the centroid with the smallest SAD in either the first iteration or the second iteration search, fractional-sample refinement is additionally applied.

For parametric error surface based sub-pixel offsets estimation, the center position cost and the costs at 4 neighboring positions from the center are used to fit a 2D parabolic error surface equation of the form:

In the above equation, (x _min , y _min ) corresponds to the fractional position with the smallest cost, and C corresponds to the minimum cost value. By solving the above equations by using the cost value of the 5 search points, (x _min , y _min ) is computed as:

The values of x _min and y _min are automatically constrained between -8 and 8 since all cost values are positive and the smallest value is E(0,0) . This corresponds to a half pel offset with 1/16-pel MV accuracy in VVC. The computed fraction (x _min , y _min ) is added to the integer distance refinement MV to obtain a sub-pixel accurate refinement delta MV.

The following describes bilinear interpolation and sample padding. For VVC, the resolution of MVs is 1/16 luma samples. Samples at fractional positions are interpolated using an 8-tap interpolation filter. For DMVR, search points enclose an initial fractional-pel MV with an integer-sample offset, and thus samples at these fractional positions may be interpolated for the DMVR search process. To reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in the DMVR. In some examples, by using a bilinear filter with a 2-sample search range, DVMR does not access more reference samples than a normal motion compensation process. After the refined MV is achieved with the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples for the normal MC process, samples that are not needed for the interpolation process based on the original MV but are needed for the interpolation process based on the refined MV will be padded from these available samples.

The following describes an example of enabling conditions for DMVR. DMVR is enabled when all of the following conditions are satisfied.

a. CU level merge mode with bi-predictive MV

b. For a current picture, one reference picture is in the past and the other reference picture is in the future.

c. Distances from both reference pictures to the current picture (ie, POC difference) are the same

d. CU has more than 64 luma samples

e. Both CU height and CU width are greater than or equal to 8 luma samples

f. BCW (bidirectional prediction with CU-level weights) weight index indicates equal weight

g. WP (Weighted Prediction) is not enabled for the current block

h. Combined Inter and Intra Prediction (CIIP) mode not used for current block

The bi-directional optical flow is described next. Bi-directional optical flow (BDOF) is used to refine the bi-directional prediction signal of the luma samples in the CU at the 4x4 sub-block level. As its name indicates, BDOF mode is based on the optical flow concept, which assumes that the object's motion is smooth. For each 4x4 subblock, the motion refinement (v _x , v _y ) is computed by minimizing the difference between the L0 and L1 prediction samples. Motion refinement is then used to adjust the bi-predicted sample values in the 4x4 subblock.

For example, for BDOF, video encoder 200 and video decoder may determine that BDOF is enabled for a block and divide the block into a plurality of subblocks if BDOF is enabled for the block. . In some examples, video encoder 200 and video decoder 300 may determine a first reference block from a first motion vector for a block and a second reference block from a second motion vector for a block. Video encoder 200 and video decoder 300 may blend (eg, weight average) the samples in the first reference block with the samples in the second reference block to generate a predictive block. Video encoder 200 and video decoder 300 may determine motion refinement and adjust samples in a predictive block to generate predictive samples used to encode or decode the samples of the subblock. In some examples, video encoder 200 and video decoder 300 may determine the same motion refinement for each sample in a subblock (ie, a subblock level motion refinement referred to as a subblock BDOF). In some examples, video encoder 200 and video decoder 300 may determine a motion refinement of each sample in a subblock (ie, sample-level motion refinement, referred to as per-pixel BDOF).

The following steps apply in the BDOF process, which may be applicable to subblock BDOF. The steps for per-pixel BDOF are described in further detail below.

First, the horizontal and vertical gradients of the two prediction signals ( and , ) is computed by directly computing the difference between two neighboring samples, i.e.

In the example above, I ^(k) (i,j) is the sample value at coordinates (i,j) of the prediction signal in list k (k = 0, 1), and shift1 is equal to shift1 equal to 6 As set, it is calculated based on the luma bit depth (bitDepth). That is, I ⁽⁰⁾ refers to samples of the first reference block, and I ⁽¹⁾ refers to samples of the second reference block, where the first reference block and the second reference block are It was used to generate a prediction block that is being adjusted according to the

Then, the autocorrelation and cross-correlation of the gradients (S ₁ , S ₂ , S ₃ , S ₅ and S ₆ ) are calculated as follows:

here,

here, is a 6x6 window around a 4x4 subblock, the value of shift2 is set equal to 4, and the value of shift3 is set equal to 1.

The motion refinement (v _x , v _y ) is then derived using the cross-correlation and autocorrelation terms using In this example, motion refinement is for subblocks. The per-pixel motion refinement computation is described in more detail below.

here, am. is the floor function.

Based on the motion refinement and gradients, the following adjustment is computed for each sample in the 4x4 subblock:

Finally, the CU's BDOF samples are calculated by adjusting the bi-prediction samples as follows:

Here, shift5 is set equal to Max(3, 15 - BitDepth), and the variable o _offset is set equal to (1 << (shift5 - 1)).

In the above examples, I ⁽⁰⁾ refers to the first reference block, I ⁽¹⁾ refers to the second reference block, and b(x,y) is the motion refinement for the subblock (v _x , v _y ) is an adjustment value determined based on In some examples, I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) may be considered as a predictive block, and thus b(x,y) may be considered as adjusting the predictive block there is. As shown in equation (1-6-6), there may be an addition of o _offset and a right shift operation by shift5 to generate prediction samples (pred _BDOF (x,y)).

The above describes an example for a subblock BDOF in which video encoder 200 and video decoder 300 determine that the same motion refinement (v _x , v _y ) is equal for all samples in the subblock do. The adjustment value b(x,y) may be different for each sample in a subblock because of the gradient, but the motion refinement may be the same.

As described in further detail below, in per-pixel BDOF, video encoder 200 and video decoder 300 may determine a per-pixel motion refinement (v _x ', v _y '). That is, rather than having one motion refinement per subblock, as with subblock BDOF, in BDOF per pixel, there may be a different motion refinement for each sample (eg, pixel). Rather than using the same motion refinement for a subblock, video encoder 200 and video decoder 300 base the adjustment value b'(x, y) can be determined.

In some examples, the values from equation (1-6-6) are selected such that multipliers in the BDOF process do not exceed 15 bits, and the maximum bit width of intermediate parameters in the BDOF process is kept within 32 bits. .

To derive the gradient values, some prediction samples (I ^(k) (i, j)) in list k (k = 0, 1) outside the current CU boundaries are used by the video encoder 200 and the video decoder ( 300) is created by As shown in FIG. 12 , BDOF uses one extended row/column around the borders of CU 1200 . To control the computational complexity of generating out-of-bounds prediction samples, video encoder 200 and video decoder 300 take reference samples at nearby integer positions directly (using floor() operations on coordinates) without interpolation. , and a normal 8-tap motion compensation interpolation filter is used to generate prediction samples (gray positions) within the CU. These extended sample values may only be used in gradient calculations. For the remaining steps in the BDOF process, if any sample and gradient values outside the CU boundaries are needed, the sample and gradient values are padded (ie, repeated) from their nearest neighbors.

BDOF is used to refine a bi-directional prediction signal (eg, the sum of a first reference block and a second reference block) of a CU at a 4×4 subblock level. BDOF is applied to a CU if all of the following conditions are met:

a. CU is coded using “true” bi-prediction mode, i.e., one of the two reference pictures is before the current picture in display order and the other is after the current picture in display order

b. CU is not coded using affine mode or ATMVP merge mode

c. CU has more than 64 luma samples

d. Both CU height and CU width are greater than or equal to 8 luma samples

e. BCW weight index shows equal weight

f. WP currently not enabled for CU

g. CIIP mode is currently not used for CU

Some problems may exist with BDOF. As described above, in the current version of VVC, the BDOF method is used to refine the bidirectional prediction signal of luma samples in a coding block at the 4x4 subblock level. The motion refinement (v _x , v _y ) is derived by minimizing the difference between L0 and L1 prediction samples in 6x6 luma sample regions. L0 prediction samples refer to samples of the first reference block, and L1 prediction samples refer to samples of the second reference block. Then, the motion refinement (v _x , v _y ) is used to adjust each prediction sample of the 4x4 subblock.

However, a luma sample in a 4x4 subblock may have different motion refinement characteristics compared to other luma samples in a 4x4 subblock. Computing the motion refinement (v' _x , v' _y ) at the pixel level can improve the accuracy of the motion refinement for each pixel, and thus improve sub-block or block prediction quality.

However, BDOF is a decoder-side process, and the complexity of BDOF is also an important aspect to consider when designing a video coding method. If the motion refinement (v' _x , v' _y ) is computed at the pixel-level, the complexity of BDOF can be 16 times as compared to the current BDOF at the 4x4 subblock level. That is, the current 4x4 subblock BDOF does not achieve the best prediction quality. BDOF per pixel has better prediction quality, but complexity is an issue for video coding.

In VVC Draft 10, when decoder-side motion vector refinement (DMVR) is preceded by BDOF, the BDOF process can be bypassed based on the minimum SAD in the DMVR search process. The DMVR process is at the 16x16 subblock level. This BDOF bypass scheme can reduce complexity.

However, the prediction signal of a sub-region within a 16x16 subblock may need to be refined by BDOF. In the BDOF bypass of the VVC draft 10 method, BDOF cannot be applied to a sub-region within a 16×16 subblock, while BDOF can be bypassed in other sub-regions. In VVC Draft 10, there is no bypass BDOF scheme when BDOF is applied to bi-directionally predicted (non-DMVR predicted) coding blocks.

The following describes example techniques that may address the above problems. However, the techniques should not be considered limited or required to address the above problems. The following techniques may, in practice, be used separately or in any combination. For convenience, the following techniques are described as various aspects, but such aspects are not to be considered as being required to be separate and, in fact, the various aspects may be combined. Example aspects may be performed by video encoder 200 and/or video decoder 300 unless specified otherwise.

A first aspect relates to bypassing the subblock BDOF. In this first aspect, when a W×H coding block is determined to apply bi-directional optical flow (BDOF), video encoder 200 and/or video decoder 300 bypass the BDOF process for sub-regions of the coding block. You may. The BDOF process for the first aspect may be as follows.

a. The BDOF process starts with an input block (named S1), where S1 has dimensions W_1 x H_1, and the dimension of S1 is less than or equal to the dimension of the coding block. If the preceding process is block-based, the dimension of S1 is equal to the coding block. If the preceding process is subblock-based (subblock partition due to hardware constraints or from previous processing stages), the dimension of S1 is less than the coding block.

b. The input block S1 is divided into N subblocks (named S2), where S2 has dimensions W_2xH_2, and the dimension of S2 is less than or equal to the dimension of S1. For each S2 determined by condition T, S2 determines whether to apply BDOF. In some examples, condition T is to check whether the SAD between the two prediction signals in reference picture 0 and reference picture 1 is less than a threshold. A subblock in this step defines a basic unit for determining whether to apply BDOF to all samples in the unit.

c. When it is decided to apply BDOF to S2, S2 is divided into M subblocks (named as S3), where S3 has dimensions W_3×H_3, and the dimension of S3 is less than or equal to the dimension of S2. For each S3, a BDOF process is applied to derive a refined motion vector (v' _x , v' _y ), using the derived motion vector (either via motion compensation or by adding an offset to the initial predicted signal). through) derive the prediction signal of S3. A subblock in this step defines a unit for the granularity of the refined motion vector, and all samples in the unit share the same refined motion.

In the BDOF process of aspect 1, blocks S1, S2 and S3 are defined. The dimension of S3 may be less than or equal to S2, and the dimension of S2 may be less than or equal to S1. That is, W_3 is less than W_2, H_3 is less than H_2, W_2 is less than W_1, and H_2 is less than H_1. Sizes may be fixed, adapted to picture resolution, or signaled in the bitstream.

One case is that W_3 equals 1 and H_3 equals 1, where S3 is pixel-based. This case may also be a per-pixel BDOF process.

In some examples, S1 is a coding block, regardless of whether the preceding subblock-based process is applied to the coding block.

A second aspect relates to per-pixel BDOF with a sub-block BDOF bypass scheme. As in the first aspect, when a W×H coding block S1 is determined to apply bidirectional optical flow (BDOF), the coding block is divided into N subblocks S2. For each subblock, whether to apply BDOF to the subblock is further determined by checking whether the SAD between two prediction signals in reference picture 0 and reference picture 1 is less than a threshold value. If it is decided to apply BDOF to a subblock, a refined motion vector (v' _x , v' _y ) is calculated for each pixel S3 in subblock S2. The refined motion vector (v' _x , v' _y ) is used to adjust the predicted signal for that pixel S3 in sub-block S2. An example of per-pixel BDOF with subblock bypass process is shown in FIG. 13 .

For example, in FIG. 13 , video encoder 200 and video decoder 300 may determine that BDOF is enabled for a block of video data, and video encoder 200 and video decoder 300 may determine that BDOF is A block may be divided into a plurality of subblocks based on the determination that is enabled for . As illustrated in FIG. 13 , deriving the number of subblocks (N) subblock index <i=0> ( 1300 ) causes video encoder 200 and video decoder 300 to divide a block into N subblocks. , where each subblock is identified by a separate index, and the first index is 0. Thus, indices range from 0 to N-1.

Video encoder 200 and video decoder 300 may determine whether predictive samples have been determined for all subblocks in the blocks, as represented by i<N ( 1302 ). If predictive samples for all subblocks have been determined (“NO” of 1302 ), video encoder 200 and video decoder 300 may end the process of determining predictive samples for subblocks. However, if prediction samples for all subblocks have not been determined (Yes in 1302), the video encoder 200 and the video decoder 300 determine the current subblock among a plurality of subblocks from which the block is divided. The process of determining prediction samples may continue.

For the current subblock, video encoder 200 and video decoder 300 may determine a distortion value ( 1304 ). Because the determination of the distortion value may be performed on a subblock-by-subblock basis, video encoder 200 and video decoder 300, for each subblock of one or more of the plurality of subblocks, separate distortion values (eg, a first distortion value for a first subblock, a second distortion value for a second subblock, etc.).

One exemplary way to determine the distortion value for the current subblock is by determining the sum of absolute differences (SAD) between the first reference block (ref0) and the second reference block (ref1). However, there may be other ways of determining the distortion value. For example, as described in further detail below, in some examples, video encoder 200 and video decoder 300 perform BDOF and video encoder 200 when video decoder 300 perform BDOF Likewise, the distortion value may be determined in such a way that the resulting values may be reused later.

As illustrated in FIG. 13 , video encoder 200 and video decoder 300 may compare the distortion value to a threshold value ( 1306 ). Based on the comparison, video encoder 200 and video decoder 300 may have two options. A first option may be to perform BDOF per pixel, and a second option may be to bypass BDOF. There may not be other options for video encoder 200 and video decoder 300, such as subblock BDOF. Accordingly, video encoder 200 and video decoder 300 divide a plurality of subblocks based on individual distortion values (e.g., based on comparison of individual distortion values to a fixed threshold value or individual threshold values). For each subblock of one or more of the subblocks, one of the BDOFs per pixel is performed or the BDOF is bypassed.

For example, if the distortion value for the current subblock is greater than the threshold (“NO” of 1306), video encoder 200 and video decoder 300 may perform per-pixel BDOF (1308). If the distortion value for the current subblock is less than the threshold (“yes” of 1306), then video encoder 200 and video decoder 300 (eg, by bypassing the BDOF for the subblock) subblock A prediction signal may be derived from (1310).

In one or more examples, video encoder 200 and video decoder 300 may determine predictive samples for each subblock of one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed there is. For example, if video encoder 200 and video decoder 300 are to perform BDOF on a current subblock, video encoder 200 and video decoder 300 will use per-pixel BDOF techniques to determine predictive samples. However, if video encoder 200 and video decoder 300 are to bypass BDOF for a current subblock, video encoder 200 and video decoder 300 may determine predictive samples that do not use BDOF techniques. there is.

The above example of FIG. 13 explained a method of determining whether per-pixel BDOF is performed for the current subblock or whether BDOF is bypassed. Video encoder 200 and video decoder 300 may perform the example techniques above on a subblock-by-subblock basis.

For example, for each sub-block of one or more of the plurality of sub-blocks, for a first sub-block of the one or more sub-blocks to determine respective distortion values, the video encoder 200 and video Decoder 300 may determine a first distortion value of the respective distortion values, and for a second subblock of the one or more subblocks, video encoder 200 and video decoder 300 may determine a second distortion value of the respective distortion values. value can also be determined.

To determine that one of per-pixel BDOF is performed or BDOF is bypassed for each sub-block of one or more sub-blocks of the plurality of sub-blocks based on respective distortion values; For a subblock, video encoder 200 and video decoder 300 determine whether the BDOF for the first subblock based on the first distortion value (eg, based on the first distortion value being greater than a threshold value) is It can also be determined that it is enabled. In this example, based on determining that BDOF is enabled for the first sub-block, video encoder 200 and video decoder 300 perform per-pixel to refine a first set of prediction samples for the first sub-block. Motion refinement may be determined (eg, per-pixel BDOF is performed). For example, video encoder 200 and video decoder 300 derive, for a first sample of a first subblock, a first motion refinement for refining a first prediction sample, and For 2 samples, we may derive a second motion refinement to refine the second prediction sample, and so on.

However, for a second sub-block of the plurality of sub-blocks, the video encoder 200 and the video decoder 300 perform a second distortion value based on a second distortion value (eg, based on the second distortion value being less than a threshold value). so) may determine that BDOF is bypassed. In this example, based on a determination that BDOF is bypassed for the second block, video encoder 200 and video decoder 300 use motion per pixel to refine a second set of prediction samples for the second sub-block. It is also possible to bypass determining refinement (eg, bypass BDOF). For example, video encoder 200 and video decoder 300 bypass, for a first sample of a first subblock, derivation of a first motion refinement for refining a first prediction sample, and For a second sample of the block, it may bypass the derivation of a second motion refinement to refine the second predictive sample, and so forth.

To determine predictive samples for each subblock of one or more subblocks based on determining whether per-pixel BDOF is performed or BDOF is bypassed, video encoder 200 and video decoder 300: For one sub-block, a first refined first set of prediction samples of the first sub-block may be determined based on the per-pixel motion refinement for the first sub-block. For the second subblock, video encoder 200 and video decoder 300 do not refine the second set of predictive samples based on per-pixel motion refinement to refine the second set of predictive samples. You can also decide on 2 sets.

Within the second aspect, the bypass subblock BDOF is described next. Given a W×H coding block determined to apply bi-directional optical flow (BDOF), the number of subblocks (N) is determined as follows:

In the above, thW represents the maximum subblock width, and thH represents the maximum subblock height. The values of thW and thH are predetermined integer values (eg, thW = thH = 8).

For each subblock, video encoder 200 and/or video decoder 300 may derive a prediction signal (predSig0) and a prediction signal (predSig1) from reference picture 0 and reference picture 1, respectively. The width (sbWidth) and height (sbHeight) of predSig0 and predSig1 are determined as follows:

Whether to bypass BDOF in a subblock is determined by checking the SAD between predSig0 and predSig1. SAD is derived as follows:

In the above expression, is a sbWidth×sbHeight subblock, and I ^(k) (i, j) is a sample value at coordinates (i, j) of the prediction signal in reference picture k (k = 0, 1).

If sbSAD is less than the threshold (sbDistTh), video encoder 200 and/or video decoder 300 may determine to bypass BDOF in the subblock; otherwise (if sbSAD is greater than or equal to sbDistTh), video encoder 200 and/or video decoder 300 may determine to apply BDOF to the subblock. The threshold (sbDistTh) is derived as follows:

In the above formula, n and s are predetermined values. For example, n can be derived as follows: n = InternalBitDepth - bitDepth + 1. In the above equation, s represents a scale factor, for example s = 1. In the current version of VVC, InternalBitDepth equals bitDepth 10 to 14, and therefore n equals 5. The scale (s) may be 1, 2, 3 other predefined values, or signaled in the bitstream.

It should be understood that the above describes one example way of determining the threshold value and one example way of determining the distortion value. However, example techniques are not limited thereto. As described in more detail below, in some examples, video encoder 200 and video decoder 300, once a determination is made that per-pixel BDOF will be performed, the calculations used to determine distortion values It is also possible to determine the distortion values in a way that can be reused to perform

Within the second aspect, the following describes BDOF per pixel. If video encoder 200 and/or video decoder 300 have determined to apply BDOF to a sbWidth × sbHeight subblock, the subblock extends into the (sbWidth + 4) × (sbHeight + 4) region. For each pixel within a subblock, video encoder 200 and/or video decoder 300, based on the gradients of the 5×5 surrounding area, a motion refinement (v′, also referred to as a refined motion vector) _x , v' _y ) can also be derived. 14 illustrates an example of BDOF per pixel of an 8x8 subblock. Thus, in per-pixel BDOF, video encoder 200 and video decoder 300 may determine a per-pixel motion refinement. In subblock BDOF, motion refinement is for a subblock and is not determined on a sample-by-sample (eg, pixel-by-pixel) basis.

In the above, given the sbWidth×sbHeight subblock, the following steps are applied in the per-pixel BDOF process.

- horizontal and vertical gradients of the two prediction signals ( and , ) is computed by directly computing the difference between two neighboring samples as in the bidirectional optical flow described above, where (i,j) is the (sbWidth + 4) This is the adjusted position in the area × (sbHeight + 4).

- For each pixel in a sub-block, the following steps are applied.

o Autocorrelation and cross-correlation of the gradients (S ₁ , S ₂ , S ₃ , S ₅ and S ₆ ) are computed as in the bi-directional optical flow described above, where: is a 5×5 window around the pixel.

o The motion refinement (v' _x , v' _y ) is then derived using the cross-correlation and autocorrelation terms.

o Based on the motion refinement and gradients, the following adjustment is calculated to derive the pixel's prediction signal:

In the above examples, I ⁽⁰⁾ refers to the first reference block, and I ⁽¹⁾ refers to the second reference block. The adjustment value (b'(x,y)) is an adjustment value determined based on the per-pixel motion refinement (v' _x , v' _y ) for each sample in the subblock. In some examples, I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) may be considered as a predictive block, and thus b'(x,y) may be considered as adjusting a predictive block. may be As shown in equation (3-1-2-1), there may be an addition of o _offset and a right shift operation by shift5 to generate prediction samples (pred _BDOF (x,y)).

A third aspect relates to alternative subblock SAD derivation. This example technique for deriving SAD may be such that the values determined for SAD derivation can be reused to perform per-pixel BDOF. That is, video encoder 200 and video decoder may first determine a distortion value (eg, SAD value) for a sub-block to determine whether to perform per-pixel BDOF. If video encoder 200 and video decoder 300 determine that per-pixel BDOF is to be performed, the calculations performed by video encoder 200 and video decoder 300 to determine whether to perform per-pixel BDOF are may be reused to perform per BDOF.

For example, one way to determine a distortion value for a subblock is to use a first reference block (e.g., identified by a first motion vector) and a first reference block (e.g., identified by a second motion vector). ) determining a second reference block, and determining a distortion value by determining a difference value between samples of the first reference block and samples of the second reference block. As an example, as described above, one way to determine the distortion value is to determine:

In the above equation, I ⁽¹⁾ (i,j) refers to samples of the first reference block, and I ⁽⁰⁾ (i,j) refers to samples of the second reference block. As described further above, to determine motion refinement, including per-pixel motion refinement (eg, v' _x , v' _y ), video encoder 200 and video decoder 300 use a gradient The autocorrelation and cross-correlation of S ₁ , S ₂ , S ₃ , S ₅ , and S ₆ may be determined. As described in equation (1-6-3), part of determining the autocorrelation and cross-correlation of the gradients is determining the median value for θ, where θ = am.

Thus, if per-pixel BDOF is to be performed for a subblock, video encoder 200 and video decoder 300 may need to be determined. In one or more examples, as part of determining a distortion value for a subblock, video encoder 200 and video decoder 300: Instead of (or in addition to) determining the distortion value based on A distortion value for a subblock may be determined based on . That is, to determine a distortion value for a subblock, e.g., whether per-pixel BDOF is to be performed, video encoder 200 and video decoder 300 use as a value for sbSAD can also decide In this way, if per-pixel BDOF is to be performed, video encoder 200 and video decoder 300 already have the value of θ and which is used to determine the motion refinement. would have determined the value for

Accordingly, in one or more examples, video encoder 200 and video decoder 300, to determine respective distortion values for each subblock of one or more of the plurality of subblocks, For each subblock of one or more subblocks of the blocks, it may be configured to determine a first reference block and a second reference block. For example, I ⁽⁰⁾ (i,j) may be a first reference block, and I ⁽¹⁾ (i,j) may be a second reference block.

Video encoder 200 and video decoder 300 may scale the samples of the first reference block and the samples of the second reference block. For example, video encoder 200 and video decoder 300 may perform the operation I ⁽⁰⁾ (i,j) >> shift2. In this example, the value of shift2 may define how much to scale the value of I ⁽⁰⁾ (i,j) to generate the scaled samples of the first reference block. For example, video encoder 200 and video decoder 300 may perform the operation I ⁽¹⁾ (i,j) >> shift2. In this example, the value of shift2 may define how much to scale the value of I ⁽¹⁾ (i,j) to generate the scaled samples of the second reference block.

Video encoder 200 and video decoder 300 may determine a difference value between the scaled samples of the first reference block and the scaled samples of the second reference block to determine the respective distortion values. For example, video encoder 200 and video decoder 300 can also decide Video encoder 200 and video decoder 300 A distortion value (eg, sbSAD) for the subblock may be determined based on the result of .

As described above, in some examples, there may be computational gains for video encoder 200 and video decoder 300; The value of may be reused for per-pixel BDOF. For example, the video encoder 200 and the video decoder 300 determine that per-pixel BDOF is performed on a first subblock among one or more subblocks from among a plurality of subblocks from which a block to be encoded or decoded is divided. Suppose you did

In this example, video encoder 200 and video decoder 300 may determine, for each sample in the first subblock, separate motion refinements. That is, video encoder 200 and video decoder 300 may, rather than or in addition to determine one motion refinement (v _x ,v _y ) that is the same for all samples in the first subblock, A motion refinement (v' _x , v' _y ) may be determined for each sample of a subblock.

Video encoder 200 and video decoder 300, for each sample in the first sub-block, a separate refined sample value from samples in the predictive block for the first sub-block based on the respective motion refinements. may be configured to determine For example, as described above, the equation for determining prediction samples for BDOF per pixel is It could be.

To determine the pred _{BDOF ,} video encoder ₂₀₀ and video decoder 300 use a per-pixel adjustment value, b'(x, y) can be determined. In some examples, a predictive block may be considered as the sum of the first reference block and the second reference block (ie, I ⁽⁰⁾ (i,j) + I ⁽¹⁾ (i,j)). As shown in the equation for determining pred _BDOF , video encoder 200 and video decoder 300 calculate b'(x,y) as I ⁽⁰⁾ (i,j) + I ⁽¹⁾ (i,j) can also be added. Thus, as part of determining pred _BDOF , video encoder 200 and video decoder 300 use (v' _x , v ' _y )) in the predictive block for the first subblock (eg, where the predictive block is equal to I ⁽⁰⁾ (i,j) + I ⁽¹⁾ (i,j))) Refinement sample values (eg, pred _BDOF ) may be determined from samples of .

Stated another way, video encoder 200 and video decoder 300 may determine a first set of sample values in a first reference block for a first one of one or more subblocks (e.g., , which determines I ⁽⁰⁾ (i,j)). Video encoder 200 and video decoder 300 may scale the first set of sample values with a scale factor to generate the first set of scaled sample values. That is, to perform I ⁽⁰⁾ (i,j) >> shift2, the video encoder 200 and the video decoder 300 sample by the scale factor defined by the values of “>>” and “shift2”. may be considered as scaling the first set of .

Video encoder 200 and video decoder 300 may determine a second set of sample values in a second reference block for a first one of the one or more subblocks (e.g., I ⁽¹⁾ i,j)). Video encoder 200 and video decoder 300 may scale the second set of sample values with a scale factor to generate the second set of scaled sample values. That is, to perform I ⁽¹⁾ (i,j) >> shift2, the video encoder 200 and the video decoder 300 sample by the scale factor defined by the values of “>>” and “shift2” may be considered as scaling the second set of .

Video encoder 200 and video decoder 300 perform, for a first subblock, based on a first set of scaled sample values and a second set of scaled sample values (e.g., I ⁽⁰⁾ ( Based on i,j) >> shift2 and I ⁽¹⁾ (i,j) >> shift2), the distortion value may be determined. For example, video encoder 200 and video decoder 300 calculate the second based on (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2)). A distortion value for 1 subblock may be determined.

In one or more examples, assume that per-pixel BDOF is performed for the first sub-block, as described above. In this example, video encoder 200 and video decoder 300 may reuse the first set of scaled sample values and the second set of scaled sample values to determine the per-pixel motion refinement for per-pixel BDOF. there is. For example, video encoder 200 and video decoder 300 use a method to determine autocorrelation and cross-correlation of gradients to determine per-pixel motion refinement (eg, ( _v'x , _v'y )). The calculation of (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2) may be reused. As described above _, video encoder 200 and video decoder 300 use b'(x ,y) may use per-pixel motion refinement to determine the adjustment value.

The above indicates that video encoder 200 and video decoder 300 may reuse the first set of scaled sample values and the second set of scaled sample values to determine the per-pixel motion refinement for per-pixel BDOF. Explain an example. However, the techniques are not limited thereto. In some examples, video encoder 200 and video decoder 300 may reuse the first set of scaled sample values and the second set of scaled sample values to determine motion refinement for BDOF. That is, example techniques may not be limited to reusing the first set of scaled sample values and the second set of scaled sample values for per-pixel motion refinement for per-pixel BDOF, but motion refinement for BDOF. (e.g., not limited to per-pixel motion refinement for per-pixel BDOF). A reduction in complexity may exist for subblock-based BDOF as well as per-pixel BDOF, as in examples where BDOF includes motion refinement for the entire subblock rather than per pixel.

Accordingly, as in the second aspect, the following describes an alternative method for deriving the subblock SAD used to determine whether to bypass a subblock (ie, whether BDOF is bypassed). As described above, the exemplary method is as in the bi-directional optical flow described above in conjunction with Equations 1-6. Calculate the difference ( diff(i,j) ) between the two reference signals in the same way as calculating .

If a subblock is determined to apply BDOF, diff(i,j) can be reused in the step for computing the autocorrelation and crosscorrelation of gradients S3 and S6, as in the bidirectional optical flow described above. there is.

The expression of (3-1-1-1) in the second aspect is modified as follows:

In the above equation, I ^(k) (i, j) is the (sbWidth + 4) × (sbHeight + 4) area of the prediction signal in the reference picture k (k = 0, 1) at coordinates (i, j). is the sample value. Shift2 is a predetermined value, for example, shift2 is equal to 4. is the sbWidth×sbHeight subblock area.

It should be noted that an alternative technique for determining a distortion value for a subblock (eg, for determining sbSAD) based on sbSAD should not be considered limited to examples in which per-pixel BDOF is performed. An alternative technique for determining a distortion value for a subblock may be applicable to examples even when subblock BDOF or some other BDOF technique is applied. For example, even for subblock BDOF, video encoder 200 and video decoder 300 may utilize alternative techniques to determine distortion values for determining whether BDOF is performed on a subblock. If BDOF is to be performed, video encoder 200 and video decoder 300 may reuse calculations for alternative techniques to determine distortion values for determining motion refinement as part of a subblock BDOF (eg, For example, there may be reuse of calculations for alternative techniques to determine distortion values).

As described above, the threshold value against which the distortion value is compared to determine whether per-pixel BDOF is performed or whether BDOF is bypassed is, as shown in equation (3-1-1-2) above (sbWidth *sbHeight*s) << sbDistTh calculated as n. However, in an alternative technique for determining the distortion value, video encoder 200 and video decoder 300 scale I ⁽⁰⁾ (i,j) >> by shift 2, as described above. and scale I ⁽¹⁾ by >> shift2. Thus, in some examples, the manner in which video encoder 200 and video decoder 300 determine sbDistTh may be modified to account for >>shift2 scaling.

The expression of (3-1-1-2) in the second aspect for calculating sbDistTh is modified as follows:

In the above formula, n and s are predetermined values. For example, n can be derived as follows: n = InternalBitDepth - bitDepth + 1. In the above equation, s represents a scale factor, for example s = 1. In the current version of VVC, InternalBitDepth equals bitDepth 10 to 14, and therefore n equals 5. The scale (s) may be 1, 2, 3 other predefined values, or signaled in the bitstream.

Accordingly, to determine the threshold value, video encoder 200 and video decoder 300 use the width of a first one of the one or more subblocks (i.e., equation (3-2-2) to generate an intermediate value. ), the height of the first subblock of the one or more subblocks (ie, sbHeight in equation (3-2-2)), and the first scale factor (ie, equation (3-2-2) It may be configured to multiply "s" in ). Video encoder 200 and video decoder 300 may be configured to perform a left-shift operation on the intermediate value based on the second scale factor to generate a threshold value. For example, the second scale factor may be (n - shift2) in equation (3-2-2), and the left-shift operation is represented as "<<" in equation (3-2-2).

In one or more examples, video encoder 200 and video decoder 300 use a distortion value for the first subblock (eg, a distortion value calculated using an alternative technique for determining the distortion value) as a threshold value (eg, sbDistTh as determined in equation (3-2-2)). Video encoder 200 and video decoder 300 may determine that either per-pixel BDOF is performed or BDOF is bypassed for the first subblock based on the comparison. For example, if the distortion value is less than the threshold value (eg, the “yes” of 1306 in FIG. 13 ), video encoder 200 and video decoder 300 may bypass BDOF. If the distortion value is greater than the threshold value (eg, “no” of 1306 in FIG. 13 ), video encoder 200 and video decoder 300 may perform per-pixel BDOF.

Fourth aspects relate to determining the values of thW and thH. As in the above aspects, example techniques may be applied to a bi-predicted coding block. The total number of subblocks is derived from the width and height of the current block and the maximum subblock width (thW) and height (thH) of the subblock.

When the current coding block applies a subblock-based method, eg, DMVR, the values of thW and thH must be equal to or less than the maximum subblock width and height of the preceding method (eg, DMVR).

The values of thW and thH can be fixed predetermined values, eg thW is equal to 8 and thH is equal to 8. The values of thW and thH may be adaptive, and the values are determined by information decoded from the bitstream. The following describes the ways in which the values of thW and thH are adapted:

a. Determined by the preceding coding method: If the current coding block applies the subblock-based method, thW and thH may be set to the same subblock dimensions as the preceding method. For example, when DMVR is applied to the current coding block, thW is set equal to the DMVR maximum subblock width, eg 16, and thH is set equal to the DMVR maximum subblock height, eg 16. . Otherwise (if the current coding block does not apply any subblock-based method), thW and thH may be set to predetermined values, eg 8.

b. Determined by the current coding block dimension: in this example, the larger value of thW and thH is set to the coding block with the total number of luma samples greater than the threshold T (eg, T = 128). When a W×H coding block is given: If W*H is greater than T, the values of thW and thH are set equal to 16. Otherwise (if W*H is less than or equal to T), set the values of thW and thH equal to 8.

A fifth aspect relates to an example decoder process that applies per-pixel BDOF with subblock bypass. The above aspects may be applied at an encoder (eg, video encoder 200 ) and/or a decoder (eg, video decoder 300 ). A decoder (e.g., video decoder 300) may execute the methods described herein by all or a subset of the following steps to decode an inter-predicted block in a picture from a bitstream:

1. Deriving the position component (cbX, cbY) as the upper-left luma position of the current block by decoding the syntax elements in the bitstream.

2. Deriving the size of the current block as a width value (W) and a height value (H) by decoding syntax elements in the bitstream.

3. Determine that the current block is an inter predicted block from decoding elements in the bitstream.

4. Derive motion vector components (mvL0 and mvL1) and reference indices (refPicL0 and refPicL1) of the current block from the decoding elements in the bitstream.

5. Infer a flag from the decoding elements in the bitstream, where the flag indicates whether decoder-side motion vector derivation (eg, DMVR, bilateral merging, template matching) is applied to the current block. The reasoning method of the flag may be the same as the examples described above with respect to the enabling condition for when the DMVR is enabled, but is not limited thereto. In another example, this flag can be explicitly signaled in the bitstream to avoid complex condition checking at the decoder.

6. If it is determined to apply DMVR to the current block, refined motion vectors are derived.

7. Derive two (W + 6) × (H + 6) luma prediction sample arrays (predSampleL0 and predSampleL1) from the decoded refPicL0, refPicL1 and motion vectors, where it is determined to apply DMVR, the motion vector are the refined motion vectors, otherwise the motion vectors are mvL0, mvL1.

8. Infer a flag from the decoding elements in the bitstream, where the flag indicates whether bi-directional optical flow is applied to the current block. The reasoning method of the flag may be the same as that of the bi-directional optical flow, but is not limited thereto. In another example, this flag can be explicitly signaled in the bitstream to avoid complex condition checking at the decoder.

9. If, according to the above flag value, the decision is to apply BDOF to the current block, the number of subblocks in the horizontal direction (numSbX) and the number of subblocks in the vertical direction (numSbY), subblocks as follows: Derive the width (sbWidth) and height (sbHeight):

where thW and thH are predetermined integer values (e.g. thW = thH = 8)

10. Derive the variable sbDistTh as follows:

here,

shift2 is a predetermined value, for example, shift2 is equal to 4.

n is a predetermined value, for example n = InternalBitDepth - bitDepth + 1 = 5.

s is a scale factor, eg s = 1.

11. Set the position component (sbX, sbY) = (0, 0) as the upper-left luma position of the first subblock of the current block.

12. For each subblock in (sbX, sbY), if sbX is less than W and sbY is less than H, the following steps apply.

12.1. x = sbX - 2 . sbX + sbWidth + 1, y = sbY - 2... For sbY + sbHeight + 1, the variables diff[x][y] are derived as:

Here, shift2 is a predetermined value, for example, shift2 is equal to 4.

12.2. The variable sbDist is derived as follows:

Here, i = 　 0 … sbWidth - 1, j = 0 ... sbHeight - 1.

12.3. (Bypass Subblock BDOF) If sbDist is smaller than sbDistTh, the prediction signal of the subblock is derived as follows.

12.3.1. x = sbX... sbX + sbWidth - 1, y = sbY... For sbY + sbHeight - 1,

here,

shift5 is set equal to Max(3, 15 - BitDepth).

offset5 is set equal to (1 << (shift5 - 1)).

12.4. Otherwise (if sbDist is greater than or equal to sbDistTh), the following steps apply.

12.4.1. x = sbX - 2 . sbX + sbWidth + 1, y = sbY - 2... For sbY + sbHeight + 1, the variables (gradientHL0[　x　][　y　], gradientVL0[　x　][　y　], gradientHL1[　x　][　y　] and gradientVL1[　x　][　y　]) are derived as follows:

Here, shift1 is a predetermined value, and for example, shift1 is set equal to 6.

12.4.2. x = sbX - 2 . sbX + sbWidth + 1, y = sbY - 2... For sbY + sbHeight + 1, the variables (tempH[　x　][　y　] and tempV[　x　][　y　]) are derived as follows:

Here, shift3 is a predetermined value, and for example, shift3 is set equal to 1.

12.4.3. For each pixel in (piX, piY) (where piX = sbX ... sbX + sbWidth - 1, piY = sbY ... sbY + sbHeight - 1), the following steps apply.

12.4.3.1. The variables (sGx2, sGy2, sGxGy, sGxdI and sGydI) are derived as follows:

Here, i = 　 -2... 2, and j = -2... is 2.

12.4.3.2. The horizontal and vertical motion offsets of the current pixel are derived as follows:

Here, mvRefineThres is a predetermined value, for example, mvRefineThres is set equal to (1 << 4).

12.4.3.3. The prediction signal of the current pixel is derived as follows:

here,

shift5 is set equal to Max(3, 15 - BitDepth).

offset5 is set equal to (1 << (shift5 - 1)).

12.5. Update the upper-left luma position of the subblock as follows.

13. A prediction block is derived using the derived prediction signal of each subblock, and the derived prediction block is used for video decoding.

15 is a flowchart illustrating an example method for decoding video data, in accordance with the techniques of this disclosure. A current block may contain a current CU. Although described with respect to the video decoder 300 ( FIGS. 1 and 4 ), it should be understood that other devices may be configured to perform a method similar to that of FIG. 15 . For example, prediction processing unit 304 and/or motion compensation unit 316 may be configured to perform the example techniques of FIG. 15 . Prediction processing unit 304 and/or motion compensation unit 316 may be coupled to a memory, such as DPB 314 , or other memory of video decoder 300 . In some examples, video decoder 300 may be coupled to memory 120 that stores information used by video decoder 300 to perform the example techniques of FIG. 15 .

Video decoder 300 may determine that bidirectional optical flow (BDOF) is enabled for the block of video data ( 1500 ). For example, video decoder 300 may receive signaling indicating that BDOF is enabled for the block. In some examples, video decoder 300 may infer (eg, determine without receiving signaling) that BDOF is enabled for a block, eg, based on certain criteria being satisfied.

Video decoder 300 may divide a block into a plurality of subblocks based on determining that BDOF is enabled for the block ( 1502 ). For example, video decoder 300 may divide a block into N subblocks. In some cases, two or more of the subblocks may be of different sizes, but it is possible for the subblocks to have the same size. Video decoder 300 may determine how to divide a block based on signaled information or by inference.

Video decoder 300 may determine, for each subblock of one or more of the plurality of subblocks, respective distortion values ( 1504 ). There may be a variety of ways in which video decoder 300 may determine individual distortion values. As an example, video decoder 300 determines a first reference block (e.g., I ⁽⁰⁾ (i,j)) and determines a second reference block (e.g., I ⁽¹⁾ (i,j) )) can be determined. Video decoder 300 may calculate a sum of absolute differences (SAD) between I ⁽⁰⁾ (i,j) and I ⁽¹⁾ (i,j).

However, example techniques are not limited thereto. In some examples, video decoder 300 may perform an alternative technique to determine the distortion value, as described above. For example, video decoder 300 may determine a first set of sample values in a first reference block for a first one of the one or more subblocks (e.g., I ⁽⁰⁾ (i,j) decide). Video decoder 300 may scale the first set of sample values with a scale factor to generate the first set of scaled sample values (e.g., I ⁽⁰ ) to generate the first set of scaled sample values. ⁾ (i,j) << determines shift2). Video decoder 300 may determine a second set of sample values in a second reference block for a first one of the one or more subblocks (eg, determine I ⁽¹⁾ (i,j) box). Video decoder 300 may scale the second set of sample values with a scale factor to generate the second set of scaled sample values (e.g., I ^{(1 )} (i,j) << determines shift2). In one or more examples, to determine respective distortion values for each sub-block of one or more of the plurality of sub-blocks, video decoder 300, for a first sub-block, determines the number of scaled sample values determine a distortion value of individual distortion values based on the first set and the second set of scaled sample values (e.g., determine a SAD based on the first set of scaled sample values and the second set of scaled sample values); ) may be configured.

Video decoder 300 may determine that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of one or more subblocks of a plurality of subblocks based on the respective distortion values (1506). For example, as described with respect to FIG. 13 , for video decoder 300 there may be two options: performing per-pixel BDOF on a subblock or bypassing BDOF. In some examples, there may be no other option for video decoder 300 when evaluating a subblock.

In some examples, to determine whether to perform per-pixel BDOF or bypass BDOF, video encoder 200 and video decoder 300 may determine a threshold value. One exemplary way to determine the threshold is am. However, in examples where alternative techniques for determining the distortion value are utilized, video decoder 300 can also be determined as

That is, video decoder 300 uses a width (eg, sbWidth) of a first one of the one or more subblocks, a height (eg, sbHeight) of a first one of the one or more subblocks, and a first scale factor (eg, “s”). Video decoder 300 may perform a left-shift operation on the intermediate value based on the second scale factor to generate a threshold value (e.g., performing << (n - shift2), where (n - shift2) is the second scale factor).

Video decoder 300 may compare the distortion value of the individual distortion values for the first subblock to a threshold value. To determine that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of one or more subblocks of a plurality of subblocks based on the respective distortion values, video decoder 300, FIG. 13 As illustrated in decision block 1306 of , it may be determined based on the comparison that either per-pixel BDOF is performed or BDOF is bypassed for the first sub-block.

Video decoder 300 may be configured to determine predictive samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed ( 1508 ). As an example, to determine predictive samples, video decoder 300 may determine that per-pixel BDOF is performed on a first one of the one or more subblocks. In this example, video decoder 300 may determine, for each sample in the first subblock, separate motion refinements, and for each sample in the first subblock, based on the respective motion refinements. Individual refined sample values may be determined from samples in a prediction block for the first subblock.

For example, video decoder 300 operations can be performed. pred _BDOF may indicate refined sample values. In this example, I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) may be considered as the predictive block. The value for b'(x,y) may be determined by the respective motion refinements (v' _x , v' _y ) for each sample in the subblock. Thus, individual refinement sample values (eg, pred _BDOF ) are based on the predictive block and individual motion refinements.

There may be various ways of determining the motion refinements (v' _x , v' _y ). As part of determining motion refinements, video decoder 300 Including can also determine autocorrelation and cross-correlation. In one or more examples, such as when an alternative technique for determining distortion values is used, video decoder 300 may determine a distortion value for the first subblock using may have already been decided. In such examples, video decoder 300 uses a first set of scaled sample values (e.g., ) and a second set of scaled sample values (eg, ) can also be reused (e.g., the value for and can be determined without recalculating ).

Video decoder 300 may reconstruct the block based on the predictive samples ( 1510 ). For example, reconstructing a block based on prediction samples involves video decoder 300 receiving residual values representing differences between the prediction samples and samples of the block, and the residual values to the prediction samples. It may also involve restoring blocks by adding them.

The above gives examples for individual subblocks of a block. The following is an example in which there are two subblocks and per-pixel BDOF is performed for one subblock and BDOF is bypassed for the other subblock.

For example, for a first subblock of the one or more subblocks, video decoder 300 may determine a first distortion value of the respective distortion values, and for a second subblock of the one or more subblocks, video decoder 300 may determine a first distortion value of the respective distortion values. Decoder 300 may determine a second distortion value of the individual distortion values.

For a first subblock of the plurality of subblocks, video decoder 300 determines that a BDOF is assigned to the first subblock based on the first distortion value (eg, based on a comparison of the first distortion value with a threshold value). may be determined to be enabled for Based on determining that BDOF is enabled for the first subblock, video decoder 300 may determine a per-pixel motion refinement to refine the first set of predictive samples for the first subblock. For example, video decoder 300 derives, for a first sample of a first subblock, a first motion refinement for refining a first prediction sample, and for a second sample of a first subblock, a first motion refinement for refining a first prediction sample. may derive a second motion refinement to refine 2 prediction samples, and so forth.

For a second subblock of the plurality of subblocks, video decoder 300 may determine that the BDOF is bypassed based on the second distortion value (eg, based on a comparison of the second distortion value to a threshold value). there is. Based on determining that BDOF is bypassed for the second block, video decoder 300 may bypass determining a per-pixel motion refinement for refining the second set of predictive samples for the second sub-block. . For example, video decoder 300 bypasses, for a first sample of a first subblock, derivation of a first motion refinement to refine a first prediction sample, and for a second sample of a first subblock , may bypass the derivation of the second motion refinement for refining the second prediction sample, and so forth.

For a first sub-block, video decoder 300 may determine a refined first set of predictive samples of the first sub-block based on the per-pixel motion refinement for the first sub-block (e.g., as described in this disclosure). determine pred _BDOF using example techniques described above). For the second subblock, video decoder 300 may determine the second set of predictive samples without refining the second set of predictive samples based on per-pixel motion refinement to refine the second set of predictive samples. . That is, for the second subblock, BDOF is bypassed. Video decoder 300 may determine predictive samples for the second subblock based on various techniques, such as determining the predictive block based on a weighted average of reference blocks.

16 is a flowchart illustrating an example method of encoding video data, in accordance with the techniques of this disclosure. A current block may contain a current CU. Although described with respect to video encoder 200 ( FIGS. 1 and 3 ), it should be understood that other devices may be configured to perform a method similar to that of FIG. 16 . For example, motion selection unit 202 and/or motion compensation unit 224 may be configured to perform the example techniques of FIG. 16 . Motion selection unit 202 and/or motion compensation unit 224 may be coupled to a memory, such as DPB 218 , or other memory of video encoder 200 . In some examples, video encoder 200 may be coupled to memory 106 that stores information used by video encoder 200 to perform the example techniques of FIG. 16 . In general, video encoder 200 may perform the same operations as video decoder 300 to generate predictive samples.

Video encoder 200 may determine that bidirectional optical flow (BDOF) is enabled for the block of video data ( 1600 ). For example, video encoder 200 may determine rate-distortion costs associated with the different coding modes and, based on the rate-distortion costs, may determine that BDOF is enabled for the block.

Video encoder 200 may divide a block into a plurality of subblocks when BDOF is enabled for the block ( 1602 ). Video encoder 200 may determine, for each subblock of one or more of the plurality of subblocks, respective distortion values ( 1604 ). Video encoder 200 may perform the same techniques as those described for video decoder 300 to determine individual distortion values.

Video encoder 200 may determine that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of one or more subblocks of a plurality of subblocks based on the respective distortion values (1606). For example, since video encoder 200 may not signal information indicating whether per-pixel BDOF is performed or whether BDOF is bypassed, video encoder 200 may perform per-pixel BDOF for each subblock. may perform the same operations as video decoder 300 to determine whether .

Video encoder 200 may determine predictive samples for each subblock of one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed ( 1608 ). Video encoder 200 may signal residual values between prediction samples and samples of a block (eg, individual subblocks) ( 1610 ).

The following describes some example techniques that may be applied together or separately.

Article 1. A method of decoding video data, the method comprising: determining that bi-directional optical flow (BDOF) is enabled for a block of video data; dividing the block into a plurality of subblocks based on a determination that BDOF is enabled for the block; for each sub-block of one or more sub-blocks of the plurality of sub-blocks, determining individual distortion values; determining that one of the per-pixel BDOFs is performed or the BDOFs are bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values; determining prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed; and reconstructing the block based on the predicted samples.

Article 2. The method of clause 1, wherein for each sub-block of one or more sub-blocks of the plurality of sub-blocks, determining individual distortion values comprises: for a first sub-block of the one or more sub-blocks, an individual distortion value determining a first distortion value of ?; and determining, for a second subblock of the one or more subblocks, a second distortion value of the respective distortion values, wherein each of the one or more subblocks of the plurality of subblocks based on the respective distortion values. Determining that one of the BDOFs per pixel is performed or the BDOF is bypassed for a subblock of , wherein, for a first subblock among a plurality of subblocks, based on a first distortion value, the BDOF is selected as the first subblock. determining that it is enabled for; based on determining that BDOF is enabled for the first sub-block, determining a per-pixel motion refinement for refining a first set of prediction samples for the first sub-block; determining that BDOF is bypassed for a second subblock among the plurality of subblocks based on a second distortion value; and based on the determination that the BDOF is bypassed for the second block, bypassing determining a per-pixel motion refinement for refining a second set of prediction samples for the second sub-block, wherein: Determining prediction samples for each sub-block of the one or more sub-blocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed comprises, for a first sub-block, per-pixel for the first sub-block. determining a refined first set of prediction samples of a first subblock based on motion refinement; and, for the second sub-block, determining a second set of predictive samples without refining the second set of predictive samples based on the per-pixel motion refinement to refine the second set of predictive samples.

Article 3. The method of any one of clauses 1 and 2, wherein either per-pixel BDOF is performed for each subblock of one or more subblocks of the plurality of subblocks based on individual distortion values, or BDOF is bypassed. The determining step includes determining that per-pixel BDOF is performed for a first subblock of the one or more subblocks, the method determining, for each sample in the first subblock, respective motion refinements. further comprising, wherein determining prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed comprises: For each sample, determining respective refined sample values from samples in the prediction block for the first subblock based on the respective motion refinements.

Article 4. The method of any one of clauses 1 to 3 uses a width of a first one of the one or more subblocks, a height of a first one of the one or more subblocks, and a first scale factor to generate an intermediate value. multiplying; performing a left-shift operation on the intermediate value based on the second scale factor to generate a threshold value; and comparing a distortion value of the individual distortion values for the first sub-block with a threshold value, wherein for each sub-block of one or more sub-blocks of the plurality of sub-blocks based on the individual distortion values. Determining that one of the BDOFs per pixel is performed or the BDOF is bypassed includes determining that one of the BDOFs per pixel is performed or the BDOF is bypassed for the first subblock based on the comparison.

Article 5. The method of any of clauses 1 to 4 includes determining a first set of sample values in a first reference block for a first subblock of the one or more subblocks; scaling the first set of sample values with a scale factor to produce a first set of scaled sample values; determining a second set of sample values in a second reference block for a first one of the one or more subblocks; and scaling the second set of sample values with a scale factor to generate a second set of scaled sample values, wherein for each subblock of one or more subblocks of the plurality of subblocks. , determining individual distortion values comprises, for the first subblock, determining a distortion value of the individual distortion values based on the first set of scaled sample values and the second set of scaled sample values. .

Article 6. The method of clause 5, wherein determining that one of the per-pixel BDOF is performed or the BDOF is bypassed for each sub-block of one or more sub-blocks of the plurality of sub-blocks based on respective distortion values comprises: determining that per-pixel BDOF is performed for the sub-block, the method comprising: a first set of scaled sample values and a second set of scaled sample values to determine a per-pixel motion refinement for the per-pixel BDOF; It further includes the step of reusing.

Article 7. The method of clause 5, wherein determining that one of the per-pixel BDOF is performed or the BDOF is bypassed for each sub-block of one or more sub-blocks of the plurality of sub-blocks based on respective distortion values comprises: determining that per-pixel BDOF is performed for the subblock, the method comprising: reusing the first set of scaled sample values and the second set of scaled sample values to determine motion refinement for the BDOF; more includes

Article 8. The method of any one of clauses 1 to 7, wherein reconstructing the block comprises: receiving residual values representing differences between prediction samples and samples of the block; and adding the residual values to the prediction samples to reconstruct the block.

Article 9. A device for decoding video data, the device comprising: a memory configured to store video data; and processing circuitry coupled to the memory, wherein the processing circuitry determines that bi-directional optical flow (BDOF) is enabled for the block of video data; divide the block into a plurality of subblocks based on determining that BDOF is enabled for the block; for each sub-block of one or more sub-blocks of the plurality of sub-blocks, determining respective distortion values; determine that one of the per-pixel BDOF is performed or the BDOF is bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values; determine prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed; and reconstruct the block based on the predicted samples.

Article 10. The device of clause 9, wherein for each sub-block of one or more sub-blocks of the plurality of sub-blocks, to determine respective distortion values, the processing circuitry comprises: for a first sub-block of the one or more sub-blocks: determine a first distortion value of the respective distortion values; and for a second subblock of the one or more subblocks, determine a second distortion value of the respective distortion values, wherein each subblock of the one or more subblocks of the plurality of subblocks is configured to determine a second distortion value of the respective distortion values. To determine that one of the per-pixel BDOFs are performed for the block or that the BDOFs are bypassed, processing circuitry determines, for a first one of the plurality of subblocks, that the BDOF determines, based on a first distortion value, that the BDOF is bypassed. determine that it is enabled for the block; determine a per-pixel motion refinement for refining a first set of prediction samples for the first sub-block based on the determination that BDOF is enabled for the first sub-block; For a second subblock of the plurality of subblocks, determine that the BDOF is bypassed based on the second distortion value; and, based on the determination that BDOF is bypassed for the second block, bypass determining a per-pixel motion refinement for refining a second set of prediction samples for the second sub-block, where: To determine predictive samples for each sub-block of the one or more sub-blocks based on a determination that BDOF is performed or BDOF is bypassed, the processing circuitry determines, for the first sub-block, a pixel for the first sub-block determine a refined first set of prediction samples of the first subblock based on the per motion refinement; and for the second subblock, determine a second set of prediction samples without refining the second set of prediction samples based on the per-pixel motion refinement to refine the second set of prediction samples.

Article 11. The device of any of clauses 9 and 10, wherein for each sub-block of one or more sub-blocks of the plurality of sub-blocks based on respective distortion values, one of the per-pixel BDOFs is performed or the BDOF is bypassed. To determine, the processing circuitry is configured to determine that per-pixel BDOF is performed for a first one of the one or more subblocks, wherein the processing circuitry is further configured to: for each sample in the first subblock: processing circuitry configured to determine individual motion refinements, wherein processing circuitry to determine prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed; and for each sample in the first subblock, determine respective refined sample values from samples in the prediction block for the first subblock based on the respective motion refinements.

Article 12. The device of any of clauses 9-11, wherein processing circuitry comprises: a width of a first of the one or more sub-blocks, a height of a first of the one or more sub-blocks, and multiply by a first scale factor; perform a left-shift operation on the intermediate value based on the second scale factor to generate a threshold value; and compare a distortion value of the respective distortion values for the first sub-block to a threshold value, wherein the per-pixel BDOF for each sub-block of one or more sub-blocks of the plurality of sub-blocks based on the respective distortion values. To determine that one of the BDOFs is performed or the BDOF is bypassed, the processing circuitry is configured to determine that one of the BDOFs is performed or the BDOF is bypassed per pixel for the first subblock based on the comparison.

Article 13. The device of any of clauses 9-12, wherein the processing circuitry is configured to: determine a first set of sample values in a first reference block for a first one of the one or more subblocks; scaling the first set of sample values with a scale factor to produce a first set of scaled sample values; determine a second set of sample values in a second reference block for a first one of the one or more subblocks; and scale the second set of sample values with a scale factor to generate a second set of scaled sample values, wherein, for each subblock of one or more subblocks of the plurality of subblocks, a respective distortion. To determine the values, the processing circuitry is configured to, for the first sub-block, determine a distortion value of the individual distortion values based on the first set of scaled sample values and the second set of scaled sample values.

Article 14. The device of clause 13, wherein processing circuitry is configured to determine that one of the per-pixel BDOF is performed or the BDOF is bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values. is configured to determine that per-pixel BDOF is performed for the first subblock, wherein the processing circuitry includes a first set of scaled sample values and a set of scaled sample values to determine a per-pixel motion refinement for the per-pixel BDOF. configured to reuse the second set.

Article 15. The device of clause 13, wherein processing circuitry is configured to determine that one of the per-pixel BDOF is performed or the BDOF is bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values. is configured to determine that per-pixel BDOF is performed for the first sub-block, wherein the processing circuitry determines a first set of scaled sample values and a second set of scaled sample values to determine a motion refinement for the BDOF. configured to be reused.

Article 16. The device of any of clauses 9 to 15, wherein to reconstruct a block, processing circuitry receives residual values representing differences between predicted samples and samples of the block; and reconstruct the block by adding the residual values to the prediction samples.

Article 17. The device of any of clauses 9 to 16 further comprises a display configured to display the decoded video data.

Article 18. The device of any of clauses 9-17, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set top box.

Article 19. A computer-readable storage medium storing instructions that, when executed, cause one or more processors to: determine that bi-directional optical flow (BDOF) is enabled for a block of video data; divide the block into a plurality of subblocks based on a determination that BDOF is enabled for the block; for each sub-block of one or more sub-blocks of the plurality of sub-blocks, determine respective distortion values; determine that one of the BDOFs per pixel is performed or the BDOF is bypassed for each subblock of one or more subblocks of the plurality of subblocks based on the respective distortion values; determine prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed; and to reconstruct a block based on the predicted samples.

Article 20. The computer-readable storage medium of clause 19, wherein the instructions causing the one or more processors to determine, for each sub-block of the one or more sub-blocks of the plurality of sub-blocks, individual distortion values include: determine, for a first subblock of the one or more subblocks, a first distortion value of the respective distortion values; and instructions to cause, for a second one of the one or more subblocks, to determine a second distortion value of the respective distortion values, wherein the one or more processors determine a plurality of subblocks based on the respective distortion values. instructions that cause one or more processors to determine that one of the per-pixel BDOFs is performed or that BDOF is bypassed for each subblock of one or more subblocks of the plurality of subblocks , determine that BDOF is enabled for the first subblock based on the first distortion value; determine a per-pixel motion refinement for refining a first set of prediction samples for the first sub-block based on the determination that BDOF is enabled for the first sub-block; for a second subblock of the plurality of subblocks, determine that the BDOF is bypassed based on the second distortion value; and instructions to bypass determining a per-pixel motion refinement for refining a second set of prediction samples for a second sub-block based on a determination that the BDOF is bypassed for the second block, wherein: , instructions that cause one or more processors to determine prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed, for a first sub-block, determine a refined first set of prediction samples of the first sub-block based on the per-pixel motion refinement for the first sub-block; and instructions to cause, for the second subblock, to determine a second set of predictive samples without refining the second set of predictive samples based on per-pixel motion refinement to refine the second set of predictive samples.

Article 21. The computer-readable storage medium of any one of clauses 19 and 20, wherein the one or more processors cause the one or more processors to generate a per-pixel BDOF of the BDOF for each subblock of one or more subblocks of the plurality of subblocks based on respective distortion values. instructions that cause one to determine that BDOF is performed or that BDOF is bypassed include instructions that cause one or more processors to determine that per-pixel BDOF is performed for a first subblock of the one or more subblocks; further include instructions that cause the one or more processors to determine, for each sample in the first subblock, individual motion refinements, wherein the one or more processors perform per-pixel BDOF or BDOF instructions that cause the one or more processors to determine, for each sample in the first subblock, a respective motion refinement and instructions to determine individual refined sample values from samples in a predictive block for the first sub-block based on .

Article 22. The computer-readable storage medium of any of clauses 19-21, which causes one or more processors to: a width of a first one of the one or more subblocks, a width of a first one of the one or more subblocks, a second one of the one or more subblocks to generate an intermediate value. multiply the height of 1 subblock, and the first scale factor; perform a left-shift operation on the intermediate value based on the second scale factor to generate a threshold value; and further comprising instructions for causing the one or more processors to compare a distortion value of the respective distortion values for the first subblock to a threshold value, wherein the one or more subblocks of the plurality of subblocks based on the respective distortion values. Instructions that cause one or more processors to determine that either a BDOF per pixel is performed for each subblock of , or a BDOF is bypassed, the one or more processors perform a BDOF per pixel for the first subblock based on the comparison. or contains instructions that determine that BDOF is bypassed.

Article 23. The computer-readable storage medium of any of clauses 19-22, which causes one or more processors to: determine a first set of sample values in a first reference block for a first sub-block of the one or more sub-blocks. make; scale the first set of sample values with a scale factor to produce a first set of scaled sample values; determine a second set of sample values in a second reference block for a first one of the one or more subblocks; and further comprising instructions that cause the one or more processors to scale the second set of sample values with a scale factor to generate a second set of scaled sample values, wherein: one or more subblocks of the plurality of subblocks. For each subblock of the , instructions to determine individual distortion values may cause one or more processors to, for a first subblock, a first set of scaled sample values and a second set of scaled sample values. and instructions for determining a distortion value of individual distortion values based on

Article 24. A device for decoding video data, the device comprising: means for determining that bi-directional optical flow (BDOF) is enabled for a block of video data; means for dividing the block into a plurality of subblocks based on determining that BDOF is enabled for the block; means for determining respective distortion values for each sub-block of one or more sub-blocks of the plurality of sub-blocks; means for determining that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of one or more subblocks of the plurality of subblocks based on respective distortion values; means for determining prediction samples for each subblock of the one or more subblocks based on a determination that per-pixel BDOF is performed or BDOF is bypassed; and means for reconstructing the block based on the prediction samples.

Article 25. A method of coding video data, the method comprising: dividing an input block into a plurality of subblocks, wherein a size of the input block is less than or equal to a size of a coding unit; determining that bidirectional optical flow (BDOF) is to be applied to a subblock of the plurality of subblocks based on a condition being satisfied; dividing a subblock into a plurality of sub-subblocks; Determining a refined motion vector for one or more of the sub-subblocks, wherein the refined motion vector for a sub-subblock of the one or more sub-subblocks is dependent on a plurality of samples in the sub-subblock. determining the refined motion vector, which is the same for ; and performing BDOF on a subblock based on the refined motion vector for one or more sub-subblocks.

Article 26. A method of coding video data, the method comprising: dividing an input block into a plurality of subblocks, wherein a size of the input block is less than or equal to a size of a coding unit; determining that bidirectional optical flow (BDOF) is to be applied to a subblock of the plurality of subblocks based on a condition being satisfied; dividing a subblock into a plurality of sub-subblocks; determining a refined motion vector for each of one or more samples in the subblock; and performing BDOF on the subblock based on the refined motion vector for each of one or more samples in the subblock.

Article 27. The method of any one of clauses 25 and 26 further comprises bypassing BDOF for other subblocks of the plurality of subblocks.

Article 28. The method of any one of clauses 25 to 27, wherein the condition being satisfied determines whether a sum of absolute differences (SAD) between two prediction signals in reference picture 0 and reference picture 1 is smaller than a threshold. includes

Article 29. The method of any of clauses 25 to 28, wherein the size of the input block is thWxthH, where thW and thH are fixed predetermined values; based on one or more of the values decoded from the bitstream; or based on the size of blocks used before BDOF in encoding or decoding a coding unit.

Article 30. A method of coding video data, the method comprising any one or combination of clauses 25-29.

Article 31. The method of any of clauses 25-30, wherein performing BDOF comprises performing BDOF as part of decoding the video data.

Article 32. The method of any of clauses 25-31, wherein performing BDOF comprises performing BDOF as part of encoding the video data by including in a recovery loop of the encoding.

Article 33. A device for coding video data, the device comprising: a memory for storing video data; and processing circuitry coupled to the memory, the processing circuitry configured to perform any one or combination of clauses 25-32.

Article 34. Device for coding video data, the device comprising one or more means for performing the method of any of clauses 25 to 32.

Article 35. The device of any of clauses 33 and 34 further comprises a display configured to display the decoded video data.

Article 36. The device of any of clauses 33-35, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set top box.

Article 37. The device of any of clauses 33 to 36, wherein the processing circuitry or means for performing comprises a video decoder.

Article 38. The device of any of clauses 33 to 37, wherein the processing circuitry or means for performing comprises a video encoder.

Article 39. A computer-readable storage medium having stored thereon instructions, which, when executed, cause one or more processors to perform the method of any of clauses 25-32.

Depending on the example, any particular acts or events of the techniques described herein may be performed in a different sequence, and may be added, merged, or removed entirely (e.g., all acts or events described may be not essential for the practice of the techniques). Moreover, in certain examples, actions or events may be performed simultaneously rather than sequentially, eg, via multi-threaded processing, interrupt processing, or multiple processors.

In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer readable media are computer readable storage media that correspond to tangible media such as data storage media, or any medium that facilitates transfer of a computer program from one place to another, eg, according to a communication protocol. Communication media including In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media that is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer readable medium.

By way of example and not limitation, such computer readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or desired program code instructions. or any other medium that can be used for storage in the form of data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. Coax, for example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave. Cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. However, it should be understood that computer readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory tangible storage media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy discs, and Blu-ray discs, where the disc Disks typically reproduce data magnetically, but discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuitry. Accordingly, the terms “processor” and “processing circuitry” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or set of ICs (eg, a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined into a codec hardware unit including one or more processors as described above along with suitable software and/or firmware or provided by a collection of interoperable hardware units. .

Various examples have been described. These and other examples are within the scope of the following claims.

Claims (24)

A method of decoding video data, comprising:
determining that bi-directional optical flow (BDOF) is enabled for the block of video data;
dividing the block into a plurality of subblocks based on the determination that BDOF is enabled for the block;
determining individual distortion values for each sub-block of one or more sub-blocks of the plurality of sub-blocks;
determining that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values;
determining prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed; and
Reconstructing the block based on the prediction samples. According to claim 1,
For each subblock of one or more subblocks of the plurality of subblocks, determining individual distortion values comprises:
determining, for a first subblock of the one or more subblocks, a first distortion value of the individual distortion values; and
For a second subblock of the one or more subblocks, determining a second distortion value of the respective distortion values;
Determining that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values;
determining, for the first sub-block among the plurality of sub-blocks, that BDOF is enabled for the first sub-block based on the first distortion value;
based on the determination that BDOF is enabled for the first sub-block, determining a per-pixel motion refinement for refining a first set of prediction samples for the first sub-block;
determining that BDOF is bypassed for the second subblock among the plurality of subblocks based on the second distortion value; and
based on the determination that BDOF is bypassed for a second block, bypassing determining a per-pixel motion refinement for refining a second set of prediction samples for the second sub-block;
Determining prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed comprises:
for the first sub-block, determining a refined first set of prediction samples of the first sub-block based on the per-pixel motion refinement for the first sub-block; and
for the second sub-block, determining a second set of prediction samples without refining the second set of prediction samples based on the per-pixel motion refinement for refining the second set of prediction samples; A method for decoding video data, comprising: According to claim 1,
Determining that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values may include: Determining that per-pixel BDOF is performed for a first subblock of
The method further comprises determining, for each sample in the first subblock, respective motion refinements;
Determining prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed comprises: for each sample in the first subblock, determining individual refined sample values from samples in a predictive block for the first subblock based on the individual motion refinements. According to claim 1,
multiplying a width of a first one of the one or more subblocks, a height of the first one of the one or more subblocks, and a first scale factor to generate an intermediate value;
performing a left-shift operation on the intermediate value based on a second scale factor to generate a threshold value; and
Comparing a distortion value of the individual distortion values for the first sub-block with the threshold value;
Determining that one of per-pixel BDOF is performed or BDOF is bypassed for each sub-block of the one or more sub-blocks of the plurality of sub-blocks based on the individual distortion values may include: A method of decoding video data comprising determining that either per-pixel BDOF is performed or BDOF is bypassed for a first subblock. According to claim 1,
determining a first set of sample values in a first reference block for a first one of the one or more subblocks;
scaling the first set of sample values with a scale factor to produce a first set of scaled sample values;
determining a second set of sample values in a second reference block for the first one of the one or more subblocks; and
scaling the second set of sample values with the scale factor to produce a second set of scaled sample values;
For each subblock of one or more subblocks of the plurality of subblocks, determining individual distortion values comprises, for the first subblock, the first set of scaled sample values and the scaled sample and determining a distortion value of the individual distortion values based on a second set of values. According to claim 5,
Determining that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values may be performed in the first subblock. determining that per-pixel BDOF is performed for
wherein the method further comprises reusing the first set of scaled sample values and the second set of scaled sample values to determine a per-pixel motion refinement for per-pixel BDOF. . According to claim 5,
Determining that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values may be performed in the first subblock. determining that per-pixel BDOF is performed for
wherein the method further comprises reusing the first set of scaled sample values and the second set of scaled sample values to determine a motion refinement for BDOF. According to claim 1,
The step of restoring the block is,
receiving residual values representing differences between the prediction samples and samples of the block; and
and adding the residual values to the prediction samples to reconstruct the block. A device for decoding video data,
a memory configured to store the video data; and
processing circuitry coupled to the memory;
The processing circuitry,
determine that bi-directional optical flow (BDOF) is enabled for the block of video data;
divide the block into a plurality of subblocks based on the determination that BDOF is enabled for the block;
for each sub-block of one or more sub-blocks of the plurality of sub-blocks, determine respective distortion values;
determine that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values;
determine prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed; and
To reconstruct the block based on the prediction samples
A device for decoding video data, configured. According to claim 9,
For each sub-block of one or more sub-blocks of the plurality of sub-blocks, to determine respective distortion values, the processing circuitry comprises:
for a first one of the one or more subblocks, determine a first distortion value of the individual distortion values; and
For a second subblock of the one or more subblocks, determine a second distortion value of the respective distortion values.
constituted,
To determine based on the respective distortion values, for each sub-block of the one or more sub-blocks of the plurality of sub-blocks, per-pixel BDOF is performed or BDOF is bypassed, the processing circuitry comprises:
for the first sub-block of the plurality of sub-blocks, determine that BDOF is enabled for the first sub-block based on the first distortion value;
based on the determination that BDOF is enabled for the first sub-block, determine a per-pixel motion refinement for refining a first set of prediction samples for the first sub-block;
determine that BDOF is bypassed for the second subblock among the plurality of subblocks based on the second distortion value; and
Based on the determination that BDOF is bypassed for a second block, bypass determining a per-pixel motion refinement for refining a second set of prediction samples for the second sub-block.
constituted,
To determine prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed, the processing circuitry comprises:
for the first sub-block, determine a refined first set of prediction samples of the first sub-block based on the per-pixel motion refinement for the first sub-block; and
For the second sub-block, determine a second set of predictive samples without refining the second set of predictive samples based on the per-pixel motion refinement for refining the second set of predictive samples.
A device for decoding video data, configured. According to claim 9,
To determine that one of per-pixel BDOF is performed or BDOF is bypassed for each sub-block of the one or more sub-blocks of the plurality of sub-blocks based on the respective distortion values, the processing circuitry comprises: be configured to determine that per-pixel BDOF is performed for a first subblock of the one or more subblocks;
the processing circuitry is further configured to determine, for each sample in the first subblock, respective motion refinements;
To determine prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed, the processing circuitry comprises: each of the respective subblocks in the first subblock; For a sample, determine individual refined sample values from samples in a predictive block for the first subblock based on the individual motion refinements. According to claim 9,
The processing circuitry,
multiply a width of a first one of the one or more subblocks, a height of the first one of the one or more subblocks, and a first scale factor to generate an intermediate value;
perform a left-shift operation on the intermediate value based on a second scale factor to generate a threshold value; and
To compare a distortion value of the individual distortion values for the first sub-block with the threshold value
constituted,
To determine that one of per-pixel BDOF is performed or BDOF is bypassed for each sub-block of the one or more sub-blocks of the plurality of sub-blocks based on the respective distortion values, the processing circuitry comprises: and determine that one of per-pixel BDOF is performed or BDOF is bypassed for the first subblock based on the comparison. According to claim 9,
The processing circuitry,
determine a first set of sample values in a first reference block for a first one of the one or more subblocks;
scaling the first set of sample values with a scale factor to produce a first set of scaled sample values;
determine a second set of sample values in a second reference block for the first one of the one or more subblocks; and
to scale the second set of sample values with the scale factor to generate a second set of scaled sample values;
constituted,
For each sub-block of one or more sub-blocks of the plurality of sub-blocks, to determine respective distortion values, the processing circuitry, for the first sub-block, the first set of scaled sample values and and determine a distortion value of the individual distortion values based on the second set of scaled sample values. According to claim 13,
To determine that one of per-pixel BDOF is performed or BDOF is bypassed for each sub-block of the one or more sub-blocks of the plurality of sub-blocks based on the respective distortion values, the processing circuitry comprises: be configured to determine that per-pixel BDOF is performed for the first sub-block;
a device for decoding video data, wherein the processing circuitry is configured to reuse the first set of scaled sample values and the second set of scaled sample values to determine a per-pixel motion refinement for per-pixel BDOF . According to claim 13,
To determine that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values, the processing circuitry comprises: be configured to determine that per-pixel BDOF is performed for the first sub-block;
wherein the processing circuitry is configured to reuse the first set of scaled sample values and the second set of scaled sample values to determine motion refinement for BDOF. According to claim 9,
To recover the block, the processing circuitry comprises:
receive residual values representing differences between the prediction samples and samples of the block; and
Reconstruct the block by adding the residual values to the prediction samples
A device for decoding video data, configured. According to claim 9,
A device for decoding video data, further comprising a display configured to display the decoded video data. According to claim 9,
Wherein the device comprises one or more of a camera, computer, mobile device, broadcast receiver device, or set top box. A computer readable storage medium storing instructions,
The instructions, when executed, cause one or more processors to:
determine that bi-directional optical flow (BDOF) is enabled for the block of video data;
divide the block into a plurality of subblocks based on the determination that BDOF is enabled for the block;
for each sub-block of one or more sub-blocks of the plurality of sub-blocks, determine respective distortion values;
determine that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values;
determine prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed; and
Reconstruct the block based on the prediction samples. According to claim 19,
Instructions that cause the one or more processors to determine, for each subblock of one or more subblocks of the plurality of subblocks, individual distortion values, cause the one or more processors to:
for a first one of the one or more subblocks, determine a first distortion value of the individual distortion values; and
determining a second distortion value of the individual distortion values for a second subblock of the one or more subblocks;
contains commands,
cause the one or more processors to determine based on the respective distortion values that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks. Instructions cause the one or more processors to:
for the first sub-block of the plurality of sub-blocks, determine that BDOF is enabled for the first sub-block based on the first distortion value;
determine, based on the determination that BDOF is enabled for the first sub-block, a per-pixel motion refinement for refining a first set of prediction samples for the first sub-block;
for the second subblock of the plurality of subblocks, determine that BDOF is bypassed based on the second distortion value; and
Bypass determining a per-pixel motion refinement for refining a second set of prediction samples for the second sub-block based on the determination that BDOF is bypassed for the second block.
contains commands,
instructions that cause the one or more processors to determine prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed; cause:
for the first sub-block, determine a refined first set of prediction samples of the first sub-block based on the per-pixel motion refinement for the first sub-block; and
For the second subblock, determine a second set of prediction samples without refining the second set of prediction samples based on the per-pixel motion refinement for refining the second set of prediction samples.
A computer readable storage medium containing instructions. According to claim 19,
cause the one or more processors to determine based on the respective distortion values that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks. the instructions comprising instructions that cause the one or more processors to determine that per-pixel BDOF is performed for a first one of the one or more subblocks;
the instructions further comprising instructions to cause the one or more processors to determine, for each sample in the first subblock, respective motion refinements;
instructions that cause the one or more processors to determine prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed; For each sample in the first subblock, determine individual refined sample values from samples in a predictive block for the first subblock based on the individual motion refinements , a computer readable storage medium. According to claim 19,
Causes the one or more processors to:
multiply a width of a first one of the one or more subblocks, a height of the first one of the one or more subblocks, and a first scale factor to generate an intermediate value;
perform a left-shift operation on the intermediate value based on a second scale factor to generate a threshold value; and
Comparing a distortion value of the individual distortion values for the first subblock with the threshold value
contains more commands,
cause the one or more processors to determine based on the respective distortion values that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks. The instructions include instructions that cause the one or more processors to determine, based on the comparison, that one of the BDOFs per pixel is performed or BDOF is bypassed for the first subblock. According to claim 19,
Causes the one or more processors to:
determine a first set of sample values in a first reference block for a first one of the one or more subblocks;
scale the first set of sample values with a scale factor to produce a first set of scaled sample values;
determine a second set of sample values in a second reference block for the first one of the one or more subblocks; and
Scale the second set of sample values with the scale factor to generate a second set of scaled sample values.
contains more commands,
Instructions that cause the one or more processors to determine, for each subblock of one or more subblocks of the plurality of subblocks, respective distortion values, cause the one or more processors to: , instructions to determine a distortion value of the respective distortion values based on the first set of scaled sample values and the second set of scaled sample values. A device for decoding video data,
means for determining that bi-directional optical flow (BDOF) is enabled for the block of video data;
means for dividing the block into a plurality of subblocks based on the determination that BDOF is enabled for the block;
means for determining respective distortion values for each subblock of one or more of the plurality of subblocks;
means for determining that one of per-pixel BDOF is performed or BDOF is bypassed for each subblock of the one or more subblocks of the plurality of subblocks based on the respective distortion values;
means for determining prediction samples for each subblock of the one or more subblocks based on the determination that per-pixel BDOF is performed or BDOF is bypassed; and
and means for reconstructing the block based on the prediction samples.