TW202243480A - Template matching based affine prediction for video coding - Google Patents

Abstract

A video decoder can be configured to determine that a current block in a current picture of the video data is coded in an affine prediction mode; determine one or more control-point motion vectors (CPMVs) for the current block; identify an initial prediction block for the current block in a reference picture using the one or more CPMVs; determine a current template for the current block in the current picture; and determine an initial reference template for the initial prediction block in the reference picture; and perform a motion vector refinement process to determine a modified prediction block based on a comparison of the current template to the initial reference template.

Description

Template-matching-based affine prediction for video decoding

This patent application claims the benefit of U.S. Provisional Patent Application No. 63/173,861, filed April 12, 2021, and U.S. Provisional Patent Application No. 63/173,949, filed April 12, 2021, The entire contents of each of the above applications are incorporated by reference.

The content of this case is about video coding and video decoding.

Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers devices, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radiotelephones (so-called "smart phones"), video conferencing devices, video streaming devices, etc. Digital video equipment implements video coding technology (such as in MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 (Part 10, Advanced Video Coding (AVC)) , the standards defined by ITU-T H.265/High Efficiency Video Coding (HEVC) and those techniques described in extensions to such standards). By implementing such video decoding technology, video equipment can more efficiently send, receive, encode, decode and/or store digital video information.

Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video decoding, a video slice (e.g., a video picture or a portion of a video picture) can be divided into video blocks, which can also be referred to as coding tree units (CTUs), coding units (CUs), and /or decoding node. Video blocks in an intra-coded (I) slice of a picture are coded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be called a frame, and a reference picture may be called a reference frame.

This document describes techniques related to affine prediction mode, a type of inter-frame prediction mode that potentially takes into account possible rotations of objects in a series of pictures. The affine motion model of the block may be decided based on the motion vector of the block's control points, which may be referred to as a control point motion vector (CPMV). In some implementations, the control points of a block are the upper left and upper right corners of the block. In some implementations, the control points of a block also include the bottom left corner of the block. A video coder (ie, a video transcoder or video decoder) may calculate motion vectors for sub-blocks of a block based on the CPMV of the block to locate predictive sub-blocks in a reference picture. The predictive sub-blocks may form a predictive block.

This disclosure describes decoder-side techniques that can refine the predictive sub-block and thus the predictive block. That is, the techniques of this disclosure may cause the video decoder to use different sub-blocks to form the predictive block than the sub-blocks originally determined or located using CPMV. By performing a motion vector refinement procedure in the manner described in this application to determine a modified prediction block for an affine-coded block, a video decoder can determine a prediction block that is more accurate than normal affine prediction. Determining a more accurate prediction block using the techniques presented in this case can improve the overall decoding quality without increasing the burden of signaling management.

According to an example of the content of this case, a method for decoding video information includes: determining to use an affine prediction mode to decode a current block in a current picture of the video information; determining one or more controls for the current block point motion vector (CPMV); use the one or more CPMVs to identify an initial prediction block for the current block in a reference picture; determine a current template for the current block in the current picture; determine a current template for the reference an initial reference template for the initial prediction block in a picture; and performing a motion vector refinement procedure based on a comparison of the current template and the initial reference template to determine a modified prediction block.

According to another example of the content of this case, a device for decoding video data includes: a memory; and one or more processors, which are implemented in a circuit, coupled to the memory, and configured to: decide to simulate Decoding the current block in the current picture of the video data in a projective prediction mode; determining one or more control point motion vectors (CPMV) for the current block; using the one or more CPMVs to identify reference pictures the initial prediction block of the current block in the current picture; determine the current template for the current block in the current picture; determine the initial reference template for the initial prediction block in the reference picture; and based on the current template and the initial The motion vector refinement procedure is performed by comparing the reference templates to determine the modified prediction block.

According to another example of the content of this application, a computer-readable storage medium stores instructions, and the instructions, when executed by one or more processors, cause the one or more processors to perform the following operations: determine an affine prediction mode for Decoding a current block in a current picture of the video data; determining one or more control point motion vectors (CPMVs) for the current block; using the one or more CPMVs to identify the current block in a reference picture an initial prediction block for a block; determining a current template for the current block in the current picture; determining an initial reference template for the initial prediction block in the reference picture; and based on a comparison between the current template and the initial reference template to perform the motion vector refinement procedure to determine the modified prediction block.

According to another example of the content of this application, a device for decoding video data includes: a unit for determining to decode a current block in a current picture of the video data in an affine prediction mode; A unit of one or more control point motion vectors (CPMVs) of the current block; a unit for identifying an initial prediction block for the current block in a reference picture using the one or more CPMVs; a unit of a current example of the current block in the current picture; a unit for determining an initial reference example for the initial prediction block in the reference picture; and a unit for determining an initial reference example based on a comparison of the current example and the initial reference example A motion vector refinement procedure is performed to determine units of modified prediction blocks.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

Video decoding (e.g., video encoding and/or video decoding) typically involves predicting blocks of video data from already decoded blocks of video data in the same picture (e.g., intra-frame prediction) or from video data in a different picture Blocks of video data are predicted from already decoded blocks (eg, inter-frame prediction). In some cases, video transcoders also calculate residual data by comparing the predicted block with the original block. The residual data therefore represent the difference between the predicted block and the original block. To reduce the number of bits required to signal the residual data, a video transcoder transforms and quantizes the residual data and signals the transformed and quantized residual data in an encoded bit stream. Compression via transform and quantization procedures can be lossy, which means that the transformation and quantization procedures can introduce distortion in the decoded video data.

A video decoder decodes the residual data and adds it to the prediction block to produce a reconstructed video block that matches the original video block more closely than the prediction block alone. The first reconstructed block may have distortions or artifacts due to losses introduced by transforming and quantizing the residual data. A common type of artifact or distortion is known as blockiness, where the boundaries of the blocks used to decode the video data are visible.

To further improve the quality of the decoded video, a video decoder may perform one or more filtering operations on the reconstructed video blocks. Examples of these filtering operations include deblocking filtering, sample self-adjusting offset (SAO) filtering, and self-adjusting loop filtering (ALF). The parameters of these filtering operations can be determined by the video transcoder and signaled explicitly in the encoded video bitstream, or they can be determined implicitly by the video decoder without requiring an explicit signal in the encoded video bitstream. These parameters are explicitly signaled in the metastream.

This document describes techniques related to affine prediction mode, a type of inter-frame prediction mode that potentially takes into account possible rotations of objects in a series of pictures. The affine motion model of the block may be decided based on the motion vector of the block's control points, which may be referred to as a control point motion vector (CPMV). In some implementations, the control points of a block are the upper left and upper right corners of the block. In some implementations, the control points of a block also include the bottom left corner of the block. A video coder (ie, a video transcoder or video decoder) may calculate motion vectors for sub-blocks of a block based on the CPMV of the block to locate predictive sub-blocks in a reference picture. The predictive sub-blocks may form a predictive block.

This disclosure describes decoder-side techniques that can refine the predictive sub-block and thus the predictive block. That is, the techniques of this disclosure may cause the video decoder to use different sub-blocks to form the predictive block than the sub-blocks originally determined or located using CPMV. By performing a motion vector refinement procedure in the manner described in this application to determine a modified prediction block for an affine-coded block, a video decoder can determine a prediction block that is more accurate than normal affine prediction. Determining a more accurate prediction block using the techniques presented in this case can improve the overall decoding quality without increasing the burden of signaling management.

Although the techniques in this disclosure are generally described as being performed by a video decoder, it should be understood that the techniques described herein may also be performed by a video transcoder. For example, the techniques of this disclosure may be performed by a video transcoder as part of a process for deciding how to encode a video block and for generating reference pictures that may be used to encode subsequent pictures of the video.

1 is a block diagram illustrating an example video encoding and decoding system 100 that may implement techniques of this disclosure. Generally speaking, the technology involved in this case involves decoding (encoding and/or decoding) video data. In general, video data includes any data used to process video. Thus, video data may include raw unencoded video, encoded video, decoded (eg, reconstructed) video, and video metadata (eg, signaling data).

As shown in FIG. 1 , in this example, system 100 includes source device 102 that provides encoded video material to be decoded and displayed by destination device 116 . Specifically, the source device 102 provides video data to the destination device 116 via the computer-readable medium 110 . Source device 102 and destination device 116 may comprise any of a wide variety of devices, including desktop computers, notebook computers (i.e., laptops), mobile devices, tablet computers, set-top boxes, such as Telephone handsets such as smart phones, televisions, cameras, display devices, digital media players, video game consoles, video data streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be equipped for wireless communication, and thus may be referred to as wireless communication devices.

In the example of FIG. 1 , the source device 102 includes a video source 104 , a memory 106 , a video transcoder 200 and an output interface 108 . The destination device 116 includes an input interface 122 , a video decoder 300 , a memory 120 and a display device 118 . In accordance with the present disclosure, the video transcoder 200 of the source device 102 and the video decoder 300 of the destination device 116 may be configured to apply techniques for performing template-based affine prediction. Thus, source device 102 represents an instance of a video encoding device, and destination device 116 represents an instance of a video decoding device. In other examples, the source and destination devices may include other components or arrangements. For example, source device 102 may receive video data from an external video source, such as an external camera. Likewise, destination device 116 may interface with an external display device rather than include an integrated display device.

System 100 as shown in FIG. 1 is but one example. In general, any digital video encoding and/or decoding device can implement the techniques for performing template-based affine prediction. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates decoded video material for transmission to destination device 116 . The context of this case refers to a "decoding" device as a device that performs decoding (eg, encoding and/or decoding) of data. Accordingly, the video transcoder 200 and the video decoder 300 represent examples of coding devices, specifically, a video transcoder and a video decoder, respectively. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner such that source device 102 and destination device 116 each include components for video encoding and decoding. Thus, system 100 can support one-way or two-way video transmission between source device 102 and destination device 116, eg, for video streaming, video playback, video broadcasting, or video telephony.

In general, video source 104 represents a source of video data (ie, raw unencoded video data), and provides video transcoder 200 with a sequence of pictures (also called "frames") of the video data, The video transcoder 200 encodes data for pictures. Video source 104 of source device 102 may include a video capture device, such as a video camera, a video archive unit containing previously captured raw video, and/or a video feed interface for receiving video from a video content provider. As a further alternative, video source 104 may generate computer graphics-based material as the source video, or a combination of live video, archived video, and computer-generated video. In each case, video transcoder 200 encodes captured, pre-captured, or computer-generated video data. Video transcoder 200 may rearrange pictures from a received order (sometimes referred to as "display order") into a decoding order for decoding. The video transcoder 200 can generate a bit stream including encoded video data. Source device 102 may then output the encoded video data via output interface 108 to computer-readable medium 110 for receipt and/or retrieval by, for example, input interface 122 of destination device 116 .

Memory 106 of source device 102 and memory 120 of destination device 116 represent general purpose memory. In some examples, the memories 106 , 120 may store raw video data, eg, raw video from the video source 104 and raw decoded video data from the video decoder 300 . Additionally or alternatively, the memories 106, 120 may store software instructions executable by, for example, the video transcoder 200 and the video decoder 300, respectively. Although memory 106 and memory 120 are shown in this example as being separate from video transcoder 200 and video decoder 300, it should be understood that video transcoder 200 and video decoder 300 may also be included for use in Internal memory for a functionally similar or equivalent purpose. In addition, the memories 106 , 120 can store encoded video data output from the video transcoder 200 and input to the video decoder 300 , for example. In some examples, portions of memory 106, 120 may be allocated as one or more video buffers, eg, to store raw decoded and/or encoded video data.

Computer-readable medium 110 may represent any type of medium or device capable of transporting encoded video material from source device 102 to destination device 116 . In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to transmit encoded video data directly to destination device 116 in real-time, such as over a radio frequency or computer-based network. The output interface 108 can modulate the transmission signal including the encoded video data according to a communication standard such as a wireless communication protocol, and the input interface 122 can modulate the received transmission signal according to a communication standard such as a wireless communication protocol. to demodulate. Communication media may include any wireless or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other device that may be useful for facilitating communication from source device 102 to destination device 116 .

In some examples, source device 102 may output encoded data from output interface 108 to depository device 112 . Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122 . Storage device 112 may comprise any of a variety of distributed or locally accessed data storage media, such as hard disks, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or Any other suitable digital storage medium for storing encoded video data.

In some examples, source device 102 may output the encoded video data to file server 114 or another intermediate storage device that may store the encoded video data generated by source device 102 . Destination device 116 may access the stored video data from file server 114 via streaming or download.

File server 114 may be any type of server device capable of storing encoded video data and sending the encoded video data to destination device 116 . File server 114 may represent a web server (eg, for a website), a server configured to provide file transfer protocol services such as file transfer protocol (FTP) or file delivery over unidirectional transfer (FLUTE) protocol. server, Content Delivery Network (CDN) device, Hypertext Transfer Protocol (HTTP) server, Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and/or Network Attached Storage (NAS) device . File server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Self-Adjusting Streaming over HTTP (DASH), HTTP Instant Streaming (HLS), Real Time Streaming Protocol (RTSP ), HTTP dynamic data streaming, etc.

Destination device 116 may access encoded video data from file server 114 via any standard data connection, including an Internet connection. This may include wireless channels (e.g., a Wi-Fi connection), wired connections (e.g., Digital Subscriber Line (DSL), cable modem, etc.) suitable for accessing encoded video data stored on the file server 114 ), or a combination of the two. Input interface 122 may be configured to operate in accordance with any one or more of the various protocols discussed above for retrieving or receiving media data from file server 114 , or other such protocols for retrieving media data.

Output interface 108 and input interface 122 may represent a wireless transmitter/receiver, a modem, a wired networking unit (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In instances where the output interface 108 and the input interface 122 include wireless components, the output interface 108 and the input interface 122 may be configured to conform to a cellular communication standard (such as 4G, 4G-LTE (Long Term Evolution), LTE-Advanced, 5G, etc.) to transmit data (such as encoded video data). In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 may be configured to comply with other wireless standards (such as IEEE 802.11 specifications, IEEE 802.15 specifications (e.g., ZigBee™), Bluetooth™ standards, etc.) to transmit data (such as encoded video data). In some examples, source device 102 and/or destination device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include a SoC device for performing the functions assigned to video transcoder 200 and/or output interface 108, and destination device 116 may include a device for performing functions assigned to video decoder 300 and/or input interface 108. 122 function SoC devices.

The technology in this case can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (such as HTTP-based dynamic self-adjusting data streaming (DASH)), encoding of digital video onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

The input interface 122 of the destination device 116 receives an encoded video bitstream from the computer-readable medium 110 (eg, communication medium, storage device 112 , file server 114 , etc.). The encoded video bitstream may include signaling information defined by video transcoder 200 (which is also used by video decoder 300) such as the following syntax elements: The value of the properties and/or processing of a unit (eg, slice, picture, group of pictures, sequence, etc.). The display device 118 displays the decoded pictures of the decoded video data to the user. Display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

Although not shown in FIG. 1 , in some examples video transcoder 200 and video decoder 300 may each be integrated with an audio encoder and/or audio decoder, and may include appropriate MUX-DEMUX units or other Hardware and/or software to process multiplexed streams including both audio and video in the common data stream. If applicable, the MUX-DEMUX unit may follow the ITU H.223 multiplexer protocol or other protocols such as User Datagram Protocol (UDP).

Each of video transcoder 200 and video decoder 300 may be implemented as any of a variety of suitable encoder and/or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), individual logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in hardware to perform the present invention content technology. Each of the video transcoder 200 and the video decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as a combined encoding in the corresponding device part of the codec/decoder (CODEC). Devices including video transcoder 200 and/or video decoder 300 may include integrated circuits, microprocessors, and/or wireless communication devices (such as cellular phones).

Video transcoder 200 and video decoder 300 may be based on video coding standards such as ITU-T H.265 (also known as High Efficiency Video Coding (HEVC) standard) or extensions thereof such as multi-view and/or or Scalable Video Decoding Extension)) to operate. Alternatively or additionally, video transcoder 200 and video decoder 300 may be based on other proprietary or industry standards such as ITU-T H.266 (also known as Universal Video Coding (VVC) and extensions thereto ( Such as the expansion of screen content or high dynamic range)). A draft of the VVC standard is described in the following documents: Bross et al., "Versatile Video Coding Draft 10", ITU-T SG 16 WP 3 and ISO/IEC Joint Video Expert Team (JVET) of JTC 1/SC 29/WG 11, 18th meeting: via conference call, 22 June - 1 July 2020, JVET-S2001-v17 (hereinafter referred to as " VVC draft 10"). However, the techniques in this case are not limited to any particular coding standard.

In general, video transcoder 200 and video decoder 300 may perform block-based coding of pictures. The term "block" generally refers to a structure comprising data to be processed (eg, to be encoded, decoded, or otherwise used in an encoding and/or decoding process). For example, a block may include a two-dimensional matrix of samples of luma and/or chrominance data. Generally, the video transcoder 200 and the video decoder 300 can decode video data expressed in YUV (eg, Y, Cb, Cr) format. That is, instead of decoding sampled red, green, and blue (RGB) data for a picture, video transcoder 200 and video decoder 300 can decode luma and chrominance components, where chrominance The components may include both red hue and blue hue chroma components. In some examples, video transcoder 200 converts received RGB formatted data to a YUV representation prior to encoding, and video decoder 300 converts the YUV representation to RGB format. Alternatively, preprocessing and postprocessing units (not shown) may perform these transformations.

In general terms, this disclosure may relate to the decoding (eg, encoding and decoding) of pictures to include procedures for encoding or decoding data of the pictures. Similarly, the present disclosure may relate to the coding of blocks of pictures to include procedures for encoding or decoding (eg, predictive and/or residual coding) material for the blocks. An encoded video bitstream typically includes a series of values for syntax elements representing coding decisions (eg, coding modes) and partitioning a picture into blocks. Therefore, references to coding a picture or block should generally be understood as coding the values of the syntax elements used to form the picture or block. In this case, the current block or current picture generally refers to a block or picture that is currently being encoded or decoded, rather than a block or picture that has been decoded or a block or picture that has not yet been decoded.

HEVC defines various blocks, including Coding Units (CUs), Prediction Units (PUs), and Transform Units (TUs). According to HEVC, a video coder, such as video transcoder 200 , partitions coding tree units (CTUs) into CUs according to a quadtree structure. That is, the video coder partitions the CTU and CU into four equal, non-overlapping squares, and each node of the quadtree has either zero or four child nodes. A node without child nodes may be referred to as a "leaf node," and a CU of such a leaf node may include one or more PUs and/or one or more TUs. Video decoders can further partition PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents a partition of a TU. In HEVC, PU represents inter-frame prediction data, and TU represents residual data. An intra-predicted CU includes intra-prediction information, such as an intra-mode indication.

As another example, video transcoder 200 and video decoder 300 may be configured to operate according to VVC. According to VVC, a video coder such as video transcoder 200 partitions a picture into a plurality of CTUs. The video transcoder 200 may partition a CTU according to a tree structure such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concept of multiple partition types, such as partitions between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level divided according to quadtree division, and a second level divided according to binary tree division. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to CUs.

In an MTT partition structure, blocks can be partitioned using quadtree (QT) partitions, binary tree (BT) partitions, and one or more types of ternary tree (TT) (also known as ternary tree (TT)) partitions. segmentation. A ternary tree or ternary tree partition is a partition in which a block is divided into three sub-blocks. In some examples, ternary tree or ternary tree partitioning divides a block into three sub-blocks without dividing the original block via the center. Segmentation types in MTT (eg, QT, BT, and TT) can be symmetric or asymmetric.

In some examples, video transcoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chrominance components, while in other examples video transcoder 200 and video decoding The processor 300 may use two or more QTBT or MTT structures, such as one QTBT/MTT structure for the luma component and another QTBT/MTT structure for the two chroma components (or a QTBT/MTT structure for the corresponding chroma components two QTBT/MTT structures).

Video transcoder 200 and video decoder 300 may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, or other partitioning structures per HEVC. For purposes of explanation, a description of the techniques in this case is provided with respect to QTBT segmentation. However, it should be understood that the techniques disclosed herein can also be applied to video decoders configured to use quadtree partitioning, or to use other types of partitioning as well.

In some examples, a CTU includes a coding tree block (CTB) of luma samples for a picture with three sample arrays, two corresponding CTBs of chroma samples, or a monochrome picture or coded using three separate color planes The CTB of the samples of the picture and the syntax structure used to code the samples. A CTB may be a block of NxN samples for some value of N, such that dividing a component into a CTB is a partition. Components are either an array from three arrays (luminance and two chrominances) containing pictures in 4:2:0, 4:2:2, or 4:4:4 color format or a single array from one of the three arrays A sample, or an array consisting of pictures in monochrome format, or a single sample of the array. In some examples, a coding block is an MxN block of samples for some values of M and N, such that dividing the CTB into coding blocks is a partition.

Blocks (eg, CTUs or CUs) can be grouped in a picture in various ways. As an example, a brick may represent a rectangular area of CTU rows within a particular tile in a picture. A tile may be a rectangular area of a CTU within a specific tile column and a specific tile row in a picture. A tile column represents a rectangular area of a CTU with a height equal to that of a picture and a width specified by a syntax element (eg, such as in a picture parameter set). A tile row represents a rectangular area of a CTU with a height specified by a syntax element (eg, such as in a picture parameter set) and a width equal to the width of the picture.

In some examples, a tile can be partitioned into multiple bricks, each brick can include one or more rows of CTUs within the tile. A tile that is not divided into multiple bricks may also be called a brick. However, bricks that are a true subset of tiles may not be called tiles.

The bricks in the picture can also be arranged in slices. A slice may be an integer number of bricks of a picture that may be uniquely contained within a single Network Abstraction Layer (NAL) unit. In some examples, a slice includes multiple full tiles or a contiguous sequence of full bricks that includes only one tile.

This document uses "NxN" and "N by N" interchangeably to refer to the sample size of a block (such as a CU or other video block) in vertical and horizontal dimensions, eg, 16x16 samples or 16 by 16 samples. Typically, a 16x16 CU has 16 samples vertically (y=16) and will have 16 samples horizontally (x=16). Likewise, an NxN CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily need to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may include NxM samples, where M is not necessarily equal to N.

Video transcoder 200 encodes video data representing prediction and/or residual information, among other information, for a CU. The prediction information indicates how the CU will be predicted in order to form the prediction block for the CU. The residual information typically represents the sample-by-sample difference between the samples of the CU before encoding and the prediction block.

In order to predict a CU, the video transcoder 200 may generally form a prediction block for the CU through inter-prediction or intra-prediction. Inter prediction generally refers to predicting a CU based on previously coded data for a picture, while intra prediction generally refers to predicting a CU based on previously coded data for the same picture. To perform inter-frame prediction, video transcoder 200 may use one or more motion vectors to generate a prediction block. Video transcoder 200 can typically perform a motion search to identify reference blocks that closely match the CU, for example, with a difference pattern between the CU and the reference block. Video transcoder 200 may calculate the difference metric using sum of absolute difference (SAD), sum of square difference (SSD), mean absolute difference (MAD), mean square difference (MSD), or other such difference calculations to determine the reference block Whether it closely matches the current CU. In some examples, video transcoder 200 may use uni-prediction or bi-prediction to predict the current CU.

Some examples of VVC also provide an affine motion compensation mode, which can be considered an inter-frame prediction mode. In the affine motion compensation mode, video transcoder 200 may determine two or more motion vectors representing non-translational motion, such as zooming in or out, rotation, perspective motion, or other irregular motion types.

In order to perform intra prediction, the video transcoder 200 can select an intra prediction mode to generate a prediction block. Some examples of VVC provide sixty-seven intra-frame prediction modes, including various directional modes, as well as planar and DC modes. In general, video transcoder 200 selects an intra prediction mode that describes neighboring samples of the current block (eg, a block of a CU) from which samples of the current block (eg, a block of a CU) are to be predicted. Assuming video transcoder 200 codes CTUs and CUs in raster scan order (left-to-right, top-to-bottom), such samples may typically be above the current block in the same picture as the current block , Top Left, or Left.

The video transcoder 200 encodes data representing the prediction mode for the current block. For example, for an inter prediction mode, video transcoder 200 may encode data indicating which of various available inter prediction modes to use and motion information for the corresponding mode. For uni-directional or bi-directional inter-frame prediction, for example, video transcoder 200 may use advanced motion vector prediction (AMVP) or merge mode to encode motion vectors. Video transcoder 200 may use a similar scheme to encode motion vectors for affine motion compensation mode.

After prediction, such as intra prediction or inter prediction for a block, video transcoder 200 may compute residual data for the block. Residual data, such as a residual block, represents the sample-by-sample difference between a block and the prediction block for that block, which was formed using the corresponding prediction mode. Video transcoder 200 may apply one or more transforms to the residual block to generate transformed data in the transform domain rather than the sample domain. For example, video transcoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. In addition, the video transcoder 200 may apply a secondary transform after the first transform, such as mode-dependent non-separable secondary transform (MDNSST), signal-dependent transform, Karhunen-Loeve transform (KLT), and so on. Video transcoder 200 generates transform coefficients after applying one or more transforms.

As before, following any transform to generate transform coefficients, video transcoder 200 may perform quantization of the transform coefficients. Quantization generally refers to a procedure in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, thereby providing further compression. By performing a quantization process, video transcoder 200 may reduce the bit depth associated with some or all transform coefficients. For example, video transcoder 200 may round down an n- bit value to an m -bit value during quantization, where n is greater than m . In some examples, to perform quantization, video transcoder 200 may perform a bitwise right shift of the value to be quantized.

After quantization, video transcoder 200 may scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including quantized transform coefficients. The sweep can be designed to place higher energy (and therefore lower frequency) transform coefficients at the front of the vector, and lower energy (and thus higher frequency) transform coefficients at the back of the vector. In some examples, the video transcoder 200 may scan the quantized transform coefficients using a predefined scan order to generate a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other examples, video transcoder 200 may perform self-adjusting scans. After scanning the quantized transform coefficients to form a 1D vector, the video transcoder 200 may entropy code the 1D vector, for example, with Context Self-Adjusting Binary Arithmetic Coding (CABAC). The video transcoder 200 may also entropy encode values of syntax elements used to describe metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

To perform CABAC, video transcoder 200 may assign contexts within a context model to symbols to be transmitted. The context may relate, for example, to whether a neighboring value of a symbol is zero-valued. Probabilistic decisions may be based on the context assigned to the symbol.

The video transcoder 200 can also generate syntax data (such as block-based syntax data, picture-based syntax data, and sequence-based syntax data) for the video decoder 300, for example, in the picture header, block header, slice header, or other syntax data (such as sequence parameter set (SPS), picture parameter set (PPS) or video parameter set (VPS)). Likewise, the video decoder 300 can decode such syntax data to determine how to decode the corresponding video data.

In this way, video transcoder 200 may generate a bitstream that includes encoded video data, e.g., describing the partitioning of a picture into blocks (e.g., CUs) and the prediction and/or residual for that block. Syntactic elements for information. Finally, the video decoder 300 can receive the bit stream and decode the encoded video data.

Generally, the video decoder 300 performs the reverse process of the process performed by the video transcoder 200 to decode the encoded video data of the bitstream. For example, video decoder 300 may use CABAC to decode the values of the syntax elements for the bitstream in a substantially similar, but reversed, manner to the CABAC encoding process of video transcoder 200 . The syntax elements may define partition information for partitioning a picture into CTUs, and partitioning each CTU according to a corresponding partition structure (such as a QTBT structure) to define the CUs of the CTUs. Syntax elements may also define prediction and residual information for a block (eg, CU) of video data.

The residual information may be represented by, for example, quantized transform coefficients. Video decoder 300 may inverse quantize and inverse transform the quantized transform coefficients of a block to reproduce a residual block for the block. Video decoder 300 uses the signaled prediction mode (intra prediction or inter prediction) and associated prediction information (eg, motion information for inter prediction) to form a prediction block for the block. Video decoder 300 may then combine the predicted and residual blocks (on a sample-by-sample basis) to reconstruct the original block. Video decoder 300 may perform additional processing, such as performing a deblocking procedure to reduce visual artifacts along block boundaries.

Broadly speaking, the content of this case may involve "signaling" certain information (such as grammatical elements). The term "signaling" may generally refer to the transmission of values for syntax elements and/or other data used to decode encoded video data. That is, video transcoder 200 may signal the values for the syntax elements in the bitstream. Typically, signaling represents the generation of values in the bitstream. As previously mentioned, the source device 102 may transmit the bitstream to the destination device substantially instantaneously or not (such as may occur when syntax elements are stored to the depository device 112 for later retrieval by the destination device 116) 116.

2A and 2B are conceptual diagrams illustrating an example quadtree-binary-tree (QTBT) structure 130 and corresponding CTU 132 . Solid lines indicate quadtree splits, while dashed lines indicate binary tree splits. In each split (i.e. non-leaf) node of the binary tree, a flag is signaled to indicate which type of split to use (i.e., horizontal or vertical), where in this instance 0 indicates a horizontal split and 1 indicates a vertical split. For quadtree splitting, since the quadtree node splits the block horizontally and vertically into 4 sub-blocks of equal size, there is no need to indicate the splitting type. Thus, video transcoder 200 can encode, and video decoder 300 can decode, syntax elements (such as split information ), and syntax elements (such as split information) for the predicted tree level (ie dashed line) of the QTBT structure 130 . Video transcoder 200 may encode video data, such as prediction and transform data, for a CU represented by a terminal leaf node of QTBT structure 130 , and video decoder 300 may decode the video data.

In general, the CTU 132 of FIG. 2B may be associated with parameters defining the size of the blocks corresponding to the nodes of the QTBT structure 130 at the first and second levels. These parameters may include the CTU size (representing the size of the CTU 132 in the sample), the minimum quadtree size (MinQTSize, which represents the minimum allowable quadtree leaf node size), the maximum binary tree size (MaxBTSize, which represents the maximum allowable binary tree root node size ), the maximum binary tree depth (MaxBTDepth, which represents the maximum allowable binary tree depth), and the minimum binary tree size (MinBTSize, which represents the minimum allowable binary tree leaf node size).

The root node of the QTBT structure corresponding to the CTU may have four child nodes at the first level of the QTBT structure, and each child node may be divided according to quadtree division. That is, nodes at the first level are either leaf nodes (no child nodes) or have four child nodes. The example of QTBT structure 130 represents such nodes as including parent nodes and child nodes with solid line branches. If the first-level nodes are not larger than the maximum allowable binary tree root node size (MaxBTSize), then these nodes can be further divided through the corresponding binary tree. The binary tree split of a node can be iterated until the node resulting from the split reaches the minimum allowable binary tree leaf node size (MinBTSize) or the maximum allowable binary tree depth (MaxBTDepth). The example of QTBT structure 130 shows such nodes as having dashed branches. The binary tree leaf nodes are called CUs, which are used for prediction (eg, intra-picture or inter-picture prediction) and transformation without any further partitioning. As discussed above, a CU may also be referred to as a "video block" or "block."

In an instance of the QTBT partition structure, the CTU size is set to 128x128 (a luma sample and two corresponding 64x64 chroma samples), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to produce quadtree leaf nodes. Quadtree leaf nodes can have a size from 16x16 (i.e. MinQTSize) to 128x128 (i.e. CTU size). If the quadtree leaf node is 128x128, the leaf quadtree node may not be further split by the binary tree since this size exceeds MaxBTSize (ie, 64x64 in this example). Otherwise, the quadtree leaf nodes can be further split by the binary tree. Therefore, the quadtree leaf node is also the root node for the binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (4 in this instance), no further splits are allowed. A binary tree node having a width equal to MinBTSize (4 in this example) means that no further vertical splits (ie, divisions of width) are allowed for that binary tree node. Similarly, a binary tree node having a height equal to MinBTSize means that no further horizontal splits (ie, division of heights) are allowed for that binary tree node. As before, the leaf nodes of the binary tree are called CUs and are further processed according to prediction and transformation without further partitioning.

As mentioned above, the video transcoder 200 and the video decoder 300 may be configured to perform motion vector prediction. In HEVC, for prediction units (PUs), there are two inter-frame prediction modes called merge (skipping is considered a special case of merge) and AMVP mode. In AMVP and merge modes, video transcoder 200 and video decoder 300 maintain motion vector (MV) candidate lists for multiple motion vector predictors. The motion vector of the current PU and the reference index in merge mode are generated by selecting one candidate from the MV candidate list.

In an HEVC implementation, the MV candidate list contains at most five candidates for merge mode and two candidates for AMVP mode. Merge candidates may contain sets of motion information, eg, motion vectors corresponding to both reference picture lists (list 0 and list 1 ) and reference indices. By receiving the merge candidates identified by the merge indices, the video decoder 300 determines the reference pictures and associated motion vectors for predicting the current block. In another aspect, in AMVP mode for each potential prediction direction from list 0 or list 1, video decoder 300 receives the MV predictor (MVP) index of the MV candidate list, since AMVP candidates only contain motion vectors. The video decoder 300 additionally receives a motion vector difference (MVD) and a reference index to unambiguously identify the reference picture. In AMVP mode, the predicted motion vectors can be further refined.

Candidates for both modes can be similarly derived from the same spatial and temporal neighboring blocks. In HEVC, as shown in FIGS. 3A and 3B , for a specific PU (PU ₀ ), video transcoder 200 and video decoder 300 can derive spatial MV candidates from neighboring blocks, although the techniques used to generate candidates from blocks It is different for Merge and AMVP modes.

FIG. 3A is a conceptual diagram illustrating spatial neighbor candidates of a block 140 for a merge mode. FIG. 3B is a conceptual diagram illustrating spatial neighbor candidates for a block 142 in AMVP mode. In merge mode, video transcoder 200 and video decoder 300 can derive up to four spatial MV candidates in the sequence shown in FIG. 3A . The order is as follows: left block (0, A1), upper block (1, B1), upper right block (2, B0), lower left block (3, A0) and upper left block (4, B2).

In AMVP mode, video transcoder 200 and video decoder 300 may divide adjacent blocks into two groups: the left group includes blocks 0 and 1, and the upper group includes blocks 2, 3 and 4, as shown in FIG. 3B. For each group, potential candidates in neighboring blocks that refer to the same reference picture as the reference picture indicated by the signaled reference index have the highest priority selected to form the final candidates for the group. All neighboring blocks may not contain motion vectors pointing to the same reference picture. Therefore, if no such candidate can be found, video transcoder 200 and video decoder 300 may scale the first available candidate to form a final candidate. Therefore, the temporal distance difference can be compensated.

Temporal motion vector prediction in HEVC will now be discussed. Video transcoder 200 and video decoder 300 may be configured to add temporal motion vector predictor (TMVP) candidates (if enabled and available) to the MV candidate list after spatial motion vector candidates. The procedure for motion vector derivation for TMVP candidates is the same for merge and AMVP modes. However, in HEVC, the target reference index for TMVP candidates in merge mode is set to 0.

FIG. 4A illustrates example TMVP candidates for block 154 (PUO), and FIG. 4B illustrates motion vector scaling procedure 156 . The main block location for TMVP candidate derivation is the lower right block outside the co-located PU. This candidate is shown as block "T" in FIG. 4A. The location of block T is used to compensate for the bias to the above and left blocks used to generate spatially neighboring candidates. However, if the block is located outside the current CTB row or motion information is not available, the block will be replaced by the center block of the PU.

Video transcoder 200 and video decoder 300 may derive motion vectors for TMVP candidates from co-located PUs located in co-located pictures, as indicated at the slice level. A motion vector for a co-located PU is called a co-located MV. Similar to the temporal direct mode in AVC, to derive TMVP candidate motion vectors, the co-located MVs can be scaled to compensate for the temporal distance difference, as shown in Figure 4B.

Other aspects of motion prediction in HEVC that relate to the techniques described herein will now be described. Video transcoder 200 and video decoder 300 may be configured to perform motion vector scaling. The value of the motion vector is assumed to be proportional to the distance of the picture in presentation time. A motion vector associates two pictures (ie, a reference picture and a picture containing the motion vector (ie, a containing picture)). When predicting another motion vector using a motion vector, the distance of the containing picture and the reference picture is calculated based on a picture order count (POC) value.

For a motion vector to be predicted, the associated containing picture may be different from the reference picture. Therefore, the video transcoder 200 and the video decoder 300 can calculate the new distance based on the POC. Video transcoder 200 and video decoder 300 can scale motion vectors based on these two POC distances. For spatially adjacent candidates, the containing pictures for the two motion vectors are the same, but the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporal neighbor candidates.

The video transcoder 200 and the video decoder 300 may be configured to perform manual motion vector candidate generation. If the motion vector candidate list is not complete, artificial motion vector candidates are generated and inserted at the end of the list until the list is full.

In merge mode, there are two types of artificial MV candidates: combined candidates derived only for B slices; and zero candidates for AMVP only if the first type does not provide enough artificial candidates. For each pair of candidates already in the candidate list and having the necessary motion information, a bidirectional combination is derived via the combination of the motion vector of the first candidate referencing a picture in list 0 and the motion vector of the second candidate referencing a picture in list 1 motion vector candidates.

Video transcoder 200 and video decoder 300 may be configured to perform a pruning procedure for candidate insertion. Candidates from different blocks may happen to be the same, which reduces the efficiency of merging/AMVP candidate lists. Apply a trimmer to fix this. When implementing the pruning procedure, the video transcoder 200 or the video decoder 300 compares one candidate in the current candidate list with other candidates to avoid inserting the same candidate to a certain extent. To reduce complexity, only a limited number of pruners are applied, rather than comparing each potential pruner with all other existing pruners.

The video transcoder 200 and the video decoder 300 may be configured to perform template matching prediction. Frame Match Prediction is a special merge mode based on Frame Rate Up-Conversion (FRUC) technology. In this mode, part of the motion information of the block is not signaled but derived at the decoder side. Template matching can be applied to both AMVP mode and general merge mode. In AMVP mode, MVP candidate selection is decided based on template matching to select the MVP candidate that achieves the minimum difference between the current block template and the reference block template. In normal merge mode, signal the template matching mode flag to indicate that template matching is used. Then, the video transcoder 200 and the video decoder 300 can apply template matching to the merge candidates indicated by the merge indexes for MV refinement.

As shown in FIG. 5 , template matching is used to derive the motion information of the current CU 160 by looking at the closest match between the current template 162 in the current picture and the reference template 164 (same size as the template) in the reference picture. In case AMVP candidates are selected based on initial matching errors, video transcoder 200 and video decoder 300 may use template matching to refine the MVP. In the case of a merge candidate indicated by the signaled merge index, the video transcoder 200 and the video decoder 300 may be configured to independently refine the MVs corresponding to L0 and L1 via template matching, and then based on More accurate MVs to further refine less accurate MVs.

和

分別指示當前測試MV和初始MV（亦即，AMVP模式下的MVP候選或合併模式下的合併運動）。SAD被用作範本匹配的匹配成本。 The video transcoder 200 and the video decoder 300 can be configured to determine the cost function. Motion compensated interpolation is required when motion vectors point to fractional sample locations. In order to reduce the complexity, bilinear interpolation can be used instead of the general 8-point DCT-IF interpolation for template matching to generate a template on the reference picture. The matching cost C of template matching is calculated as follows: C = SAD + w * ( | MV _x - MV _x ^s | + | MV _y - MV _y ^s | ), where w is a weighting factor, which can be set as an integer, such as 0, 1, 2, 3 or 4,

with

Indicates the current test MV and initial MV (ie, MVP candidate in AMVP mode or merge motion in merge mode), respectively. SAD is used as the matching cost for template matching.

When template matching is used, video transcoder 200 and video decoder 300 may be configured to refine motion using only luma samples. The derived motion can be used for both luma and chroma for motion compensated (MC) inter-frame prediction. After deciding the MV, a final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chrominance.

The video transcoder 200 and the video decoder 300 can be configured to determine and implement a search procedure. MV refinement is a pattern-based MV search with criteria and hierarchy of template matching costs. Two search modes are supported for MV refinement: diamond search and cross search. The hierarchy specifies the iterative procedure for refining the MVs, starting with coarse MVD precision (eg, quarter primitive) and ending with fine MVD precision (eg, 1/8 primitive). MVs are sought directly using diamond mode to sample MVD precision at quarter luma, then sample MVD precision with interleave mode to quarter luma, and then sample MVD refinement with interleave mode to eighth luma. The search range for MV refinement can be set equal to (-8, +8) luma samples around the original MV. When the current block is bi-predictive, two MVs are refined independently, and the best of them (in terms of matching cost) is then set as a priori to further refine the other MV with the BCW weight value.

Video transcoder 200 and video decoder 300 may be configured to perform affine prediction. In HEVC, only translational motion models are applied for motion compensated prediction (MCP). In the real world, however, there are many kinds of motion, such as zoom in/out, rotation, perspective motion, and other irregular motions. In VTM-6, block-based affine transform motion compensated prediction is applied. As shown in Figure 6A, the affine motion field of a block is described by the motion information of two control points (170A and 170B), also known as a four-parameter model. As shown in FIG. 6B, the affine motion field of a block is described by motion information of three control points (172A-172C) and three control point motion vectors, which is also called a 6-parameter model.

(2-1) For a 4-parameter affine motion model, the motion vector at the sampled position ( x, y ) in the block is derived as:

(2-1)

(2-2) For a 6-parameter affine motion model, the motion vector at the sampled position ( x, y ) in the block is derived as:

(2-2)

In the above equation, ( mv _0x , mv _0y ) represents the upper-left CPMV, and ( mv _1x , mv _1y ) and ( mv _2x , mv _2y ) represent the upper-right and lower-left CPMVs, respectively.

To simplify motion compensated prediction, video transcoder 200 and video decoder 300 may be configured to apply block-based affine transform prediction. Figure 7 illustrates block 170, which is a 16x16 luma block comprising sixteen 4x4 luma sub-blocks. In order to derive the motion vector for each 4×4 luma sub-block, the video transcoder 200 and the video decoder 300 calculate the motion vector of the center sample of each sub-block according to the above equation, as shown in FIG. 7, and is Rounds to 1/16 fractional precision. Arrows 172A and 172B identify two of the sixteen motion vectors for the sub-block. The other 14 arrows also correspond to motion vectors, but are not labeled in FIG. 7 . A motion compensated interpolation filter is applied to generate a prediction for each sub-block using the derived motion vectors. The sub-block size of the chroma component is also set to 4x4. The MV of a 4x4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4x4 luma sub-blocks.

Video transcoder 200 and video decoder 300 may be configured to perform prediction refinement using affine mode optical flow. Predictive Refinement Using Optical Flow (PROF) is used to refine sub-block based affine motion compensated predictions without increasing the memory access bandwidth for motion compensation. In VVC, after sub-block based affine motion compensation is performed, the luma prediction samples are refined by adding a difference derived via the optical flow equation.

In an example implementation of PROF, video decoder 300 may be configured to perform the following four steps: Step 1) Perform sub-block based affine MC to generate sub-block prediction I ( i , j ). Step 2) Compute the spatial gradients g _x ( i , j ) and g _y ( i , j ) of the operator block prediction at each sampling location using a 3-tap filter [−1, 0, 1]. The gradient calculation is exactly the same as the gradient calculation in BDOF. g _x ( i , j ) = ( I ( i + 1, j ) ＞＞ shift1) - ( I ( i - 1, j ) ＞＞ shift1), g _y ( i , j ) = ( I ( i , j + 1) ＞＞ shift1) - ( I ( i , j - 1) ＞＞ shift1), where shift1 is used to control the accuracy of the gradient. Subblock (eg, 4x4) predictions are extended by one sample on each side of the gradient computation. To avoid extra memory bandwidth and extra interpolation calculations, those outsamples on the outbound boundaries are copied from the nearest integer primitive position in the reference picture. Step 3) Compute the brightness prediction refinement via the following optical flow equation: ∆ I ( i , j ) = g _x ( i , j ) * ∆ _x ( i , j ) + g _y ( i , j ) * ∆ _y ( i , j ), where ∆ v ( i , j ) is the sub-block of the sample MV (denoted by v ( i , j )) computed for the sample location ( i , j ) and the sub-block to which the sample ( i , j ) belongs The difference between the MVs is shown in Figure 8. Quantize ∆v ( i , j ) with 1/32 luma sampling precision. FIG. 8 illustrates sub-block MV V _SB and primitive ∆ v ( i , j ) (arrow 190 ).

Since the affine model parameters and sampling locations relative to the sub-patch centers do not change between sub-plots, ∆v ( i , j ) can be computed for the first sub-patch, and ∆v ( i , j ) can be reused for other subblocks in the same CU. Let dx ( i , j ) and dy ( i , j ) be the horizontal and vertical offsets from the sampling position ( i , j ) to the center of the sub-block ( x _SB , y _SB ), then it can be derived via the following equation ∆ v ( i , j ): dx ( i , j ) = i - x _SB dy ( i , j ) = j - y _SB ∆ v _x ( i , j ) = C * dx ( i , j ) + D * dy ( i , j ) ∆ v _y ( i , j ) = E * dx ( i , j ) + F * dy ( i , j )

To maintain accuracy, the center of the sub-block ( x _SB , y _SB ) is calculated as ( ( W _SB − 1 )/2, ( H _SB − 1 ) / 2 ), where _WSB and H _SB are the sub-block width and high.

For a 4-parameter affine model, C = F = ( v _{1 x} – v _{0 x} ) / w , E = - D = ( v _{1 y} – v _{0 y} ) / w .

For a 6-parameter affine model, C = ( v _{1 x} – v _{0 x} ) / w , D = ( v _{2 x} – v _{0 x} ) / h , E = ( v _{1 y} – v _{0 y} ) / w , F = ( v _2y – v _{0 y} ) / h , where ( v _{0 x} , v _{0 y} ), ( v _{1 x} , v _{1 y} ) and ( v _{2 x} , v _{2 y} ) are the upper left, upper right and lower left The control point motion vector of the corner, w and h are the width and height of the CU. Step 4) Finally, the luma prediction refinement ∆I ( i , j ) is added to the subblock prediction I ( i , j ). The final forecast I ' is produced by the following equation: I' ( i , j ) = I ( i , j ) + ∆ I ( i , j ).

For an affine-coded CU, PROF is not applied in two cases: 1) all control points MV are the same, which indicates that the CU has only translational motion; 2) the affine motion parameter is larger than the specified limit, because the Affine MC is downgraded to CU-based MC to avoid large memory access bandwidth requirements.

The video transcoder 200 and the video decoder 300 can be configured to apply a fast encoding procedure to reduce the encoding complexity of the affine motion estimation using PROF. In the following two cases, PROF is not applied in the affine motion estimation stage: a) If the CU is not the root block, and its parent block is not decoded with affine mode as its best mode, since the current CU uses The possibility of affine mode as the best mode is very low, so PROF is not applied; b) If the size of the four affine parameters (C, D, E, F) is less than the predefined threshold, and the current picture is not low-latency image, PROF is not applied because for this case the improvement introduced by PROF is minimal. In this way, affine motion estimation with PROF can be accelerated.

The prior art has several potential problems. For blocks, the signaling management burden of CPMV may be significantly increased when compared to that of the translation model for inter-frame prediction. Therefore, decoder-side refinement for CPMV can improve CPMV accuracy and reduce signaling management burden. This case describes techniques that may address some of these problems.

Affine Prediction Based on Template Matching (hereinafter referred to as AffTM) is a decoder-side inter-frame prediction mode for refining CPMV of affinely decoded blocks. Similar to template matching, as mentioned above, the video decoder 300 can determine an initial reference template block based on the initially determined CPMV, and then search for other reference templates with reduced matching costs within the search area. Then, the video decoder 300 can determine the best CPMV set to replace the original CPMV.

The video transcoder 200 and the video decoder 300 can be configured to determine a reference template block. The sampling of the reference template block is generated on a sub-block basis using the CPMV-derived motion field. Under the assumption that the current block and the corresponding current template block 192 are located in the same affine motion field, the video transcoder 200 and the video decoder 300 can use equation (2-1) or (2-2) to determine the sub-block ( For example, the MVs of A ₀ , A ₁ , ..., A _n-1 and L ₀ , L ₁ , ..., L _n-1 ) in FIG. 9A on the current template block 192, wherein the sampling position ( x , y ) is The centroid of each corresponding subblock. Subsequently, the video transcoder 200 and the video decoder 300 acquire or interpolate samples of the sub-blocks of the reference template block based on the corresponding sub-block MV. As shown by reference example 194A in example FIG. 9B, the reference example sub-block does not need to be immediately adjacent to any boundary sub-block of the prediction block. In addition, the interpolation filter used to generate sub-block samples on the reference sample block can be one or more of the following: No filter (thus clipping or rounding the sub-block MV to integer precision), 2-tap bilinear filter, 6-tap DCTIF (as in AVC), 8-tap DCTIF (as in HEVC or VVC), or switchable filter (as in VVC) .

In another example, as shown in the reference template 194B in FIG. 9C , the reference template sub-block may be adjacent to the boundary sub-block of the corresponding prediction block. Therefore, the MV of each sub-block (A ₀ , . . . , _n-1 and L ₀ , . . . , _n-1 ) is the same as the corresponding immediately adjacent sub-block located on the boundary of the current block.

In another example, the MVs of sub-blocks other than A ₀ and L ₀ on the current sample block may be at the sampling position ( x , y ) is calculated via equation (2-1) or (2-2). For A ₀ and L ₀ , if both A ₀ and L ₀ exist, the sampling position ( x , y ) can be (0,0); if only A ₀ exists, the sampling position ( x , y ) can be A ₀ and the centroid between the first sub-block on the current block; if only L ₀ exists, the sampling position ( x , y ) can be the centroid between L ₀ and the first sub-block on the current block.

In another example, video transcoder 200 and video decoder 300 may be configured to apply PROF to reference template blocks.

In another example, when all CPMVs are identical to each other, the video transcoder 200 and the video decoder 300 can replace the AffTM prediction procedure with the aforementioned general block-based template matching prediction procedure. One of the CPMVs can be regarded as the initial MV and used for block-based template matching.

In another example, when all initial CPMVs are identical to each other, the video transcoder 200 and video decoder 300 may be configured to perform the general block-based template matching as described above before AWTM to refine the initial CPMV. One of the CPMVs can be considered as the initial MV for the general template matching procedure. This example can be further extended to translational model search as described above.

The video transcoder 200 and the video decoder 300 can be configured to perform a search procedure. This section provides several search procedures for AffTM. Without loss of generality, all algorithms are provided using a 6-parameter affine model. These algorithms can be directly translated for 4-parameter affine mode by simply removing the bottom left CPMV from the description. The search range can be predefined or signaled, eg, ±2, ±4, ±6, ±8 picture elements. The initial search point for the CPMV may be any of the following: an AMVP candidate, a CPMV corresponding to a reference picture list of a merge candidate, or a CPMV corresponding to a reference picture list of a block.

Video transcoder 200 and video decoder 300 may be configured to perform square search. Square seeking applies a square pattern to sequentially refine the CU's CPMVs, one vector at a time, starting with the upper left CPMV, followed by the upper right CPMV, and ending at the lower left CPMV. It is worth noting that the bottom left CPMV is searched to have this third CPMV only if the CU has a 6-parameter model. The square search pattern can be specified as a series of incremental motion vectors, dMv = {(0,0), (-1,1), (0,1), (1,1), (1,0), (1 ,-1), (0,-1), (-1,-1), (-1,0)} or any other sequence based on these nine incremental motion vectors. Without loss of generality, this section takes the above dMv as an example, where the initial search step s ₀ and the minimum search step s _min are Based on indication of AMVR index or 1/16. The value of s ₀ may be set equal to or greater than s _min , and s _i+1 is set equal to s _i for all i ∈ {0, 1, . . . , min }. The square search procedure is a 7-step procedure specified as follows: 1. Given a set of search steps { s ₀ , s ₁ , …, s _min }, dMv and mv ₀ ^{( 0)} , mv ₁ ⁽⁰⁾ and mv ₂ ⁽⁰⁾ , the search procedure starts from the repeated operation of i =0. 2. For s _i , dMv and {mv ₀ ^{( i )} , mv ₁ ^{( i )} , mv ₂ ^{( i )} }, the search subroutine starts the sequential procedure to begin with the search for mv ₀ ^{( i )} , followed by mv ₁ ^{( i )} , and finally ends with mv ₁ ^{( i )} (note that in some instances, the order may be {mv ₂ ^{( i )} , mv ₁ ^{( i )} , mv ₀ ^{( i )} }). 3. For s _i , dMv and mv ₀ ^{( i )} , the search subroutine separately calculates the corresponding template matching costs for all these CPMV sets: S={mv ₀ ^{( i )} + d * s _i , mv ₁ ^{( i )} , mv ₂ ^{( i )} , for all d∈dMv}. The search subroutine can be expressed as mv ₀ ^{( i )*} = mv ₀ ^{( i )} + argmin _d {cost(S ₀ ), cost(S ₁ ), …, cost(S ₈ )} * s _i . 4. Similar to step 3, the search subroutine calculates the corresponding template matching cost for all d∈dMv} for S={mv ₀ ^{( i )*} , mv ₁ ^{( i )} + d * s _i , mv ₂ ^{( i )} , and the best result is denoted as mv ₁ ^{( i )*} . 5. Similar to step 3, the search subroutine calculates the corresponding template matching for all d∈dMv} for S={mv ₀ ^{( i )*} , mv ₁ ^{( i )*} , mv ₂ ^{( i )} + d * s _i cost, and the best result is expressed as mv ₂ ^{( i )*} . 6. Before searching all CPMVs in steps 3-5, the output of the search subroutine is {mv ₀ ^{( i )*} , mv ₁ ^{( i )*} , mv ₂ ^{( i )*} }. ·If the search program accesses step 6 multiple times on the predefined threshold when the search step size is si , the search program will be {mv ₀ ^{( i +1)} _, mv ₁ ^{( i +1)} , mv ₂ ^{( i +1)} } is set equal to the subroutine output, and go to step 7. · Otherwise, if the subroutine output is exactly the same as {mv ₀ ^{( i )} , mv ₁ ^{( i )} , mv ₂ ^{( i )} }, the search routine will be {mv ₀ ^{( i +1)} , mv ₁ ^{( i +1)} , mv ₂ ^{( i +1 )} } is set equal to the subroutine output, and go to step 7. ·Otherwise (if the subroutine output is not exactly the same as {mv ₀ ^{( i )} , mv ₁ ^{( i )} , mv ₂ ^{( i )} }), set {mv ₀ ^{( i )} , mv ₁ ^{( i )} , mv ₂ ^{( i )} } is set equal to {mv ₀ ^{( i )*} , mv ₁ ^{( i )*} , mv ₂ ^{( i )*} }, and the search procedure continues at step 2. 7. If s _i is not equal to s _min , the search program sets i to i+1, and returns to step 2. Otherwise, the search procedure terminates with the output {mv ₀ ^{( i +1)} , mv ₁ ^{( i +1)} , mv ₂ ^{( i +1)} }.

The video transcoder 200 and the video decoder 300 may be configured to perform a cross search. Cross-seeking utilizes cross-patterns to refine the CPMV. The search procedure is the same as for the square search, except that the incremental motion vectors are defined differently. The incremental motion vector for this search mode is defined as: dMv = {(0,0), (-1,0), (0,-1) (0,1), (1,0)}.

The video transcoder 200 and the video decoder 300 may be configured to perform a diagonal search. Diagonal search utilizes diagonal patterns to refine the CPMV. The search procedure is the same as for the square search, except that the incremental motion vectors are defined differently, as follows: dMv = {(0,0), (-1,-1), (-1,1), (1 ,1), (1,-1)}.

Video transcoder 200 and video decoder 300 may be configured to perform a diamond search. Diamond search utilizes diagonal patterns to refine the CPMV. The search procedure is the same as for the square search, except that the incremental motion vectors are defined differently, as follows: dMv = {(0,0), (0, 2), (1,1), (2,0) , (1,-1), (0,-2), (-1,-1), (-2,0), (-1,1)}.

In another example, the output of the diamond search can be used as the input of the cross search, and the output of the cross search is considered as the final output of the combined search procedure.

The video transcoder 200 and the video decoder 300 may be configured to perform a two-pass eight-point search. A dual-pass eight-point hunt is a search procedure in which two search patterns (i.e., cross pattern and diagonal pattern) are conditionally used during the search procedure. The search procedure is the same as the square search, except for steps 3-5. In a two-path eight-point search, dMV includes two sets of incremental motion vectors: dMv ₀ = {(0,0), (-1,0), (0,-1) (0,1), (1, 0)} and dMv ₁ = {(-1,-1), (-1,1), (1,1), (1,-1)}. The difference with respect to square search is illustrated below. 1-2. These steps are the same as the square search. 3. For s _i , dMv and mv ₀ ^{( i )} , the search subroutine separately calculates the corresponding template matching costs for all these CPMV sets: S={mv ₀ ^{( i )} + d * s _i , mv ₁ ^{( i )} , mv ₂ ^{( i )} , for all d∈dMv ₀ }. The search subroutine can be expressed as d ₀ ^* = argmin _d {cost(S ₀ ), cost(S ₁ ), …, cost(S ₅ )}. Then, if d ₀ ^* is equal to (0,0), mv ₀ ^{( i )*} is set equal to mv ₀ ^{( i )} . Otherwise, the subroutine calculates the corresponding template matching costs for S={mv ₀ ^{( i )} + d * s _i , mv ₁ ^{( i )} , mv ₂ ^{( i )} , for all d∈dMv ₁ U d ₀ ^* }, and Its optimal incremental motion vector is denoted as d ₁ ^* . The result is mv ₀ ^{( i )*} = mv ₀ ^{( i )} + d ₁ ^* * s _i . 4. Similar to step 3, the search subroutine calculates the corresponding template matching for all d∈dMv ₀ } for S={mv ₀ ^{( i )*} , mv ₁ ^{( i )} + d * s _i , mv ₂ ^{( i )} _cost ^, _{and when necessary} ^{, compute the} corresponding _{_ _} template matching cost for . The best search result is expressed as mv ₁ ^{( i )*} = mv ₁ ^{( i )} + d ₁ ^* * s _i (if d ₀ ^* ≠(0,0)) or mv ₁ ^{( i )} (if d ₀ ^* =( 0,0)). 5. Similar to step 3, the search subroutine calculates corresponding templates for S={mv ₀ ^{( i )*} , mv ₁ ^{( i )*} , mv ₂ ^{( i )} + d * s _i , for all d∈dMv ₀ } Matching cost, and when necessary, computed against another S={mv ₀ ^{( i )*} , mv ₁ ^{( i )} , mv ₂ ^{( i )} + d * s _i , for all d∈dMv ₁ U d ₀ ^* } The corresponding template matching cost. The best search result is expressed as mv ₂ ^{( i )*} = mv ₂ ^{( i )} + d ₁ ^* * s _i (if d ₀ ^* ≠(0,0)) or mv ₂ ^{( i )} (if d ₀ ^* =( 0,0)). 6-7. These steps are the same as the square search.

Video transcoder 200 and video decoder 300 may be configured to perform a gradient-based search to update all CPMVs simultaneously. Assuming that the initial CPMV is {mv ₀ ⁽⁰⁾ , mv ₁ ⁽⁰⁾ , mv ₂ ⁽⁰⁾ }, the CPMV is used to generate a reference sample block, which is used to calculate the gradient value of the sampling domain in the horizontal and vertical directions and prediction residuals (that is, the delta between the current sample block and the reference sample block). These values are then used in a gradient-based search to update a given CPMV. The new CPMV (denoted as {mv ₀ ⁽¹⁾ , mv ₁ ⁽¹⁾ , mv ₂ ⁽¹⁾ }) is then used as input for another iterative operation of the gradient-based search. The iterative procedure can be terminated when the condition is met. A condition may eg be that the number of iterations exceeds a predefined (or signaled) threshold, or that the CPMV does not change between two iterations.

Video transcoder 200 and video decoder 300 may be configured to perform translational model search. When all CPMVs happen to be the same before, during, or after the application of the above-mentioned search procedure, all search procedures of AffTM are terminated and one of the best CPMVs is used in a general block-based template matching (e.g., since all CPMVs are the same, so Use random CPMV), as described above for template matching prediction, as the initial search point for its further motion vector refinement.

The video transcoder 200 and the video decoder 300 can be configured to calculate the template matching cost. Template matching cost can be defined (or signaled) as one of the following metrics: SAD, sum of absolute transformed differences (SATD), sum of squared errors (SSE), mean reduced sum of absolute differences (MRSAD), mean reduced sum of absolute transformed differences (MRSATD). MRSAD can be used conditionally if illumination compensation is used for the current processing block.

In another example, video transcoder 200 and video decoder 300 may assign per-sample weight values to each sample on the template block. For example, for a WxH template block, the per-sample weight value can be expressed as N * w _{x , y} , and applied to the corresponding samples c _{x , y} and p _{x , y} of the current block template and the reference block template, where N can be a positive integer (eg, 1, 2, 3, 4, 5, etc.). For simplicity, template matching cost can be defined as: N ^-1 * ∑ _{x,y ϵ template} ( N * w _{x , y} * | c _{x , y} - p _{x , y} | ) or ∑ _{x,y ϵ template} ( N * w _{x , y} * | c _{x , y} - p _{x , y} | )

When using local illumination compensation (LIC) or MRSAD, for simplicity, the equation can be: N ^-1 * ∑ _{x,y ϵ template} ( N * w _{x , y} * | c _{x , y} - p _{x , y} - ∆ _{x , y} | ) or ∑ _{x,y ϵ template} ( N * w _{x , y} * | c _{x , y} - p _{x , y} - ∆ _{x , y} | ) In these equations, ∆ _{x , y} is p _{x , the mean of y} minus the mean of c _{x , y} (in short, mean( p _{x , y} ) – mean( c _{x , y} )). Since the assignment of the weight value of the left template is the transposition of the assignment of the weight value of the upper template, only the assignment of the weight value of the upper template needs to be determined.

In another example, the per-sample weight value may be region-based, with the sample block being equally split into 16 regions, and the sample samples within a region sharing a single weight value.

10 is a conceptual diagram illustrating an example of per-sample weights that may be assigned to samples of neighboring blocks to compute template matching costs. In some examples, video transcoder 200 and video decoder 300 may assign larger weight values to regions closer to the current block, and/or assign smaller weight values to regions closer to the upper left corner of the current block . Figure 10 illustrates two examples. In both instances of current CUs 198A and 198B, video transcoder 200 and video decoder 300 may assign larger weight values to regions closer to the current block, while for the instance of CU 198A, video transcoder 200 and The video decoder 300 additionally lowers the weight of the region closer to the upper left corner of the current block.

In another example, the above-mentioned metrics can be added in a weighted manner, as described above for case-matching prediction, where delta motion vectors for all CPMVs are derived via AffTM.

The video transcoder 200 and the video decoder 300 may be configured to perform a bi-predictive search procedure. In some examples, video transcoder 200 and video decoder 300 may be configured to perform refinement via AW™ CPMVs respectively corresponding to each reference picture list of the bi-predictive block.

In some examples, video transcoder 200 and video decoder 300 may be configured to first refine using AW™ CPMVs respectively corresponding to each reference picture list of the bi-predictive block. Subsequently, the video transcoder 200 and the video decoder 300 may additionally refine the CPMV corresponding to the reference picture list, wherein the CPMV corresponding to another reference picture list is used as a priori. For example, the video transcoder 200 and the video decoder 300 can select the bi-prediction weight w for the CPMV of L1 and L1 to be further refined. First, the current template block used during refinement becomes a weighted delta between the original current template block C and the reference template block _R0 corresponding to L0. C' = ( C – (1- w ) R ₀ ) / w

This subtraction procedure is also called high frequency removal and uses C' in the same way as the current template block used during the L1 CPMV search procedure. Note that this high frequency removal can be performed in another way, namely C' = ( C - w R ₀ ) / (1 - w ).

In some examples, video transcoder 200 and video decoder 300 may be configured to apply high frequency removal when the CPMV of reference picture list Lx is to be refined, where x may be 0 or 1 . In some examples, video transcoder 200 and video decoder 300 may be configured to apply high frequency removal when the CPMV of reference picture list Lx is to be refined, where x may be 0 or 1 . After AWTM is performed on the CPMV of Lx, high frequency removal is applied based on the CPMV of Lx, and then AWTM may be performed on the CPMV of another reference picture list. The iterative procedure terminates until no CPMV changes during the AWTM's search procedure.

In some examples, video transcoder 200 and video decoder 300 may be configured to first apply high frequency removal to the CPMV of Lx according to the BWC weight value. Rules may apply. The rule may be, for example: when the BCW weight of L0 is larger, the CPMV of L0 is first refined, or when the BCW weight of L0 is smaller, the CPMV of L0 is first refined.

In some examples, the video transcoder 200 and the video decoder 300 may be configured to first provide the Lx The CPMV applies high frequency removal. Rules may apply. A rule could eg be: when the flag is true, refine the CPMV of L0 first, or when the flag is false, refine the CPMV of L0 first.

In some examples, the video transcoder 200 and the video decoder 300 may be configured to refine the CPMV of the reference picture list Lx first when the template matching cost is higher than that of other reference picture lists described in the above examples.

In some examples, video transcoder 200 and video decoder 300 may be configured to first refine the CPMVs of the reference picture lists Lx, which achieve higher than The template matching cost of another reference image list's template matching cost.

In some examples, video transcoder 200 and video decoder 300 may be configured to convert bi-predictive CPMV to uni-predictive CPMV. After performing AffTM, before applying high frequency removal, there should be two template matching cost values, cost ₀ corresponding to the CPMV of L0 and cost ₁ corresponding to the CPMV of L1. The third cost value comes from the cost value generated by AWTM after applying high frequency removal. If the third cost value is higher than one of the other two cost values, the CPMV corresponding to the reference L0 or L1 is discarded, depending on which of cost ₀ and cost ₁ is greater.

Video transcoder 200 and video decoder 300 may be configured to perform model conversion from a 4-parameter model to a 6-parameter model. The affine model can be converted from a 4-parameter affine model to a 6-parameter affine model. Using the coordinate position of the lower left corner on the current block (ie (0, block height)), the CPMV of the lower left corner can be calculated based on Equation (2-1). Then, the motion model of the current block is considered as a 6-parameter affine model for AffTM.

Video transcoder 200 and video decoder 300 may be configured to perform model conversion from translational models to affine models. In some examples, in the template matching merge mode, the previously described AffTM procedure can be applied on top of the general block-based template matching, the initial CPMVs are all set equal to the translated MVs generated via the template matching procedure. If the template matching cost of applying the additional AffTM procedure is less than the cost of general template matching, then the CPMV of the AffTM procedure is used for affine motion compensation for the current block instead of the translational motion model from the original template matching procedure.

In some instances, the transformation is only applied if neither bilateral matching (or decoder-side motion refinement (DMVR) in VVC) nor bi-directional optical flow (BDOF) is applied to the current block.

In some examples, video decoder 300 may be configured to always use a 4-parameter affine model as the target transformation model. In some examples, video decoder 300 may be configured to use a 6-parameter affine model as the target transformation model. In some examples, the video decoder 300 may determine the final motion model by minimizing the template matching cost.

FIG. 11 is a block diagram illustrating an example video transcoder 200 that may implement the techniques of this disclosure. FIG. 11 is provided for purposes of explanation and should not be considered limiting of the techniques broadly illustrated and described in this context. For purposes of explanation, this case describes a video transcoder 200 according to VVC (ITU-T H.266 under development) and HEVC (ITU-T H.265) technologies. However, the techniques of this disclosure may be implemented by video encoding devices configured to other video encoding standards.

In the example of FIG. 11 , the video transcoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transformation processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transformation processing unit 212, Reconstruction unit 214 , filter unit 216 , decoded picture buffer (DPB) 218 , and entropy encoding unit 220 . Video data memory 230, mode selection unit 202, residual generation unit 204, transformation processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transformation processing unit 212, reconstruction unit 214, filter unit 216, DPB 218 and Any or all of entropy encoding units 220 may be implemented in one or more processors or in processing circuitry. For example, the elements of video transcoder 200 may be implemented as one or more circuits or logic elements, as part of a hardware circuit, or as part of a processor, ASIC or FPGA. Furthermore, video transcoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

The video data memory 230 can store video data to be encoded by the components of the video transcoder 200 . Video transcoder 200 may receive video data stored in video data memory 230 from, for example, video source 104 ( FIG. 1 ). The DPB 218 may act as a reference picture memory, which stores reference video data for use by the video transcoder 200 in predicting subsequent video data. Video data memory 230 and DPB 218 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM ( RRAM), or other types of memory devices. Video data memory 230 and DPB 218 may be provided by the same memory device or separate memory devices. In various examples, video data memory 230 may be on-die with other components of video transcoder 200 (as shown), or off-die with respect to those components.

In this context, references to video data memory 230 should not be construed as limited to memory internal to video codec 200 (unless so specifically described), or to memory external to video codec 200 (unless so specifically described). Specifically, the reference to the video data memory 230 should be understood as a reference memory for storing video data received by the video transcoder 200 for encoding (eg, video data for the current block to be encoded). The memory 106 of FIG. 1 may also provide temporary storage of outputs from various units of the video transcoder 200 .

The various units of FIG. 11 are illustrated to illustrate and understand the operations performed by the video transcoder 200 . These units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. A fixed-function circuit represents a circuit that provides a specific function and is preset as to operations that can be performed. Programmable circuits represent circuits that can be programmed to perform various tasks and provide flexible functionality in terms of operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions of the software or firmware. Fixed-function circuits can execute software instructions (for example, to receive parameters or output parameters), but the types of operations performed by fixed-function circuits are usually immutable. In some examples, one or more of these units may be different circuit blocks (fixed function or programmable), and in some examples, one or more of these units may be integrated circuits.

The video transcoder 200 may include an arithmetic logic unit (ALU), an elementary functional unit (EFU), a digital circuit, an analog circuit and/or a programmable core formed by programmable circuits. In instances where the operations of video transcoder 200 are performed using software executed by programmable circuitry, memory 106 (FIG. 1) may store instructions (e.g., object code) for the software that video transcoder 200 receives and executes. ), or another memory (not shown) in the video transcoder 200 can store such instructions.

The video data memory 230 is configured to store the received video data. The video transcoder 200 can retrieve the picture of the video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202 . The video data in the video data memory 230 may be the original video data to be encoded.

The mode selection unit 202 includes a motion estimation unit 222 , a motion compensation unit 224 and an intra prediction unit 226 . The mode selection unit 202 may include additional functional units to perform video prediction according to other prediction modes. As examples, mode selection unit 202 may include a palette unit, an intra-block replication unit (which may be part of motion estimation unit 222 and/or motion compensation unit 224 ), an affine unit, a linear model (LM) unit, and the like.

Mode selection unit 202 typically coordinates multiple encoding passes to test combinations of encoding parameters and resulting rate-distortion values for such combinations. The encoding parameters may include partitioning of a CTU into CUs, prediction modes for CUs, transform types for residual data of CUs, quantization parameters for residual data of CUs, and the like. The mode selection unit 202 may finally select a combination of encoding parameters that has a better rate-distortion value than other tested combinations.

The video transcoder 200 can divide the picture retrieved from the video data memory 230 into a series of CTUs, and pack one or more CTUs into slices. The mode selection unit 202 may divide the CTUs of the picture according to a tree structure (such as the above-mentioned QTBT structure or quadtree structure of HEVC). As mentioned above, the video transcoder 200 can form one or more CUs by dividing the CTU according to the tree structure. Such CUs may also be commonly referred to as "video blocks" or "blocks."

Typically, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra prediction unit 226) to generate overlaps) prediction blocks. For inter-prediction of the current block, motion estimation unit 222 may perform a motion search to identify one or more reference pictures (eg, one or more previously decoded pictures stored in DPB 218) or multiple closely matching reference blocks. Specifically, the motion estimation unit 222 may, for example, calculate the sum of absolute difference (SAD), sum of squared difference (SSD), mean absolute difference (MAD), mean square difference (MSD), etc. similar value of . Motion estimation unit 222 may typically perform these calculations using the sample-by-sample difference between the current block and the reference block under consideration. Motion estimation unit 222 may identify the reference block resulting from these calculations with the lowest value, which indicates the reference block that most closely matches the current block.

Motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of the current block in the current picture. Motion estimation unit 222 may then provide the motion vectors to motion compensation unit 224 . For example, for unidirectional inter prediction, motion estimation unit 222 may provide a single motion vector, and for bidirectional inter prediction, motion estimation unit 222 may provide two motion vectors. Motion compensation unit 224 may then use the motion vectors to generate a prediction block. For example, the motion compensation unit 224 may use motion vectors to retrieve reference block information. As another example, if the motion vector has fractional sampling precision, motion compensation unit 224 may interpolate the values used to predict the block according to one or more interpolation filters. Furthermore, for bidirectional inter-prediction, motion compensation unit 224 may retrieve data for two reference blocks identified by corresponding motion vectors and combine the retrieved data, eg, via sample-wise averaging or weighted averaging.

According to the techniques described herein, motion estimation unit 222 and motion compensation unit 224 may be configured to encode and decode blocks of video data using an affine prediction mode. Additionally, motion estimation unit 222 and motion compensation unit 224 may be configured to perform the motion vector refinement procedure described herein.

As another example, for intra prediction or intra prediction coding, the intra prediction unit 226 may generate a prediction block according to samples adjacent to the current block. For example, for directional modes, the intra prediction unit 226 may generally mathematically combine values of adjacent samples and pad these calculated values in a defined direction across the current block to produce a predicted block. As another example, for DC mode, the intra prediction unit 226 may calculate an average value of neighboring samples of the current block, and generate a predicted block to include the resulting average value for each sample of the predicted block.

The mode selection unit 202 provides the prediction block to the residual generation unit 204 . The residual generation unit 204 receives the original unencoded version of the current block from the video data memory 230 and receives the predicted block from the mode selection unit 202 . The residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample differences define the residual block for the current block. In some examples, residual generation unit 204 may determine the difference between sample values in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

In examples where mode select unit 202 partitions a CU into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. The video transcoder 200 and the video decoder 300 can support PUs with various sizes. As noted above, the size of a CU may represent the size of a luma coding block of the CU, and the size of a PU may represent the size of a luma prediction unit of the PU. Assuming that a particular CU has a size of 2Nx2N, the video transcoder 200 may support a PU size of 2Nx2N or NxN for intra prediction, and 2Nx2N, 2NxN, Nx2N, NxN, or similar symmetric PU size for inter prediction. PU size. Video transcoder 200 and video decoder 300 may also support asymmetric partitioning for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

In examples where mode select unit 202 does not further partition the CU into PUs, each CU may be associated with a luma coding block and a corresponding chroma coding block. As mentioned above, the size of a CU may represent the size of a luma coding block of the CU. The video transcoder 200 and the video decoder 300 can support a CU size of 2Nx2N, 2NxN or Nx2N.

For other video coding techniques (such as intra-block copy mode coding, affine mode coding, and linear model (LM) mode coding, to name a few examples), the mode selection unit 202 selects A predictive block is generated for the current block being encoded. In some examples, such as palette mode coding, mode select unit 202 may not generate a prediction block, but instead generate syntax elements indicating the manner in which the block is reconstructed based on the selected palette. In such modes, mode selection unit 202 may provide these syntax elements to entropy encoding unit 220 for encoding.

As mentioned above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. Subsequently, the residual generation unit 204 generates a residual block for the current block. To generate a residual block, the residual generation unit 204 calculates a sample-by-sample difference between the predicted block and the current block.

Transform processing unit 206 applies one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form a block of transform coefficients. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, transform processing unit 206 may perform multiple transforms on the residual block, eg, a primary transform and a secondary transform (such as a rotation transform). In some examples, transform processing unit 206 applies no transform to the residual block.

Quantization unit 208 may quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 may quantize transform coefficients of a block of transform coefficients according to a quantization parameter (QP) value associated with the current block. Video transcoder 200 (eg, via mode selection unit 202 ) may adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may cause loss of information, and thus, the quantized transform coefficients may have lower precision compared to the original transform coefficients generated by transform processing unit 206 .

The inverse quantization unit 210 and the inverse transform processing unit 212 may apply inverse quantization and inverse transform to quantized transform coefficient blocks, respectively, to reconstruct a residual block from the transform coefficient blocks. Reconstruction unit 214 may generate a reconstructed block corresponding to the current block (although potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by mode selection unit 202 . For example, reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples from the prediction block produced by mode selection unit 202 to produce a reconstructed block.

Filter unit 216 may perform one or more filter operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce blocking artifacts along the edges of a CU. In some examples, the operation of filter unit 216 may be skipped.

Video transcoder 200 stores the reconstructed blocks in DPB 218 . For example, in instances where the operations of filter unit 216 are not performed, reconstruction unit 214 may store the reconstructed block into DPB 218 . In instances where the operations of filter unit 216 are performed, filter unit 216 may store the filtered reconstruction block into DPB 218 . Motion estimation unit 222 and motion compensation unit 224 may retrieve reference pictures formed from reconstructed (and potentially filtered) blocks from DPB 218 for inter-prediction of blocks of subsequently encoded pictures. In addition, the intra prediction unit 226 may use the reconstructed block of the current picture in the DPB 218 to perform intra prediction on other blocks in the current picture.

Generally, the entropy coding unit 220 can entropy code the syntax elements received from other functional components of the video transcoder 200 . For example, entropy encoding unit 220 may entropy encode the quantized transform coefficient block from quantization unit 208 . As another example, the entropy encoding unit 220 may entropy encode the prediction syntax elements (eg, motion information for inter prediction or intra mode information for intra prediction) from the mode selection unit 202 . The entropy encoding unit 220 may perform one or more entropy encoding operations on syntax elements, which are another example of video data, to generate entropy encoded data. For example, the entropy coding unit 220 may perform context self-adjusting variable length coding (CAVLC) operations, CABAC operations, variable-to-variable (V2V) length coding operations, syntax-based context self-adjusting binary arithmetic coding (SBAC) operation, a Probability Interval Partitioning Entropy (PIPE) decoding operation, an Exponential Golomb encoding operation, or another type of entropy encoding operation on the data. In some examples, entropy encoding unit 220 may operate in a bypass mode in which syntax elements are not entropy encoded.

Video transcoder 200 may output a bitstream comprising entropy-encoded syntax elements required for reconstructing the blocks of the slice or picture. Specifically, the entropy coding unit 220 can output a bit stream.

The above operations are described with respect to blocks. Such descriptions should be understood as operations for luma coding blocks and/or chroma coding blocks. As previously mentioned, in some examples, the luma and chroma coding blocks are the luma and chroma components of the CU. In some examples, the luma coding block and the chroma coding block are the luma and chroma components of the PU.

In some examples, operations performed on luma coding blocks need not be repeated for chroma coding blocks. As an example, the operations of identifying motion vectors (MVs) and reference pictures for luma coded blocks need not be repeated to identify MVs and reference pictures for chroma blocks. Specifically, the MV for luma coded blocks may be scaled to decide the MV for chroma blocks, and the reference pictures may be the same. As another example, the intra prediction procedure may be the same for luma and chroma coded blocks.

12 is a block diagram illustrating an example video decoder 300 that may perform techniques of this disclosure. FIG. 12 is provided for purposes of explanation, and not limitation, of the techniques broadly illustrated and described in this context. For purposes of explanation, this document describes video decoder 300 in terms of VVC (ITU-T H.266 under development) and HEVC (ITU-T H.265) technologies. However, the techniques in this disclosure may be performed by video decoding devices configured to other video coding standards.

In the example of FIG. 12 , the video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, filter unit 312 and decoded picture buffer (DPB) 134 . Any one or all of CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312 and DPB 134 can be in one or more implemented in a processor or in a processing circuit. For example, the elements of video decoder 300 may be implemented as one or more circuits or logic components, as part of a hardware circuit, or as part of a processor, ASIC or FPGA. Furthermore, video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

The prediction processing unit 304 includes a motion compensation unit 316 and an intra-frame prediction unit 318 . The prediction processing unit 304 may include an addition unit that performs prediction according to other prediction modes. As examples, prediction processing unit 304 may include a palette unit, an intra-block replication unit (which may form part of motion compensation unit 316 ), an affine unit, a linear model (LM) unit, and the like. In other examples, video decoder 300 may include more, fewer or different functional components.

The CPB memory 320 may store video data, such as an encoded video bit stream, to be decoded by components of the video decoder 300 . For example, video data stored in CPB memory 320 may be obtained from computer readable medium 110 (FIG. 1). CPB memory 320 may include a CPB that stores encoded video data (eg, syntax elements) from an encoded video bitstream. In addition, the CPB memory 320 may store video data other than the syntax elements of the decoded pictures, such as temporary data representing outputs from various units of the video decoder 300 . The DPB 314 typically stores decoded pictures, and the video decoder 300 may output the decoded pictures and/or use the decoded pictures as reference video data when decoding subsequent data or pictures of the encoded video bitstream. CPB memory 320 and DPB 314 may be formed from any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, CPB memory 320 may be on-die with other components of video decoder 300, or off-die with respect to those components.

Additionally or alternatively, in some examples, video decoder 300 may retrieve decoded video data from memory 120 ( FIG. 1 ). That is, memory 120 may utilize CPB memory 320 to store data as discussed above. Likewise, the memory 120 may store instructions to be executed by the video decoder 300 when some or all of the functions of the video decoder 300 are implemented by software to be executed by the processing circuit of the video decoder 300 .

The various units shown in FIG. 12 are illustrated to illustrate and understand the operations performed by the video decoder 300 . These units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Similar to FIG. 11 , a fixed-function circuit represents a circuit that provides a specific function and is preset as to operations that can be performed. Programmable circuits represent circuits that can be programmed to perform various tasks and provide flexible functionality in terms of operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions of the software or firmware. Fixed-function circuits can execute software instructions (eg, to receive parameters or output parameters), but the types of operations performed by fixed-function circuits are typically immutable. In some examples, one or more of these units may be different circuit blocks (fixed function or programmable), and in some examples, one or more of these units may be integrated circuits.

The video decoder 300 may include an ALU, an EFU, a digital circuit, an analog circuit and/or a programmable core formed by programmable circuits. In instances where the operations of video decoder 300 are performed by software executing on programmable circuitry, on-chip or off-chip memory may store instructions (eg, object code) for the software that video decoder 300 receives and executes.

The entropy decoding unit 302 may receive encoded video data from the CPB, and perform entropy decoding on the video data to recover syntax elements. The prediction processing unit 304 , inverse quantization unit 306 , inverse transform processing unit 308 , reconstruction unit 310 , and filter unit 312 can generate decoded video data based on syntax elements extracted from the bitstream.

In general, video decoder 300 reconstructs a picture block by block. The video decoder 300 may perform reconstruction operations on each block individually (wherein the block currently being reconstructed (ie, decoded) may be referred to as a "current block").

The entropy decoding unit 302 may entropy decode syntax elements defining quantized transform coefficients of a block of quantized transform coefficients, as well as transform information such as quantization parameters (QP) and/or transform mode indications. Inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine the degree of quantization, and as such, determine the degree of inverse quantization for inverse quantization unit 306 to apply. Inverse quantization unit 306 may, for example, perform a bitwise left shift operation to inverse quantize the quantized transform coefficients. The inverse quantization unit 306 may thus form a transform coefficient block comprising transform coefficients.

After inverse quantization unit 306 forms the block of transform coefficients, inverse transform processing unit 308 may apply one or more inverse transforms to the block of transform coefficients to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotation transform, an inverse direction transform, or another inverse transform to the transform coefficient block.

In addition, the prediction processing unit 304 generates a prediction block according to the prediction information syntax element entropy-decoded by the entropy decoding unit 302 . For example, motion compensation unit 316 may generate a prediction block if the prediction information syntax element indicates that the current block is inter-predicted. In this case, the prediction information syntax element may indicate the reference picture in the DPB 314 from which the reference block is to be retrieved, and identify the position of the reference block in the reference picture relative to the position of the current block in the current picture motion vector. Motion compensation unit 316 may generally perform inter-frame prediction procedures in a manner substantially similar to that described with respect to motion compensation unit 224 (FIG. 11). According to the techniques described herein, the motion compensation unit 316 may be configured to decode blocks of video data using an affine prediction mode, and may be configured to perform the motion vector refinement procedure described herein.

As another example, if the prediction information syntax element indicates that the current block is intra-predicted, the intra prediction unit 318 may generate the prediction block according to the intra prediction mode indicated by the prediction information syntax element. Again, intra prediction unit 318 may generally perform the intra prediction process in a manner substantially similar to that described with respect to intra prediction unit 226 (FIG. 11). The intra-frame prediction unit 318 can retrieve data of neighboring samples of the current block from the DPB 314 .

The reconstruction unit 310 may reconstruct the current block using the prediction block and the residual block. For example, the reconstruction unit 310 may add the samples of the residual block and the corresponding samples of the prediction block to reconstruct the current block.

Filter unit 312 may perform one or more filter operations on the reconstructed block. For example, filter unit 312 may perform a deblocking operation to reduce blocking artifacts along edges of reconstructed blocks. The operations of filter unit 312 are not necessarily performed in all examples.

Video decoder 300 may store the reconstructed blocks in DPB 314 . For example, in instances where the operations of filter unit 312 are not performed, reconstruction unit 310 may store the reconstructed block into DPB 314 . In examples where the operations of filter unit 312 are performed, filter unit 312 may store the filtered reconstruction block into DPB 314 . As discussed above, DPB 314 may provide reference information, such as samples of the current picture for intra prediction and previously decoded pictures for subsequent motion compensation, to prediction processing unit 304 . Additionally, video decoder 300 may output decoded pictures (eg, decoded video) from DPB 314 for subsequent presentation on a display device, such as display device 118 of FIG. 1 .

13 is a flowchart illustrating an example procedure for encoding a current block in accordance with the techniques of this disclosure. A current block may include a current CU. Although described with respect to video transcoder 200 ( FIGS. 1 and 11 ), it should be understood that other devices may be configured to perform procedures similar to those of FIG. 13 .

In this example, video transcoder 200 initially predicts the current block (350). For example, video transcoder 200 may use template-based affine prediction to form a predicted block for a current block, as described in this disclosure. Video transcoder 200 may then compute a residual block for the current block (352). To calculate the residual block, video transcoder 200 may calculate the difference between the original uncoded block and the predicted block for the current block. Video transcoder 200 may then transform the residual block and transform and quantize the coefficients of the residual block (354). Next, video transcoder 200 may scan the residual block for quantized transform coefficients (356). During scanning or after scanning, video transcoder 200 may entropy encode the transform coefficients (358). For example, video transcoder 200 may use CAVLC or CABAC to encode transform coefficients. Video transcoder 200 may then output entropy encoded data for the block (360).

14 is a flowchart illustrating an example process for decoding a current block of video material in accordance with the techniques of this disclosure. A current block may include a current CU. Although described with respect to video decoder 300 ( FIGS. 1 and 12 ), it should be understood that other devices may be configured to perform procedures similar to those of FIG. 14 .

Video decoder 300 may receive entropy-coded data for the current block (eg, entropy-coded prediction information and entropy-coded data for transform coefficients of a residual block corresponding to the current block) (370) . Video decoder 300 may entropy decode the entropy encoded data to determine prediction information for the current block and to recover transform coefficients for the residual block (372). Video decoder 300 may predict the current block (374), eg, using an intra or inter prediction mode as indicated by the prediction information for the current block, to compute a predicted block for the current block. For example, video decoder 300 may predict the current block using template-based affine prediction as described in this disclosure. Video decoder 300 may then inverse scan (376) the reconstructed transform coefficients to create blocks of quantized transform coefficients. Video decoder 300 may then inverse quantize the transform coefficients and apply the inverse transform to the transform coefficients to produce a residual block (378). Finally, video decoder 300 may decode the current block by combining the prediction block and the residual block (380).

15 is a flowchart illustrating an example process for decoding a current block of video material in accordance with the techniques of this disclosure. A current block may include a current CU. Although described with respect to video decoder 300 ( FIGS. 1 and 12 ), it should be understood that other devices may be configured to perform procedures similar to those of FIG. 15 .

The video decoder 300 may decide to decode a current block in a current picture of the video data in an affine prediction mode (400). The affine prediction mode may, for example, be a 4-parameter affine prediction mode, a 6-parameter affine prediction mode, or some other such affine prediction mode.

Video decoder 300 may determine one or more control point motion vectors (CPMV) for the current block (402). Video decoder 300 may use one or more CPMVs to identify an initial predictive block for a current block in a reference picture (404). In order to identify the initial predictive block of the current block, the video decoder 300 may, for example, use CPMV to locate a plurality of sub-blocks in the reference frame.

Video decoder 300 may determine a current template for a current block in a current picture (406). The current template may include a plurality of sub-blocks located above or to the left of the current block, for example, as shown in FIG. 9A .

Video decoder 300 may determine an initial reference example for an initial prediction block in a reference picture (408). The initial reference template may include a plurality of sub-blocks located above or to the left of the initial prediction block, for example, as shown in FIGS. 9B and 9C .

Video decoder 300 may perform a motion vector refinement procedure based on the comparison of the current template and the initial reference template to determine a modified prediction block (410). To perform the motion vector refinement process to further determine the modified prediction block, the video decoder 300 may, for example, search for a subsequent reference template that matches the current template more closely than the initial reference template in a search area around the initial reference template. For example, the comparison of the current template and the initial reference template may be a template matching cost, and video decoder 300 may determine the template matching cost based on a weighted per-sample comparison of samples in the current template and samples in the initial reference template.

The video decoder 300 may determine a prediction block based on the modified prediction block; add the prediction block to the residual block to determine a reconstructed block; apply one or more filtering operations to the reconstructed block; A picture of the decoded video data of the filtered reconstructed block.

The following numbered clauses illustrate one or more aspects of the devices and techniques described in this disclosure.

Clause 1A. A method of decoding video material, the method comprising: determining one or more control point motion vectors (CPMVs) for a current block, wherein the one or more CPMVs correspond to initial a prediction block; and performing a motion vector refinement procedure to determine a modified prediction block.

Clause 2A. The method of Clause 1A, wherein the motion vector refinement procedure comprises: performing a template matching procedure.

Clause 3A. The method as recited in Clause 2A, wherein the one or more CPMVs comprise an initial set of CPMVs, and the template matching procedure comprises: determining a refined set of CPMVs.

Clause 4A. The method of Clause 3A, wherein determining the refined set of CPMVs comprises: adding one or more incremental motion vector values to the one or more CPMVs to determine the refined set of CPMVs.

Clause 5A. The method as recited in clause 3A or 4A, further comprising: determining a search area based on the one or more CPMVs; and wherein determining the set of refined CPMVs comprises limiting the refined CPMVs to the search area within the area.

Clause 6A. The method as recited in any one of clauses 3A-5A, further comprising: determining a search pattern; and determining the refined set of CPMVs based on the search pattern.

Clause 7A. The method of any one of clauses 1A-6A, wherein performing the motion vector refinement procedure to determine the modified prediction block comprises: performing one or more template matching cost calculations.

Clause 8A. The method of any one of clauses 1A-7A, wherein the method of decoding is performed as part of an encoding procedure.

Clause 9A. An apparatus for decoding video material, the apparatus comprising one or more means for performing the method of any one of clauses 1A-8A.

Clause 10A. The apparatus of Clause 9A, wherein the one or more units comprise one or more processors implemented in circuitry.

Clause 11A. The apparatus according to any one of clauses 9A and 10A, further comprising: memory for storing the video data.

Clause 12A. The apparatus of any one of clauses 9A-11A, further comprising: a display configured to display the decoded video data.

Clause 13A. The device of any one of clauses 9A-12A, wherein the device comprises one or more of a camera, computer, mobile device, broadcast receiver device, or set-top box.

Clause 14A. The apparatus of any one of clauses 9A-13A, wherein the apparatus comprises a video decoder.

Clause 15A. The apparatus of any one of clauses 9A-14A, wherein the apparatus comprises a video transcoder.

Clause 16A. A computer-readable storage medium having stored thereon instructions which, when executed, cause one or more processors to perform the method of any one of clauses 1A-8A.

Clause 1B. A method of decoding video data, the method comprising: determining an affine prediction mode for decoding a current block in a current picture of the video data; determining one or more control points for the current block motion vector (CPMV); use the one or more CPMVs to identify an initial prediction block for the current block in a reference picture; determine a current template for the current block in the current picture; determine a current template for the reference picture an initial reference template for the initial prediction block; and performing a motion vector refinement procedure based on a comparison between the current template and the initial reference template to determine a modified prediction block.

Clause 2B. The method as recited in Clause 1B, wherein performing the motion vector refinement procedure to determine the modified prediction block further comprises: searching a search area around the initial reference template that is more closely related to the initial reference template than the initial reference template The subsequent reference template that this current template matches.

Clause 3B. The method of Clause 1B, wherein the comparison of the current template and the initial reference template includes a template matching cost.

Clause 4B. The method of Clause 3B, further comprising: determining the template matching cost based on a weighted per-sample comparison of samples in the current template and samples in the initial reference template.

Clause 5B. The method of Clause 1B, wherein the initial reference example comprises a plurality of sub-blocks located above or to the left of the initial prediction block.

Clause 6B. The method of Clause 1B, wherein the affine prediction mode comprises a 4-parameter affine prediction mode.

Clause 7B. The method of Clause 1B, wherein the affine prediction mode comprises a 6-parameter affine prediction mode.

Clause 8B. The method of Clause 1B, further comprising: determining a prediction block based on the modified prediction block; adding the prediction block to the residual block to determine a reconstructed block; adding to the reconstructed block applying one or more filtering operations; and outputting a picture of decoded video data comprising the filtered reconstructed block.

Clause 9B. The method of Clause 1B, wherein the decoding is performed as part of a video encoding process.

Clause 10B. An apparatus for decoding video data, the apparatus comprising: a memory; and one or more processors implemented in a circuit, coupled to the memory and configured to: determine an affine predictive mode decoding a current block in a current picture of the video data; determining one or more control point motion vectors (CPMVs) for the current block; using the one or more CPMVs to identify a control point motion vector (CPMV) for a reference picture an initial prediction block of the current block; determining a current template for the current block in the current picture; determining an initial reference template for the initial prediction block in the reference picture; and based on the current template and the initial reference template The comparison to perform the motion vector refinement procedure to determine the modified prediction block.

Clause 11B. The apparatus of Clause 10B, wherein for performing the motion vector refinement procedure to determine the modified prediction block, the one or more processors are also configured to: search regions around the initial reference template A subsequent reference template that more closely matches the current template than the initial reference template is searched within.

Clause 12B. The apparatus of Clause 10B, wherein the comparison of the current template and the initial reference template includes a template matching cost.

Clause 13B. The apparatus of Clause 12B, wherein the one or more processors are further configured to: determine the template matching cost based on a weighted per-sample comparison of samples in the current template with samples in the initial reference template .

Clause 14B. The apparatus of Clause 10B, wherein the initial reference template comprises a plurality of sub-blocks located above or to the left of the initial predictive block.

Clause 15B. The apparatus of Clause 10B, wherein the affine prediction mode comprises a 4-parameter affine prediction mode.

Clause 16B. The apparatus of Clause 10B, wherein the affine prediction mode comprises a 6-parameter affine prediction mode.

Clause 17B. The apparatus of Clause 10B, wherein the one or more processors are also configured to: determine a prediction block based on the modified prediction block; add the prediction block to the residual block to determine the reconstructed applying one or more filtering operations to the reconstructed block; and outputting a picture of decoded video data comprising the filtered reconstructed block.

Clause 18B: The device of Clause 10B, wherein the device comprises a wireless communication device and also comprises a receiver configured to receive the encoded video data.

Clause 19B: The device of Clause 18B, wherein the wireless communication device comprises a telephone handset, and wherein the receiver is configured to demodulate a signal comprising the encoded video data according to a wireless communication standard.

Clause 2OB. The apparatus of Clause 1OB, further comprising: a display configured to display the decoded video data.

Clause 21B. The device of clause 10B, wherein the device comprises one or more of a camera, computer, mobile device, broadcast receiver device, or set-top box.

Clause 22B. The device of Clause 10B, wherein the device comprises a video encoding device.

Clause 23B. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determine an affine prediction mode for the video data Decoding the current block in the current picture; determining one or more control point motion vectors (CPMVs) for the current block; using the one or more CPMVs to identify an initial prediction for the current block in a reference picture block; determine a current template for the current block in the current picture; determine an initial reference template for the initial prediction block in the reference picture; and perform motion vectoring based on a comparison between the current template and the initial reference template Refinement procedure to determine modified prediction blocks.

Clause 24B. The computer-readable storage medium of clause 23B, wherein to execute the motion vector refinement procedure to determine the modified prediction block, the instructions also cause the one or more processors to: A subsequent reference template that more closely matches the current template than the initial reference template is searched in a search area around the initial reference template.

Clause 25B. The computer-readable storage medium of Clause 23B, wherein the comparison of the current template and the initial reference template includes a template matching cost.

Clause 26B. The computer-readable storage medium of clause 25B, wherein the instructions cause the one or more processors to: each Sampling comparisons are used to determine the template matching cost.

Clause 27B. The computer-readable storage medium of Clause 23B, wherein the initial reference template includes a plurality of sub-blocks located above or to the left of the initial predicted block.

Clause 28B. The computer-readable storage medium of clause 23B, wherein the instructions cause the one or more processors to: determine a prediction block based on the modified prediction block; add the prediction block to the residual block to determine a reconstructed block; apply one or more filtering operations to the reconstructed block; and output a picture of decoded video data including the filtered reconstructed block.

Clause 29B. An apparatus for decoding video data, the apparatus comprising: means for determining to decode a current block in a current picture of the video data in an affine prediction mode; A unit of one or more control point motion vectors (CPMVs) of a block; a unit used to identify an initial prediction block for the current block in a reference picture using the one or more CPMVs; used to determine a block for the current block A unit for the current example of the current block in the picture; a unit for determining an initial reference example for the initial prediction block in the reference picture; and for performing motion based on a comparison of the current example and the initial reference example Vector refinement procedure to determine elements of the modified prediction block.

Clause 30B. The apparatus of Clause 29B, wherein the comparison of the current template and the initial reference template includes a template matching cost, the means further comprising: The weighted per-sample comparisons are used to determine the cost of matching the template to the unit.

Clause 1C. A method of decoding video data, the method comprising: determining an affine prediction mode to decode a current block in a current picture of the video data; determining one or more control points for the current block motion vector (CPMV); use the one or more CPMVs to identify an initial prediction block for the current block in a reference picture; determine a current template for the current block in the current picture; determine a current template for the reference picture an initial reference template for the initial prediction block; and performing a motion vector refinement procedure based on a comparison between the current template and the initial reference template to determine a modified prediction block.

Clause 2C. The method of Clause 1C, wherein performing the motion vector refinement procedure to determine the modified prediction block further comprises: searching a search area around the initial reference template that is more closely related to the initial reference template than the initial reference template The subsequent reference template that this current template matches.

Clause 3C. The method of Clause 1C or 2C, wherein the comparison of the current template and the initial reference template includes a template matching cost.

Clause 4C. The method of Clause 3C, further comprising: determining the template matching cost based on a weighted per-sample comparison of samples in the current template and samples in the initial reference template.

Clause 5C. The method of any one of clauses 1C-4C, wherein the initial reference example comprises a plurality of sub-blocks located above or to the left of the initial predicted block.

Clause 6C. The method of any one of clauses 1C-5C, wherein the affine prediction mode comprises a 4-parameter affine prediction mode.

Clause 7C. The method of any one of clauses 1C-5C, wherein the affine prediction mode comprises a 6-parameter affine prediction mode.

Clause 8C. The method of any one of clauses 1C-7C, further comprising: determining a prediction block based on the modified prediction block; adding the prediction block to the residual block to determine a reconstructed block; Applying one or more filtering operations to the reconstructed block; and outputting a picture of decoded video data comprising the filtered reconstructed block.

Clause 9C. The method of any one of clauses 1C-8C, wherein the decoding is performed as part of a video encoding process.

Clause 10C. An apparatus for decoding video data, the apparatus comprising: a memory; and one or more processors implemented in a circuit, coupled to the memory, and configured to: determine an affine prediction mode to decode a current block in a current picture of the video data; determine one or more control point motion vectors (CPMVs) for the current block; use the one or more CPMVs to identify a control point motion vector (CPMV) for a reference picture an initial prediction block of the current block; determining a current template for the current block in the current picture; determining an initial reference template for the initial prediction block in the reference picture; and based on the current template and the initial reference template The comparison to perform the motion vector refinement procedure to determine the modified prediction block.

Clause 11C. The apparatus of Clause 10C, wherein for performing the motion vector refinement procedure to determine the modified prediction block, the one or more processors are also configured to: search regions around the initial reference template A subsequent reference template that more closely matches the current template than the initial reference template is searched within.

Clause 12C. The apparatus of Clause 10C or 11C, wherein the comparison of the current template and the initial reference template includes a template matching cost.

Clause 13C. The apparatus of Clause 12C, wherein the one or more processors are further configured to: determine the template matching cost based on a weighted per-sample comparison of samples in the current template with samples in the initial reference template .

Clause 14C. The apparatus of any one of clauses 10C-13C, wherein the initial reference example comprises a plurality of sub-blocks located above or to the left of the initial prediction block.

Clause 15C. The apparatus of any one of clauses 10C-14C, wherein the affine prediction mode comprises a 4-parameter affine prediction mode.

Clause 16C. The apparatus of any one of clauses 10C-14C, wherein the affine prediction mode comprises a 6-parameter affine prediction mode.

Clause 17C. The apparatus of any one of clauses 10C-16C, wherein the one or more processors are also configured to: determine a prediction block based on the modified prediction block; add the prediction block to the residual block to determine a reconstructed block; apply one or more filtering operations to the reconstructed block; and output a picture of decoded video data including the filtered reconstructed block.

Clause 18C: The device according to any one of clauses 10C-17C, wherein the device comprises a wireless communication device and also comprises a receiver configured to receive the encoded video data.

Clause 19C: The device of Clause 18C, wherein the wireless communication device comprises a telephone handset, and wherein the receiver is configured to demodulate a signal comprising the encoded video data according to a wireless communication standard.

Clause 20C. The apparatus according to any one of clauses 10C-19C, further comprising: a display configured to display the decoded video data.

Clause 21C. The device of any one of clauses 10C-20C, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set top box.

Clause 22C. The apparatus of any one of clauses 10C-20C, wherein the apparatus comprises a video encoding apparatus.

Clause 23C. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determine an affine prediction mode for the video data Decoding the current block in the current picture; determining one or more control point motion vectors (CPMVs) for the current block; using the one or more CPMVs to identify an initial prediction for the current block in a reference picture block; determine a current template for the current block in the current picture; determine an initial reference template for the initial prediction block in the reference picture; and perform motion vectoring based on a comparison between the current template and the initial reference template Refinement procedure to determine modified prediction blocks.

Clause 24C. The computer-readable storage medium of Clause 23C, wherein to execute the motion vector refinement procedure to determine the modified prediction block, the instructions cause the one or more processors to: A search region around the initial reference template is searched for subsequent reference templates that more closely match the current template than the initial reference template.

Clause 25C. The computer-readable storage medium of Clause 23C, wherein the comparison of the current template and the initial reference template includes a template matching cost.

Clause 26C. The computer-readable storage medium of clause 25C, wherein the instructions cause the one or more processors to: each Sampling comparisons are used to determine the template matching cost.

Clause 27C. The computer-readable storage medium of Clause 23C, wherein the initial reference template includes a plurality of sub-blocks located above or to the left of the initial predicted block.

Clause 28C. The computer-readable storage medium of clause 23C, wherein the instructions cause the one or more processors to: determine a prediction block based on the modified prediction block; add the prediction block to the residual block to determine a reconstructed block; apply one or more filtering operations to the reconstructed block; and output a picture of decoded video data including the filtered reconstructed block.

Clause 29C. An apparatus for decoding video data, the apparatus comprising: means for determining to decode a current block in a current picture of the video data in an affine prediction mode; A unit of one or more control point motion vectors (CPMVs) of a block; a unit used to identify an initial prediction block for the current block in a reference picture using the one or more CPMVs; used to determine a block for the current block A unit for the current example of the current block in the picture; a unit for determining an initial reference example for the initial prediction block in the reference picture; and for performing motion based on a comparison of the current example and the initial reference example Vector refinement procedure to determine elements of the modified prediction block.

Clause 30C. The apparatus of Clause 29C, wherein the comparison of the current template and the initial reference template includes a template matching cost, the means further comprising: The weighted per-sample comparisons are used to determine the cost of matching the template to the unit.

It is to be appreciated that, depending on the example, certain acts or events of any of the techniques described herein may be performed in a different order, added to, combined, or omitted entirely (e.g., not all described acts or events are essential for implementing such technology is necessary). Furthermore, in some instances, actions or events may be performed concurrently rather than sequentially, eg, via multi-thread processing, interrupt handling, or multiple processors.

In one or more instances, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or communication media including, for example, communication protocols to facilitate transfer of computer programs from one place to another. any media. In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to obtain instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer readable medium.

By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or Any other medium that stores desired code in the form of instructions or data structures and can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, and microwave), then coaxial cable, fiber optic Fiber optic cable, twisted pair, DSL, or wireless technologies (eg, infrared, radio, and microwave) are included in the definition of media. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, compact disc, digital versatile disc (DVD), floppy disc, and Blu-ray disc, where disks usually reproduce data magnetically, while discs use lasers to Optically reproduce data. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Accordingly, the terms "processor" and "processing circuitry," as used herein, may represent any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a combined transcoder. In addition, the techniques may be implemented entirely in one or more circuits or logic components.

The technology at issue may be implemented in a wide variety of devices or devices, including a wireless handset, an integrated circuit (IC), or a group of ICs (eg, a chipset). Various components, modules, or units are described in this context to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization via different hardware units. Specifically, as mentioned above, various units may be combined in a transcoder hardware unit, or a collection of hardware units (including one or more processors as described above) combined with appropriate software and /or firmware to provide.

Various examples have been described. These and other examples are within the scope of the appended claims.

100: system 102: source device 104: video source 106: memory 108: output interface 110: computer readable medium 112: storage device 114: file server 116: destination device 118: display device 120: memory 122: Input interface 130: QTBT structure 132: CTU 140: block 142: block 154: block 156: motion vector scaling program 160: current CU 162: current template 164: reference template 170A: control point 170B: control point 172A: control point 172B: Control point 172C: control point 190: arrow 192: current template block 194A: reference template 194B: reference template 198A: current CU 198B: current CU 200: video transcoder 202: mode selection unit 204: residual generation unit 206: transformation Processing unit 208: Quantization unit 210: Inverse quantization unit 212: Inverse transform processing unit 214: Reconstruction unit 216: Filter unit 218: Decoded picture buffer (DPB) 220: Entropy encoding unit 222: Motion estimation unit 224: Motion compensation Unit 226: intra-frame prediction unit 230: video data memory 300: video decoder 302: entropy decoding unit 304: prediction processing unit 306: inverse quantization unit 308: inverse transform processing unit 310: reconstruction unit 312: filter unit 314: Decoded Picture Buffer (DPB) 316: Motion Compensation Unit 318: Intra-Frame Prediction Unit 320: Decoded Picture Buffer (CPB) Memory 350: Block 352: Block 354: Block 356: Block 358: Block 360: Block 370: block 372: block 374: block 376: block 378: block 380: block 400: block 402: block 404: block 406: block 408: block 410: block A ₀ : sub-block A ₁ : sub-block A ₂ : Sub-block A ₃ : Sub-block L ₀ : Sub-block L ₁ : Sub-block L ₂ : Sub-block L ₃ : Sub-block PU0: Prediction unit PU1: Prediction unit TMVP: Temporal motion vector predictor V ₀ : Vector V ₁ : Vector V _SB : vector v ₂ : vector

1 is a block diagram illustrating an example video encoding and decoding system that may implement the techniques of this disclosure.

2A and 2B are conceptual diagrams illustrating example quadtree binary tree (QTBT) structures and corresponding coding tree units (CTUs).

FIG. 3A is a conceptual diagram illustrating spatially adjacent motion vector candidates for merge mode.

FIG. 3B is a conceptual diagram illustrating spatially adjacent motion vector candidates for an advanced motion vector prediction (AMVP) mode.

FIG. 4A is a conceptual diagram showing temporal motion vector candidates.

FIG. 4B is a conceptual diagram illustrating motion vector scaling.

FIG. 5 illustrates an example of template matching performed on a search area around an initial motion vector.

FIG. 6A is a conceptual diagram illustrating a control point-based 6-parameter affine motion model.

FIG. 6B is a conceptual diagram illustrating a control point-based 4-parameter affine motion model.

FIG. 7 illustrates an example of an affine motion vector field for each sub-block.

FIG. 8 illustrates an example of a sub-block motion vector.

9A-9C illustrate a current template block and a reference template block.

10 is a conceptual diagram illustrating an example of per-sample weights that may be assigned to samples of neighboring blocks to compute template matching costs.

11 is a block diagram illustrating an example video transcoder that may implement the techniques of this disclosure.

12 is a block diagram illustrating an example video decoder that may perform techniques of this disclosure.

13 is a flowchart illustrating an example procedure for encoding a current block in accordance with the techniques of this disclosure.

14 is a flowchart illustrating an example procedure for decoding a current block in accordance with the techniques of this disclosure.

15 is a flowchart illustrating an example procedure for decoding a current block in accordance with the techniques of this disclosure.

Domestic deposit information (please note in order of depositor, date, and number) none Overseas storage information (please note in order of storage country, institution, date, and number) none

400: block

402: block

404: block

406: block

408: block

410: block

Claims (30)

A method for decoding video data, the method includes the following steps: determining to decode a current block in a current picture of the video data in an affine prediction mode; determine one or more control point motion vectors (CPMV) for the current block; using the one or more CPMVs to identify an initial prediction block for the current block in a reference picture; determining a current template for the current block in the current picture; determining an initial reference sample for the initial prediction block in the reference picture; and A motion vector refinement procedure is performed based on a comparison of the current template and the initial reference template to determine a modified prediction block. The method according to claim 1, wherein performing the motion vector refinement procedure to determine the modified prediction block also includes the following steps: A subsequent reference template that matches the current template more closely than the initial reference template is searched in a search region around the initial reference template. The method according to claim 1, wherein the comparison of the current template and the initial reference template includes a template matching cost. The method according to claim 3 also includes the following steps: The template matching cost is determined based on a weighted per-sample comparison of samples in the current template with samples in the initial reference template. The method according to claim 1, wherein the initial reference template includes a plurality of sub-blocks located above or to the left of the initial prediction block. The method according to claim 1, wherein the affine prediction mode comprises a 4-parameter affine prediction mode. The method according to claim 1, wherein the affine prediction mode comprises a 6-parameter affine prediction mode. The method according to Claim 1 also includes the following steps: determining a prediction block based on the modified prediction block; adding the predicted block to a residual block to determine a reconstructed block; applying one or more filtering operations to the reconstructed block; and A picture of decoded video data including the filtered reconstructed blocks is output. The method according to claim 1, wherein the decoding method is performed as part of a video encoding process. A device for decoding video data, the device comprising: a memory; and One or more processors implemented in circuitry, coupled to the memory and configured to: determining to decode a current block in a current picture of the video data in an affine prediction mode; determine one or more control point motion vectors (CPMV) for the current block; using the one or more CPMVs to identify an initial prediction block for the current block in a reference picture; determining a current template for the current block in the current picture; determining an initial reference sample for the initial prediction block in the reference picture; and A motion vector refinement procedure is performed based on a comparison of the current template and the initial reference template to determine a modified prediction block. The apparatus according to claim 10, wherein for performing the motion vector refinement procedure to determine the modified prediction block, the one or more processors are also configured to: A subsequent reference template that matches the current template more closely than the initial reference template is searched in a search region around the initial reference template. The apparatus according to claim 10, wherein the comparison of the current template and the initial reference template includes a template matching cost. The apparatus according to claim 12, wherein the one or more processors are also configured to: The template matching cost is determined based on a weighted per-sample comparison of samples in the current template with samples in the initial reference template. The apparatus according to claim 10, wherein the initial reference template includes a plurality of sub-blocks located above or to the left of the initial prediction block. The apparatus according to claim 10, wherein the affine prediction mode comprises a 4-parameter affine prediction mode. The apparatus according to claim 10, wherein the affine prediction mode comprises a 6-parameter affine prediction mode. The device according to claim 10, wherein the one or more processors are also configured to: determining a prediction block based on the modified prediction block; adding the predicted block to a residual block to determine a reconstructed block; applying one or more filtering operations to the reconstructed block; and A picture of decoded video data including the filtered reconstructed blocks is output. The apparatus according to claim 10, wherein the apparatus comprises a wireless communication device and also comprises a receiver configured to receive encoded video data. The apparatus according to claim 18, wherein the wireless communication device comprises a telephone handset, and wherein the receiver is configured to demodulate a signal including the encoded video data according to a wireless communication standard. The equipment according to claim 10 also includes: A display configured to display decoded video data. The device according to claim 10, wherein the device includes one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box. The device according to claim 10, wherein the device comprises a video encoding device. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determining to decode a current block in a current picture of the video data in an affine prediction mode; determine one or more control point motion vectors (CPMV) for the current block; using the one or more CPMVs to identify an initial prediction block for the current block in a reference picture; determining a current template for the current block in the current picture; determining an initial reference sample for the initial prediction block in the reference picture; and A motion vector refinement procedure is performed based on a comparison of the current template and the initial reference template to determine a modified prediction block. The computer-readable storage medium according to claim 23, wherein in order to execute the motion vector refinement procedure to determine the modified prediction block, the instructions also cause the one or more processors to: A subsequent reference template that matches the current template more closely than the initial reference template is searched in a search region around the initial reference template. The computer-readable storage medium according to claim 23, wherein the comparison of the current template and the initial reference template includes a template matching cost. The computer-readable storage medium according to claim 25, wherein the instructions cause the one or more processors to perform the following operations: The template matching cost is determined based on a weighted per-sample comparison of samples in the current template with samples in the initial reference template. The computer-readable storage medium according to claim 23, wherein the initial reference template includes a plurality of sub-blocks located above or to the left of the initial prediction block. The computer-readable storage medium according to claim 23, wherein the instructions cause the one or more processors to perform the following operations: determining a prediction block based on the modified prediction block; adding the predicted block to a residual block to determine a reconstructed block; applying one or more filtering operations to the reconstructed block; and A picture of decoded video data including the filtered reconstructed blocks is output. A device for decoding video data, the device comprising: means for determining to decode a current block in a current picture of the video data in an affine prediction mode; means for determining one or more control point motion vectors (CPMV) for the current block; a unit for identifying an initial prediction block for the current block in a reference picture using the one or more CPMVs; means for determining a current example for the current block in the current picture; means for determining an initial reference example for the initial prediction block in the reference picture; and A motion vector refinement procedure is performed based on a comparison of the current template and the initial reference template to determine a modified prediction block unit. The apparatus according to claim 29, wherein the comparison of the current template and the initial reference template includes a template matching cost, the apparatus further comprising: A unit for determining the template matching cost based on a weighted per-sample comparison of samples in the current template and a sample in the initial reference template.

Images