JP2024513066A - A scheme for adjusting adaptive resolution of motion vector differences - Google Patents

Abstract

The present disclosure generally relates to video coding, and in particular to methods and systems for providing a scheme for setting different values of allowable motion vectors when implementing adaptive resolution for motion vector difference. An exemplary method for processing a current video block of a video stream is disclosed. The method may include receiving a video stream and determining that the current video block is inter-coded based on a predictive block and a motion vector (MV), where the MV should be derived from a reference motion vector (RMV) and a motion vector difference (MVD) of the current video block. The method further includes, in response to determining that the MVD is coded at an adaptive MVD pixel resolution, determining a reference MVD pixel precision for the current video block, identifying a maximum allowable MVD pixel precision, determining a set of allowable MVD levels for the current video block based on the reference MVD pixel precision and the maximum allowable MVD pixel precision, and deriving the MVD from the video stream according to at least one MVD parameter signaled in the video stream for the current video block and the set of allowable MVD levels.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is a U.S. nonprovisional application entitled "Schemes for Adjusting Adaptive Resolution for Motion Vector Difference," filed on May 25, 2022. No. 17/824,193, filed on January 24, 2022, entitled “Further Improvement for Adaptive MVD Resolution,” U.S. Provisional Patent Application No. Claiming the benefit of priority under No. 63/302, 518. These prior applications are incorporated herein by reference in their entirety.

TECHNICAL FIELD This disclosure relates generally to video coding and, more particularly, to methods and systems for providing a scheme for setting different values of allowable motion vectors when implementing adaptive resolution for motion vector differences.

The background description provided herein is for the purpose of broadly presenting the context of the disclosure. The inventors' work, insofar as it is described in this Background section, together with other aspects of the description that may not be recognized as prior art at the time of filing this application, is disclosed in this disclosure. It is not recognized as prior art, either expressly or implicitly.

Video coding and decoding may be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a series of pictures, each picture having a spatial dimension of, for example, 1920x1080 luma samples and associated fully sampled or subsampled chroma samples. The series of pictures may have a fixed or variable picture rate (also referred to as frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920 x 1080, a frame rate of 60 frames/s, and chroma subsampling of 4:2:0 with 8 bits per pixel per color channel has a bandwidth close to 1.5 Gbit/s. I need. One hour of such video requires over 600GByte of storage space.

One purpose of video coding and decoding may be to reduce redundancy in an uncompressed input video signal through compression. Compression can help reduce the aforementioned bandwidth and/or storage space requirements by more than two orders of magnitude in some cases. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to a coding/decoding process in which the original video information is not completely preserved during coding and cannot be fully recovered during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal can be reduced, albeit with some information loss. It is small enough to make the reconstructed signal useful for its intended use. In the case of video, lossy compression is widely employed in many applications. The amount of strain that can be tolerated depends on the application. For example, users for certain consumer video streaming applications may tolerate higher distortion than users for movie or television broadcast applications. The compression ratio achievable by a particular coding algorithm may be selected or adjusted to reflect different distortion tolerances. That is, in general, higher distortion tolerance allows for coding algorithms that yield higher losses and higher compression ratios.

Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

Video codec technology may include a technique known as intra-coding. In intra-coding, sample values are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the picture can be called an intra picture. Intra pictures and their derived pictures, such as independent decoder refresh pictures, may be used to reset the decoder state and thus used as the first picture in a coded video bitstream and video session or as a still image. can be done. The samples of the block after intra-prediction can then be subjected to a transform to the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra prediction refers to a technique that minimizes sample values in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficient, the fewer bits are needed at a given quantization step size to represent the block after entropy coding.

Conventional intra-coding, for example, as known from MPEG-2 generative coding techniques, does not use intra-prediction. However, some newer video compression techniques utilize, for example, spatially adjacent surroundings obtained during encoding and/or decoding that precede in the decoding order the block of data being intra-coded or intra-decoded. techniques that attempt to code/decode blocks based on sample data and/or metadata. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that in at least some cases, intra prediction uses reference data only from the current picture being reconstructed and not from other reference pictures.

Intra prediction can take many different forms. If more than one such technique is available in a given video coding technique, the technique used can be referred to as an intra-prediction mode. One or more intra prediction modes may be provided with a particular codec. In certain cases, a mode may have sub-modes and/or be associated with various parameters, and the mode/sub-mode information and intra-coding parameters of a block of video may be individually coded. can be included in the mode codeword or all together in the mode codeword. Which codewords are used for a given mode, submode, and/or parameter combination can impact coding efficiency gains through intra-prediction, and therefore The entropy coding technique used to transform may also have an impact.

The specific mode of intra-prediction is H. 264, H. 265 and further improvements in newer coding techniques such as Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Sets (BMS). In general, intra-prediction may use adjacent sample values as they become available to form a predictor block. For example, the available values of a particular set of adjacent samples along a particular direction and/or line may be copied to a predictor block. The reference to the direction in use may be coded in the bitstream or may itself be predicted.

Referring to FIG. 1A, shown in the lower right corner are the H. is a subset of 9 predictor directions specified by 265 33 possible intra-predictor directions. The point where the arrows converge (101) represents the sample being predicted. The arrow represents the direction from which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples to the bottom left of sample (101) at an angle of 22.5 degrees from the horizontal.

Still referring to FIG. 1A, in the top left is shown a square block (104) of 4×4 samples (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S", its location in the Y dimension (eg, row index), and its location in the X dimension (eg, column index). For example, sample S21 is the second sample (from the top) in the Y dimension, and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions in block (104). The block is 4x4 samples in size, so S44 is at the bottom right. Exemplary reference samples following a similar numbering scheme are also shown. The reference sample is labeled with R for block (104), its Y position (eg, row index), and X position (column index). H. 264 and H. In both 265 and 265 prediction samples adjacent to the block being reconstructed are used.

Intra-picture prediction of block 104 may begin by copying reference sample values from neighboring samples according to a signaled prediction direction. For example, the coded video bitstream includes, for this block 104, signaling indicating the prediction direction of the arrow (102), i.e., the samples move from one or more predicted samples to the top right, at an angle of 45 degrees from the horizontal. Assume that it is predicted by In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 is then predicted from reference sample R08.

In certain cases, in order to calculate the reference samples, the values of multiple reference samples may be combined, for example by interpolation, especially if the orientation is not evenly divisible by 45 degrees.

The number of possible directions has increased as video coding technology continues to evolve. H. 264 (2003), for example, nine different directions are available for intra-prediction. This is H. 265 (2013) increases to 33, and JEM/VVC/BMS can support up to 65 directions at the time of this disclosure. Experimental studies have been carried out to help identify the most appropriate intra-prediction directions, using certain techniques of entropy coding and accepting certain bit penalties for the directions so that those most appropriate directions are can be encoded in bits. Furthermore, the direction itself may be predicted from the neighboring directions used in the intra-prediction of the decoded neighboring blocks.

FIG. 1B shows a schematic diagram (180) showing 65 intra-prediction directions by JEM to illustrate the increasing number of prediction directions in various encoding techniques that have evolved over time.

The method for mapping of bits representing intra-prediction direction to prediction direction in a coded video bitstream can vary depending on the video coding technology, e.g. from a simple direct mapping of prediction direction to intra-prediction mode; It can range from codewords, complex adaptation schemes including most probable modes, and similar techniques. However, in all cases, there may be certain directions of intro prediction that are statistically less likely to occur in video content than certain other directions. Since the goal of video compression is redundancy reduction, in well-designed video coding techniques those less likely directions may be represented by a greater number of bits than the more likely directions.

Inter picture prediction, or inter prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part of it (reference picture) is spatially shifted in the direction indicated by a motion vector (hereafter MV) and then reconstructed anew. may be used to predict a picture or a picture portion (eg, a block) that has been modified. In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third dimension being an indication of the reference picture in use (analogous to the temporal dimension).

In some video compression techniques, the current MV applicable to a particular area of sample data is separated from other MVs, e.g. spatially adjacent to the area under reconstruction and preceding the current MV in the decoding order. can be predicted from other MVs related to other areas of the sample data. By doing so, the overall amount of data required to code MVs can be significantly reduced by relying on the removal of redundancy in correlated MVs, thereby increasing compression efficiency. MV prediction can work effectively when, for example, coding an input video signal derived from a camera (known as natural video) than in the area where a single MV is applicable. This is because large areas have a statistical likelihood of moving in a similar direction in the video sequence and can therefore, in some cases, be predicted using similar motion vectors derived from the MVs of neighboring areas. be. As a result, the actual MV of a given area will be similar or identical to the MV predicted from the surrounding MVs. Such MVs may further be represented by fewer bits after entropy coding than would be used if the MVs were coded directly rather than predicted from neighboring MVs. In some cases, MV prediction may be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., a sample stream). In other cases, the MV prediction itself may be lossy, e.g. due to rounding errors when calculating the predictor from several surrounding MVs.

Various MV prediction mechanisms are available in H. 265/HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). H. Among the many MV prediction mechanisms specified by H.265, the one described below is a technique hereafter referred to as "spatial merging."

Specifically, referring to Figure 2, the current block (201) contains samples detected by the encoder during the motion search process to be predictable from a spatially shifted previous block of the same size. include. Instead of directly coding that MV, the MV uses the MV associated with any one of the five surrounding samples, denoted A0, A1, and B0, B1, B2 (202-206, respectively). may be derived from metadata associated with one or more reference pictures, for example from the most recent reference picture (in decoding order). H. In H.265, MV prediction may use predictors from the same reference pictures that neighboring blocks are using.

This disclosure relates generally to video coding, and more particularly, to signaling various motion vector or motion vector difference related syntax based on whether magnitude-dependent adaptive resolution for motion vector differences in inter prediction is used. METHODS AND SYSTEM FOR.

In an example implementation, a method is disclosed for processing a current video block of a video stream. The method includes the steps of receiving a video stream and determining that a current video block is intercoded based on a predictive block and a motion vector (MV), where MV is a reference motion vector of the current video block. vector (RMV) and motion vector difference (MVD). The method includes, in response to determining that the MVD is coded with an adaptive MVD pixel resolution, determining a reference MVD pixel precision for a current video block; identifying a maximum allowable MVD pixel precision; determining a set of acceptable MVD levels for the current video block based on an MVD pixel precision and a maximum allowed MVD pixel precision; deriving an MVD from the video stream according to at least one MVD parameter.

In the above implementations, the reference MVD pixel precision of the current video block is specified/signaled/derived at the sequence level, picture level, frame level, superblock level, or coding block level.

In any one of the above implementations, the reference MVD pixel precision of the current video block depends on the MVD class associated with the MVD of the current video block.

In any one of the above implementations, the reference MVD pixel precision of the current video block depends on the MVD size of the MVD of the current video block. In any one of the above implementations, the maximum allowed MVD pixel accuracy is predefined.

In any one of the above implementations, the method may further include determining a current MVD class from among a predefined set of MVD classes. Determining a set of acceptable MVD levels for the MVD based on the reference MVD pixel accuracy and the maximum allowable MVD pixel accuracy includes determining the set of acceptable MVD levels for the current video block based on the reference MVD pixel accuracy. and excluding from the reference MVD level set determined based on the current MVD class, an MVD level associated with an MVD pixel precision that is greater than or equal to a maximum allowed MVD pixel precision.

In any one of the above implementations, the maximum allowed MVD pixel precision is 1/4 pixel. In any one of the above implementations, MVD levels associated with accuracy of ⅛ pixel or better are excluded from the set of allowable MVD levels for the current video block.

In any one of the above implementations, the method may further include determining a current MVD class from among a predefined set of MVD classes. MVD levels associated with fractional MVD precision may be included in the set of acceptable MVD levels, regardless of the reference MVD precision, if the current MVD class is less than or equal to the threshold MVD class.

In any one of the above implementations, the threshold MVD class may be the lowest MVD class among a predefined set of MVD classes.

In any one of the above implementations, the method may further include determining a magnitude of the MVD, wherein the MVD level associated with an MVD accuracy greater than a threshold MVD precision is such that the magnitude of the MVD is greater than the threshold. It is acceptable in the set of acceptable MVD levels only if the MVD size is less than or equal to the MVD size.

In any one of the above implementations, the threshold MVD magnitude is 2 pixels or less. In any one of the above implementations, the threshold MVD accuracy is 1 pixel. In any one of the above implementations, MVD levels associated with MVD accuracy of 1/4 pixel or greater are only allowed if the MVD magnitude is 1/2 pixel or less. In any one of the above implementations, the maximum allowed MVD pixel precision may be no greater than the reference MVD pixel precision.

In other example implementations, a method for processing a current video block of a video stream is disclosed. The method includes the steps of: receiving a video stream; determining that a current video block is intercoded and associated with a plurality of reference frames; and determining whether an adaptive motion vector difference (MVD) pixel resolution is applied to at least one of the pixels.

In the above implementations, the signaling may include a single bit flag to indicate whether the adaptive MVD pixel resolution applies to all or none of the multiple reference frames.

In any one of the above implementations, the signaling may include separate flags each corresponding to one of the plurality of reference frames to indicate whether adaptive MVD pixel resolution is applied.

In any one of the above implementations, the signaling includes, for each of the plurality of reference frames, an implicit indication that the adaptive MVD pixel resolution is not applied when the MVD corresponding to each of the plurality of reference frames is zero. , and a single-bit flag to indicate whether adaptive MVD pixel resolution is applied when the MVD corresponding to each of the plurality of reference frames is non-zero.

In other example implementations, a method for processing a current video block of a video stream is disclosed. The method includes the steps of receiving a video stream and determining that a current video block is inter-coded based on a predictive block and a motion vector (MV), where MV is a reference motion vector for the current video block. the current MVD class of the MVD from among a predefined set of MVD classes, which should be derived from the motion vector difference (RMV) and the motion vector difference (MVD); deriving at least one context for entropy decoding at least one explicit signaling in the video stream based on the at least one explicit signaling for at least one component of the MVD; a step included in the video stream to specify the MVD pixel resolution of the at least one component from the video stream using at least one context to determine the MVD pixel resolution for at least one component of the MVD; entropy decoding the explicit signaling.

In the above implementation, the at least one component of the MVD may include a horizontal component and a vertical component of the MVD, and the at least one context is associated with one of the horizontal and vertical components of the MVD, respectively. The horizontal and vertical components are associated with separate MVD pixel resolutions.

Aspects of the present disclosure also provide a video encoding or decoding device or apparatus that includes a circuit configured to perform any of the method implementations described above.

Aspects of the present disclosure also include a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and/or encoding, cause the computer to perform a method for video decoding and/or encoding. provide.

Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings.

FIG. 2 is a schematic diagram of an exemplary subset of intra-prediction directional modes; FIG. 3 is a diagram illustrating an example intra prediction direction. 2 is a schematic diagram illustrating a current block and its surrounding spatial merging candidates for motion vector prediction in an example; FIG. 1 is a schematic diagram of a simplified block diagram of a communication system (300), according to an example embodiment. FIG. 1 is a schematic diagram of a simplified block diagram of a communication system (400), according to an example embodiment. FIG. 1 is a schematic diagram of a simplified block diagram of a video decoder, according to an example embodiment; FIG. 1 is a schematic diagram of a simplified block diagram of a video encoder, according to an example embodiment; FIG. FIG. 2 is a block diagram of a video encoder, according to another example embodiment. FIG. 3 is a block diagram of a video decoder, according to another example embodiment. FIG. 3 is a diagram illustrating a scheme of coding block partitioning according to an example embodiment of the present disclosure. FIG. 7 illustrates another scheme of coding block partitioning according to an example embodiment of the present disclosure. FIG. 7 illustrates another scheme of coding block partitioning according to an example embodiment of the present disclosure. FIG. 3 illustrates an example partitioning of a base block into coding blocks according to an example partitioning scheme. FIG. 3 illustrates an exemplary trisection scheme. FIG. 2 is a diagram illustrating an example quadtree binary tree coding block partitioning scheme. FIG. 3 illustrates a scheme for dividing a coding block into multiple transform blocks and a coding order of the transform blocks, according to an example embodiment of the present disclosure. FIG. 7 is a diagram illustrating another scheme for dividing a coding block into multiple transform blocks and coding order of transform blocks, according to an example embodiment of the present disclosure. FIG. 6 illustrates another scheme for dividing a coding block into multiple transform blocks, according to an example embodiment of the present disclosure. FIG. 3 illustrates a flowchart of a method, according to an example embodiment of the disclosure. FIG. 7 illustrates another flowchart of a method, according to an example embodiment of the disclosure. FIG. 3 illustrates a flowchart of a method, according to an example embodiment of the disclosure. 1 is a schematic diagram of a computer system, according to an example embodiment of the present disclosure. FIG.

Throughout this specification and claims, terms may have nuanced meanings that are suggested or implied within the context beyond those explicitly stated. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and "in other embodiments" as used herein The phrases "in other embodiments" or "in other embodiments" are not necessarily referring to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and are not necessarily referring to "other implementations" as used herein. The phrases "in one embodiment" or "in other implementations" are not necessarily referring to different implementations. For example, claimed subject matter is intended to include any combination of all or a portion of the example embodiments/implementations.

Generally, terminology may be understood at least in part from its usage in context. For example, terms such as "and," "or," or "and/or" as used herein have various meanings that may depend, at least in part, on the context in which such terms are used. May include. Typically, "or" when used to relate a list such as A, B, or C is used here in an inclusive sense, A, B, and C, and here exclusively. intended to mean A, B, or C used in that sense. Additionally, the terms "one or more" or "at least one" as used herein may describe any feature, structure, or characteristic in the singular sense, depending at least in part on the context. or may be used in more than one sense to describe a feature, structure, or combination of characteristics. Similarly, terms such as "a," "an," or "the" are also understood to convey singular or plural usage, depending at least in part on the context. may be done. Additionally, the terms "based on" or "determined by" may be understood to not necessarily convey an exclusive set of factors, but instead also at least partially dependent on the context. may also allow for the existence of additional factors that are not necessarily explicitly stated. FIG. 3 depicts a simplified block diagram of a communication system (300), according to one embodiment of the present disclosure. The communication system (300) includes, for example, a plurality of terminal devices that can communicate with each other via a network (350). For example, a communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via a network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform unidirectional transmission of data. For example, a terminal device (310) codes video data (e.g., of a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) over a network (350). It is possible. Encoded video data may be sent in the form of one or more coded video bitstreams. A terminal device (320) may receive coded video data from a network (350), decode the coded video data to recover video pictures, and display the video pictures according to the recovered video data. Unidirectional data transmission may be implemented, such as in media serving applications.

In other examples, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data, which may occur, for example, during video conferencing applications. include. For bidirectional transmission of data, in one example, each terminal device of terminal devices (330) and (340) transmits to the other terminal device of terminal devices (330) and (340) via the network (350). may code video data (e.g., of a stream of video pictures captured by the terminal device) for video processing. Each of the terminal devices (330) and (340) also receives the coded video data transmitted by the other terminal device of the terminal devices (330) and (340), and decodes the coded video data. The video picture may be encoded to recover the video picture and displayed on an accessible display device according to the recovered video data.

In the example of FIG. 3, terminal devices (310), (320), (330), and (340) may be implemented as servers, personal computers, and smartphones, but the application of the principles underlying this disclosure need not be so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, specialized video conferencing equipment, and the like. Network (350) may include any wired and/or wireless communication network that conveys coded video data between terminal devices (310), (320), (330), and (340). represents the number or type of network. The communication network (350) may exchange data over circuit-switched channels, packet-switched channels, and/or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For purposes of this description, the architecture and topology of network (350) may not be important to the operation of the present disclosure unless explicitly described herein.

FIG. 4 illustrates the placement of a video encoder and video decoder in a video streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter is equally applicable to other video applications, including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc. It's possible.

The video streaming system may include a video capture subsystem (413) that may include a video source (401), e.g. a digital camera, for creating a stream (402) of uncompressed video pictures or images. good. In one example, the stream of video pictures (402) includes samples recorded by a digital camera of video source 401. The stream of video pictures (402), shown as a thick line to emphasize the large amount of data when compared to the encoded video data (404) (or coded video bitstream), is the video source (401) The video encoder (403) may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video encoder (403). Video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. Encoded video data (404) (or encoded video bitstream (404)) is shown with thin lines to emphasize the low amount of data when compared to the stream of uncompressed video pictures (402). , may be stored for future use on the streaming server (405) or directly on a downstream video device (not shown). One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, access the streaming server (405) for copies (407) of encoded video data (404) and (409) can be extracted. Client subsystem (406) may include, for example, a video decoder (410) within electronic device (430). A video decoder (410) decodes an input copy (407) of the encoded video data and displays it uncompressed on a display (412) (e.g., a display screen) or other rendering device (not shown). Create an output stream (411) of video pictures that can be rendered. Video decoder 410 may be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, encoded video data (404), (407), and (409) (eg, video bitstreams) may be encoded according to a particular video coding/compression standard. Examples of these standards include ITU-T Recommendation H. 265 included. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video coding standards.

Note that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) can include a video decoder (not shown), and electronic device (430) can also include a video encoder (not shown).

FIG. 5 depicts a block diagram of a video decoder (510), according to any embodiment of the present disclosure below. A video decoder (510) may be included in an electronic device (530). The electronic device (530) can include a receiver (531) (eg, a receiving circuit). A video decoder (510) may be used in place of the video decoder (410) in the example of FIG.

A receiver (531) may receive one or more coded video sequences to be decoded by a video decoder (510). In the same or other embodiments, one coded video sequence may be decoded at a time, and the decoding of each coded video sequence is independent of other coded video sequences. Each video sequence may be associated with multiple video frames or video images. A coded video sequence may be received from a channel (501), where the channel (501) is a hardware/software link to a storage device that stores encoded video data or a streaming source that transmits encoded video data. It can be. A receiver (531) may receive encoded video data along with other data, such as coded audio data and/or ancillary data stream, which may be transferred to respective processing circuits (not shown). A receiver (531) may separate the coded video sequence from other data. To combat network jitter, a buffer memory (515) may be placed between the receiver (531) and the entropy decoder/parser (520) (hereinafter "parser (520)"). In certain applications, buffer memory (515) may be implemented as part of a video decoder (510). In other applications, the buffer memory (515) may be separate and external to the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), e.g. to counter network jitter, or a buffer memory (not shown) external to the video decoder (510), e.g. to handle playback timing. There may be other additional buffer memories (515) inside. When the receiver (531) is receiving data from a storage/transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may be unnecessary or small. be able to. For use in best effort packet networks such as the Internet, a buffer memory (515) of sufficient size may be required, and its size can be relatively large. Such buffer memory may be implemented with an adaptive size and may be at least partially implemented in an operating system or similar element (not shown) external to the video decoder (510).

The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. Those symbol categories contain information used to manage the operation of a video decoder (510) and potentially even if it is an integral part of an electronic device (530), as shown in Figure 5. and information for controlling a rendering device, such as a display (512) (eg, a display screen), which may be coupled to the electronic device (530), although it may not be possible to do so. The control information for the rendering device may be in the form of a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). Parser (520) may parse/entropy decode coded video sequences received by parser (520). Entropy coding of a coded video sequence may be in accordance with a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. can. A parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the subgroup. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser (520) may also extract information such as transform coefficients (eg, Fourier transform coefficients), quantization parameter values, motion vectors, etc. from the coded video sequence.

A parser (520) may perform entropy decoding/parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

The reconstruction of the symbols (521) can be performed in several different processing units or functional units, depending on the type of coded video picture or parts thereof (inter-pictures and intra-pictures, inter-blocks and intra-blocks, etc.), as well as other factors. can include. The units included and how the units are included may be controlled by subgroup control information parsed from the coded video sequence by a parser (520). The flow of such subgroup control information between the parser (520) and the following processing units or functional units is not illustrated for the sake of brevity.

Besides the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units, as explained below. In actual implementations operating under commercial constraints, many of these functional units will interact closely with each other and may be, at least partially, integrated with each other. However, for the purpose of clearly describing the various features of the disclosed subject matter, a conceptual subdivision into functional units is employed in the following disclosure.

The first unit may include a scaler/inverse transform unit (551). The scaler/inverse transform unit (551) provides control information to the parser, including quantized transform coefficients as well as information indicating what type of inverse transform to use, block size, quantized coefficients/parameters, quantized scaling matrix, etc. (520) as a symbol (521). The scaler/inverse transform unit (551) can output blocks containing sample values that can be input to the aggregator (555).

In some cases, the output samples of the scaler/inverse transform (551) are intra-coded blocks, i.e., do not use prediction information from previously reconstructed pictures, but do not use prediction information from previously reconstructed pictures of the current picture. It may concern blocks that can use prediction information from parts. Such prediction information may be provided by an intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) uses information from surrounding blocks that have already been reconstructed and stored in the current picture buffer (558) to create a block of the same size and shape as the block being reconstructed. blocks may be generated. A current picture buffer (558) buffers, for example, a partially reconstructed current picture and/or a fully reconstructed current picture. The aggregator (555) may, in some implementations, add the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler/inverse transform unit (551) on a sample-by-sample basis. good.

In other cases, the output samples of the scaler/inverse transform unit (551) may relate to inter-coded and potentially motion-compensated blocks. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion compensating the fetched samples according to the symbols (521) associated with the block, these samples are added to the output of the scaler/inverse transform unit (551) by the aggregator (555) to generate output sample information. (The output of unit 551 may be referred to as a residual sample or a residual signal). The address in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the predicted samples may have, for example, an X component, a Y component (shift), and a reference picture component (time). (521) available to the motion compensated prediction unit (553). Motion compensation may also include interpolation of sample values fetched from a reference picture memory (557) when subsample accurate motion vectors are used, and may also be associated with a motion vector prediction mechanism, etc. good.

The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). may include in-loop filter techniques that respond not only to meta-information obtained during decoding of previous parts (in decoding order) of a coded picture or coded video sequence; It can also respond to reconstructed and loop-filtered sample values. As described in further detail below, several types of loop filters may be included as part of loop filter unit 556 in various orders.

The output of the loop filter unit (556) may be a sample stream that can be output to the rendering device (512) as well as stored in the reference picture memory (557) for use in future inter-picture predictions. .

Once a particular coded picture is fully reconstructed, it may be used as a reference picture for future inter-picture prediction. For example, once the coded picture corresponding to the current picture has been completely reconstructed and the coded picture has been identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) The unused current picture buffer, which can be part of the reference picture memory (557), can be reallocated before starting reconstruction of the next coded picture.

The video decoder (510) may be configured according to, for example, ITU-T Recommendation H. Decoding operations may be performed according to a predetermined video compression technique adopted in standards such as H.265. A coded video sequence is defined by the video compression technology used, in the sense that the coded video sequence adheres both to the syntax of the video compression technology or standard and to the profile documented in the video compression technology or standard. or may conform to the syntax specified by the standard. Specifically, a profile may select a particular tool from all available tools in a video compression technology or standard as the only tool that can be used under that profile. To comply with a standard, the complexity of the coded video sequence may be within a range defined by the video compression technology or level of the standard. In some cases, the level limits maximum picture size, maximum frame rate, maximum reconstruction sample rate (eg, measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may in some cases be further limited by the virtual reference decoder (HRD) specification and HRD buffer management metadata signaled in the coded video sequence.

In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. Additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and/or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

FIG. 6 shows a block diagram of a video encoder (603), according to an example embodiment of the present disclosure. A video encoder (603) may be included in an electronic device (620). Electronic device (620) may further include a transmitter (640) (eg, transmit circuitry). A video encoder (603) can be used in place of the video encoder (403) in the example of FIG.

A video encoder (603) receives video samples from a video source (601) (not part of the electronic device (620) in the example of Figure 6) that may capture video images to be coded by the video encoder (603). obtain. In other examples, video source (601) may be implemented as part of electronic device (620).

The video source (601) can be any suitable bit depth (e.g. 8-bit, 10-bit, 12-bit,…), any color space (e.g. BT.601 YCrCb, RGB, XYZ…), and any suitable may provide a source video sequence to be coded by a video encoder (603) in the form of a digital video sample stream, which may have a sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). . In a media serving system, a video source (601) may be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. Video data may be provided as multiple individual pictures or images that impart motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc. being used. Those skilled in the art can easily understand the relationship between pixels and samples. The following discussion focuses on the sample.

According to some example embodiments, a video encoder (603) converts the pictures of the source video sequence into a coded video sequence in real time or under any other time constraints required by the application. (643). Enforcing appropriate coding speeds constitutes one function of the controller (650). In some embodiments, controller (650) may be operably coupled to and control other functional units, as described below. For the sake of brevity, the coupling is not shown. Parameters set by the controller (650) include rate control related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search. This may include ranges, etc. Controller (650) may be configured with other suitable functionality related to video encoder (603) optimized for a particular system design.

In some example embodiments, video encoder (603) may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop includes a source coder (630) (e.g., involved in generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture). and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to match what the (remote) decoder would produce, even though the embedded decoder 633 processes the video stream coded by the source coder 630 without entropy coding. Create sample data in a similar manner (as in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the coded video bitstream may be lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream yields bit-exact results regardless of the location of the decoder (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values as reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained due to channel errors, for example) is used to improve coding quality.

The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with FIG. Referring also briefly to Figure 5, however, since the symbols are available and the encoding/decoding of symbols into the coded video sequence by the entropy coder (645) and parser (520) can be reversible, the buffer memory The entropy decoding portion of the video decoder (510), including (515) and the parser (520), may not be fully implemented in the local decoder (633) within the encoder.

What can be said at this point is that any decoder technique other than parsing/entropy decoding, which can exist only in the decoder, must also necessarily exist in substantially the same functional form in the corresponding encoder. It means getting. As such, the disclosed subject matter may focus on decoder operation, which is similar to the decoding portion of an encoder. Thus, the description of the encoder technology may be omitted since it is the opposite of the decoder technology that is generally described. A more detailed description of the encoder is provided below, only in certain areas or aspects.

In operation, in some example implementations, the source coder (630) refers to one or more previously coded pictures from the video sequence designated as "reference pictures" to predictively code the Motion compensated predictive coding may be performed to code the input picture. In this way, the coding engine (632) codes the color channel difference (or residual) between the pixel block of the input picture and the pixel block of the reference picture, which may be selected as a predictive reference to the input picture. . The term "residual" and its adjective form "residual" may be used interchangeably.

A local video decoder (633) may decode coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a non-reversible process. When the coded video data may be decoded with a video decoder (not shown in Figure 6), the reconstructed video sequence is typically a replica of the source video sequence with some errors. could be. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference picture and cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) creates a copy of the reconstructed reference picture that has content in common with the reconstructed reference picture obtained by the far-end (remote) video decoder (without transmission errors). Can be stored locally.

Predictor (635) may perform a predictive search for coding engine (632). That is, for a new picture to be coded, the predictor (635) uses sample data (as candidate reference pixel blocks) or reference picture motion vectors, block shapes, etc., which can serve as suitable prediction references for the new pixels. A reference picture memory (634) may be searched for specific metadata. A predictor (635) may operate on the sample block by pixel block to find an appropriate prediction reference. In some cases, the input picture has a predicted reference drawn from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635). Good too.

Controller (650) may manage coding operations of source coder (630), including, for example, setting parameters and subgroup parameters used to encode video data.

The outputs of all the aforementioned functional units can be subjected to entropy coding in an entropy coder (645). The entropy coder (645) converts the symbols generated by the various functional units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

The transmitter (640) may buffer the coded video sequence created by the entropy coder (645) to prepare it for transmission over a communication channel (660), the communication channel (660) , a hardware/software link to a storage device that stores the encoded video data. The transmitter (640) merges the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and/or an auxiliary data stream (source not shown). It is possible.

A controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures are often assigned as one of the following picture types:

Intra pictures (I pictures) may be those that can be coded and decoded without using other pictures in the sequence as sources of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of those variations of I-pictures and their respective uses and characteristics.

A predicted picture (P picture) is a picture that may be coded and decoded using intra or inter prediction using at most one motion vector and reference index to predict the sample values of each block. obtain.

Bidirectionally predicted pictures (B pictures) are pictures that may be coded and decoded using intra or inter prediction using up to two motion vectors and reference indices to predict the sample values of each block. It can be. Similarly, multi-predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

A source picture is generally spatially subdivided into multiple sample coding blocks (eg, blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and may be coded block by block. A block may be predictively coded with reference to other (already coded) blocks, as determined by a coding assignment applied to each picture of the block. For example, blocks of an I picture may be coded non-predictively or predictively with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A block of B pictures may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures. A source picture or an intermediately processed picture may be subdivided into other types of blocks for other purposes. Partitioning of coding blocks and other types of blocks may or may not follow the same method, as explained in more detail below.

The video encoder (603) complies with ITU-T Recommendation H. The coding operation may be performed according to a predetermined video coding technique or standard, such as H.265. In its operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technology or standard being used.

In some example embodiments, transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. Additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

Video may be captured as multiple source pictures (video pictures) in chronological order. Intra-picture prediction (often abbreviated as intra-prediction) takes advantage of spatial correlations in a given picture, and inter-picture prediction takes advantage of temporal or other correlations between pictures. For example, a particular picture being encoded/decoded, called the current picture, may be divided into blocks. A block in the current picture may be coded by a vector called a motion vector if it is similar to a reference block in a previously coded still buffered reference picture in the video. A motion vector points to a reference block within a reference picture and can have a third dimension that identifies the reference picture if multiple reference pictures are used.

In some example embodiments, bi-prediction techniques may be used for inter-picture prediction. According to such bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, which both advance the current picture in the video in the decoding order (but (in the display order, each can be in the past or future). A block in the current picture is coded by a first motion vector pointing to a first reference block in a first reference picture, and a second motion vector pointing to a second reference block in a second reference picture. can be done. A block may be predicted cooperatively by a combination of the first reference block and the second reference block.

Additionally, merge mode techniques may be used to improve coding efficiency in inter-picture prediction.

According to some example embodiments of the present disclosure, predictions, such as inter-picture prediction and intra-picture prediction, are performed on a block-by-block basis. For example, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. may have. Generally, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or 32x32 pixel four CUs. Each of the one or more of the 32x32 blocks may be further divided into four CUs of 16x16 pixels. In some example embodiments, each CU may be analyzed during encoding to determine its prediction type among various prediction types, such as inter-prediction type and intra-prediction type. A CU may be divided into one or more prediction units (PUs) depending on temporal and/or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding/decoding) are performed in units of prediction blocks. The division of CU into PUs (or PBs of different color channels) can be done in various spatial patterns. A luma PB or chroma PB may include a matrix of sample values (eg, luma values), eg, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

FIG. 7 shows a diagram of a video encoder (703), according to another example embodiment of the present disclosure. A video encoder (703) receives a processing block (e.g., a prediction block) of sample values in a current video picture within a sequence of video pictures, and converts the processing block into a coded block that is part of a coded video sequence. Configured to encode into a picture. The example video encoder (703) may be used in place of the example video encoder (403) of FIG.

For example, the video encoder (703) receives a matrix of sample values for a processing block, such as an 8x8 sample prediction block. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode. to decide. If it is determined that the processing block is coded in intra mode, the video encoder (703) encodes the processing block into a coded picture using an intra prediction technique, and if the processing block is determined to be coded in inter mode or bi-prediction mode. If determined to be coded, a video encoder (703) may encode the processing block into a coded picture using inter-prediction techniques or bi-prediction techniques, respectively. In some example embodiments, as a submode of inter-picture prediction, motion vectors are derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. A merge mode can be used. In some other example embodiments, there may be motion vector components applicable to the current block. Accordingly, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode determination module, to determine the prediction mode of the processing block.

In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), coupled to each other as shown in the example arrangement of FIG. It includes a switch (726), a residual encoder (724), a general purpose controller (721), and an entropy encoder (725).

An inter-encoder (730) receives samples of a current block (e.g., a processing block) and combines that block with one or more reference blocks in a reference picture (e.g., in a previous picture and in a later picture in display order). (blocks within), generate inter-prediction information (e.g., redundant information by inter-encoding techniques, motion vectors, descriptions of merge mode information), and generate inter-prediction information based on the inter-prediction information using any suitable technique. The method is configured to calculate a prediction result (eg, a predicted block). In some examples, the reference picture is a decoding unit 633 incorporated in the example encoder 620 of FIG. 6 (depicted as residual decoder 728 in FIG. 7, as described in further detail below). is a decoded reference picture that is decoded based on the encoded video information using .

The intra-encoder (722) receives samples of the current block (e.g., a processing block), compares the block with already coded blocks in the same picture, produces transformed quantization coefficients, and optionally Intra-prediction information (eg, intra-prediction direction information due to one or more intra-encoding techniques) is also configured to generate. Intra encoder (722) may calculate an intra prediction result (eg, a predicted block) based on the intra prediction information and reference blocks within the same picture.

The general-purpose controller (721) may be configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. In one example, the general purpose controller (721) determines a prediction mode for the block and provides a control signal to the switch (726) based on the prediction mode. For example, if the prediction mode is intra mode, the general purpose controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and the entropy encoder (725 ) to select intra prediction information and include the intra prediction information in the bitstream, and if the prediction mode of the block is inter mode, the general purpose controller (721) controls the switch (726) to A residual calculator (723) selects an inter-prediction result for use and controls an entropy encoder (725) to select inter-prediction information and include the inter-prediction information in the bitstream.

The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result for the selected block from the intra-encoder (722) or the inter-encoder (730). can be done. A residual encoder (724) may be configured to encode residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). A residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter encoder (730) may generate a decoded block based on the decoded residual data and inter prediction information, and the intra encoder (722) may generate a decoded block based on the decoded residual data and inter prediction information. A decoded block can be generated based on intra prediction information. The decoded blocks are suitably processed to generate decoded pictures, which may be buffered in memory circuitry (not shown) and used as reference pictures.

An entropy encoder (725) may be configured to format the bitstream to include encoded blocks and perform entropy coding. The entropy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general purpose control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. . Residual information may not be present when coding a block in merge submode of either inter mode or bi-prediction mode.

FIG. 8 shows a diagram of an example video decoder (810), according to another embodiment of the present disclosure. A video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, video decoder (810) may be used in place of video decoder (410) in the example of FIG.

In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), coupled to each other as shown in the exemplary arrangement of FIG. Includes a reconstruction module (874), and an intra decoder (872).

The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols representing syntactic elements of which the coded picture is composed. Such symbols are e.g. Prediction information (eg, intra-prediction information or inter-prediction information) that can identify particular samples or metadata used for prediction, such as residual information in the form of quantized transform coefficients, etc., may be included. In one example, if the prediction mode is inter-prediction mode or bi-prediction mode, inter-prediction information is provided to the inter-decoder (880), and if the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). provided. The residual information may be subjected to inverse quantization and provided to a residual decoder (873).

An inter-decoder (880) may be configured to receive inter-prediction information and generate inter-prediction results based on the inter-prediction information.

An intra decoder (872) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

The residual decoder (873) may be configured to perform inverse quantization to extract inverse quantized transform coefficients and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. good. The residual decoder (873) may also utilize certain control information (to include the quantizer parameter (QP)), which information may be provided by the entropy decoder (871) (this The data path is not shown since it may only be a small amount of control information).

The reconstruction module (874) combines, in the spatial domain, the residual as an output by the residual decoder (873) and the prediction result (possibly as an output by an inter-prediction module or an intra-prediction module) to perform reconstruction. may be configured to form a reconstructed block that forms part of a reconstructed picture as part of a reconstructed video. Note that other suitable operations, such as deblocking operations, may be performed to improve visual quality.

Note that video encoders (403), (603), and (703) and video decoders (410), (510), and (810) may be implemented using any suitable technique. . In some exemplary embodiments, video encoders (403), (603), and (703) and video decoders (410), (510), and (810) incorporate one or more integrated circuits. can be implemented using In other embodiments, video encoders (403), (603), and (603) and video decoders (410), (510), and (810) implement one or more processors that execute software instructions. can be implemented using

Turning to block partitioning for coding and decoding, a general partitioning may start from a base block, using a predefined set of rules, a specific pattern, a partitioning tree, or an arbitrary partitioning structure or You may follow the method. Partitioning may be hierarchical and recursive. After separating or partitioning the base blocks according to any of the example partitioning procedures described below or other procedures, or combinations thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels within the partitioning hierarchy and may be partitions of various shapes. Each of the partitions may be called a coding block (CB). In various exemplary partitioning implementations described further below, each resulting CB may be a CB of any allowed size and partitioning level. Such a partition forms a unit for which some basic coding/decoding decisions may be made and coding/decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. Therefore, it is called a coding block. The highest or deepest level in the final partition represents the depth of the coding block partitioning tree structure. The coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color is sometimes called a coding block tree (CBT).

The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The splitting patterns or splitting structures of the various color channels within a CTU may or may not be the same.

In some implementations, the splitting tree scheme or structure used for luma and chroma channels may not need to be the same. In other words, the luma channel and chroma channel may have separate coding tree structures or patterns. Additionally, whether the luma and chroma channels use the same or different coding split tree structures, and the actual coding split tree structure to be used, depends on whether the slice being coded is a P slice or not. It may depend on whether it is a B slice or an I slice. For example, for I-slices, chroma and luma channels may have separate coding partition tree structures or coding partition tree structure modes, whereas for P-slices or B-slices, luma and chroma channels may have the same coding partition The tree method may be shared. When separate coding split tree structures or modes are applied, luma channels may be split into CBs by one coding split tree structure, and chroma channels may be split into chroma CBs by another coding split tree structure. good.

In some example implementations, a predetermined partitioning pattern may be applied to the base block. As shown in Figure 9, an exemplary four-way splitting tree may start from a first predefined level (e.g., 64x64 block level or other size as the base block size), and the base Blocks may be divided hierarchically down to a predefined lowest level (eg, 4x4 level). For example, a base block can follow four predefined partitioning options or patterns denoted by 902, 904, 906, and 908, and a partition designated as R can follow the same partitioning options shown in Figure 9. Recursive partitioning is possible in that the can be repeated at lower scales down to the lowest level (e.g. 4x4 level). In some implementations, additional restrictions may be placed on the partitioning scheme of FIG. 9. In the implementation of Figure 9, rectangular partitions (e.g., 1:2/2:1 rectangular partitions) may be allowed but not recursive, whereas square partitions are recursive. It is permissible. If necessary, subsequent partitioning of FIG. 9 by recursion generates the final set of coding blocks. A coding tree depth may be further defined to indicate the division depth from the root node or root block. For example, the coding tree depth for the root node or root block of a 64×64 block may be set to 0, and the coding tree depth increases by 1 after the root block is divided one more time according to FIG. The maximum or deepest level from the 64x64 base block to the 4x4 minimum partition is 4 in the above scheme (starting from level 0). Such a partitioning scheme may be applied to one or more of the color channels. Each color channel may be independently split according to the scheme of FIG. ). Alternatively, two or more color channels may share the same hierarchical pattern tree of Figure 9 (e.g., for two or more color channels at each hierarchical level, the same dividing pattern within the predefined pattern or options may be selected).

FIG. 10 shows another example predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As shown in FIG. 10, ten exemplary partitioning structures or patterns may be predefined. The root block may start from a predefined level (eg, from a 128x128 level or a 64x64 level base block). The exemplary partition structure of FIG. 10 includes various 2:1/1:2 and 4:1/1:4 rectangular partitions. A partition type with three subpartitions, shown at 1002, 1004, 1006, and 1008 in the second column of FIG. 10, may be referred to as a "T-type" partition. "T" partitions 1002, 1004, 1006, and 1008 may be referred to as left T, top T, right T, and bottom T. In some example implementations, none of the rectangular partitions in FIG. 10 can be further subdivided. A coding tree depth may be further defined to indicate the division depth from the root node or root block. For example, the coding tree depth for the root node or root block of a 128x128 block may be set to 0, and the coding tree depth increases by 1 after the root block is divided one more time according to FIG. 10. In some implementations, only 1010 all-square partitions may be capable of recursive partitioning to the next level of the partitioning tree following the pattern of FIG. 10. In other words, recursive partitioning may not be possible for square partitions within T-shaped patterns 1002, 1004, 1006, and 1008. If necessary, the partitioning procedure following FIG. 10 by recursion generates the final set of coding blocks. Such a scheme may be applied to one or more of the color channels. In some implementations, more flexibility may be added to the use of partitions with less than 8x8 levels. For example, in some cases, 2x2 chroma inter prediction may be used.

In some other example implementations for coding block partitioning, a quadtree structure may be used to partition base blocks or intermediate blocks into quadtree partitions. Such a quadtree partition may be applied hierarchically and recursively to arbitrary square partitions. Whether a base block or intermediate block or partition is further quadtree partitioned may be adapted to various local characteristics of the base block or intermediate block/partition. Quadtree splitting at picture boundaries may also be suitable. For example, implicit quadtree splitting may occur at picture boundaries such that blocks continue to be quadtree split until their size fits within the picture boundaries.

In some other example implementations, hierarchical bipartition from the base block may be used. In such a scheme, the base block or intermediate level block may be divided into two partitions. The bisection can be either horizontal or vertical. For example, a horizontal bipartition may divide a base block or intermediate block into equal left and right partitions. Similarly, vertical bisection may divide a base block or intermediate block into equal upper and lower partitions. Such bipartition may be hierarchical and recursive. Whether the bisection scheme should be continued, and if the scheme continues further, whether horizontal bisection or vertical bisection should be used may be determined in each of the base blocks or intermediate blocks. . In some implementations, further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further segmentation may be stopped upon reaching a predefined segmentation level or depth from the base block. In some implementations, the aspect ratio of a partition may be limited. For example, the aspect ratio of a partition may not be smaller than 1:4 (or larger than 4:1). Therefore, a vertical strip partition with a 4:1 vertical-to-horizontal aspect ratio can only be subdivided vertically into upper and lower partitions each having a 2:1 vertical-to-horizontal aspect ratio.

In still some other examples, a split-by-third scheme may be used to partition the base block or any intermediate blocks, as shown in FIG. 13. The ternary pattern may be implemented vertically, as shown at 1302 in FIG. 13, or horizontally, as shown at 1304 in FIG. The exemplary split ratios in FIG. 13 are shown as 1:2:1 either vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. While it is possible for such ternary tree partitions to capture objects located at block centers in one consecutive partition, quadtrees and binary trees always partition along block centers, and thus objects Such a tripartition scheme may be used to complement a quadtree or bipartite structure in that it divides the ``input'' into separate partitions. In some implementations, the width and height of the exemplary ternary tree partitions are always powers of two to avoid further transformations.

The above partitioning schemes can be combined in any way at different partitioning levels. As an example, the quadtree and bipartition schemes described above may be combined to partition the base block into a quadtree-binary tree (QTBT) structure. In such a scheme, the base block or intermediate blocks/partitions may be either quadtree partitioned or bipartitioned, subject to a set of predefined conditions, if specified. A specific example is shown in FIG. In the example of FIG. 14, the base block is first quadtree partitioned into four partitions, as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then quadtree-divided into four further partitions (such as 1408) or at the next level (such as horizontal or It is either bisected into two further partitions (either vertically) or not (such as 1404). Bipartition or quadtree partitioning may be enabled recursively for square partitions, as shown by the overall exemplary partitioning pattern at 1410 and the corresponding tree structure/representation at 1420, where the solid line It represents a quadtree partition, and the dashed line represents a bipartition. A flag may be used for each bipartition node (non-leaf binary partition) to indicate whether the bipartition is horizontal or vertical. For example, a flag "0" may represent a horizontal bisection, and a flag "1" may represent a vertical bisection, as shown at 1420, which matches the partition structure of 1410. For quadtree partitions, there is no need to indicate the partition type, as quadtree partitions always partition a block or partition both horizontally and vertically to produce four subblocks/partitions of equal size. In some implementations, a flag '1' may represent a horizontal bisection and a flag '0' may represent a vertical bisection.

In some example implementations of QTBT, the quadtree and binary splitting rule sets may be represented by the following predefined parameters and corresponding functions associated therewith.
- CTU size: Quadtree root node size (base block size)
-MinQTSize: Minimum allowable quadtree leaf node size -MaxBTSize: Maximum allowable binary tree root node size -MaxBTDepth: Maximum allowable binary tree depth -MinBTSize: Minimum allowable binary tree leaf node size
In some example implementations of QTBT partitioning structures, the CTU size may be set as 128x128 luma samples with two corresponding 64x64 blocks of chroma samples (example chroma sub If sampling is considered and used), MinQTSize may be set as 16×16, MaxBTSize may be set as 64×64, and MinBTSize (for both width and height) is 4× MaxBTDepth may be set as 4. Quadtree splitting may be first applied to the CTU to generate quadtree leaf nodes. A quadtree leaf node may have a size from its minimum allowed size of 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If the node is 128x128, it will not be initially split by the binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize may be split by the binary tree. In the example of FIG. 14, the base block is 128×128. Base blocks can only be quadtree-split according to a predefined set of rules. The base block has a division depth of 0. Each of the four resulting partitions is 64x64, no more than MaxBTSize, and may be further quadtree-split or binary-split at level 1. The process continues. Once the binary tree depth reaches MaxBTDepth (i.e. 4), no further splits may be considered. When the width of a binary tree node is equal to MinBTSize (i.e., 4), no further horizontal splits may be considered. Similarly, when the height of a binary tree node is equal to MinBTSize, no further vertical splits are considered.

In some example implementations, the QTBT scheme described above may be configured to support flexibility for luma and chroma to have the same QTBT structure or separate QTBT structures. For example, for P slices and B slices, luma CTB and chroma CTB within one CTU may share the same QTBT structure. However, for I-slices, luma CTBs may be divided into CBs by QTBT structures, and chroma CTBs may be partitioned into chroma CBs by other QTBT structures. This means that CU may be used to refer to different color channels within an I-slice, for example, an I-slice may consist of a luma component coding block or two chroma component coding blocks, and P This means that a CU within a slice or B-slice may consist of coding blocks of all three color components.

In some other implementations, the QTBT scheme may be complemented with the ternary scheme described above. Such an implementation may be referred to as a multi-type tree (MTT) structure. For example, in addition to dividing the node into two, one of the three-division patterns of FIG. 13 may be selected. In some implementations, only square nodes can undergo trisection. Additional flags may be used to indicate whether the thirds are horizontal or vertical.

The design of two-level trees or multi-level trees, such as QTBT implementations and QTBT implementations supplemented with tripartitions, may be primarily motivated by complexity reduction. Theoretically, the complexity of traversing the tree is T ^D , where T represents the number of split types and D is the depth of the tree. A trade-off may be made by using multi-type (T) while reducing depth (D).

In some implementations, the CB may be further divided. For example, a CB may be further divided into multiple prediction blocks (PB) for the purpose of intra-frame prediction or inter-frame prediction during coding and decoding processes. In other words, the CB may be further divided into different sub-partitions, where individual prediction decisions/configurations may be made. In parallel, the CB may be further divided into multiple transform blocks (TB) for the purpose of describing the levels at which the video data is transformed or inversely transformed. The partitioning scheme of CB into PB and TB may or may not be the same. For example, each partitioning scheme may be performed using a unique procedure based on, for example, various characteristics of the video data. The PB and TB partitioning schemes may be independent in some example implementations. The PB and TB partitioning schemes and boundaries may be correlated in some other example implementations. In some implementations, for example, TBs may be divided after PB partitioning, and in particular, each PB may be determined subsequently after coding block partitioning and then further partitioned into one or more TBs. may be done. For example, in some implementations, a PB may be divided into 1, 2, 4, or other number of TBs.

In some implementations, the luma channel and chroma channel may be processed differently to divide the base block into coding blocks and further into prediction blocks and/or transform blocks. For example, in some implementations, partitioning of coding blocks into prediction blocks and/or transform blocks may be allowed for luma channels; Splitting may not be allowed for chroma channels. In such implementations, the transformation and/or prediction of luma blocks may thus be performed only at the coding block level. In other examples, the minimum transform block sizes for the luma and chroma channels may be different, e.g., the coding block for the luma channel may be allowed to be divided into smaller transform and/or prediction blocks than for the chroma channel. . In yet other examples, the maximum depth of partitioning of coding blocks into transform blocks and/or prediction blocks may differ between luma and chroma channels, e.g., coding blocks in a luma channel may be deeper than in a chroma channel. It may be allowed to be divided into transform blocks and/or prediction blocks. As a concrete example, a luma coding block may be divided into transform blocks of multiple sizes, which can be represented by recursive partitioning down by up to two levels: square, 2:1/1:2, and 4:1/1. Transform block shapes such as :4 and transform block sizes from 4x4 to 64x64 may be acceptable. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.

In some example implementations for partitioning coding blocks into PBs, the depth, shape, and/or other characteristics of the PB partitions depend on whether the PBs are intra-coded or inter-coded. Good too.

Partitioning a coding block (or prediction block) into transform blocks may be performed recursively or non-recursively, in various exemplary manners including, but not limited to, quadtree partitioning and predefined pattern partitioning. It may also be implemented to further consider transform blocks at the boundaries of blocks or prediction blocks. In general, the resulting transform blocks may be at different decomposition levels, may not have the same size, and may not need to be square in shape (e.g. they may have some permissible (can be rectangular with size and aspect ratio). Additional examples are described in further detail below in connection with FIGS. 15, 16, and 17.

However, in some other implementations, the CB obtained via any of the above partitioning schemes may be used as a basic or minimal coding block for prediction and/or transformation. In other words, no further splitting is performed for the purpose of performing inter/intra prediction and/or for the purpose of transformation. For example, the CB obtained from the QTBT method described above may be used as is as a unit for making predictions. Specifically, such a QTBT structure removes the concept of multiple partition types, i.e., removes the separation of CU, PU, and TU, and allows for additional flexibility regarding the CU/CB partition shape, as discussed above. to support. In such a QTBT block structure, the CU/CB can have either square or rectangular shape. Leaf nodes of such QTBT are used as units for prediction and transformation processing without further division. This means that CU, PU, and TU have the same block size in such an exemplary QTBT coding block structure.

The various CB partitioning schemes described above and further partitioning of the CB into PB and/or TB (including without PB/TB partitioning) may be combined in any manner. The following specific implementations are provided as non-limiting examples.

Specific example implementations of coding block and transform block partitioning are described below. In such an exemplary implementation, the base block is divided into coding blocks using recursive quadtree decomposition or a predefined partitioning pattern as described above (such as the partitioning patterns of Figures 9 and 10). may be divided into At each level, whether to continue further quadtree splitting of a particular partition may be determined by local video data characteristics. The resulting CBs may be at different quadtree decomposition levels and of different sizes. The decision whether to code a picture area using inter-picture (temporal) or intra-picture (spatial) prediction is made at the CB level (or at the CU level for all three color channels). May be done. Each CB may be further divided into 1, 2, 4, or other number of PBs according to a predefined PB partition type. Within one PB, the same prediction process may be applied and relevant information may be sent to the decoder on a PB basis. After obtaining the residual block by applying a prediction process based on the PB partition type, the CB can be partitioned into TBs according to other quadtree structures similar to the coding tree for CBs. In this particular implementation, CB or TB may not be limited to squares. Furthermore, in this particular example, the PB may be square or rectangular for inter-prediction, and only square for intra-prediction. The coding block may be divided into 4 square TBs, for example. Each TB may be further partitioned recursively (using quadtree splitting) into smaller TBs called residual quadtrees (RQTs).

Other example implementations for dividing base blocks into CBs, PBs, and/or TBs are described further below. For example, rather than using multiple partition unit types such as the types shown in Figure 9 or Figure 10, nested A quadtree with a multi-type tree may also be used. Separation of CBs, PBs, and TBs (i.e., splitting CBs into PBs and/or TBs, and splitting PBs into TBs) is subject to maximum conversion lengths if such CBs require further splitting. may be abandoned unless necessary for a CB with too large a size. This example partitioning scheme may be designed to support further flexibility on the CB partitioning shape, such that both prediction and transformation can be done at the CB level without further partitioning. In such a coding tree structure, the CB may have either a square or rectangular shape. Specifically, a coding tree block (CTB) may first be partitioned by a quadtree structure. The quadtree leaf nodes may then be further divided by a nested multi-type tree structure. An example of a nested multi-type tree structure using a bipartite or a tripartite is shown in FIG. 11. Specifically, the example multi-type tree structure of FIG. Contains four split types called Split (SPLIT_TT_HOR) (1108). CB then corresponds to the leaf of the multitype tree. In this example implementation, this segmentation is used for both prediction and transform processing without further partitioning, as long as the CB is not too large for the maximum transform length. This means that in most cases CB, PB, and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is less than the width or height of the CB color components. In some implementations, in addition to a bipartition or a tripartition, the nested pattern of FIG. 11 may further include a quadtree partition.

One specific example for a quadtree with a nested multi-type tree coding block structure of block partitioning (including options of quadtree partitioning, bipartition, and tripartite partitioning) for one base block is shown in Figure 12. has been done. More specifically, FIG. 12 shows that base block 1200 is quadtree partitioned into four square partitions 1202, 1204, 1206, and 1208. The decision to further use the multi-type tree structure and quadtree of FIG. 11 for further partitioning is made for each of the quadtree partitions. In the example of FIG. 12, partition 1204 is not further divided. Partition 1202 and partition 1208 each employ other quadtree partitions. In partition 1202, the second level quadtree partitioned top left, top right, bottom left, and bottom right partitions are quadtree partitioned, horizontal bipartition 1104 in Figure 11, unsplit, and horizontal partition in Figure 11, respectively. Adopt the third level division of tripartite 1108. Partition 1208 adopts other quadtree partitions, and the second level quadtree partitions top left, top right, bottom left, and bottom right partitions are vertical third partition 1106, nonpartition, Unsplit, and employ the third level of splitting of horizontal bisection 1104 in FIG. Two of the subpartitions of the third level upper left partition of 1208 are further divided according to horizontal bipartition 1104 and horizontal third partition 1108 of FIG. 11, respectively. Partition 1206 adopts a second level division pattern into two partitions, following vertical bisection 1102 of FIG. further divided into levels. A fourth level of division is further applied to one of them according to the horizontal bisection 1104 of FIG.

In the above example, the maximum luma transform size may be 64x64, and the maximum supported chroma transform size may be different than luma, for example 32x32. Even if the above exemplary CB of FIG. 12 is generally not further divided into smaller PBs and/or TBs, the width or height of the luma coding block or chroma coding block is greater than the maximum transform width or maximum transform height. When large, a luma or chroma coding block may be automatically split in that direction to meet horizontal and/or vertical transform size constraints.

In the specific example for base block partitioning into CBs above, as mentioned above, the coding tree scheme may support the ability for luma and chroma to have separate block tree structures. For example, for P slices and B slices, luma CTB and chroma CTB within one CTU may share the same coding tree structure. For I-slices, for example, luma and chroma may have separate coding block tree structures. When separate block tree structures are applied, a luma CTB may be divided into luma CBs by one coding tree structure, and a chroma CTB is divided into chroma CBs by another coding tree structure. This means that a CU in an I-slice may consist of a luma component coding block or two chroma-component coding blocks, and a CU in a P-slice or B-slice always consists of all three chroma components unless the video is monochrome. This means that it is composed of coding blocks.

When the coding block is further divided into multiple transform blocks, the transform blocks therein may be ordered within the bitstream according to various orders or scanning schemes. Exemplary implementations for dividing a coding block or prediction block into transform blocks and the coding order of the transform blocks are described in further detail below. In some example implementations, as described above, the transform partitioning may include multiple shapes, e.g., 1:1 (square ), 1:2/2:1, and 1:4/4:1 transform blocks. In some implementations, if the coding block is smaller than or equal to 64x64, transform block splitting is applied only to the luma component such that for chroma blocks, the transform block size is the same as the coding block size. You can. Otherwise, if the width or height of the coding block is greater than 64, both the luma and chroma coding blocks are min(W, 64) × min(H, 64) and min(W, 32 )×min(H, 32).

In some example implementations of transform block partitioning, the coding block has a partitioning depth of up to a predefined number of levels (e.g., 2 levels) for both intra-coded blocks and inter-coded blocks. may be further divided into multiple transform blocks having . The partition depth and size of transform blocks may be related. For some example implementations, the mapping from current depth transform size to next depth transform size is shown below in Table 1.

Based on the example mapping in Table 1, for a 1:1 square block, the next level of transform partitioning may create four 1:1 square sub-transform blocks. The conversion division may be stopped at 4×4, for example. Therefore, the transform size of the current depth of 4x4 corresponds to the same size of 4x4 of the next depth. In the example in Table 1, for a 1:2/2:1 non-square block, the next level of transform splitting may create two 1:1 square sub-transform blocks, but a 1:4/4:1 non-square block. For blocks, the next level of transform partitioning may create two 1:2/2:1 sub-transform blocks.

In some example implementations, additional restrictions may be applied on transform block partitioning for the luma components of intra-coded blocks. For example, for each level of transform partitioning, all sub-transform blocks may be constrained to have equal size. For example, for a 32x16 coding block, a level 1 transform split creates two 16x16 sub-transform blocks, and a level 2 transform split creates eight 8x8 sub-transform blocks. . In other words, the second level decomposition must be applied to all first level sub-blocks to keep the transform units equal in size. An example of transform block partitioning for a square block intra-coded according to Table 1 is shown in FIG. 15 with the coding order indicated by the arrows. Specifically, 1502 indicates a square coding block. The first level partitioning into four equally sized transform blocks according to Table 1 is shown at 1504 with the coding order indicated by the arrows. The second level partitioning of all first level equal size blocks into 16 equal size transform blocks according to Table 1 is shown at 1506 with the coding order indicated by the arrows.

In some example implementations, the above restrictions on intra-coding may not apply to the luma components of inter-coded blocks. For example, after a first level of transform partitioning, any one of the sub-transform blocks may be further independently partitioned at another level. Therefore, the resulting transform blocks may or may not be blocks of the same size. An exemplary partitioning of inter-coded blocks into translation locks according to their coding order is shown in FIG. 16. In the example of FIG. 16, inter-coded block 1602 is divided into transform blocks at two levels according to Table 1. At the first level, the intercoded block is divided into four transform blocks of equal size. Then only one of the four transform blocks (but not all of them) is further divided into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as shown by 1604 . An exemplary coding order for these seven transform blocks is indicated by arrows at 1604 in FIG.

In some example implementations, some additional restrictions on the transform block may be applied for chroma components. For example, for the chroma component, the transform block size can be as large as the coding block size, but cannot be smaller than a predefined size, eg, 8×8.

In some other example implementations, for a coding block where either the width (W) or the height (H) is greater than 64, both the luma coding block and the chroma coding block are W, 64) × min (H, 64) and min (W, 32) × min (H, 32) may be implicitly divided into multiple transformation units. Here, in the present disclosure, "min(a, b)" can return the smaller value between a and b.

FIG. 17 further illustrates another alternative example scheme for dividing a coding block or prediction block into transform blocks. As shown in FIG. 17, instead of using recursive transform partitioning, a predefined set of partition types may be applied to the coding block according to the transform type of the coding block. In the particular example shown in FIG. 17, one of six example partitioning types may be applied to partition the coding block into various numbers of transform blocks. Such a scheme of generating transform block partitions may be applied to either coding blocks or prediction blocks.

More specifically, the partitioning scheme of Figure 17 provides up to six exemplary partitioning types for any given transform type (transform type refers to the type of primary transform, e.g., ADST). . In this scheme, every coding block or prediction block may be assigned a transform partition type based on rate-distortion cost, for example. In one example, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. As illustrated by the six transform partition types illustrated in FIG. 17, a particular transform partition type may correspond to a partition size and pattern of transform blocks. Correspondences between various transformation types and various transformation partition types may be predefined. An example is shown below where the uppercase label indicates the transform partition type that may be assigned to a coding block or prediction block based on rate-distortion cost.

- PARTITION_NONE: Allocate transform size equal to block size.

-PARTITION_SPLIT: Allocate a transform size of 1/2 width of block size and 1/2 height of block size.

-PARTITION_HORZ: Allocate a conversion size with the same width as the block size and 1/2 the height of the block size.

-PARTITION_VERT: Allocate a transformation size with a width of 1/2 the block size and a height equal to the block size.

-PARTITION_HORZ4: Allocate a conversion size with the same width as the block size and 1/4 height of the block size.

- PARTITION_VERT4: Allocate a transform size with a width of 1/4 of the block size and a height equal to the block size.

In the above example, the transform partition type shown in FIG. 17 includes a uniform transform size for all partitioned transform blocks. This is an example only, not a limitation. In some other implementations, mixed transform block sizes may be used for partitioned transform blocks in a particular partition type (or pattern).

The PB obtained from any of the above partitioning schemes (or the CB, also called PB if not further partitioned into predictive blocks) is divided into individual blocks for coding via either intra-prediction or inter-prediction. It can be. For inter prediction in the current PB, residuals between the current block and the predicted block may be generated, coded, and included in the coded bitstream.

Inter prediction may be performed in a single reference mode or a combined reference mode, for example. In some implementations, a skip flag may be initially included in the current block's bitstream (or at a higher level) to indicate whether the current block is intercoded and not skipped. If the current block is inter-coded, other flags may be further included in the bitstream to signal whether single reference mode or mixed reference mode is used for prediction of the current block. . For single reference mode, one reference block may be used to generate a predictive block for the current block. For composite reference mode, two or more reference blocks may be used to generate a predictive block, eg, by weighted averaging. Combined reference mode is sometimes referred to as multiple reference mode, two reference mode, or multiple reference mode. The one or more reference blocks correspond to one another using one or more reference frame indices to further indicate a shift between the reference block and the current block in position, e.g., in horizontal and vertical pixels. May be identified using one or more motion vectors. For example, the inter-prediction block of the current block may be generated from a single reference block identified by one motion vector in the reference frame as the prediction block in single reference mode, whereas in the case of composite reference mode, the prediction block is , may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. Motion vectors may be coded and included in the bitstream in a variety of ways.

In some implementations, an encoding or decoding system may maintain a decoded picture buffer (DPB). Some images/pictures may be maintained in the DPB waiting to be displayed (in the decoding system), and some images/pictures in the DPB may undergo inter prediction (in the decoding or encoding system). May be used as a reference frame for enabling. In some implementations, reference frames in the DPB may be tagged as either short-term or long-term references for the current image being encoded or decoded. For example, a short-term reference frame includes a frame in decoding order that is used for inter-prediction of a block within the current frame or a predefined number (e.g., two) of subsequent video frames that are closest to the current frame. But that's fine. A long-term reference frame may include frames within a DPB that can be used to predict image blocks within a predefined number of frames from the current frame in decoding order. Information about such tags for short-term and long-term reference frames is called a reference picture set (RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in an encoded video stream may be identified by a picture order counter (POC), which is numbered in an absolute manner or according to the playback sequence, e.g., relative to a group of pictures starting from an I frame. will be attached.

In some example implementations, one or more reference picture lists that include identification of short-term and long-term reference frames for inter prediction may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter prediction, denoted as L0 reference (or reference list 0), and two picture reference lists may be formed for L0 reference (or reference list 0) for each of the two prediction directions. (or reference list 0) and L1 (or reference list 1) may be formed for bidirectional inter prediction. The reference frames included in the L0 list and the L1 list may be ordered in various predetermined ways. The lengths of L0 and L1 lists may be signaled in the video bitstream. Unidirectional inter prediction can be either single reference mode or composite reference mode, if multiple references for the generation of a predicted block by weighted average in composite prediction mode are on the same side of the block to be predicted . Bidirectional inter-prediction may only be a compound mode in that bidirectional inter-prediction includes at least two reference blocks.

In some implementations, merge mode (MM) for inter prediction may be implemented. In general, for merge mode, one or more of the motion vectors in the single reference prediction or the motion vectors in the composite reference prediction of the current PB are derived from other motion vectors rather than being computed and signaled independently. may be done. For example, in an encoding system, the current motion vector of the current PB is represented by the difference between the current motion vector and one or more other already encoded motion vectors (called reference motion vectors). Good too. Such differences in motion vectors, rather than the entire current motion vector, may be encoded and included in the bitstream and may be linked to a reference motion vector. Correspondingly, in the decoding system, a motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the decoded reference motion vector linked thereto. As a specific form of general merge mode (MM) inter prediction, such inter prediction based on motion vector differences is sometimes referred to as merge mode with motion vector differences (MMVD). Therefore, MM in general or MMVD in particular may be implemented to exploit the correlation between motion vectors associated with different PBs to improve coding efficiency. For example, neighboring PBs may have similar motion vectors, so the MVD may be small and can be efficiently coded. In other examples, motion vectors may be temporally correlated (between frames) for similarly located/located blocks in space.

In some example implementations, an MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included during the encoding process and signaled in the bitstream to indicate whether the current PB is in MMVD mode. MM and/or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. In a particular example, both the MM and MMVD flags may be included for the current CU, and the MMVD flag may be combined with the skip flag and Can be signaled immediately after the MM flag.

In some example implementations of MMVD, for a block to be predicted, a list of reference motion vectors (RMVs) or MV predictor candidates for motion vector prediction may be formed. The list of RMV candidates may include a predetermined number (eg, two) of MV predictor candidate blocks whose motion vectors may be used to predict the current motion vector. RMV candidate blocks may include blocks selected from adjacent blocks within the same frame and/or temporal blocks (eg, blocks co-located in ongoing or subsequent frames of the current frame). These options represent blocks at spatial or temporal positions relative to the current block that are likely to have similar or identical motion vectors to the current block. The size of the list of MV predictor candidates may be predetermined. For example, a list may include more than one candidate. To be on the list of RMV candidates, a candidate block may e.g. need to have the same reference frame (or frames) as the current block and must be present (e.g. if the current block is a frame (needs to be bounds checked), must already be encoded during the encoding process and/or must already be decoded during the decoding process. In some implementations, the list of merge candidates is first filled with spatially adjacent blocks (traversed in a certain predefined order) if they are available and satisfy the above conditions; If space is still available in the list, it may then be filled with time blocks. Adjacent RMV candidate blocks may be selected, for example, from blocks to the left and above the current block. The list of RMV predictor candidates may be dynamically formed at various levels (sequence, picture, frame, slice, superblock, etc.) as a dynamic reference list (DRL). DRL may be signaled in the bitstream.

In some implementations, the actual MV predictor candidate being used as a reference motion vector to predict the motion vector of the current block may be signaled. If the RMV candidate list includes two candidates, a 1-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. For a current block being predicted in composite mode, each of the multiple motion vectors predicted using the MV predictor may be associated with a reference motion vector from the merge candidate list. The encoder may decide which RMV candidate more closely predicts the current coding block and signal the selection as an index to the DRL.

In some exemplary implementations of MMVD, an RMV candidate is selected and used as a base motion vector predictor for the motion vector to be predicted, and then the motion vector difference (between the motion vector to be predicted and the reference candidate motion vector) is MVD or delta MV), which represents the difference between Such an MVD may include information representing the MV difference magnitude and the MV difference direction, both of which may be signaled in the bitstream. The magnitude of the motion difference and the direction of the motion difference may be signaled in various ways.

In some exemplary implementations of MMVD, a distance index is used to specify motion vector difference magnitude information and to represent a predefined motion vector difference from a starting point (reference motion vector). may indicate one of a set of offsets. An MV offset depending on the signaled index may then be added to either the horizontal or vertical component of the starting (reference) motion vector. Whether the horizontal or vertical component of the reference motion vector should be offset may be determined by the direction information of the MVD. An example predefined relationship between distance index and predefined offset is specified in Table 2.

In some example implementations of MMVD, a direction index may also be signaled and used to represent the direction of the MVD relative to the reference motion vector. In some implementations, the orientation may be limited to either horizontal or vertical. An exemplary 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may vary according to the information of the starting/reference MV. For example, if the start/reference MV corresponds to a uni-predicted block or both reference frame lists correspond to bi-predicted blocks pointing to the same side of the current picture (i.e. the POCs of the two reference pictures are both (or both are less than the current picture's POC), the code in Table 3 may specify the sign (direction) of the MV offset that is added to the starting/reference MV. The start/reference MV corresponds to a bi-predictive block with two reference pictures on different sides of the current picture (i.e. the POC of one reference picture is larger than the POC of the current picture and the POC of the other reference picture is larger than the POC of the other reference picture). is less than the current picture's POC), and the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame , the sign in Table 3 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may be , may have opposite values (opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, then the sign in Table 3 is It may be specified that the sign of the MV offset added to the reference MV associated with picture reference list 1 and the sign of the offset to the reference MV associated with picture reference list 0 have opposite values.

In some example implementations, the MVD may be scaled according to the difference in POC in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the difference in POC in reference list 0 is greater than the difference in reference list 1, then the MVD of reference list 1 is scaled. If the POC difference of reference list 1 is greater than list 0, then the MVD of list 0 may be scaled as well. If the starting MV is single predicted, the MVD is added to the available or reference MV.

In some example implementations of MVD coding and signaling for bidirectional composite prediction, only one MVD requires signaling in addition to or instead of coding and signaling two MVDs separately. , and symmetric MVD coding can be implemented such that other MVDs can be derived from the signaled MVD. In such implementations, motion information is signaled that includes reference picture indices for both list 0 and list 1. However, for example, only the MVD associated with reference list 0 is signaled, and the MVD associated with reference list 1 is not signaled and is derived. Specifically, at the slice level, a flag called "mvd_l1_0_flag" may be included in the bitstream to indicate whether reference list 1 is not signaled in the bitstream. If this flag is 1, indicating that reference list 1 is equal to zero (and therefore not signaled), the bidirectional prediction flag called "BiDirPredFlag" may be set to 0, which indicates that there is no bidirectional prediction. It means that. Otherwise, if mvd_l1_0_flag is zero, then the nearest reference picture in list 0 and the nearest reference picture in list 1 are forward and backward pairs of reference pictures or backward and forward pairs of reference pictures , BiDirPredFlag may be set to 1, and the reference pictures of list 0 and list 1 are both short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. BiDirPredFlag of 1 may indicate that a symmetric mode flag is additionally signaled in the bitstream. The decoder may extract the symmetric mode flag from the bitstream if BiDirPredFlag is 1. A symmetric mode flag may, for example, be signaled at the CU level (if necessary) and may indicate whether symmetric MVD coding mode is used for the corresponding CU. If the symmetric mode flag is 1, it indicates the use of symmetric MVD coding mode, and only the reference picture indices of both list 0 and list 1 (called "mvp_l0_flag" and "mvp_l1_flag") were associated with list 0. MVD (referred to as "MVD0") to indicate that another motion vector difference "MVD1" should be derived rather than signaled. For example, MVD1 may be derived as -MVD0. Therefore, in the exemplary symmetric MVD mode, only one MVD is signaled. Some other example implementations for MV prediction include the general merge mode MMVD, for both single reference mode MV prediction and combined reference mode MV prediction, and some other types of A harmonic scheme may be used to implement MV prediction. Various syntax elements may be used to signal how the MV of the current block is predicted.

For example, for single reference mode, the following MV prediction modes may be signaled:

NEARMV - Use one of the motion vector predictors (MVPs) in the list indicated by the DRL (Dynamic Reference List) index directly without MVD.

NEWMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and apply a delta to the MVP (eg, using MVD).

GLOBALMV - Use motion vectors based on frame-level global motion parameters.

Similarly, for a composite reference inter prediction mode that uses two reference frames corresponding to two MVs to be predicted, the following MV prediction modes may be signaled.

NEAR_NEARMV - For each of the two MVs to be predicted, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD.

NEAR_NEWMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD as the reference MV to predict the first of the two motion vectors. , to predict the second of the two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index, and an additionally signaled delta MV (MVD) to be used as a reference MV.

NEW_NEARMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD as the reference MV to predict the second of the two motion vectors. , to predict the first of the two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index, and an additionally signaled delta MV (MVD) to be used as a reference MV.

NEW_NEWMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV and use it in conjunction with the additionally signaled delta MV to Make predictions for each.

GLOBAL_GLOBALMV - Use MV from each reference based on frame-level global motion parameters.

Therefore, the term "NEAR" above refers to MV prediction using the reference MV without MVD as the general merging mode, whereas the term "NEW" refers to the MV prediction using the reference MV and with MMVD mode. refers to MV prediction, which involves offsetting it with a signaled MVD. For composite inter-prediction, both the reference base motion vector and the motion vector delta described above are correlated, and such correlation is utilized to reduce the amount of information required to signal the two motion vector deltas. may be generally different or independent between the two references. In such a situation, joint signaling of two MVDs may be implemented and indicated in the bitstream.

The dynamic reference list (DRL) described above may be used to maintain a set of indexed motion vectors that are dynamically maintained and considered candidate motion vector predictors.

In some example implementations, predefined resolutions of MVDs may be allowed. For example, a motion vector precision (or accuracy) of 1/8 pixel may be acceptable. The MVDs described above in various MV prediction modes can be constructed and signaled in various ways. In some implementations, various syntax elements may be used to signal the above motion vector differences in reference frame list 0 or list 1.

For example, a syntax element called "mv_joint" may specify which components of the motion vector difference associated with it are non-zero. For MVD, this is signaled together for all non-zero components. For example, mv_joint has the following values:
0 can indicate no non-zero MVD along either the horizontal or vertical direction,
1 can show that there is non-zero MVD only along the horizontal direction,
2 can show that there is a non-zero MVD only along the vertical direction,
3 can indicate that there is non-zero MVD along both horizontal and vertical directions.

If the "mv_joint" syntax element for MVD signals that there are no non-zero MVD components, no further MVD information may be signaled. However, if the "mv_joint" syntax signals that there are one or two non-zero components, additional syntax elements may be further signaled for each of the non-zero MVD components, as described below.

For example, a syntax element called "mv_sign" may be used to further specify whether the corresponding motion vector difference component is positive or negative.

In another example, a syntax element called "mv_class" may be used to specify the class of motion vector differences between a predefined set of classes of corresponding non-zero MVD components. Predefined classes of motion vector differences may be used, for example, to separate a continuous magnitude space of motion vector differences into non-overlapping ranges, each range corresponding to an MVD class. Therefore, the signaled MVD class indicates the magnitude range of the corresponding MVD component. In the exemplary implementation shown in Table 4 below, higher classes correspond to motion vector differences that have larger magnitude ranges. In Table 4, the symbol (n, m] is used to represent the range of motion vector differences greater than n pixels and less than or equal to m pixels.

Some other examples further use a syntax element called "mv_bit" to specify an integer of the offset between the non-zero motion vector difference component and the correspondingly signaled starting magnitude of the MV class magnitude range. You may also specify a portion. Therefore, mv_bit may indicate the magnitude or amplitude of MVD. The number of bits required for "my_bit" to signal the full range of each MVD class may vary depending on the MV class. For example, MV_CLASS 0 and MV_CLASS 1 in the implementation of Table 4 may require only a single bit to indicate an integer pixel offset of 1 or 2 from a starting MVD of 0, and each Higher MV_CLASSes may require progressively more bits for "mv_bit" than previous MV_CLASSes.

In some other examples, a syntax element called "mv_fr" may be further used to specify the first two fractional bits of the motion vector difference of the corresponding non-zero MVD components, and " A syntax element called "mv_hp" may be used to specify the third fractional bit (high resolution bit) of the motion vector difference of the corresponding non-zero MVD component. The 2-bit "mv_fr" essentially provides 1/4 pixel MVD resolution, while the "mv_hp" bit may additionally provide 1/8 pixel resolution. In some other implementations, more than one "mv_hp" bit may be used to provide MVD pixel resolution finer than 1/8 pixel. In some example implementations, additional flags may be signaled at one or more of various levels to indicate whether MVD resolutions of 1/8 pixel or higher are supported. If an MVD resolution does not apply to a particular coding unit, the above syntax elements for the corresponding unsupported MVD resolution may not be signaled.

In some example implementations above, fractional resolution may be independent of different classes of MVDs. In other words, a predefined number of 'mv_fr' and 'mv_hp' bits are used to signal the fractional MVD of the non-zero MVD component, regardless of the magnitude of the motion vector difference, and the same for the motion vector resolution. options may be provided.

However, in some other example implementations, the resolution of motion vector differences in various MVD size classes may be differentiated. Specifically, high-resolution MVD for large MVD sizes in higher MVD classes may not result in statistically significant improvements in compression efficiency. Therefore, MVDs may be coded with reduced resolution (integer pixel resolution or fractional pixel resolution) for larger MVD size ranges corresponding to higher MVD size classes. Similarly, MVD may generally be coded at a lower resolution (integer pixel resolution or fractional pixel resolution) for larger MVD values. Such MVD class-dependent or MVD magnitude-dependent MVD resolution may generally be referred to as adaptive MVD resolution, amplitude-dependent adaptive MVD resolution, or magnitude-dependent MVD resolution. The term "resolution" is sometimes further referred to as "pixel resolution." Adaptive MVD resolution may be implemented in various ways, as illustrated by example implementations below, to achieve better overall compression efficiency. In particular, reducing the number of signaling bits by aiming for a less precise MVD may result in a non-adaptive processing of the MVD resolution of a large scale or high class MVD to a similar level as a low scale or low class MVD. Due to the statistical observation that inter-prediction residual coding efficiency cannot be significantly increased for blocks with large or high-class MVDs, coding inter-prediction residuals as a result of such less accurate MVDs can be larger than the additional bits needed to do so. In other words, using a higher MVD resolution for large or high class MVDs may not result in more coding gain than using a lower MVD resolution.

In some common example implementations, the pixel resolution or precision of the MVD may or may not increase with increasing MVD class. Reducing the pixel resolution of the MVD corresponds to a coarser MVD (or a larger step from one MVD level to the next). In some implementations, the correspondence between MVD pixel resolution and MVD class may be specified, predefined, or preconfigured, and thus does not need to be signaled in the encoded bitstream.

In some example implementations, each of the MV classes in Table 3 may be associated with a different MVD pixel resolution.

In some example implementations, each MVD class may be associated with a single allowed resolution. In some other implementations, one or more MVD classes may be associated with more than one optional MVD pixel resolution. Therefore, a signal in the bitstream of the current MVD component having such MVD class is followed by additional signaling to indicate which optional pixel resolution is selected for the current MVD component. Can be done.

In some example implementations, the adaptively allowed MVD pixel resolutions are (in descending order of resolution) 1/64 pel, 1/32 pel, 1/16 pel, 1/8 pel, 1-4 pel, It may include, but is not limited to, 1/2 pel, 1 pel, 2 pel, 4 pel... Thus, each of the ascending MVD classes may be associated with one of these resolutions in non-ascending order. In some implementations, an MVD class may be associated with more than one of the above resolutions, and the higher resolution may be less than or equal to the lower resolution of the preceding MVD class. For example, if MV_CLASS_3 in Table 4 can be associated with the optional 1pel and 2pel resolutions, then the highest resolution MV_CLASS_4 in Table 4 can be associated will be 2pel. In some other implementations, the highest allowed resolution of an MV class may be higher than the lowest allowed resolution of a preceding (lower) MV class. However, the average resolution allowed for ascending MV classes may only be non-ascending.

In some implementations, "mv_fr" and "mv_hp" signaling may be extended to accommodate more than 3 fractional bits in total if fractional pixel resolution higher than 1/8 pel is allowed.

In some example implementations, fractional pixel resolution may be allowed only for MVD classes below a threshold MVD class. For example, fractional pixel resolution may only be allowed for MVD-CLASS 0 and not for all other MV classes in Table 4. Similarly, fractional pixel resolution may only be allowed for MVD classes less than or equal to any one of the other MV classes in Table 4. For other MVD classes above the threshold MVD class, only integer pixel resolutions of MVD are allowed. In this way, fractional resolution signaling, such as one or more of the "mv-fr" and/or "mv-hp" bits, is signaled for MVDs that are signaled in MVD classes equal to or greater than the threshold MVD class. There is no need to For MVD classes with resolutions less than 1 pixel, the number of bits of "mv-bit" signaling may be further reduced. For example, for MV_CLASS_5 in Table 4, the range of MVD pixel offsets is (32, 64], so 5 bits are required to signal the entire range at 1 pel resolution. However, if MV_CLASS_5 is 2 pel MVD resolution (1 pixel resolution), ``mv-bit'' may require 4 bits instead of 5, and ``mv-fr'' and ``mv-hp'' are both MV-CLASS_5. does not need to be signaled following the signaling of "mv_class".

In some example implementations, fractional pixel resolution may be allowed only for MVDs with integer values less than a threshold integer pixel value. For example, fractional pixel resolution may only be allowed for MVDs smaller than 5 pixels. Corresponding to this example, fractional resolution is allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4, and may not be allowed for all other MV classes. In other examples, fractional pixel resolution may only be allowed for MVDs smaller than 7 pixels. Corresponding to this example, if fractional resolution is allowed for MV_CLASS_0 and MV_CLASS_1 (with a range of less than 5 pixels) in Table 4, but not for MV_CLASS_3 and above (with a range of more than 5 pixels) There is. For MVDs belonging to MV_CLASS_2 whose pixel range includes 5 pixels, the fractional pixel resolution of the MVD may be allowed according to the "mv-bit" value. If the "m-bit" value is signaled as 1 or 2 (the integer part of the signaled MVD is calculated as the start of the pixel range of MV_CLASS_2 with offset 1 or 2 as indicated by "m-bit") is 5 or 6), fractional pixel resolution may be acceptable. Otherwise, if the "mv-bit" value is signaled as 3 or 4 (such that the integer part of the signaled MVD is 7 or 8), then fractional pixel resolution may not be allowed.

In some other implementations, only a single MVD value may be allowed for MV classes above the threshold MV class. For example, such a threshold MV class may be MV_CLASS 2. Therefore, MV_CLASS_2 and above can only be allowed to have a single MVD value and no sub-pixel resolution. A single allowed MVD value for these MV classes may be predefined. In some examples, the single value allowed may be the upper value of the range for each of these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be greater than or equal to the threshold class of MV_CLASS 2, and the single allowed MVD values for these classes are 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively. may be predefined as In some other examples, the single value allowed may be the median of the ranges for each of these MV classes in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 may be above the class threshold, and the single allowed MVD values for these classes are predefined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. may be defined. Any other value within the range may also be defined as a single allowable resolution for each MVD class.

In the above implementation, only "mv_class" signaling is sufficient to determine the MVD value if the signaled "mv_class" is greater than or equal to a predefined MVD class threshold. The magnitude and direction of the MVD is then determined using "mv_class" and "mv_sign".

Therefore, if the MVD is signaled for just one reference frame (either from reference frame list 0 or list 1, but not both) or for two reference frames together, the MVD The accuracy (or resolution) may depend on the relevant motion vector difference class and/or the magnitude of the MVD in Table 3.

In some other implementations, the pixel resolution or precision of the MVD may or may not increase with increasing MVD size. For example, pixel resolution may depend on the integer part of the MVD size. In some implementations, sub-pixel resolution may be allowed only for MVD magnitudes that are less than or equal to an amplitude threshold. For the decoder, the integer part of the MVD size may be extracted from the bitstream first. The pixel resolution can then be determined, and then a decision can be made as to whether any fractional MVDs are present in the bitstream and need to be analyzed (e.g., if the fractional pixel resolution of a particular extracted MVD is The bitstream that requires extraction may not contain fractional MVD bits if this is not allowed for the integer size). The above example implementations for MVD class-dependent adaptive MVD pixel resolution apply to MVD size-dependent adaptive MVD pixel resolution. In certain examples, MVD classes that exceed or contain a magnitude threshold may be allowed to have only one predefined value.

The various example implementations described above apply to single reference mode. These implementations also apply to the example NEW_NEARMV, NEAR_NEWMV, and/or NEW_NEWMV modes in composite prediction under MMVD. These implementations generally apply to any MVD adaptive resolution.

In a particularly exemplary implementation for adaptive MVD pixel resolution, the MVD pixel resolution for MVD magnitudes less than 1 may be fractional, and for MV classes greater than or equal to MV_CLASS_1, the corresponding MVD magnitude in Table 4 Only a single MVD magnitude equal to the end value of the range may be allowed. In such an example, allowable MVD values are shown in Table 4 for allowable fractional pixel resolutions of 1/8, 1/4, or 1/2 pixel.

For coding blocks, whether adaptive MVD pixel resolution is used may be explicitly or implicitly signaled (derived). If adaptive MVD pixel resolution is signaled as not being used, it indicates that different MVD classes may follow the MVD ranges shown in Table 4, and non-adaptive MVD pixel resolution may be defined or signaled. Such non-adaptive resolution may be fractional (e.g., 1/8, 1/4, or 1/2 pixel) or non-fractional (e.g., 1, 2, 4, ... pixels), and all MVD Applies to classes. The non-adaptive resolution basically determines the number of bits required to signal mv_bit, mv_fr, and mv_hp mentioned above. If the non-adaptive resolution is fractional, it is only possible to determine the number of bits needed to signal mv_fr and mv_hp for all MVD classes (independently of the class of MVD) and signal Mv_bit. The number of bits to do may depend on the MVD class.

When adaptive MVD pixel resolution is signaled to be used, allowable MVD levels or values may be predefined or signaled in an adaptive manner such as those shown in Table 5. For example, they may be signaled within the bitstream in various ways depending on the particular scheme for adaptive MVD resolution. In the example in Table 5, a set of signaling syntax is used to indicate the fractional resolution (e.g., 1/8 pixel) and the magnitude threshold at which the signaled fractional resolution applies (e.g., MVD magnitude of 1 pixel). can be done. Other sets of syntax (which may be more complex) may be used to signal other adaptive MVD resolution schemes. Such an indication of adaptive MVD pixel resolution scheme may be signaled at one of various coding levels, such as sequence level, picture level, frame level, slice level, superblock level, or coding block level.

In some example implementations, the overall adaptive MVD pixel resolution scheme, including but not limited to those shown in Table 5, is based on specific coding levels (e.g., sequence level, picture level, frame level, slice level). , superblock level). Such an adaptive MVD pixel resolution scheme may be further modified at the same or other coding levels, such that the allowed MVD pixel resolution values for various MVD classes may be adjusted or modified at the same or other coding levels. If no adjustment is made at a particular coding level, a signaled or predefined adaptive MVD pixel resolution scheme is applied without modification. For example, a comprehensive adaptive MVD pixel resolution scheme may be defined or signaled at the frame level, while adjustments may be made at one or more superblock or coding block levels, and vice versa. The same is true.

Such adjustments may be implemented as limiting MVD accuracy or expanding MVD accuracy. Information related to such adjustments may be predefined or signaled. Predefined adjustments may be applied to all coding blocks. Alternatively, predefined adjustments may be activated at various coding levels by signaling.

In some implementations, such adjustment may be embodied as a maximum allowed MVD accuracy. For a particular coding block, when adaptive MVD resolution is applied, such maximum allowed MVD accuracy may be specified/signaled/derived at the picture level, or superblock level, or coded block level, as described above. The MVD pixel accuracy of the adaptive MVD pixel resolution method may be different. In such situations, allowable MVD resolution values for various MVD classes may be determined by taking both the allowable value specified by or derived from the adaptive MVD pixel resolution scheme and the maximum allowable MVD precision.

For example, assume that the adaptive MVD pixel resolution scheme of Table 5 is predefined/signaled/derived for a particular coding level. Furthermore, assuming the maximum allowed precision is 1/4 pixel, this means that no precision greater than 1/8 pixel is allowed for any MVD class, regardless of the adaptive MVD resolution associated with Table 5. means. Then, by indiscriminately applying the maximum allowed pixel accuracy as the limit in Table 5, the allowed MVD pixel levels or values for various MVD classes can be modified as follows:

Defining/signaling a maximum allowed MVD pixel precision of 1/4 pixel, but not allowing more than 1/8 pixel precision for all MVD classes, is just one example. In other examples, the maximum allowed pixel precision may be defined/signaled as 1/2 pixel. The allowable MVD values corresponding to MV_CLASS_0 above are (1/2, 1, 2) for adaptive resolution schemes with fractional pixel resolutions of 1/8 pixel, 1/4 pixel, and 1/2 pixel, and 1 It can be (1, 2) in case of pixel resolution of pixels.

As exemplified in the implementation above in connection with Table 6, use an adaptive resolution scheme defined/signaled/derived at a particular coding level, with an additional defined/signaled maximum allowed at the same or a different coding level. When adaptive MVD resolution with MVD precision is applied, it may be required/restricted that such maximum allowed MVD precision is not greater than the MVD resolution of the adaptive resolution scheme. In other words, the actually applied MVD accuracy derived considering both the adaptive resolution scheme and the maximum allowed accuracy is clipped by the MVD resolution in the adaptive resolution scheme (i.e. the maximum allowed accuracy is definition/signaling/derived resolution).

However, in some other implementations, such clipping may not be required and the defined/signaled maximum allowed MVD precision may control the actual MVD resolution of at least some MVD classes. In these implementations, if adaptive MVD resolution is applied (as indicated by the definition/signaling/derivation at various coding levels, as described above), the adjustment of the MVD level for at least some MVD classes will be , may include increasing rather than limiting the adaptive MVD resolution defined/signaled/derived with an adaptive MVD pixel resolution scheme such as that associated with Table 5. For example, adjustments may be made to allow higher accuracy than specified/signaled/derived from an adaptive resolution scheme for MVD classes below a defined or signaled threshold MVD class level. Such higher accuracy may be defined/signaled as the maximum allowed MVD accuracy, as described above. Such maximum allowed accuracy may be imposed below the threshold MVD class level, regardless of the specified/signaled/derived MVD resolution in the adaptive resolution scheme. Specifically, such threshold MVD class level may (but need not be) MV_CLASS_0 (or the lowest MVD class level of a set of MVD classes, such as the set of MVD classes in Table 5). ). The maximum allowed pixel accuracy may be predefined/signaled. The maximum allowed pixel precision may be a fractional number. For a particular example, the threshold MVD class is MV_CLASS_0 in the adaptive resolution scheme of Table 5, the MVD pixel resolution of MV_CLASS_0 is 1 non-fractional pixel, and the maximum allowed fractional pixel precision for adjustment is 1/8, 1/ 4, or 1/2 pixel, the acceptable MVD values for adjustment are as follows:

In some example implementations alternative to the implementation of Table 7, a threshold MVD amplitude may be used instead of a threshold MVD class level. In these implementations, higher accuracy may be imposed by the maximum allowed MVD accuracy specified/signaled for MVDs with magnitudes below the threshold MVD amplitude rather than the threshold MVD class level. In such implementations, the mv_bit information in addition to the mv_class information is signaled early enough in the video stream so that the MVD magnitude can be determined in time to determine the allowable MVD value. Good too. For example, by replacing the threshold MVD class with a threshold MVD magnitude of 1/2 pixel and still assuming that the adaptive MVD resolution of MV_CLASS_0 is 1 pixel in the adaptive resolution scheme, Table 7 becomes Table 8 below:

In some other example implementations, the above adjustment adjusts a certain precision and a lower precision (e.g., fractional precision 1/8, 1/8 4, or 1/2 or less). In such implementations, again, the mv_bit information in addition to the mv_class information is signaled early enough in the video stream so that the magnitude of the MVD can be determined in time to determine the allowable MVD value. may be done.

In such implementations, no additional resolution can be imposed on the MVD values derived from the adaptive resolution scheme (such as Table 5). Alternatively, if the amplitude of the MVD is greater than a threshold MVD magnitude, MVD values associated with a resolution equal to or greater than the defined/signaled precision level may not be allowed. Again, the example in Table 5 further states that for MVD magnitudes greater than the threshold MVD magnitude of 1/2 pixel, the definition of 1/8 pixel precision/MVD values associated with resolutions greater than or equal to the signaled precision are acceptable. Assume that it is not. Then adjust Table 5 as follows:

In particular, as shown above, for MV_CLASS_0 and 1/8 pixel decimal resolution (1/8, 2/8, 3/8, 1/2, 5/8, 6/8, 7/8, 1, 2) The allowed MVD values are adjusted to (1/8, 2/8, 3/8, 1/2, 6/8, 1, 2), and MVD values associated with 1/8 precision are allowed and 1/2 It is only kept below the threshold MVD magnitude of the pixel. Beyond the 1/2 pixel size, MVD values associated with 1/8 precision, such as 5/8 pixel values and 7/8 pixel values, are not allowed.

Similarly, in the example in Table 5, if the MVD magnitude is greater than the threshold magnitude of 1/2 pixel, then the MVD value associated with a resolution equal to or greater than the 1/4 pixel precision definition/signaled precision is not allowed. Assume. Then adjust Table 5 as follows:

In some of the above implementations, the threshold MVD size may be 2 pixels or less, such as the 1/2 pixel size threshold given in the example above.

The example implementations above are described for a particular MVD, regardless of whether the inter-prediction mode is in single reference mode or combined reference mode. In some other example implementations in a composite reference mode where the MV is predicted by multiple reference frames, whether adaptive MVD resolution is applied and the reference among the multiple reference frames to which it is applied A set of definitions/signaling may be used to indicate which of the frames it applies to.

In some example implementations, when MVD is signaled for multiple reference frames, one (or more) flags/indexes are signaled to indicate whether adaptive MVD resolution is applied. obtain.

For example, when MVD is signaled for multiple reference frames (e.g., in the NEW_NEWMV mode described above, or other composite reference inter-prediction modes), an adaptive MVD resolution is applied to the MVD signaling for all of the multiple reference frames. One flag/index may be signaled within the video stream to indicate whether or not. If this flag/index is 1 (or 0), it indicates that the adaptive MVD resolution is applied to the MVD signaling for all of the multiple reference frames. Otherwise, if this flag/index is 0 (or 1), then adaptive MVD coding is not applied to the MVD signaling for any of the multiple reference frames. In such implementations, adaptive MVD resolution is applied in an all-or-nothing manner with respect to multiple inter-predicted reference frames.

In some other examples, when MVD is signaled for multiple reference frames (e.g., in the NEW_NEWMV mode described above for 2-reference frame composite inter-prediction mode or other composite inter-prediction modes), the adaptive MVD resolution is One flag/index may be signaled separately for each reference frame to indicate whether it applies to the frame. In such implementations, whether adaptive MVD resolution is applied may be determined individually for each of the reference frames. The decision to apply adaptive MVD resolution may be made at the encoder independently for each of the multiple reference frames and signaled separately within the video stream.

In some example implementations, when the MVD is signaled for multiple reference frames, for each of the multiple reference frames, if the MVD for that reference frame is non-zero, the adaptive MVD resolution is applied to that reference frame. One flag/index may be signaled to indicate whether it applies or not. Otherwise, the flag/index does not need to be signaled. In other words, if the MVD of a particular reference frame is signaled/indicated as zero, there is no need to determine whether adaptive MVD resolution is applied or not, and therefore no corresponding signaling in the video stream is required. However, in such implementations, an indication that MVD is zero needs to be signaled before a determination is made as to whether adaptive resolution is applied.

Turning further to MVD resolution signaling, in some example implementations a flag/index may be signaled to explicitly indicate the MVD resolution of the current coding block; The context used to entropy code an index may depend on the MVD class associated with the MVD. Such a flag/index may be either the MVD resolution used to derive the adaptive resolution scheme such as Table 5, or the maximum allowed MVD precision described above.

In some example implementations of signaling MVD resolution, various components of the MVD may be signaled separately. The MVD may include, for example, a horizontal component and a vertical component. A flag/index may be signaled for each of the horizontal and vertical components of the MVD to indicate the MVD resolution of the horizontal and vertical components, respectively.

In some example implementations, the MVD resolution flag/index may be signaled after the MVD class information. Depending on the value of the signaled MVD class information such as MV_CLASS_0, MV_CLASS_1, MV_CLASS_2, a context value may be derived and used to signal the MVD resolution flag/index to indicate the MVD resolution. In other words, the syntax(es) for signaling MVD resolution may be entropy coded using different contexts for different MVD classes or different MVD class groups.

FIG. 18 shows a flowchart 1800 of an example method that follows the principles underlying the above implementations for adaptive MVD resolution. The flow of the example decoding method begins at S1801. At S1810, a video stream is received. In S1820, it is determined that the video block is inter-coded based on the predictive block and the motion vector (MV), and the MV should be derived from the reference motion vector (RMV) and motion vector difference (MVD) of the video block. It is. In S1830, in response to determining that the MVD is coded with an adaptive MVD pixel resolution, a reference MVD pixel precision for the current video block is determined, a maximum allowable MVD pixel precision is identified, and a The set of acceptable MVD levels is determined based on the reference MVD pixel accuracy and the maximum allowed MVD pixel accuracy, and the MVD from the video stream is determined based on at least one MVD parameter signaled in the video stream for the current video block and the tolerance derived according to a set of possible MVD levels. The example method stops at S1899.

FIG. 19 shows a flowchart 1900 of an example method that follows the principles underlying the above implementations for adaptive MVD resolution. The flow of the example decoding method begins at S1901. In S1910, a video stream is received. At S1920, it is determined that the current video block is inter-coded and associated with multiple reference frames. At S1930, it is further determined whether adaptive motion vector difference (MVD) pixel resolution is applied to at least one of the plurality of reference frames based on signaling in the video stream. The example method stops at S1999.

FIG. 20 shows a flowchart 2000 of an example method that follows the principles underlying the above implementations for adaptive MVD resolution. The flow of the exemplary decoding method starts at S2001. In S2010, a video stream is received. In S2020, it is determined that the video block is inter-coded based on the predictive block and the motion vector (MV), and the MV should be derived from the reference motion vector (RMV) and motion vector difference (MVD) of the video block. It is. At S2030, the current MVD class of the MVD in the predefined set of MVD classes is determined. In S2040, at least one context for entropy decoding at least one explicit signaling in the video stream is derived based on the current MVD class, and the at least one explicit signaling is included in the video stream to specify the MVD pixel resolution of the two components. At S2050, at least one explicit signaling from the video stream is entropy decoded using the at least one context to determine an MVD pixel resolution of at least one component of the MVD. The example method stops at S2099.

In embodiments and implementations of the present disclosure, any steps and/or acts may be combined or arranged in any amount or order as desired. Two or more of the steps and/or actions may be performed in parallel. Embodiments and implementations of the present disclosure may be used separately or combined in any order. Additionally, each of the methods (or embodiments), encoder, and decoder may be implemented by processing circuitry (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium. Embodiments of the present disclosure may be applied to luma blocks or chroma blocks. The term block may be interpreted as a predictive block, coding block, or coding unit, or CU. The term block herein may also be used to refer to a transform block. In the following items, when we say block size, we mean the width or height of the block, or the maximum width and height, or the minimum width and height, or the size of the area (width * height), or Can refer to either the block's aspect ratio (width:height or height:width).

The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 21 depicts a computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter.

Computer software is coded using any suitable machine code or computer language that is amenable to assembly, compilation, linking, etc. mechanisms and is executed by one or more computer central processing units (CPUs), graphics processing units, etc. Code can be created that includes instructions that can be executed directly, such as by a GPU (GPU), or through interpretation, microcode execution, etc.

The instructions may be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

The components illustrated in FIG. 21 with respect to computer system (2100) are exemplary in nature and are not intended to suggest any limitations as to the scope of use or functionality of computer software implementing embodiments of the present disclosure. do not have. The configuration of components should not be construed as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (2100).

Computer system (2100) may include certain human interface input devices. Such human interface input devices may include, for example, tactile input (such as keystrokes, swipes, data glove movements, etc.), audio input (such as voice, applause, etc.), visual input (such as gestures), ) may respond to input by one or more human users via olfactory input. Human interface devices may include audio (e.g., voice, music, environmental sounds), images (e.g., scanned images, photographic images obtained from still image cameras), video (e.g., 2D video, 3D video, including stereoscopic video), etc. It can also be used to capture specific media that are not directly related to conscious human input.

Input human interface devices include keyboard (2101), mouse (2102), trackpad (2103), touch screen (2110), data glove (not shown), joystick (2105), microphone (2106), and scanner (2107). ), cameras (2108) (only one of each is depicted).

Computer system (2100) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human interface output devices include tactile output devices (e.g., tactile feedback from a touch screen (2110), data glove (not shown), or joystick (2105), but which do not function as an input device). audio output devices (such as speakers (2109), headphones (not depicted)), and touch screen inputs (such as screens (2110), including CRT screens, LCD screens, plasma screens, and OLED screens, respectively) Each with or without haptic feedback function, some of them include stereographic output, virtual reality glasses (not depicted), holographic display and smoke tank (not depicted), etc. Visual output devices (which may be capable of outputting two-dimensional visual output or three-dimensional or more dimensional output via means of) as well as printers (not depicted) may be included.

The computer system (2100) includes optical media including a CD/DVD ROM/RW (2120) with a CD/DVD or similar media (2121), a thumb drive (2122), a removable hard drive or solid state drive (2123), Human-accessible storage devices and their associated media, such as legacy magnetic media such as tapes and floppy disks (not depicted), specialized ROM/ASIC/PLD-based devices such as security dongles (not depicted) can also be included.

Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

Computer system (2100) may also include an interface (2154) to one or more communication networks (2155). The network can be wireless, wired, optical, for example. Networks can also be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable television, satellite television, and television wired or wireless wide area television, including terrestrial broadcast television. This includes digital networks, vehicular and industrial applications including CANbus. Certain networks typically require an external network interface adapter attached to a specific general-purpose data port (e.g., a USB port on a computer system (2100)) or a peripheral bus (2149); other networks typically require , integrated into the core of the computer system (2100) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, computer system (2100) can communicate with other entities. Such communications may be unidirectional reception only (e.g. broadcast TV), unidirectional transmission only (e.g. CANbus to specific CANbus devices), or other computer systems using local or wide area digital networks, e.g. It can be both ways. Specific protocols and protocol stacks may be used in each of those networks and network interfaces, as described above.

The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (2140) of the computer system (2100).

A core (2140) consists of one or more central processing units (CPUs) (2141), graphics processing units (GPUs) (2142), field programmable gate arrays (FPGAs) (2143), and hardware for specific tasks. Special programmable processing devices in the form of accelerators (2144), graphics adapters (2150), etc. may be included. These devices are connected via the system bus (2148), along with read-only memory (ROM) (2145), random access memory (2146), internal mass storage (2147) such as internal hard drives, SSDs, etc. that are not accessible to the user. May be connected. In some computer systems, the system bus (2148) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (2148) or via the peripheral bus (2149). In one example, a screen (2110) can be connected to a graphics adapter (2150). Architectures for peripheral buses include PCI, USB, etc.

The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) may be combined to execute specific instructions that may constitute the aforementioned computer code. The computer code can be stored in ROM (2145) or RAM (2146). Transitional data may also be stored in RAM (2146) and persistent data may be stored, for example, in internal mass storage (2147). Fast storage and retrieval for any of the memory devices can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage (2147), ROM (2145), RAM (2146), etc. Can be made possible using cache memory.

The computer-readable medium can have computer code for performing various computer-implemented operations. The media and computer code may be of the type specifically designed and constructed for the purposes of this disclosure, or of the available types well known to those skilled in the computer software arts.

By way of non-limiting example, a computer system (2100) having an architecture, specifically a core (2140), may include a processor (including a CPU, GPU, FPGA, accelerator, etc.) having one or more tangible computer-readable media. Functionality may be provided as a result of executing software embodied in the software. Such computer-readable media can include the user-accessible mass storage introduced above, as well as certain types of core (2140) of a non-transitory nature, such as core internal mass storage (2147) or ROM (2145). It can be a medium associated with storage. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (2140). The computer-readable medium can include one or more memory devices or chips, depending on particular needs. Software allows cores (2140), and specifically processors (including CPUs, GPUs, FPGAs, etc.) among them, to define data structures stored in RAM (2146) and processes defined by the software. Certain processes or certain portions of certain processes described herein may be performed, including modifying such data structures in accordance with the invention. Additionally or alternatively, the computer system may provide functionality as a result of hardwired or otherwise embodied logic in the circuitry (e.g., accelerator (2144)), which is described herein. may operate in place of, or in conjunction with, software to perform certain processes, or certain portions of certain processes, described in the book. References to software can include logic, and vice versa, where appropriate. Reference to a computer-readable medium may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. can. This disclosure encompasses any suitable combination of hardware and software.

Although this disclosure describes several exemplary embodiments, there are modifications, substitutions, and various alternative equivalents that fall within the scope of this disclosure. Thus, those skilled in the art will be able to devise numerous systems and methods not expressly illustrated or described herein that embody the principles of this disclosure, and are therefore within the spirit and scope of this disclosure. Understand what you can do.
Appendix A: Acronyms
JEM: Joint exploration model
VVC: Versatile Video Coding
BMS: Benchmark set
MV: motion vector
HEVC: High Efficiency Video Coding
SEI: Supplementary Extended Information
VUI: Video usability information
GOPs: Picture groups
TUs: conversion units
PUs: Prediction units
CTUs: Coding Tree Units
CTBs: Coding Tree Blocks
PBs: Predicted blocks
HRD: Virtual Reference Decoder
SNR: signal-to-noise ratio
CPUs: central processing units
GPUs: Graphics processing units
CRT: cathode ray tube
LCD: Liquid crystal display
OLED: Organic light emitting diode
CD: compact disc
DVD: Digital video disc
ROM: Read-only memory
RAM: Random access memory
ASIC: Application-specific integrated circuit
PLD: Programmable logic device
LAN: Local area network
GSM: Global System for Mobile Communications
LTE: Long Term Evolution
CANBus: Controller Area Network Bus
USB: Universal Serial Bus
PCI: Peripheral Component Interconnect
FPGA: Field programmable gate area
SSD: solid state drive
IC: integrated circuit
HDR: High dynamic range
SDR: Standard dynamic range
JVET: Joint Video Exploration Team
MPM: Most probable mode
WAIP: Wide-angle intra prediction
CU: Coding unit
PU: Prediction unit
TU: conversion unit
CTU: Coding Tree Unit
PDPC: Position-dependent prediction combination
ISP: intra subpartition
SPS: Sequence parameter settings
PPS: Picture parameter set
APS: Adaptive parameter set
VPS: Video parameter set
DPS: Decoding parameter set
ALF: Adaptive loop filter
SAO: Sample adaptive offset
CC-ALF: Cross-component adaptive loop filter
CDEF: Constrained Directional Enhancement Filter
CCSO: Cross component sample offset
LSO: Local sample offset
LR: Loop restoration filter
AV1：AOMedia Video 1
AV2: AOMedia Video 2
MVD: motion vector difference
CfL: Chroma from Luma
SDT: Semi-separated tree
SDP: Semi-separate split
SST: Semi-separate tree
SB: Super Block
IBC (or IntraBC): Intra block copy
CDF: cumulative density function
SCC: Screen Content Coding
GBI: Generalized Biprediction
BCW: Bi-prediction with CU-level weights
CIIP: Combined Intra-Inter Prediction
POC: Picture order count
RPS: Reference Picture Set
DPB: decoded picture buffer
MMVD: Merge mode with motion vector difference

101 samples
102 Arrow
103 Arrow
104 blocks
201 Current block
202 samples
203 samples
204 samples
205 samples
206 samples
300 communication system
310 terminal device
320 terminal device
330 terminal device
340 terminal device
350 network
400 Communication System
401 video source
402 Stream of video pictures
403 video encoder
404 encoded video data
405 Streaming Server
406 Client Subsystem
407 video data
408 Client Subsystem
409 video data
410 video decoder
411 Video picture output stream
412 display
413 Video Capture Subsystem
420 Electronic devices
430 Electronic devices
501 channels
510 video decoder
512 rendering device
515 Buffer memory
520 parser
521 symbols
530 Electronic devices
531 Receiver
551 Scaler/inverse conversion unit
552 Intra prediction unit
553 Motion Compensated Prediction Unit
555 Aggregator
556 Loop filter
557 Reference picture memory
558 Current picture buffer
601 Video Source
603 video encoder
620 Electronic devices
630 source coder
632 coding engine
633 decoder
634 Reference picture memory
635 Predictor
640 transmitter
645 Entropy coder
650 controller
660 communication channels
703 video encoder
721 General purpose controller
722 Intra encoder
723 Residual Calculator
724 Residual Encoder
725 entropy encoder
726 switch
728 Residual Decoder
730 inter encoder
810 video decoder
871 Entropy Decoder
872 Intra decoder
873 Residual Decoder
874 Rebuild module
880 interdecoder
2100 computer system
2101 keyboard
2102 Mouse
2103 trackpad
2105 joystick
2106 Microphone
2107 Scanner
2108 camera
2109 speaker
2110 touch screen
2120 CD/DVD ROM/RW
2121 CD/DVD or similar media
2122 thumb drive
2123 Removable Hard Drive or Solid State Drive
2140 cores
2141 Central Processing Unit (CPU)
2142 Graphics processing unit (GPU)
2143 Field Programmable Gate Area (FPGA)
2144 Accelerator
2145 Read-only memory (ROM)
2146 Random Access Memory (RAM)
2147 Mass storage
2148 system bus
2149 Surrounding Bus
2150 graphics adapter
2154 network interface
2155 Communication Network

Claims (22)

A method for processing a current video block of a video stream, the method comprising:
receiving the video stream;
determining that the current video block is inter-coded based on a predictive block and a motion vector (MV), the MV being a reference motion vector (RMV) of the current video block and a motion vector difference ( MVD) should be derived from
In response to determining that the MVD is coded with an adaptive MVD pixel resolution;
determining a reference MVD pixel precision of the current video block;
identifying a maximum allowed MVD pixel accuracy;
determining a set of allowable MVD levels for the current video block based on the reference MVD pixel precision and the maximum allowed MVD pixel precision;
deriving the MVD from the video stream according to at least one MVD parameter signaled within the video stream for the current video block and the set of allowable MVD levels. 2. The method of claim 1, wherein the reference MVD pixel precision of the current video block is specified/signaled/derived at sequence level, picture level, frame level, superblock level or coding block level. 3. The method of claim 2, wherein the reference MVD pixel accuracy of the current video block depends on an MVD class associated with the MVD of the current video block. 3. The method of claim 2, wherein the reference MVD pixel precision of the current video block depends on the MVD size of the MVD of the current video block. 3. The method of claim 2, wherein the maximum allowed MVD pixel accuracy is predefined. further comprising determining a current MVD class from among a predefined set of MVD classes, and determining the set of allowable MVD levels for the MVD based on the reference MVD pixel precision and the maximum allowed MVD pixel precision. The steps to decide are:
from a set of reference MVD levels determined based on the reference MVD pixel precision and the current MVD class to determine the set of allowable MVD levels for the current video block that is greater than or equal to the maximum allowable MVD pixel precision; 6. A method according to any one of claims 1 to 5, comprising excluding MVD levels associated with MVD pixel accuracy of . 7. The method of claim 6, wherein the maximum allowed MVD pixel precision is 1/4 pixel. 6. A method according to any one of claims 1 to 5, wherein MVD levels associated with an accuracy of ⅛ pixel or better are excluded from the set of acceptable MVD levels of the current video block. further comprising determining a current MVD class from among the predefined set of MVD classes;
5. An MVD level associated with a fractional MVD precision is included in the set of acceptable MVD levels, regardless of the reference MVD precision, if the current MVD class is less than or equal to a threshold MVD class. The method described in any one of the above. 10. The method of claim 9, wherein the threshold MVD class is the lowest MVD class among the predefined set of MVD classes. further comprising determining a magnitude of the MVD, wherein the MVD level associated with an MVD accuracy higher than a threshold MVD accuracy is within the acceptable range only if the magnitude of the MVD is less than or equal to a threshold MVD magnitude. 6. A method according to any one of claims 1 to 5, which is permissible in a set of MVD levels. 12. The method of claim 11, wherein the threshold MVD magnitude is 2 pixels or less. 13. The method of claim 12, wherein the threshold MVD accuracy is 1 pixel. 12. The method of claim 11, wherein an MVD level associated with an MVD accuracy of 1/4 pixel or greater is only allowed if the magnitude of the MVD is 1/2 pixel or less. 6. A method according to any preceding claim, wherein the maximum allowed MVD pixel precision is less than or equal to the reference MVD pixel precision. A method for processing a current video block of a video stream, the method comprising:
receiving the video stream;
determining that the current video block is inter-coded and associated with multiple reference frames;
determining whether an adaptive motion vector difference (MVD) pixel resolution is applied to at least one of the plurality of reference frames based on signaling in the video stream. 17. The method of claim 16, wherein the signaling includes a single-bit flag to indicate whether adaptive MVD pixel resolution is applied to all or none of the plurality of reference frames. 17. The method of claim 16, wherein the signaling includes separate flags each corresponding to one of the plurality of reference frames to indicate whether adaptive MVD pixel resolution is applied. The signaling may include, for each of a plurality of reference frames,
an implicit indication that no adaptive MVD pixel resolution is applied when the MVD corresponding to each of the plurality of reference frames is zero;
and a single-bit flag for indicating whether adaptive MVD pixel resolution is applied when the MVD corresponding to the each of the plurality of reference frames is non-zero. A method for processing a current video block of a video stream, the method comprising:
receiving the video stream;
determining that the current video block is inter-coded based on a predictive block and a motion vector (MV), where the MV is a reference motion vector (RMV) and a motion vector difference (RMV) for the current video block; MVD) should be derived from
determining a current MVD class of said MVD from a predefined set of MVD classes;
deriving at least one context for entropy decoding at least one explicit signaling in the video stream based on the current MVD class, the at least one explicit signaling comprising: included in the video stream to specify an MVD pixel resolution for at least one component of the MVD;
entropy decoding the at least one explicit signaling from the video stream using the at least one context to determine the MVD pixel resolution for the at least one component of the MVD; Including, methods. The at least one component of the MVD includes a horizontal component and a vertical component of the MVD, and the at least one context includes two contexts each associated with one of the horizontal component and the vertical component of the MVD. 21. The method of claim 20, including separate contexts, wherein the horizontal component and the vertical component are associated with separate MVD pixel resolutions. 22. A method according to any one of claims 1 to 5 and 16 to 21, wherein the video processing device comprises a memory for storing computer instructions and a processor, the processor executing the computer instructions. A video processing device configured to:

Images