HK40080098B - Methods, apparatus and storage medium for decoding video data - Google Patents

Description

Methods, apparatus, and storage media for decoding video data

Incorporation

This application is based on and claims priority interests in U.S. Provisional Application No. 63/175,897, filed April 16, 2021, and U.S. Non-Provisional Application No. 17/568,275, filed January 4, 2022, both of which are incorporated herein by reference in their entirety.

Technical Field

This disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques relate to the interaction between transform partitioning schemes and primary/secondary transform type selection in video coding and decoding.

Background Technology

The background description provided herein is for the purpose of presenting the general content of this disclosure. The extent of the work performed by the currently named inventors as described in this background section and in various aspects of this specification does not imply that it was qualified as prior art at the time of filing of this application, nor is it expressly or implied that it was recognized as prior art of this disclosure.

Video encoding and decoding can be performed using inter-frame picture prediction with motion compensation. Uncompressed digital video can comprise a series of pictures, each with a spatial size of, for example, a 1920×1080 luminance sample and an associated full or subsampled chrominance sample. This series of pictures can have a fixed or variable picture rate (alternatively referred to as the frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, a chrominance subsampling of 4:2:0, and 8 bits per pixel/color channel requires bandwidth close to 1.5 Gbit/s. One hour of such video would require more than 600 GB of storage space.

One objective of video encoding and decoding is to reduce redundancy in the uncompressed input video signal through compression. Compression can help reduce the aforementioned bandwidth and/or storage space requirements, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations thereof can be employed. Lossless compression refers to a technique that reconstructs an exact copy of the original signal from the compressed original signal through a decoding process. Lossy compression refers to an encoding/decoding process where the original video information is not fully preserved during encoding and cannot be fully recovered during decoding. When using lossy compression, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application, despite some information loss. In the case of video, lossy compression is widely used in many applications. The tolerable amount of distortion depends on the application. For example, users of some consumer video streaming applications may tolerate higher distortion than users of movie or television broadcasting applications. The compression ratio achievable through a specific encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows the encoding algorithm to produce higher losses and higher compression ratios.

Video encoders and decoders can utilize techniques and steps from a wide range of categories, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

Video codec techniques may include techniques called intra-frame coding. In intra-frame coding, sample values are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture may be called an intra-frame picture. Intra-frame pictures and their derivatives (e.g., independent decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in an encoded video stream and video session, or as a still image. Samples of blocks following intra-frame prediction can then be transformed into the frequency domain, and the transformed coefficients thus generated can be quantized before entropy coding. Intra-frame prediction represents a technique that minimizes the sample values in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block at a given quantization step size.

For example, traditional intra-frame coding techniques known from MPEG-2 generation coding technologies do not use intra-frame prediction. However, some newer video compression techniques include those that attempt to encode/decode blocks based on, for example, surrounding sample data and/or metadata, which is obtained during spatially adjacent encoding and/or decoding and is placed before the data block being intra-frame encoded or decoded in decoding order. Such techniques are referred to below as "intra-frame prediction" techniques. It should be noted that, at least in some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and not reference data from a reference picture.

Intra-frame prediction can take many different forms. When more than one such technique is available in a given video coding technique, the technique in use is called an intra-frame prediction mode. One or more intra-frame prediction modes can be provided in a particular codec. In some cases, a mode may have sub-modes and/or may be associated with various parameters, and the mode/sub-mode information and the intra-frame coding parameters of the video block may be encoded separately or contained together in the mode codeword. Which codeword is used for a given combination of mode, sub-mode, and/or parameters can affect the coding efficiency gain through intra-frame prediction, and therefore may affect the entropy coding technique used to convert the codeword into a bitstream.

H.264 introduced an intra-frame prediction mode, which was refined in H.265 and further improved in newer coding techniques such as the Joint Exploration Model (JEM), Next-Generation Video Coding (VVC), and Baseline Spectrum (BMS). Typically, for intra-frame prediction, predictor blocks are formed using neighboring sample values that have become available. For example, available values from a specific neighboring sample set can be copied along a certain direction and/or line into the predictor block. The reference to the direction used can be encoded in the bitstream or can itself be predicted.

Referring to Figure 1A, a subset of nine prediction directions specified in the 33 possible intra-frame prediction directions of H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes specified in H.265) is depicted in the lower right. The point (101) where the arrows converge represents the sample being predicted. The arrows indicate the direction along which the sample at 101 is predicted using neighboring samples. For example, arrow (102) indicates that sample (101) is predicted based on one or more neighboring samples in the upper right at a 45-degree angle to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more neighboring samples in the lower left of sample (101) at a 22.5-degree angle to the horizontal.

Referring again to Figure 1A, a 4×4 square block (104) of samples is depicted in the upper left (indicated by a bold dashed line). The square block (104) comprises 16 samples, each labeled with “S”, its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. An exemplary reference sample following a similar numbering scheme is also shown. The reference sample is labeled with R, its Y position (e.g., row index) relative to the block (104), and its X position (column index). In H.264 and H.265, predicted samples adjacent to the block being reconstructed are used.

Intra-frame picture prediction for block 104 can begin by copying reference sample values from neighboring samples based on the prediction direction indicated by a signal. For example, suppose the encoded video stream includes signaling that indicates the prediction direction of the arrow (102) for block 104, i.e., predicting samples based on one or more prediction samples in the upper right corner at a 45-degree angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Then sample S44 is predicted based on reference sample R08.

In some cases, the values of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample; especially when the direction is not divisible by 45 degrees.

As video coding techniques continue to evolve, the number of possible directions increases. In H.264 (2003), for example, nine different directions were available for intra-frame prediction. In H.265 (2013), this increased to 33 directions, and at the time of this disclosure, JEM/VVC/BMS supports up to 65 directions. Experimental studies have been conducted to help identify the most suitable intra-frame prediction directions, and some techniques in entropy coding can be used to encode those most suitable directions with small bit counts, accepting a certain bit cost for each direction. Furthermore, sometimes the direction itself can be predicted based on the adjacent directions used in the intra-frame prediction of decoded adjacent blocks.

Figure 1B shows a schematic diagram (180) depicting 65 intra-frame prediction directions according to JEM to illustrate the increasing number of prediction directions in various coding techniques developed over time.

In an encoded video stream, the way bits representing an intra-prediction direction are mapped to that prediction direction can vary depending on the video coding technique. For example, the range can vary from a simple direct mapping of the prediction direction to an intra-prediction mode, to mapping the prediction direction to codewords, to complex adaptive schemes involving the most probable mode, and similar techniques. However, in all cases, there may be some intra-prediction directions that are statistically less likely to appear in the video content compared to others. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, less probable directions can be represented by more bits than more probable directions.

Inter-frame image prediction, or inter-prediction, can be based on motion compensation. In motion compensation, sample data from a previously reconstructed image or a portion thereof (the reference image), after being spatially offset along a direction indicated by a motion vector (hereafter referred to as MV), can be used to predict a newly reconstructed image or image portion (e.g., a patch). In some cases, the reference image may be the same as the image currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference image being used (similar to a temporal dimension).

In some video compression techniques, the current MV applicable to a region of sample data can be predicted based on other MVs, such as those related to another region of sample data spatially adjacent to the region being reconstructed and preceding the current MV in decoding order. This significantly reduces the total amount of data required to encode the MV by eliminating redundancy in related MVs, thereby increasing compression efficiency. MV prediction works effectively, for example, because when encoding the input video signal (called natural video) from the camera, there is a statistical probability that a larger region than the region applicable to a single MV moves along similar directions in the video sequence. Therefore, in some cases, the larger region can be predicted using similar motion vectors derived from the MVs of neighboring regions. This makes the actual MV for a given region similar to or the same as the MV predicted based on the surrounding MVs. Consequently, after entropy coding, the MV can be represented using fewer bits than when directly encoding the MV (rather than predicting it based on neighboring MVs). In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, such as when rounding errors occur while calculating the predicted value based on multiple surrounding MVs, the MV prediction itself can be lossy.

H.265/HEVC (ITU-T H.265 Recommendation, “High Efficiency Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms specified in H.265, the following describes a technique referred to below as “spatial merge”.

Specifically, referring to Figure 2, the current block (201) includes samples found by the encoder during the motion search process, which can be predicted based on previous blocks of the same size that have generated spatial offsets. Instead of directly encoding the MV, it can be derived from metadata associated with one or more reference images, for example, using the MV associated with any of five surrounding samples labeled A0, A1 and B0, B1, B2 (corresponding to 202 to 206 respectively), derived from the nearest reference image (in decoding order). In H.265, MV prediction can use predictions from the same reference image being used by adjacent blocks.

Summary of the Invention

This disclosure describes various embodiments of methods, apparatuses, and computer-readable storage media for video encoding and/or decoding.

According to one aspect, one embodiment of this disclosure provides a method for encoding/decoding video data in a decoder. The method includes: receiving an encoded video stream of data blocks; extracting a transform partition type associated with the data blocks from the encoded video stream; and in response to a transform partition type belonging to a subset of a predetermined set of transform partition types, each transform partition type in the predetermined set specifying a splitting pattern for dividing the data blocks into transform blocks: extracting a transform type associated with the transform blocks split from the data blocks, the transform type being represented by a signal in the encoded video stream, wherein the transform type belongs to a first predetermined set of transform types; and performing an inverse transform on the transform blocks according to the transform type.

According to another aspect, one embodiment of this disclosure provides a method for encoding/decoding video data. The method includes: receiving an encoded video stream of data blocks; extracting a transform partition type associated with a data block of video data from the encoded video stream; extracting a transform type associated with a transform block of the data block from the encoded video stream in response to the transform partition type belonging to a subset of a predetermined transform partition type set; and identifying the transform type of the data block according to a default method in response to the transform partition type not belonging to the predetermined transform partition type set.

According to another aspect, one embodiment of this disclosure provides a method for encoding/decoding video data. The method includes: receiving an encoded video stream of data blocks; extracting a transform type of a transform associated with a transform block of the data blocks from the encoded video stream; and, in response to the transform type belonging to a predetermined transform type set: extracting a transform partition type associated with the data blocks from the encoded video stream.

According to another aspect, one embodiment of this disclosure provides an apparatus for video encoding and/or decoding. The apparatus includes: a memory storing instructions; and a processor communicating with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to perform the aforementioned methods for video decoding and/or encoding.

According to another aspect, one embodiment of this disclosure provides a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and/or encoding, cause the computer to perform the aforementioned methods for video decoding and/or encoding.

The above and other aspects and their implementations are described in more detail in the drawings, description and claims.

Attached Figure Description

Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

Figure 1A shows a schematic illustration of an exemplary subset of intra-predictive orientation modes.

Figure 1B illustrates an exemplary intra-frame prediction direction.

Figure 2 shows a schematic illustration of the current block and its surrounding spatial merging candidates for motion vector prediction in one example.

Figure 3 shows a schematic illustration of a simplified block diagram of a communication system (300) according to an exemplary embodiment.

Figure 4 shows a schematic illustration of a simplified block diagram of a communication system (400) according to an exemplary embodiment.

Figure 5 shows a schematic illustration of a simplified block diagram of a video decoder according to an exemplary embodiment.

Figure 6 shows a schematic illustration of a simplified block diagram of a video encoder according to an exemplary embodiment.

Figure 7 shows a block diagram of a video encoder according to another exemplary embodiment.

Figure 8 shows a block diagram of a video decoder according to another exemplary embodiment.

Figure 9 illustrates a directional intra-frame prediction mode according to an exemplary embodiment of the present disclosure.

Figure 10 illustrates a non-directional intra-frame prediction mode according to an exemplary embodiment of the present disclosure.

Figure 11 illustrates a recursive intra-frame prediction mode according to an exemplary embodiment of the present disclosure.

Figure 12 illustrates the transform block partitioning and scanning of an intra-prediction block according to an exemplary embodiment of the present disclosure.

Figure 13 illustrates the transform block partitioning and scanning of inter-frame prediction blocks according to an exemplary embodiment of the present disclosure.

Figure 14 illustrates a low-frequency non-separable transformation process according to an exemplary embodiment of the present disclosure.

Figure 15 illustrates an intra-frame prediction scheme based on individual reference lines according to an exemplary embodiment of the present disclosure.

Figure 16 illustrates a non-recursive block partitioning scheme according to an exemplary embodiment of the present disclosure.

Figure 17 shows a flowchart of an exemplary embodiment according to the present disclosure.

Figure 18 shows a schematic illustration of a computer system according to an exemplary embodiment of the present disclosure.

Detailed Implementation

Figure 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) can perform unidirectional data transmission. For example, terminal device (310) can encode video data (e.g., video data of a video picture stream captured by terminal device (310)) for transmission over the network (350) to another terminal device (320). The encoded video data can be transmitted in the form of one or more encoded video streams. Terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to recover video pictures, and display the video pictures based on the recovered video data. Unidirectional data transmission can be implemented in applications such as media services.

In another example, the communication system (300) includes a pair of second terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which may be implemented, for example, during a video conferencing application. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., video data from a video picture stream captured by the terminal device) for transmission over a network (350) to the other terminal device (330) and (340). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other terminal device (330) and (340), and may decode the encoded video data to recover video pictures, and may display the video pictures on an accessible display device based on the recovered video data.

In the example of Figure 3, terminal devices (310), (320), (330), and (340) can be implemented as servers, personal computers, and smartphones, but the applicability of the underlying principles of this disclosure is not limited thereto. Embodiments of this disclosure can be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. Network (350) refers to any number of networks that transmit encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired (connected) and/or wireless communication networks. Communication network (350) 9 can exchange data in circuit-switched channels, packet-switched channels, and/or other types of channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and/or the Internet. For the purposes of this discussion, unless explicitly stated herein, the architecture and topology of the network (350) may be irrelevant to the operation of this disclosure.

As an example of an application of the disclosed subject matter, Figure 4 illustrates the placement of a video encoder and a video decoder in a video streaming environment. The disclosed subject matter is equally applicable to other video applications, including, for example, video conferencing, digital TV, broadcasting, gaming, virtual reality, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

The video streaming system may include a video acquisition subsystem (413), which may include a video source (401) such as a digital camera, for creating uncompressed video pictures or image streams (402). In one example, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The video picture stream (402), depicted as a thick line to emphasize its high data volume, may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. Compared to the uncompressed video picture stream (402), the encoded video data (404) (or encoded video bitstream (404)) depicted as thin lines to emphasize its lower data volume may be stored on a streaming server (405) for future use or directly stored on a downstream video device (not shown). One or more streaming client subsystems, such as client subsystems (406) and (408) in FIG. 4, may access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an uncompressed output video picture stream (411) that can be presented on a display (412) (e.g., a screen) or other presentation device (not depicted). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, encoded video data (404), (407), and (409) (e.g., video streams) can be encoded according to certain video coding/compression standards. Examples of these standards include ITU-T Recommendation H.265. In one example, the video coding standard under development is informally referred to as Next Generation Video Coding (VVC). The topics disclosed can be used in the context of VVC and other video coding standards.

It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

In the following text, Figure 5 shows a block diagram of a video decoder (510) according to any embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of the video decoder (410) in the example of Figure 4.

The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence may be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence may be associated with multiple video frames or images. Encoded video sequences may be received from a channel (501), which may be a hardware/software link leading to a storage device storing the encoded video data or a streaming source transmitting the encoded video data. The receiver (531) may receive encoded video data and other data, such as encoded audio data and/or auxiliary data streams, that may be forwarded to their respective processing circuitry (not depicted). The receiver (531) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (515) may be provided between the receiver (531) and the entropy decoder/parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be located external to and separate from the video decoder (510) (not depicted). In still other applications, a buffer memory (not depicted) may be placed external to the video decoder (510) for purposes such as preventing network jitter, and another additional buffer memory (515) may be placed internally to handle playback timing, for example. When the receiver (531) receives data from a store/forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer memory (515) may not be necessary, or the buffer memory may be made smaller. For use on packet networks such as the Internet, a buffer memory (515) of sufficient size may be required, and the size of the buffer memory (515) may be relatively large. Such a buffer memory may be implemented with an adaptive size and may be implemented at least partially in a similar component (not depicted) external to the operating system or the video decoder (510).

The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (510) and potential information for controlling a presentation device such as a display (512) (e.g., a screen), which may or may not be integral to the electronic device (530), but may be coupled to the electronic device (530), as shown in Figure 5. Control information for the presentation device may be in the form of supplementary enhancement information (SEI messages) or fragments of video availability information (VUI) parameter sets (not depicted). The parser (520) may perform parsing/entropy decoding on the encoded video sequence received by the parser (520). The entropy coding of the encoded video sequence may be performed according to video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) can extract a set of subgroup parameters from the encoded video sequence for use in the video decoder, based on at least one parameter corresponding to a subgroup. Subgroups may include group of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

The parser (520) can perform entropy decoding/parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

Depending on the type of encoded video frames or portions thereof (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different processing or functional units. The units involved, and the manner in which they are involved, can be controlled by the parser (520) through subgroup control information parsed from the encoded video sequence. For simplicity, the flow of such subgroup control information between the parser (520) and the various processing or functional units described below is not depicted.

In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into multiple functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the concept of multiple functions is adopted below in this disclosure.

The first unit may include a scaler/inverse transform unit (551). The scaler/inverse transform unit (551) can receive quantization transform coefficients as symbols (521) from the parser (520) as well as control information, including information such as the type of inverse transform to be used, block size, quantization factor/parameter, quantization scaling matrix, etc. The scaler/inverse transform unit (551) can output a block containing sample values, which can be input into the aggregator (555).

In some cases, the output samples of the scaler/inverse transform (551) may belong to intra-coded blocks; that is, blocks that do not use prediction information from previously reconstructed images but can use prediction information from previously reconstructed portions of the current image. Such prediction information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use information from surrounding blocks that have been reconstructed and stored in the current picture buffer (558) to generate blocks of the same size and shape as the blocks being reconstructed. For example, the current picture buffer (558) buffers partially reconstructed current images and/or fully reconstructed current images. In some implementations, the aggregator (555) may add the prediction information generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler/inverse transform unit (551) based on each sample.

In other cases, the output samples of the scaler/inverse transform unit (551) may belong to blocks of inter-frame coding and potential motion compensation. In this case, the motion compensation prediction unit (553) may access the reference picture memory (557) to extract samples for inter-frame picture prediction. After motion compensation is performed on the extracted samples according to the symbols (521) belonging to the blocks, these samples may be added by the aggregator (555) to the output of the scaler/inverse transform unit (551) (the output of unit 551 may be referred to as residual samples or residual signals), thereby generating output sample information. The extraction of prediction samples by the motion compensation prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, which may be provided to the motion compensation prediction unit (553) in the form of symbols (521), which may have, for example, an X component, a Y component (offset), and a reference picture component (time). Motion compensation may also include interpolation of sample values extracted from a reference image memory (557) when using precise motion vectors of subsamples, and motion compensation may also be associated with motion vector prediction mechanisms, etc.

The output samples of the aggregator (555) can be subjected to various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream) and available as symbols (521) from the parser (520) to the loop filter unit (556). However, the video compression techniques may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values. Various types of loop filters may be included in various orders as part of the loop filter unit 556, as will be described in further detail below.

The output of the loop filter unit (556) can be a sample stream that can be output to the presentation device (512) and stored in the reference image memory (557) for future inter-frame image prediction.

Once fully reconstructed, certain encoded images can be used as reference images for future inter-frame image prediction. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (520)) is identified as the reference image, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

The video decoder (510) can perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under that configuration file. To conform to the standard, the complexity of the encoded video sequence may be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy may be further limited by the hypothetical reference decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

In some exemplary embodiments, the receiver (531) may receive additional (redundant) data when receiving encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and/or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

Figure 6 shows a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., transmission circuitry). The video encoder (603) may be used in place of the video encoder (403) in the example of Figure 4.

The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of Figure 6), which can capture video images that will be encoded by the video encoder (603). In another example, the video source (601) may be implemented as part of the electronic device (620).

A video source (601) can provide a sequence of source video samples to be encoded by a video encoder (603) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device capable of storing previously prepared video. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. The video data can be provided as multiple individual pictures or images, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. The relationship between pixels and samples will be readily understood by those skilled in the art. The following focuses on describing samples.

According to some exemplary embodiments, a video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units described below. For simplicity, coupling is not depicted in the figures. Parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of images (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other suitable functions related to the video encoder (603) optimized for a particular system design.

In some exemplary embodiments, the video encoder (603) may be configured to operate within an encoding loop. As an oversimplification, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how a (remote) decoder can create sample data, even though the embedded decoder 633 processes the video stream encoded by the source encoder 630 without entropy coding (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols in entropy coding and the encoded video bitstream can be lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since the decoding of the symbol stream produces bit-precise results independent of the decoder location (local or remote), the contents of the reference image memory (634) are also bit-precisely corresponding between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the encoder's prediction section are exactly the same as the sample values that the decoder will "see" during the prediction phase. This fundamental principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is used to improve coding quality.

The operation of the “local” decoder (633) can be the same as that of a “remote” decoder, such as the video decoder (510) which has been described in detail above in conjunction with Figure 5. However, referring briefly to Figure 5 again, since symbols are available and the entropy encoder (645) and parser (520) are able to encode/decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (510), which includes the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633) within the encoder.

It can be observed that any decoder technique, other than parsing/entropy decoding which can only exist in the decoder, must also necessarily exist in the corresponding encoder in essentially the same functional form. For this reason, the disclosed subject matter may sometimes focus on decoder operations, which are combined with the decoding portion of the encoder. Therefore, the description of encoder techniques can be simplified, as encoder techniques are inverses of the comprehensively described decoder techniques. In the following, only certain areas or aspects are described in more detail about the encoder.

During operation, in some exemplary implementations, the source encoder (630) may perform motion-compensated predictive coding, which predictively encodes the input image by referencing one or more previously encoded images from the video sequence designated as "reference images." In this manner, the encoding engine (632) encodes the differences (or residuals) in the color channels between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image. The term "residual" and its adjective form "residual" are used interchangeably.

The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture based on symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the encoded video data can be decoded in the video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process, which can be performed by the video decoder on the reference picture, and allows the reconstructed reference picture to be stored in a reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that shares common content (no transmission errors) with the reconstructed reference picture that will be obtained by the remote video decoder.

The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units using techniques such as Huffman coding, variable-length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware/software link to a storage device capable of storing the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (source not shown).

The controller (650) manages the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (“IDR”) pictures. Those skilled in the art will understand these variations of I-pictures and their corresponding applications and characteristics.

A predictive picture (P-picture) can be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values for each block.

A bidirectional predictive picture (B-picture) can be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values for each block. Similarly, multiple predictive pictures can be used to reconstruct a single block using more than two reference pictures and associated metadata.

Source images can typically be spatially subdivided into multiple sample coding blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and coded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or blocks of an I-image can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction. For other purposes, source images or intermediate images can be subdivided into other types of blocks. The partitioning of coding blocks and other types of blocks may or may not follow the same manner, as described in further detail below.

The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard such as ITU-T H.265 Recommendation. In operation, the video encoder (603) can perform various compression operations, including predictive coding operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

In some exemplary embodiments, the transmitter (640) may transmit additional data while transmitting encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

The captured video can be presented as multiple source images (video images) in a time-series format. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes temporal or other correlations between images. For example, a specific image being encoded/decoded can be divided into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference image.

In some exemplary embodiments, bidirectional prediction techniques can be used for inter-frame image prediction. According to this bidirectional prediction technique, two reference images are used, such as a first reference image and a second reference image that precede the current image in the video in decoding order (but may be past or future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be jointly predicted using a combination of the first and second reference blocks.

In addition, merging mode techniques can be used for inter-frame image prediction to improve coding efficiency.

According to some exemplary embodiments of this disclosure, predictions such as inter-frame picture prediction and intra-frame picture prediction are performed on a block-by-block basis. For example, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be recursively split into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU may be split into one 64×64 pixel CU, or four 32×32 pixel CUs. Each of the one or more 32×32 blocks may be further split into four 16×16 pixel CUs. In some exemplary embodiments, each CU may be analyzed during encoding to determine the prediction type used for the CU among a variety of prediction types, such as inter-frame prediction type or intra-frame prediction type. Based on temporal and/or spatial predictability, a CU may be split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in encoding (encoding/decoding) are performed on a per-prediction-block basis. The CU can be split into PUs (or PBs for different color channels) in various spatial modes. For example, a luma or chroma PB may comprise a matrix of values for samples (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

Figure 7 illustrates a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is configured to receive processing blocks (e.g., prediction blocks) of sample values within a current video picture in a video picture sequence, and to encode the processing blocks into an encoded picture that is part of an encoded video sequence. In the example, the video encoder (703) can be used instead of the video encoder (403) in the example of Figure 4.

For example, a video encoder (703) receives a matrix of sample values for a processing block, such as an 8×8 sample prediction block. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether to optimally encode the processing block using intra-frame mode, inter-frame mode, or bidirectional prediction mode. When it is determined that the processing block is to be encoded in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processing block into an encoded picture; and when it is determined that the processing block is to be encoded in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques respectively to encode the processing block into an encoded picture. In some exemplary embodiments, a merging mode can be used as a sub-mode for inter-frame picture prediction, wherein motion vectors are derived from one or more motion vector predictors without the aid of encoded motion vector components outside the predictor. In some other exemplary embodiments, motion vector components applicable to the subject block may be present. Therefore, the video encoder (703) may include components not explicitly shown in FIG7, such as a mode decision module for determining the prediction mode of the processing block.

In the example of Figure 7, the video encoder (703) includes an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in the exemplary arrangement of Figure 7.

An inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare that block with one or more reference blocks in a reference image (e.g., blocks in the order of display from previous to later images), generate inter-frame prediction information (e.g., a description of redundancy information based on the inter-frame coding technique, motion vectors, merging mode information), and compute inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image that is decoded based on encoded video information using a decoding unit 633 embedded in the exemplary encoder 620 of FIG. 6 (shown as the residual decoder 728 of FIG. 7, as described in further detail below).

The intra encoder (722) is configured to receive samples of the current block (e.g., the processed block), compare the block with encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases generate intra prediction information (e.g., intra prediction direction information based on one or more intra coding techniques). The intra encoder (722) may compute intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

A general controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on that general control data. In one example, the general controller (721) determines the prediction mode of a block and provides control signals to a switch (726) based on that prediction mode. For example, when the prediction mode is an intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; and when the prediction mode of the block is an inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

A residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result of a block selected from the intra encoder (722) or the inter encoder (730). A residual encoder (724) is configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) may generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) may generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, which can be buffered in a memory circuit (not shown) and used as a reference image.

The entropy encoder (725) is configured to format the bitstream to include encoded blocks and to perform entropy encoding. The entropy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) can be configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Residual information may be absent when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

Figure 8 illustrates an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In one example, the video decoder (810) may be used instead of the video decoder (410) in the example of Figure 4.

In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together as shown in the exemplary arrangement in Figure 8.

The entropy decoder (871) can be configured to reconstruct certain symbols from the encoded picture, which represent the syntax elements constituting the encoded picture. Such symbols may include, for example, the mode encoding the block (e.g., intra-mode, inter-mode, bidirectional prediction mode, merged sub-mode, or another sub-mode), prediction information (e.g., intra-prediction information or inter-prediction information) that can be identified for use by the intra-decoder (872) or inter-decoder (880) for prediction, residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is inter-mode or bidirectional prediction mode, inter-prediction information is provided to the inter-decoder (880); and when the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). The residual information may be inversely quantized and provided to the residual decoder (873).

The inter-frame decoder (880) can be configured to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

The intra-frame decoder (872) can be configured to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

The residual decoder (873) can be configured to perform inverse quantization to extract the dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) can also utilize certain control information (to include quantizer parameters (QP)) which can be provided by the entropy decoder (871) (the data path is not depicted because this is only low-volume control information).

The reconstruction module (874) can be configured to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module, as appropriate) in the spatial domain to form a reconstructed block, which forms part of the reconstructed image, and the reconstructed image is part of the reconstructed video. It should be noted that other suitable operations, such as deblocking, can also be performed to improve visual quality.

It should be noted that any suitable technology can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In some exemplary embodiments, one or more integrated circuits can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (403), video encoder (603), and video encoder (603), as well as the video decoder (410), video decoder (510), and video decoder (810).

Returning to the intra-prediction process, a prediction block is generated by predicting samples in a block (e.g., a luma prediction block or a chroma prediction block, or, if not further subdivided into prediction blocks, a coded block) using samples from adjacent, next adjacent, or one or more other lines, or combinations thereof. The residual between the actual block being encoded and the prediction block is then processed by a transform, followed by quantization. Various intra-prediction modes are made available, and parameters and other parameters related to the intra-prediction mode selection can be represented as signals in the bitstream. For example, various intra-prediction modes may involve the position of one or more lines used for prediction samples, the direction along which the prediction samples are selected from one or more prediction lines, and other special intra-prediction modes.

For example, a set of intra-prediction modes (interchangeably referred to as "intra-modes") may include a predetermined number of directional intra-prediction modes. As described above in relation to the exemplary implementation of Figure 1, these intra-prediction modes may correspond to a predetermined number of directions along which out-of-block samples are selected as predictions for samples predicted in a particular block. In another particular exemplary implementation, eight (8) principal directional modes may be supported and predefined, these eight principal directional modes corresponding to angles from 45 degrees to 207 degrees with respect to the horizontal axis.

In some other implementations of intra-frame prediction, to further utilize the greater variety of spatial redundancy in oriented textures, oriented intra-frame modes can be extended to have a finer-grained set of angles. For example, the 8-angle implementation described above can be configured to provide 8 nominal angles, as shown in Figure 9. These 8 nominal angles are named V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, and for each nominal angle, a predetermined number (e.g., 7) of finer angles can be added. With this extension, a larger total number of directional angles (e.g., 56 in this example) can be used for intra-frame prediction, corresponding to the same number of predetermined oriented intra-frame modes. The predicted angle can be represented by the nominal intra-frame angle plus an angle increment. For the specific example above with 7 finer angular directions for each nominal angle, the angle increment can be from -3 to 3 times a step size of 3 degrees.

In some implementations, a predetermined number of non-directional intra-prediction modes can be predefined as an alternative to or in addition to the aforementioned directional intra-prediction modes, and these non-directional intra-prediction modes can be made available. For example, five non-directional intra-prediction modes, referred to as smooth intra-prediction modes, can be specified. These non-directional intra-prediction modes can be specifically referred to as DC intra-prediction mode, PAETH intra-prediction mode, SMOOTH intra-prediction mode, SMOOTH_V intra-prediction mode, and SMOOTH_H intra-prediction mode. Figure 10 illustrates the prediction of samples for a specific block under these exemplary non-directional modes. As an example, Figure 10 shows a 4×4 block 1002 predicted by samples from the top adjacent row and/or the left adjacent row. A specific sample 1010 in block 1002 may correspond to sample 1004 directly above sample 1010 in the top adjacent row of block 1002, sample 1006 to the upper left of sample 1010 which is the intersection of the top and left adjacent rows, and sample 1008 directly to the left of sample 1010 in the left adjacent row of block 1002. For the exemplary DC intra-prediction mode, the average of the left adjacent sample 1008 and the top adjacent sample 1004 can be used as the predicted value for sample 1010. For the exemplary PAETH intra-prediction mode, the top reference sample 1004, the left reference sample 1008, and the upper left reference sample 1006 can be extracted, and then the value closest to (top + left – upper left) among these three reference samples can be set as the predicted value for sample 1010. For the exemplary SMOOTH_V intra-prediction mode, sample 1010 can be predicted by performing quadratic interpolation along the vertical direction on the upper left adjacent sample 1006 and the left adjacent sample 1008. For the exemplary SMOOTH_H intra-prediction mode, sample 1010 can be predicted by performing quadratic interpolation along the horizontal direction on its upper-left neighbor sample 1006 and its top neighbor sample 1004. For the exemplary SMOOTH intra-prediction mode, sample 1010 can be predicted by averaging the quadratic interpolations along the vertical and horizontal directions. The above non-directional intra-prediction mode implementations are merely non-limiting examples. Other adjacent rows, other non-directional selections of samples, and methods of combining predicted samples to predict specific samples in a prediction block are also considered.

At different coding levels (pictures, slices, blocks, units, etc.), the encoder selects a specific intra-prediction mode from the aforementioned directional or non-directional modes, which can be represented by signals in the bitstream. In some exemplary implementations, eight exemplary nominal directional modes and five non-angular smoothing modes (a total of 13 options) can first be represented by signals. Then, if the mode represented by a signal is one of the eight nominal angular intra-prediction modes, it is further indexed by signal representation to indicate the selected angle increment to the corresponding nominal angle represented by the signal. In some other exemplary implementations, all intra-prediction modes can be indexed together (e.g., 56 directional modes plus 5 non-directional modes to produce 61 intra-prediction modes) for use in signal representation.

In some exemplary implementations, the exemplary 56 or other number of directional intra-prediction modes can be implemented using a unified directional predictor that projects each sample of the block to a reference subsample location and interpolates the reference sample through a 2-tap bilinear filter.

In some implementations, additional filter patterns, called FILTER INTRA modes, can be designed to utilize edge-based references to capture attenuation spatial correlations. For these modes, in addition to samples outside the block, predicted samples within the block can be used as intra-prediction reference samples for some regions within the block. For example, these modes can be predefined and made available for intra-prediction of at least luma blocks (or luma blocks only). A predetermined number (e.g., five) of filter intra-prediction modes can be pre-designed, each represented by a set of n-tap filters (e.g., 7-tap filters) reflecting the correlation between samples in, for example, a 4×2 region and its n nearest neighbors. In other words, the weighting factors of the n-tap filters can be location-dependent. Taking an 8×8 block, 4×2 regions, and 7-tap filters as an example, as shown in Figure 11, the 8×8 block 1102 can be divided into eight 4×2 regions. In Figure 11, these regions are represented by B0, B1, B2, B3, B4, B5, B6, and B7. For each region, its seven nearest neighbors (indicated by R0 to R7 in Figure 11) can be used to predict samples within the current region. For region B0, all nearest neighbors may have been reconstructed. However, for other regions, some neighbors are located in the current block and may not have been reconstructed; in this case, the predicted values of the immediate nearest neighbors are used as a reference. For example, not all nearest neighbors of region B7 as indicated in Figure 11 are reconstructed; instead, the predicted samples of the nearest neighbors are used.

In some implementations of intra-frame prediction, a color component can be predicted using one or more other color components. The color component can be any component in the YCrCb, RGB, XYZ color space, etc. For example, it is possible to predict a chroma component (e.g., a chroma patch) from a luma component (e.g., a luma reference sample), a prediction referred to as luma-derived chroma or CfL. In some exemplary implementations, cross-color prediction is only allowed from luma to chroma. For example, chroma samples in a chroma patch can be modeled as a linear function of simultaneously reconstructed luma samples. CfL prediction can be implemented as follows:

CfL(α)＝α×L ^AC +DC (1)

Where LAC indicates the AC contribution of the luminance component, α indicates the parameters of the linear model, and DC indicates the DC contribution of the chrominance component. For example, the AC component is obtained for each sample of the block, while the DC component is obtained for the entire block. Specifically, the reconstructed luminance samples can be subsampled to the chrominance resolution, and then the average luminance value (DC of luminance) can be subtracted from each luminance value to form the AC contribution of luminance. The AC contribution of luminance is then used in the linear mode of equation (1) to predict the AC value of the chrominance component. In order to estimate or predict the chrominance AC component based on the luminance AC contribution (instead of requiring the decoder to calculate the scaling parameters), an exemplary CfL implementation can determine the parameter α based on the original chrominance samples and represent the parameter α as a signal in the bitstream. This reduces the complexity of the decoder and produces more accurate predictions. As for the DC contribution of the chrominance component, in some exemplary implementations, the intra-frame DC mode within the chrominance component can be used to calculate the DC contribution of the chrominance component.

In some exemplary implementations of reference rows, multi-line intra-prediction can be used. In these implementations, more than one reference row is available for selection during intra-prediction, and the encoder decides and signals which reference row to use to generate the intra-prediction. The reference row index can be signaled before the intra-prediction mode; in the case of a non-zero reference row index, only the most probable prediction mode is allowed. Referring to Figure 15, an example of four reference rows and upper-left reference samples (from reference row 0 to reference row 3) is depicted, where each reference row consists of six segments, namely segments A to F (as indicated in 1502-1512). Furthermore, segments A and F can be filled with the nearest samples from segments B and E, respectively.

Then, the residuals of the intra-frame prediction block or inter-frame prediction block can be transformed, followed by quantization of the transform coefficients. To perform the transform, the intra-frame and inter-frame coded blocks can be further divided into multiple transform blocks (sometimes used interchangeably as "transform units," even though the term "unit" is generally used to refer to a set of three color channels; for example, a "coding unit" may include a luma-coded block and a chroma-coded block). In some implementations, the maximum partitioning depth of the coded block (or prediction block) can be specified (the term "coded block" can be used interchangeably with "coded block"). For example, this partitioning will not exceed two levels. The operation of partitioning the prediction block into transform blocks can be handled differently between intra-frame and inter-frame prediction blocks. However, in some implementations, this partitioning can be similar between intra-frame and inter-frame prediction blocks.

In some exemplary implementations, for intra-coded blocks, transforms can be partitioned such that all transform blocks have the same size, and the transform blocks are encoded in raster scan order. Figure 12 illustrates an example of such transform block partitioning for an intra-coded block. Specifically, Figure 12 shows that coded block 1202 is divided into 16 transform blocks of the same size through intermediate level quadtree splitting 1204, as shown in 1206. The exemplary raster scan order for encoding is indicated by the sequence arrows in Figure 12.

In some exemplary implementations, transform units can be partitioned recursively for inter-frame coded blocks, with a partition depth of up to a predetermined number of levels (e.g., two levels). As shown in Figure 13, the partitioning can be applied to any sub-partition and stop at any level or continue recursively. Specifically, Figure 13 shows an example where block 1302 is partitioned into four quadtree sub-blocks 1304, one of which is further partitioned into four second-level transform blocks, while the partitioning of the other sub-blocks stops after the first level, resulting in a total of seven transform blocks of two different sizes. The exemplary raster scan order for encoding is further illustrated by the sequence arrows in Figure 13. While Figure 13 shows an exemplary implementation of quadtree partitioning for up to two levels of square transform blocks, in some generative implementations, transform partitioning can support 1:1 (square), 1:2/2:1, and 1:4/4:1 transform block shapes and sizes, ranging from 4×4 to 64×64. In some exemplary implementations, if the coded block is less than or equal to 64×64, the transform block partitioning can only be applied to the luma component (in other words, under this condition, the chroma transform block can be the same as the coded block). Otherwise, if the coded block width or height is greater than 64, the luma coded block and the chroma coded block can be implicitly split into multiple min(W, 64)×min(H, 64) transform blocks and min(W, 32)×min(H, 32) transform blocks, respectively.

In some exemplary implementations, as shown in Figure 16, another alternative exemplary scheme for partitioning a coding block or prediction block into transform blocks is provided. As shown in Figure 16, as an alternative to using recursive transform partitioning, a predetermined set of partition types can be applied to the coding block based on its transform type. In the specific example shown in Figure 16, one of six exemplary partition types can be applied to split the coding block into various numbers of transform blocks. This scheme can be applied to either coding blocks or prediction blocks. In this disclosure, the term "partition type" generally refers to the manner in which a block (e.g., a prediction block or coding block) is partitioned, and can refer to a "transform partition type," a "prediction block partition type," or a "coding block partition type." Furthermore, the same concept applies to the description under "transform partition type" and vice versa.

More specifically, as shown in Figure 16, the partitioning scheme of Figure 16 provides up to six partitioning types for any given transform type. In this scheme, a transform type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In one example, the partitioning type assigned to a coding block or prediction block can be determined based on its transform type. A specific partitioning type can correspond to a transform block split size and mode (or partitioning type), as shown by the four partitioning types in Figure 16. The correspondence between the various transform types and the various partitioning types can be predefined. Exemplary correspondences are shown below, where uppercase labels indicate the transform type that can be assigned to a coding block or prediction block based on rate-distortion cost:

• PARTITION_NONE: Allocates a transformation size equal to the block size.

• PARTITION_SPLIT: Allocates a transform size that is half the width of the block size and half the height of the block size.

• PARTITION_HORZ: Allocates a transform size that has the same width as the block size and is half the height of the block size.

• PARTITION_VERT(Partition_VERT): Allocates a transform size that is half the width of the block size and has the same height as the block size.

• PARTITION_HORZ4: Allocates a transform size that has the same width as the block size and is 1/4 of the block size's height.

• PARTITION_VERT4: Allocates a transform size that is 1/4 the width of the block size and has the same height as the block size.

In the examples above, all partitioned transform blocks, as shown in Figure 16, contain a uniform transform size. This is merely an example, not a limitation. In some other implementations, mixed transform block sizes may be used for partitioned transform blocks within a specific partition type (or pattern).

Then, a master transform can be performed on each transform block obtained above. The master transform is essentially moving the residuals in the transform block from the spatial domain to the frequency domain. In some implementations of the actual master transform, in order to support the exemplary extended coding block partitioning described above, various transform sizes (ranging from 4 to 64 points for each of the two dimensions) and transform shapes (squares; rectangles with aspect ratios of 2:1/1:2 and 4:1/1:4) are allowed.

Turning to the actual master transform, in some exemplary implementations, the 2-D transform process may involve the use of a hybrid transform kernel (e.g., a hybrid transform kernel may consist of different 1-D transforms for each dimension of the encoded residual transform block). Exemplary 1-D transform kernels may include, but are not limited to: a) 4-point DCT-2, 8-point DCT-2, 16-point DCT-2, 32-point DCT-2, and 64-point DCT-2; b) 4-point asymmetric DST, 8-point asymmetric DST, 16-point asymmetric DST (DST-4, DST-7) and their inverted versions; c) 4-point identity transform, 8-point identity transform, 16-point identity transform, and 32-point identity transform. The selection of the transform kernel for each dimension may be based on a rate-distortion (RD) criterion. For example, the basis functions for implementable DCT-2 and asymmetric DST are listed in Table 1.

Table 1: Exemplary master transform basis functions (DCT-2, DST-4, and DST-7 for N-point inputs)

In some exemplary implementations, the availability of a hybrid transform kernel for a particular master transform implementation can be based on the transform block size and prediction mode. Exemplary dependencies are listed in Table 2. For chroma components, transform type selection can be performed implicitly. For example, for intra-frame prediction residuals, the transform type can be selected based on the intra-frame prediction mode, as specified in Table 3. For inter-frame prediction residuals, the transform type of the chroma block can be selected based on the transform type selection of the co-located luma block. Therefore, for chroma components, there is no transform type signaling in the bitstream.

Table 2: AV1 Hybrid Transform Kernels and their Availability Based on Prediction Mode and Block Size. Here, → and ↓ indicate the horizontal and vertical dimensions; √ and × indicate kernel availability for that block size and prediction mode.

Table 3: Transform type selection for intra-frame prediction residuals of chroma components

Intra-frame prediction Vertical Transformation Horizontal transformation DC_PRED DCT DCT V_PRED ADST DCT H_PRED DCT ADST D45_PRED DCT DCT D135_PRED ADST ADST D113_PRED ADST DCT D157_PRED DCT ADST D203_PRED DCT ADST D67_PRED ADST DCT SMOOTH_PRED ADST ADST SMOOTH_V_PRED ADST DCT SMOOTH_H_PRED DCT ADST PAETH_PRED ADST ADST

In some implementations, a secondary transformation can be performed on the master transform coefficients. For example, as shown in Figure 14, LFNST (Low-Frequency Inseparable Transform) can be applied between the forward master transform and quantization (at the encoder) and between dequantization and the inverse master transform (at the decoder) to further decorrelate the master transform coefficients. LFNST is called a simplified secondary transformation. Essentially, LFNST takes a portion of the master transform coefficients, such as the low-frequency portion (thus "simplifying" from the complete set of master transform coefficients in the transform block), for a secondary transformation. In the exemplary LFNST, depending on the transform block size, a 4×4 or 8×8 inseparable transformation can be applied. For example, a 4×4 LFNST can be applied to a smaller transform block (e.g., min(width, height) < 8), while an 8×8 LFNST can be applied to a larger transform block (e.g., min(width, height) > 8). For example, if a 4×4 LFNST is performed on an 8×8 transform block, only the low-frequency 4×4 portion of the 8×8 master transform coefficients will undergo a further secondary transformation.

Specifically, as shown in Figure 14, the transform block can be 8×8 (or 16×16). Therefore, the forward principal transform 1402 of the transform block generates an 8×8 (or 16×16) principal transform coefficient matrix 1404, where each square cell represents a 2×2 (or 4×4) portion. For example, the input to the forward LFNST may not be a complete 8×8 (or 16×16) principal transform coefficient. For example, a 4×4 (or 8×8) LFNST can be used for the secondary transform. Therefore, only the 4×4 (or 8×8) low-frequency principal transform coefficients of the principal transform coefficient matrix 1404 can be used as the input to the LFNST, as indicated by the shaded portion (top left) 1406. The secondary transform may not be performed on the remaining portion of the principal transform coefficient matrix. Therefore, after the secondary transform, the portion of the principal transform coefficients affected by the LFNST becomes the secondary transform coefficient, while the remaining portion not affected by the LFNST (e.g., the unshaded portion of matrix 1404) retains the corresponding principal transform coefficients. In some exemplary implementations, the remaining parts that are not subject to the second transformation can all be set to zero coefficients.

The following describes an application example of the inseparable transform used in LFNST. To apply the exemplary 4×4 LFNST, the 4×4 input block X (e.g., representing the 4×4 low-frequency portion of the master transform coefficient block, such as the shaded portion 1406 of the master transform matrix 1404 in Figure 14) can be represented as:

The 2D input matrix can first be linearized or scanned into a vector in an exemplary sequence.

The inseparable transform of the 4×4 LFNST is then computed into a vector of transform coefficients indicating the output, where T is a 16×16 transform matrix. The resulting 16×1 coefficient vector is then scanned in reverse into 4×4 blocks using the scan order of the blocks (e.g., horizontal, vertical, or diagonal). Coefficients with smaller indices can be placed in the 4×4 coefficient blocks along with smaller scan indices. In this way, redundancy in the main transform coefficients X can be further utilized via the second transform T, providing additional compression enhancement.

The exemplary LFNST described above is based on a direct matrix multiplication method to apply the inseparable transformation, enabling LFNST to be implemented in a single pass without requiring multiple iterations. In some further exemplary implementations, the dimension of the inseparable transformation matrix (T) of the exemplary 4×4 LFNST can be further reduced to minimize computational complexity and memory requirements for storing the transformation coefficients. This implementation can be called Simplified Non-Separable Transformation (RST). More specifically, the main idea of RST is to map an N-dimensional vector (in the example above, N is 4×4 = 16, but for an 8×8 block, N can be equal to 64) to an R-dimensional vector in a different space, where N/R (R < N) represents the dimension reduction factor. Therefore, as an alternative to the N×N transformation matrix, the RST matrix becomes an R×N matrix, as follows:

Here, the R rows of the transformation matrix form a reduced R-basis for the N-dimensional space. Therefore, this transformation converts the input vector, or N-dimensional vector, into a reduced R-dimensional output vector. Thus, as shown in Figure 14, the secondary transformation coefficients 1408, transformed from the principal coefficients 1406, are reduced in dimension by a factor of N/R. The three squares surrounding 1408 in Figure 14 can be filled with zeros.

The inverse transform matrix of an RTS can be the transpose of its forward transform. For the exemplary 8×8 LFNST (which is more varied in description than the 4×4 LFNST described above), an exemplary reduction factor of 4 can be applied, thus reducing the 64×64 direct inseparable transform matrix to a 16×64 direct matrix. Furthermore, in some implementations, a portion (but not all) of the input principal coefficients can be linearized into the input vector of the LFNST. For example, only a portion of the exemplary 8×8 input principal transform coefficients can be linearized into the X vector described above. For a particular example, in the four 4×4 quadrants of the 8×8 principal transform coefficient matrix, the lower right quadrant (high-frequency coefficients) can be ignored, and only the other three quadrants are linearized into 48×1 vectors using a predetermined scan order, instead of 64×1 vectors. In such an implementation, the inseparable transform matrix can be further simplified from 16×64 to 16×48.

Therefore, an exemplary simplified 48×16 inverse RST matrix can be used on the decoder side to generate the top-left, top-right, and bottom-left 4×4 quadrants of the 8×8 kernel (master) transform coefficients. Specifically, when a further simplified 16×48 RST matrix (instead of 16×64 RST) is applied to the same transform set configuration, the inseparable quadratic transform can extract 48 vectorized matrix elements as input from the three 4×4 quadrant blocks of the 8×8 master coefficient block (excluding the bottom-right 4×4 block). In this implementation, the removed bottom-right 4×4 master transform coefficients can be ignored in the quadratic transform. This further simplified transform can convert the 48×1 vector into a 16×1 output vector, which is then scanned inversely into a 4×4 matrix to fill 1408 in Figure 14. The three squares of the quadratic transform coefficients around 1408 can be filled with zeros.

By leveraging this dimensionality reduction of RST, the memory usage for storing all LFNST matrices is reduced. For example, in the example above, compared to an implementation without dimensionality reduction, memory usage can be reduced from 10KB to 8KB, with a relatively small performance degradation.

In some implementations, to reduce complexity, LFNST can be further restricted to apply only if all coefficients outside the primary transform coefficient portion to be subjected to LFNST (e.g., outside portion 1406 of 1404 in Figure 14) are insignificant. Therefore, when LFNST is applied, all primary-only transform coefficients (e.g., the unshaded portion of the primary coefficient matrix 1404 in Figure 14) can approach zero. This restriction allows for adjustment of the LFNST index signaling at the last significant position, thus avoiding some additional coefficient scans, which might be necessary to check significant coefficients at a specific position without this restriction. In some implementations, worst-case handling of LFNST (in terms of multiplication per pixel) can restrict the inseparable transforms of 4×4 blocks and 8×8 blocks to 8×16 and 8×48 transforms, respectively. In these cases, when LFNST is applied, the last-significant-can position must be less than 8, while other sizes must be less than 16. For blocks of shape 4×N and N×4 where N>8, the above constraints mean that LFNST is now applied only once to the top-left 4×4 region. Since all primary-only coefficients are zero when LFNST is applied, the number of operations required for the primary transform is reduced in this case. From the encoder's perspective, coefficient quantization is simplified when testing the LFNST transform. For the first 16 coefficients (in scan order), rate-distortion optimized quantization (RDO) must be maximized, forcing the remaining coefficients to zero.

In some exemplary implementations, the available RST kernels can be specified as multiple transform sets, each comprising multiple inseparable transform matrices. For example, a total of four transform sets may exist, each with two inseparable transform matrices (kernels) for use by LFNST. These kernels can be pre-trained offline, thus being data-driven. Offline-trained transform kernels can be stored in memory or hard-coded in the encoding or decoding device for use during the encoding/decoding process. The selection of transform sets during the encoding or decoding process can be determined by the intra-prediction mode. A mapping from the intra-prediction mode to the transform set can be predefined. Examples of such predefined mappings are shown in Table 4. For example, as shown in Table 4, transform set 0 can be selected for the current chroma block if one of the three cross-component linear model (CCLM) modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (i.e., 81 <= predModeIntra <= 83). For each transform set, the selected inseparable quadratic transform candidate can be further specified by an LFNST index explicitly represented by a signal. For example, after the transform coefficients, the index can be represented by a signal once for each intra-frame CU in the bitstream.

Table 4: Transformation Selection Table

IntraPredMode Transform set index IntraPredMode<0 1 0 <= IntraPredMode <= 1 0 2 <= IntraPredMode <= 12 1 13 <= IntraPredMode <= 23 2 24 <= IntraPredMode <= 44 3 45 <= IntraPredMode <= 55 2 56 <= IntraPredMode <= 80 1 81 <= IntraPredMode <= 83 0

Because the application of LFNST is restricted only when all coefficients outside the first coefficient subgroup or portion are invalid in the above exemplary implementation, LFNST index encoding depends on the position of the last significant coefficient. Furthermore, LFNST indices can be context-encoded, but are independent of the intra-prediction mode, and only the first bin can be context-encoded. Additionally, LFNST can be applied to intra-CUs in intra-slices and inter-slices, as well as luma and chroma. If dual-tree is enabled, the LFNST indices for luma and chroma can be represented by signals separately. For inter-slices (with dual-tree disabled), a single LFNST index can be represented by a signal, with a single LFNST index used for both luma and chroma.

In some exemplary implementations, when the Intra-Frame Sub-Partitioning (ISP) mode is selected, LFNST can be disabled and the RST index can be omitted from the signal representation because the performance improvement may be negligible even if RST is applied to every feasible partition block. Furthermore, disabling RST on the residuals of ISP predictions reduces coding complexity. In some further implementations, when the Multivariate Linear Regression Intra-Frame Prediction (MIP) mode is selected, LFNST can also be disabled and the RST index can be omitted from the signal representation.

Considering that large CUs larger than 64×64 (or any other predetermined size representing the maximum transform block size) are implicitly split (e.g., TU tiling) due to existing maximum transform size limitations (e.g., 64×64), LFNST index search can quadruple the data buffer size for a specific number of decoding pipeline stages. Therefore, in some implementations, the maximum allowed size of LFNST can be limited to, for example, 64×64. In some implementations, LFNST is enabled only when DCT2 is the primary transform.

In some other implementations, an intra-frame quadratic transform (IST) is provided to the luma component by defining, for example, 12 quadratic transform sets, where each set contains, for example, 3 kernels. An intra-mode dependent index can be used to select the transform set. Kernel selection within a set can be based on syntax elements represented by signals. IST can be enabled when DCT2 or ADST is used for both the horizontal transform and the vertical master transform. In some implementations, a 4×4 or 8×8 non-separable transform can be selected depending on the block size. A 4×4 IST can be selected if min(tx_width, tx_height) < 8. For larger blocks, an 8×8 IST can be used. Here, tx_width and tx_height correspond to the width and height of the transform block, respectively. The input to the IST can be low-frequency master transform coefficients in a zigzag scan order.

Various transformations during video encoding or decoding (e.g., the master transform of samples in a residual block or the second transformation of a block in the master transform coefficient process) may not be very effective at capturing directional texture patterns (e.g., edges in a 45-degree direction (e.g., directions substantially away from the horizontal or vertical direction)) when using only separable transform schemes. As mentioned above, in some exemplary implementations, one or more non-separable transform designs can be used for the second transform of the master transform coefficients.

Transform block partitioning and the transform types applied to the partitioned transform blocks can be correlated. For example, certain transform types may be better suited to a particular partition type. For instance, the transform partitioning scheme shown in Figure 16 and described above provides non-recursive partition types compared to recursive partitioning (e.g., the partitioning previously described in Figure 13). If all available transform types are allowed for transform blocks partitioned under all available partitioning modes (e.g., the transform partitioning types in Figure 16), the encoder needs to perform optimization in a large parameter space when determining which transform partitioning type to use to obtain transform blocks and which transform type to use for the transform blocks of each partition. In practice, a certain set of transform types is often better suited to a particular type of transform partitioning type compared to other transform types. In the various implementations described below, the interaction between transform partitioning types and transform types can be considered, and this interaction can be used to obtain a scheme that restricts the allowed transform types for a particular partitioning type, and similarly restricts the allowed partitioning types for a particular transform type. Such implementations can reduce the encoder's optimization space when determining the transform partitioning mode and transform type selection for each partitioned transform block, especially when using non-recursive transform partitioning.

These exemplary implementations can be used individually or in any order or in any combination. In the disclosure above and below, the terms "encoded block," "encoded block," etc., can be used to refer to a picture unit that performs a prediction or transformation. An encoded block can be a luminance-encoded block or a chrominance-encoded block. In some cases, an encoded/encoded block can refer to a prediction block. The term "block size" is used to refer to the width or height of the encoded block, or the maximum of the width and height, or the minimum of the width and height, or the area size (width * height), or the aspect ratio (width:height, or height:width).

Multiple candidate master transform types

In one embodiment, a block may have multiple candidate primary transform types. A block may include transform blocks resulting from partitioning. The selection and/or signaling of the primary transform type may be limited to a predetermined set of transform partition types. The predetermined set of transform partition types may be a subset of a larger set of available transform partition types (e.g., the complete set of partition types in Figure 16). In other words, a selection of the primary transform, signaled, is made only if the block's transform partition type belongs to the predetermined set of transform partition types. Otherwise, for example, if the transform partition type is another type, a default transform type may be used instead of selecting and signaling the primary transform type.

In one implementation, the main transform type may include at least one of the following: Discrete Cosine Transform (DCT) type 1 to DCT type 8; Asymmetric Discrete Sine Transform (ADST); Discrete Sine Transform (DST) type 1 to DST type 8; Line Graph Transform (LGT); or Karhunen-Loeve Transform (KLT).

In one implementation, the predetermined transform partition type set includes only PARTITION_NONE from the various transform partition types in Figure 16, i.e., the transform block size is equal to the prediction block (or coding block) size. Therefore, the main transform type can only be selected and/or represented by a signal if the transform type belongs to this predetermined set.

In one implementation, the number of partitions can also be considered to determine the predetermined set of transform partition types. For example, a particular transform partition type may be considered part of the predetermined set of transform partition types only if the number of partitions for that particular transform partition type is less than or equal to a predetermined threshold, and the dominant transform type can be selected and/or represented by a signal for that part. In one implementation, the predetermined threshold can be an integer from 1 to 16.

Multiple candidate quadratic transformation types

In one embodiment, a block may have multiple candidate quadratic transform types. A block may include transform blocks resulting from partitioning. The selection and/or signaling of the quadratic transform type can only be applied to a predetermined set of transform partition types. The predetermined set of transform partition types may be a subset of a larger set of available transform partition types (e.g., the complete set of transform partition types in Figure 16). In other words, a selection of a quadratic transform, signaled, is made only if the block's transform partition type belongs to the predetermined set of transform partition types. Otherwise, for example, if the transform partition type is another type, a default quadratic transform type may be used instead of selecting and signaling the quadratic transform type, or the quadratic transform may not be performed.

In one implementation, the secondary transformation type may include KLT. KLT can be configured to have different kernels.

In one implementation, the predetermined transform partition type set can only include PARTITION_NONE from the various transform partition types in Figure 16, i.e., the transform block size is equal to the prediction block (or coding block) size. Therefore, the secondary transform type can only be selected and/or represented by a signal if the transform type belongs to this predetermined set.

In one implementation, the number of partitions can also be considered to determine the predetermined set of transform partition types. For example, a particular transform partition type may be considered part of the predetermined set of transform partition types only if the number of partitions for that particular transform partition type is less than or equal to a predetermined threshold. This part can then be selected and/or represented by a signal to indicate the quadratic transform type. In one implementation, the predetermined threshold can be an integer from 1 to 16.

In one implementation, the selection and/or signaling of the secondary transform type can be based on a combination of transform partition type and primary transform type. This combination may include transform partition types from a predetermined set of transform partition types and primary transform types from a predetermined set of transform types. For example, selecting and/or signaling the secondary transform type may only be necessary if the transform partition type is PARTITION_NONE and the primary transform type used by the block is DCT or ADST. Otherwise, for example, if the transform partition type is another type, a default secondary transform type may be used instead of selecting and signaling the secondary transform type, or the secondary transform may not be performed.

Transformation related signaling

This disclosure presents various signaling mechanisms aimed at improving signaling efficiency and takes into account the order of related syntax elements/parameters.

In one embodiment, transform partition type information can be represented by a signal before the primary/secondary transform type selection information. Signaling the primary/secondary transform type selection is only necessary if the transform partition belongs to a predetermined transform partition type set (e.g., PARTITION_NONE). Otherwise, if the transform partition does not belong to the predetermined transform partition type set, signaling the primary/secondary transform type selection may not be necessary. Instead, the primary/secondary transform type can be derived as a predetermined default transform type.

In one embodiment, the primary/secondary transform type selection information can be represented by a signal before the transform partition type information. In this case, the selection of the transform partition type and/or its representation by a signal may depend on the primary/secondary transform type selection information.

In one implementation, it may be necessary to represent transform type information with signals only if the primary transform type belongs to a predetermined transform type set. Otherwise, it may not be necessary to represent transform type information with signals. For example, the predetermined transform type set may include, but is not limited to, DCT type 1 to DCT type 8, ADST, DST type 1 to DST type 8, LGT, and KLT.

In one implementation, it may be necessary to represent transform partition type information with signals only if the quadratic transform type belongs to a predetermined transform type set. As an example, the predetermined transform type set may include, but is not limited to, a specific KLT with a kernel associated with a predetermined KLT index. Otherwise, if the quadratic transform type does not belong to the predetermined transform type set, it may not be necessary to represent transform partition type information with signals. Instead, the transform partition type information can be derived as a predetermined default transform partition type (e.g., PARTITION_NONE).

Figure 17 illustrates an exemplary method 1700 for decoding video data. Method 1700 may include some or all of the following steps: step 1710, receiving an encoded video stream of data blocks; step 1720, extracting a transform partition type associated with the data blocks from the encoded video stream; step 1730, in response to a transform partition type belonging to a subset of a predetermined set of transform partition types, each transform partition type in the predetermined set specifying a splitting pattern for dividing the data blocks into transform blocks: extracting a transform type associated with the transform blocks split from the data blocks, the transform type being represented by a signal in the encoded video stream, wherein the transform type belongs to a first predetermined set of transform types; and performing an inverse transform on the transform blocks according to the transform type.

In embodiments of this disclosure, any steps and/or operations may be combined or arranged in any number or order as needed. Two or more steps and/or operations may be performed in parallel.

The embodiments of this disclosure can be used individually or in any combination in any order. Furthermore, each method (or embodiment), encoder, and decoder can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments of this disclosure can be applied to luma blocks or chroma blocks.

The above-described techniques can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media. For example, Figure 18 illustrates a computer system (1800) suitable for implementing certain embodiments of the disclosed subject matter.

Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or through interpretation, microcode execution, etc.

The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

The components of the computer system (1800) shown in Figure 18 are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependencies or requirements relating to any one or a combination of components shown in the exemplary embodiments of the computer system (1800).

The computer system (1800) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, captured images from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

Human-machine interface input devices may include one or more of the following (only one of each is shown): keyboard (1801), mouse (1802), touchpad (1803), touch screen (1810), data glove (not shown), joystick (1805), microphone (1806), scanner (1807), camera (1808).

The computer system (1800) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell/taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touchscreen (1810), data gloves (not shown), or joysticks (1805), but may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers (1809), headphones (not depicted)), and visual output devices (e.g., screens (1810) including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touchscreen input functionality, each screen may or may not have tactile feedback functionality, some of which are capable of outputting two-dimensional or more three-dimensional visual outputs through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

The computer system (1800) may also include human-accessible storage devices and their associated media, such as optical media including CD/DVD ROM/RW (1820) having media such as CD/DVD (1821), finger drives (1822), removable hard disk drives or solid-state drives (1823), conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM/ASIC/PLD such as security dongles (not depicted), etc.

Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

The computer system (1800) may also include an interface (1854) leading to one or more communication networks (1855). The network may be, for example, a wireless network, a wired network, or an optical network. The network may further be a local area network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a latency-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CAN buses, etc. Some networks typically require an external network interface adapter (e.g., a USB port of the computer system (1800)) attached to some general-purpose data port or peripheral bus (1849); other network interfaces are typically integrated into the core of the computer system (1800) by being attached to a system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system). The computer system (1800) can use any of these networks to communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., a CANBus connected to certain CANBus devices), or bidirectional, such as connecting to other computer systems using a local area network (LAN) or wide area network (WAN) digital network. As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

The aforementioned human-machine interface device, human-machine accessible storage device, and network interface can be attached to the kernel (1840) of the computer system (1800).

The core (1840) may include one or more central processing units (CPU) (1841), graphics processing units (GPUs) (1842), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (1843), hardware accelerators (1844) for certain tasks, graphics adapters (1850), etc. These devices, along with read-only memory (ROM) (1845), random access memory (1846), and internal mass storage (1847) such as internal non-user-accessible hard disk drives, SSDs, etc., may be connected via the system bus (1848). In some computer systems, the system bus (1848) may be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (1848) or attached via a peripheral bus (1849). In one example, a screen (1810) may be connected to a graphics adapter (1850). Peripheral bus architectures include PCI, USB, etc.

The CPU (1841), GPU (1842), FPGA (1843), and accelerator (1844) can execute certain instructions that can be combined to form the aforementioned computer code. This computer code can be stored in ROM (1845) or RAM (1846). Transient data can also be stored in RAM (1846), while permanent data can be stored, for example, in internal mass storage (1847). Fast storage and retrieval to any storage device can be achieved by using a cache, which can be closely associated with one or more CPUs (1841), GPUs (1842), mass storage (1847), ROM (1845), RAM (1846), etc.

Computer-readable media may have computer code thereon that performs various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

As a non-limiting example, a computer system having an architecture (1800), particularly a kernel (1840), can be made functional by one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and memory of certain non-transitory kernels (1840), such as internal kernel mass storage (1847) or ROM (1845). Software implementing the various embodiments of this disclosure can be stored in such devices and executed by the kernel (1840). Depending on specific needs, the computer-readable media may include one or more storage devices or chips. The software can cause the kernel (1840), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1846) and modifying such data structures according to software-defined processes. In addition to or as an alternative, the computer system may be made functional by hard-wired or otherwise embodied logic in circuitry (e.g., an accelerator (1844)), which may replace or operate with the software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

While several exemplary embodiments have been described in this disclosure, modifications, substitutions, and various equivalent alternatives fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and thus fall within its spirit and scope.

Appendix A: Acronyms

JEM: Joint Exploration Model

VVC: Next-Generation Video Coding

BMS: Benchmark Set

MV: Motion Vector

HEVC: High-Efficiency Video Coding

SEI: Auxiliary Enhancement Information

VUI: Video Availability Information

GOP: Image Group

TU: Transformation Unit

PU: Prediction Unit

CTU: Coding Tree Unit

CTB: Coded Tree Block

PB: Prediction Block

HRD: Hypothetical Reference Decoder

SNR: Signal-to-noise ratio

CPU: Central Processing Unit

GPU: Graphics Processing Unit

CRT: Cathode Ray Tube

LCD: Liquid Crystal Display

OLED: Organic Light Emitting Diode

CD: CD-ROM

DVD: Digital Video Disc

ROM: Read-Only Memory

RAM: Random Access Memory

ASIC: Application-Specific Integrated Circuit

PLD: Programmable Logic Device

LAN: Local Area Network

GSM: Global System for Mobile Communications

LTE: Long Term Evolution

CANBus: Controller Area Network Bus

USB: Universal Serial Bus

PCI: Interconnect Peripheral Components

FPGA: Field Programmable Gate Domain

SSD: Solid State Drive

IC: Integrated Circuit

HDR: High Dynamic Range

SDR: Standard Dynamic Range

JVET: Joint Video Exploration Team

MPM: Most Likely Pattern

WAIP: Wide-angle Intra-frame Prediction

CU: Encoding Unit

PU: Prediction Unit

TU: Transformation Unit

CTU: Coding Tree Unit

PDPC: Location-Related Prediction Combination

ISP: Intra-Frame Sub-Partition

SPS: Sequence Parameter Settings

PPS: Image Parameter Set

APS: Adaptive Parameter Set

VPS: Video Parameter Set

DPS: Decoding Parameter Set

ALF: Adaptive Loop Filter

SAO: Sampling Adaptive Offset

CC-ALF: Cross-component adaptive loop filter

CDEF: Constrained Directional Enhancement Filter

CCSO: Cross-component sampling offset

LSO: Local Sampling Offset

LR: Loop Recovery Filter

AV1: AOMedia Video 1

AV2: AOMedia Video 2

Claims (11)

1. A method for decoding video data, characterized in that the method comprises: Receive the encoded video stream of the data blocks; Extract the transform partition type associated with the data block from the encoded video stream; and In response to the fact that the transformation partition type belongs to a subset of a predetermined set of transformation partition types, the subset of the predetermined set of transformation partition types includes only PARTITION_NONE which does not perform transformation block partitioning, and each transformation partition type in the predetermined set of transformation partition types specifies a splitting mode for dividing the data block into transformation blocks: Extract the transform type associated with the transform block split from the data block, the transform type being represented by a signal in the encoded video stream, wherein the transform type belongs to a first predetermined transform type set, the transform being a quadratic transform, and the first predetermined transform type set including the Caronan-Loy transform (KLT); and In response to the transformation partition type not belonging to a subset of the predetermined transformation partition type set: The transformation type associated with the transformation block is determined as a predetermined default transformation type; According to the transformation type, perform an inverse transformation on the transformation block. 2. The method according to claim 1, wherein when the transformation is a primary transformation, the first predetermined transformation type set includes: Discrete Cosine Transform (DCT) types 1 to 8; Asymmetric Discrete Sine Transform (ADST); Discrete Sine Transform (DST) types 1 to 8; Line graph transformation LGT; and Karonan-Louis Transform (KLT). 3. The method according to claim 1, characterized in that the method further comprises: The number of transformation partitions associated with each transformation partition type in a subset of the predetermined transformation partition type set is less than or equal to a predetermined threshold. 4. The method according to claim 3, wherein the predetermined threshold comprises integers from 1 to 16, including 1 and 16. 5. The method according to claim 1, wherein the transformation type further indicates that the type of the main transformation associated with the transformation block belongs to a second predetermined transformation type set. 6. The method of claim 5, wherein the second predetermined transformation type set includes DCT and ADST. 7. An apparatus for decoding video data, comprising a memory and a processor, the memory storing instructions that cause the processor to perform the method according to any one of claims 1 to 6. 8. A method for encoding video data, characterized in that the method comprises: Obtain the video bitstream of the data block; Extract the transform partition type associated with the data block from the video bitstream; and In response to the fact that the transformation partition type belongs to a subset of a predetermined set of transformation partition types, the subset of the predetermined set of transformation partition types includes only PARTITION_NONE which does not perform transformation block partitioning, and each transformation partition type in the predetermined set of transformation partition types specifies a splitting mode for dividing the data block into transformation blocks: Extract the transform type associated with the transform block split from the data block, the transform type being represented by a signal in the video bitstream, wherein the transform type belongs to a first predetermined transform type set, the transform is a quadratic transform, and the first predetermined transform type set includes the Caronan-Loy transform (KLT); and In response to the transformation partition type not belonging to a subset of the predetermined transformation partition type set: The transformation type associated with the transformation block is determined as a predetermined default transformation type; According to the transformation type, perform an inverse transformation on the transformation block. 9. An apparatus for encoding video data, comprising a memory and a processor, the memory storing instructions that cause the processor to perform the method according to claim 8. 10. A non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform the method according to any one of claims 1 to 6, and when executed by a computer for video encoding, cause the computer to perform the method according to claim 8. 11. A method for processing a video stream, characterized in that the video stream is decoded based on the method of any one of claims 1 to 6, or generated according to the method of claim 8.