RU2788835C1 - Method and apparatus for encoding video data - Google Patents

Abstract

FIELD: video data decoding.

SUBSTANCE: invention relates to methods for decoding video data in a video decoder. The method receives a first High-Level Syntax Element (HLS) indicating whether Set Selection (SS) is enabled or disabled for an intra-coding block, wherein Transform Type information indicating the type of transformation is obtained at Explicit SS; receiving a second HLS element indicating whether explicit SS is enabled or disabled for an outer coding block, wherein the first and second HLS elements control the same set of coding blocks that include said intra coding block and said outer coding block, and enabling implicit SS for an intra-coded block, when the first HLS element indicates that explicit SS is disabled for the intra-block, regardless of whether the second HLS element indicates that explicit SS is enabled or disabled for the outer-encoding block, wherein the conversion type information is not signaled by the implicit SS.

EFFECT: reducing the load on the processor by optimizing data processing for signaling the type of conversion.

22 cl, 55 dwg

Description

INCLUSION BY LINK

[0001] The present application claims priority under U.S. Patent Application No. 16/878,390 "Method and Apparatus for Encoding Video Data", filed May 19, 2020, which claimed priority under U.S. Provisional Patent Application No. 62/860,149 "Control through high-level syntax when implicit conversion selection is enabled", filed June 11, 2019. The contents of prior applications are incorporated herein by reference in their entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[0002] This application describes embodiments of the invention generally related to video coding.

BACKGROUND OF THE INVENTION

[0003] The description of the prior art is given here to represent the overall context of the invention. The work of the authors of the invention, to the extent that it is described in this section, as well as aspects of the description that cannot be qualified as state of the art at the time of filing, neither expressly nor impliedly recognized as the state of the art for the present invention.

[0004] Encoding and decoding of video data may be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a sequence of images each having a spatial size of, for example, 1920×1080 luma samples and their associated chrominance samples. The image sequence may have a fixed or variable image rate (informally also referred to as frame rate), such as 60 images per second, or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (resolution of 1920x1080 luma samples at 60Hz frame rate) requires about 1.5Gbps of bandwidth. An hour of such video requires more than 600 GB of storage.

[0005] One goal of encoding and decoding video data may be to reduce redundancy in the input video signal by compressing. Compression can help to alleviate the aforementioned bandwidth or storage requirements, in some cases by two orders of magnitude or more. You can use both lossless and lossy compression, as well as a combination of both. Lossless compression refers to techniques for reconstructing an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the discrepancy between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used for video data. The amount of distortion allowed depends on the application; for example, users of some custom streaming applications may tolerate higher distortion than users of broadcast television applications. The achievable compression ratio may reflect that a higher allowed/allowable distortion may result in a higher compression ratio.

[0006] The video encoder and decoder may use techniques based on several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0007] Video codec technologies may include techniques known as intra-coding. With internal coding, sample values are presented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, the image is spatially divided into blocks of samples. When all blocks of samples are encoded in the intra coding mode, this picture is an intra coding picture. Intra-coded pictures and their derivatives, such as independent decoder update pictures, can be used to reset the decoder state and therefore can be used as the first picture in the encoded video bitstream and video session, or as a still picture. Intra-coded block samples may be transformed and transform coefficients may be quantized prior to entropy coding. Intra-prediction may be a technique that minimizes sample values in a pre-transform region. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are required for a given quantization step size to represent a block after entropy coding.

[0008] Conventional intra coding, such as that known from MPEG-2 generation coding technologies, does not use intra prediction. However, some new video data compression technologies include techniques that use prediction from, for example, ambient sample data and/or metadata obtained during encoding/decoding of spatially adjacent and preceding data blocks in decoding order. Such methods are hereinafter referred to as "internal prediction" methods. It should be noted that, in at least some cases, intra prediction uses only the reference data from the current image being reconstructed, and not from the reference images.

[0009] There are many different forms of intra prediction. When more than one of these methods can be used in a given video coding technology, the applied method can be used in intra prediction mode. In some cases, modes may have sub-modes and/or parameters that may be encoded individually or included in the mode codeword. The codeword used for a given mode/submode/parameter combination can contribute to the increase in coding efficiency due to intra prediction, as can the entropy coding technology used to convert the codewords to a bitstream.

[0010] The intra-prediction mode was introduced in H.264, enhanced in H.265, and further enhanced in new coding technologies such as joint exploration model (JEM), versatile video coding (VVC), and set of standards (BMS, benchmark set). The predictor block can be formed using the values of adjacent samples belonging to already available samples. The values of neighboring samples are copied into the prediction block according to the direction. The reference to the direction to be used may be encoded in the bitstream, or may itself be predicted.

[0011] Motion compensation may be a lossy compression technique and may refer to techniques in which a block of sample data from a previously reconstructed image or a portion thereof (reference image) after a spatial shift in the direction indicated by a motion vector (MV, motion vector) is used to predict a newly reconstructed image or part of an image. In some cases, the reference image may be the same as the image currently being reconstructed. The motion vectors may have two dimensions X and Υ or three dimensions, with the third dimension indicating the reference image used (the latter, indirectly, may be a time dimension).

[0012] In some video data compression techniques, a motion vector applicable to a certain region of sample data may be predicted based on other motion vectors, such as those associated with and preceding another region of sample data adjacent to the reconstructed region. movements in decoding order. This can significantly reduce the amount of data required for motion vector encoding, which eliminates redundancy and increases the compression ratio. Motion vector prediction can work efficiently, for example, because when encoding the input video signal received from the camera (known as natural video), there is a statistical probability that areas larger than the area to which one motion vector is applied move in a similar direction and therefore, in some cases, can be predicted using a similar motion vector obtained from the motion vectors of the neighboring area. This results in the motion vector found for a given region being similar or identical to the motion vector predicted from the surrounding motion vectors and, in turn, can be represented after entropy encoding with fewer bits than in the case of direct encoding. motion vector coding. In some cases, motion vector prediction may be an example of lossless compression of a signal (namely, motion vectors) derived from the original signal (namely, a stream of samples). In other cases, motion vector prediction itself may be lossy, for example, due to rounding errors when computing a predictor based on multiple surrounding motion vectors.

[0013] Various motion vector prediction mechanisms are described in the H.265/HEVC standard (ITU-T Rec. H.265, High Efficiency Video Coding, December 2016). Of the many motion vector prediction mechanisms that the H.265 standard offers, a method hereinafter referred to as "spatial fusion" is described.

[0014] As shown in FIG. 1, the current block (101) contains the samples found by the encoder during the motion search process to be predictable based on a previous block of the same size that was spatially shifted. Instead of directly encoding the motion vector, this motion vector can be derived from the metadata associated with one or more reference pictures, for example, based on the most recent (in decoding order) reference picture using the motion vector associated with any of the five surrounding samples, denoted A0, A1 and B0, B1, B2 (from 102 to 106, respectively). In H.265, motion vector prediction may use predictors based on the same reference picture as used for the neighboring block.

SUMMARY OF THE INVENTION

[0015] Aspects of the invention provide a first method for decoding video data in a video decoder. The method may include receiving a first high-level syntax (HLS) element indicating whether multiple transform selection (MTS) is enabled or disabled for the intra-coded block, and receiving a second HLS element indicating whether , explicit MTS is enabled or disabled for the outer-encoded block. The first and second HLS elements manage the same set of coding blocks, which include said intra-coding block and said outer-coding block. Implicit MTS may be enabled for an intra block when the first HLS element indicates that explicit MTS is disabled for the intra block and the second HLS element indicates that explicit MTS is enabled for the outer block.

[0016] An embodiment of the method may also include applying an implicit MTS to the intra block. The type of transformation for processing the intra block may be determined according to the size of the intra block. In various examples, the first or second element of the HLS may be one of the following elements: a video parameter set (VPS) syntax element, a sequence parameter set (SPS) syntax element, a video parameter set (PPS) syntax element, picture parameter set), a slice header syntax element, a tile header syntax element, or a tile group header syntax element. In an embodiment of the invention, an intra sub-partitioning block is not encoded in intra sub-partitioning (ISP) mode. In an embodiment of the invention, the method may also include receiving a third HLS element indicating that MTS is enabled for each of the intra and outer coding blocks.

[0017] Aspects of the invention provide a second method for decoding video data in a video decoder. The second method may include receiving a first HLS element indicating whether MTS is enabled or disabled for the intra-coded block, and receiving a second HLS element indicating whether a non-separable secondary transform (NSST, non-separable secondary transform) or internal is disabled or enabled. matrix-based intra prediction (MIP) for an intra-coded block. Implicit MTS may be enabled for an intra block when the first HLS element indicates that explicit MTS is disabled for the intra block and the second HLS element indicates that NSST or MIP is disabled for the intra block.

[0018] The invention also provides a third method for decoding video data in a video decoder. The third method may include receiving an intra-coded block associated with a first block level syntax element indicating whether MTS is applied and a second block level syntax element indicating whether NSST is applied. Implicit MTS may be included for an intra-coded block when the first block-level syntax element indicates that MTS is not applied and the second block-level syntax element indicates that NSST is not applied.

[0019] Note that although the present description relates to NSST, the proposed methods and systems can be applied to variants of NSST, such as reduced size transform (RST, reduced size transform) and low-frequency non-separable secondary transform (LFNST, low-frequency non-separable secondary transform). Thus, NSST, RST and/or LFNST can be used interchangeably in the present specification.

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, principles and various advantages of the present invention will become apparent from the following detailed description and the accompanying drawings.

[0021] FIG. 1 is a schematic illustration of the current block and its surrounding candidates for spatial merging in one embodiment of the invention.

[0022] FIG. 2 is a schematic illustration of a simplified block diagram of a communications system (200) in accordance with an embodiment of the invention.

[0023] FIG. 3 is a schematic illustration of a simplified block diagram of a communications system (300) in accordance with an embodiment of the invention.

[0024] FIG. 4 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment of the invention.

[0025] FIG. 5 is a schematic illustration of a simplified block diagram of an encoder in accordance with an embodiment of the invention.

[0026] FIG. 6 shows a block diagram of an encoder in accordance with another embodiment of the invention.

[0027] FIG. 7 shows a block diagram of a decoder in accordance with another embodiment of the invention.

[0028] FIG. 8A-8D show examples of a transform kernel matrix for 4-point, 8-point, 16-point, and 32-point DCT-2 transforms, respectively, according to an embodiment of the invention.

[0029] FIG. 9A-9D show types, sizes, and positions of sub-blocks supported in a sub-block transform (SBT) according to an embodiment of the invention.

[0030] FIG. 10 shows the number of subdivisions versus block size in the subdivision intra coding (ISP) mode according to an embodiment of the invention.

[0031] FIG. 11 shows an example in which a block is divided into two subsections in the ISP coding mode.

[0032] FIG. 12 shows an example in which a block is divided into four subsections in the ISP coding mode.

[0033] FIG. 13A-13E show a 64x64 transform kernel matrix for a 64-point DCT-2 transform according to an embodiment of the invention.

[0034] Fig. 14 shows basis transform functions for DST/DCT transforms according to an embodiment of the invention.

[0035] FIG. 15 shows a table illustrating the correspondence between the value of mts_idx and the corresponding horizontal or vertical transformations according to an embodiment of the invention.

[0036] FIG. 16A-16D show DST-7 type transform kernel matrices according to an embodiment of the invention.

[0037] FIG. 17A-17D show DCT-8 type transformation kernel matrices according to an embodiment of the invention.

[0038] FIG. 18 shows an example of controlling the use of Transform Set Selection (MTS) with Sequence Parameter Set (SPS) syntax elements.

[0039] FIG. 19 shows a mapping table between intra prediction modes and transform sets in accordance with an embodiment of the invention.

[0040] FIG. 20-21 show two alternative transform encoding processes (2000) and (2100) for RST8x8 using 16x64 transform kernels and 16x48 transform kernels, respectively, according to an embodiment of the invention.

[0041] FIG. 22 shows an example of a CU level syntax table (2200) where the lfnst_idx syntax element indicating the low frequency non-shared second transform (LFNST) kernel selection is signaled at the end of the CU level syntax.

[0042] FIG. 23 shows a reduced transform process (2301) and a reduced inverse transform process (2302) according to an embodiment of the invention.

[0043] FIG. 24A shows all of the top left 8×8 coefficients (shaded sub-blocks) of the residual block (2410) used as input to calculate the secondary transform in RST8×8.

[0044] FIG. 24B shows the coefficients of the top three left 4x4 subblocks (shaded subblocks) of the residual block (2410) used as input to calculate the secondary transform to RST8x8.

[0045] FIG. 25 shows a table for selecting a transform set based on the intra prediction mode according to an embodiment of the invention.

[0046] FIG. 26 shows an example of a matrix-based intra prediction (MIP) mode process (2600).

[0047] FIG. 27 shows a CU layer syntax table where flags signaling matrix-based intra prediction (MIP) modes are shown in box (2701) according to an embodiment of the invention.

[0048] FIG. 28A-28B collectively show text (2800) specifying a transform encoding process for performing implicit or explicit transform selection for the current block based on corresponding syntax elements obtained from the bitstream.

[0049] FIG. 29 shows modifications (2900) of text (2800) that correspond to an implicit conversion inclusion scheme where implicit conversion for intra-residual blocks and explicit conversion for outer-encoded residual blocks can be combined.

[0050] FIG. 30 shows text modifications (3000) (2800) that correspond to a scenario in which implicit conversion is enabled when non-shared secondary transform (NSST) is disabled.

[0051] FIG. 31 shows text modifications (3100) (2800) that correspond to a scenario in which implicit conversion is enabled when MIP is disabled.

[0052] FIG. 32 shows text modifications (3200) (2800) that correspond to a scenario in which both MTS and NSST are not applied to the current block.

[0053] FIG. 33 shows modifications (3300) of text (2800) that correspond to a scenario in which none of the MTS, NSST, or MIP is applied to the current block.

[0054] FIG. 34-36 show flowcharts of transform coding methods (3400), (3500), and (3600) according to some embodiments of the invention.

[0055] FIG. 37 is a schematic illustration of a computer system in accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS

[0056] 1. Video encoder and decoder

[0057] FIG. 2 shows a simplified block diagram of a communication system (200) according to an embodiment of the present invention. The communication system (200) includes a plurality of terminal devices that can communicate with each other, for example, via a network (250). For example, the communication system (200) includes a first pair of terminals (210) and (220) interconnected via a network (250). In the example shown in FIG. 2, the first pair of terminals (210) and (220) performs a unidirectional data transfer. For example, a terminal device (210) may encode video data (eg, a video stream captured by a terminal device (210)) for transmission to another terminal device (220) via a network (250). The encoded video data may be transmitted as one or more encoded video bit streams. The terminal device (220) may receive the encoded video data from the network (250), decode the encoded video data to recover the video images and display them according to the recovered video data. One-way communication may be inherent in media service applications and the like.

[0058] In another example, the communication system (200) includes a second pair of terminals (230) and (240) that carry out bi-directional transmission of encoded video data occurring, for example, during a video conference. For bidirectional data transmission, for example, each terminal of terminals (230) and (240) may encode video data (eg, a video stream captured by the terminal) for transmission to the other terminal of terminals (230) and (240) via network (250). Each terminal of terminals (230) and (240) may also receive encoded video data transmitted by another terminal from terminals (230) and (240) and may decode the encoded video data to reconstruct video images and display video images on an available display device according to recovered video data.

[0059] In the example shown in FIG. 2, terminal devices (210), (220), (230), and (240) can be illustrated as servers, personal computers, and smartphones, but this does not limit the principles of the present invention. Embodiments of the present invention find application in laptop computers, tablet computers, media players and/or dedicated video conferencing equipment. Network (250) represents any number of networks that carry encoded video data between terminals (210), (220), (230) and (240), including, for example, wired and/or wireless communication networks. The communication network (250) allows data to be exchanged in a circuit-switched and/or packet-switched mode. Examples of networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For the purposes of this discussion, the architecture and topology of the network (250) may not be relevant to the present invention unless specifically noted below.

[0060] FIG. 3 shows the placement of a video encoder and a video decoder in a streaming environment as an example of the application of the present invention. The present invention is equally applicable to other video processing applications including, for example, videoconferencing, digital television, storage of compressed video on digital media such as CD, DVD, memory card, and the like.

[0061] The streaming system may include a capture subsystem (313), which may include a source (301) of video data, such as a digital camera, creating, for example, an uncompressed video stream (302). For example, the video stream (302) includes samples taken by a digital camera. The video stream (302), shown in bold to emphasize the large amount of data compared to the encoded video data (304) (or encoded video data bitstreams), may be processed by an electronic device (320) that includes a video encoder (303) connected to the source ( 301) video data. The video encoder (303) may include hardware, software, or a combination thereof to provide or implement aspects of the present invention, as described in more detail below. The encoded video data (304) (or encoded video bitstream (304)), shown as a thin line to emphasize the smaller amount of data compared to the video stream (302), may be stored on the streaming server (305) for future use. One or more streaming client subsystems, such as client subsystems (306) and (308) in FIG. 3 may access the streaming server (305) to retrieve copies (307) and (309) of the encoded video data (304). The client subsystem (306) may contain a video decoder (310), for example, in an electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and creates an outgoing video stream (311) that can be rendered on a display (312) (eg, a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (304), (307), and (309) (eg, video bitstreams) may be encoded according to one or another video coding/compression standard. Examples of these standards include ITU-T Rec. H.265. For example, a video coding standard under the official name "Versatile Video Coding" (VVC) is being developed. The present invention can be used in the context of VVC.

[0062] Note that the electronic devices (320) and (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown) and the electronic device (330) may also include a video encoder (not shown).

[0063] FIG. 4 is a block diagram of a video decoder (410) according to an embodiment of the present invention. The video decoder (410) may be included in the electronic device (430). The electronic device (430) may include a receiver (431) (eg, receiver circuits). Video decoder (410) may be used instead of video decoder (310) in the example shown in FIG. 3.

[0064] The receiver (431) may receive one or more encoded video sequences for decoding by video decoder (410), in the same or another embodiment of the invention, one encoded video sequence at a time, with the decoding of each encoded video sequence independent of other encoded video sequences. The encoded video sequence may be received from a channel (401), which may be a hardware/software link to a storage device where the encoded video data is stored. The receiver (431) may receive encoded video data with other data, such as encoded audio data and/or auxiliary data streams, which may be redirected to their respective utilizing entities (not shown). The receiver (431) may separate the encoded video sequence from other data. To combat network jitter, a buffer memory (415) can be connected between the receiver (431) and the entropy decoder/analyzer (420) (hereinafter "analyzer (420)"). In some applications, the buffer memory (415) is included in the video decoder (410). In others, it may not be part of the video decoder (410) (not shown). In other embodiments, there may be a buffer memory outside the video decoder (410) (not shown), for example, to combat network jitter, in addition to another buffer memory (415) within the video decoder (410), for example, to control playback time. When the receiver (431) receives data from a storage/redirector with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (415) may not be needed or may be small. For use in best effort packet networks, such as the Internet, buffer memory (415) may be required, may be relatively large, may have a predominantly adaptive size, and may at least partially be implemented in an operating system or similar elements (not shown) outside video decoder (410).

[0065] The video decoder (410) may include an analyzer (420) for reconstructing the symbols (421) from the encoded video sequence. The categories of these symbols include information used to control the operation of the video decoder (410) and possibly information to control a rendering device, such as a rendering device (412) (e.g., a display screen) that is not an integral part of the electronic device (430) but may be connected to an electronic device (430) as shown in FIG. 4. The control information for the rendering device(s) may be Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The analyzer (420) may analyze/entropy decode the received encoded video sequence. Video sequence coding may be in accordance with a video coding technology or standard, and may follow various principles, including variable run length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The analyzer (420) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOP, Group of Pictures), images, tiles, slices, macroblocks, coding packages (CU, Coding Unit), blocks, transformation packages (TU, Transform Unit), prediction packages (PU, Prediction Unit) and etc. The analyzer (420) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0066] The analyzer (420) may perform an entropy decoding/analysis operation of the video sequence received from the buffer memory (415) to create symbols (421).

[0067] Several different modules may be used for symbol reconstruction (421) depending on the type of encoded video image or parts thereof (eg, intra- and outer-coded images, intra- and outer-coded blocks) and other factors. Which modules are used and how can be determined by the subgroup management information obtained from the encoded video sequence by the analyzer (420). The flow of such subgroup control information between the analyzer (420) and the modules is not shown hereinafter for simplicity.

[0068] In addition to the previously mentioned functional blocks, the video decoder (410) can be fundamentally divided into several functional modules, as described below. In practical implementation under commercial constraints, many of these modules interact closely with each other and can, at least partially, be built into one another. However, for the purpose of describing the present invention, a fundamental division into the following functional modules is appropriate.

[0069] The first module is the scaling/inverse transform module (451). The scaling/inverse transform module (451) receives the quantized transform coefficient as well as control information including the transform used, block size, quantization factor, quantization scaling matrices, etc. as character(s) (421) from the analyzer (420). The scaling/inverse transform module (451) may output blocks containing sample values that can be input to the aggregator (455).

[0070] In some cases, the output samples of the scaling/inverse transform block (451) may refer to an intra-coded block; that is, to a block that does not use predicted information from previously reconstructed images, but may use predicted information from previously reconstructed portions of the current image. Such predicted information may be provided by the intra image prediction module (452). In some cases, the intra image prediction module (452) generates a block of the same size and shape as the block to be reconstructed using previously reconstructed environment information retrieved from the current image buffer (458). The current picture buffer (458) buffers, for example, a partially reconstructed current picture and/or a fully reconstructed current picture. The aggregator (455) in some cases adds, for each sample, the prediction information generated by the intra prediction module (452) to the output sample information provided by the scaling/inverse transform module (451).

[0071] In other cases, the output samples of the scaling/inverse transform module (451) may refer to an inter-coded block, possibly with motion compensation. In such a case, the motion compensation prediction module (453) may access the reference picture memory (457) to retrieve samples used for prediction. After motion compensation has been applied to the extracted samples according to the symbols (421) associated with the block, these samples can be added by the aggregator (455) to the output signal of the scaling/inverse transform module (451) (in this case called residual samples or residual signal) to generation of output sample information. The addresses in the reference picture memory (457) from where the motion compensation prediction module (453) retrieves samples for prediction may be controlled by the motion vectors available to the motion compensation prediction module (453) in the form of symbols (421), which may have, for example , Χ, Υ components, and reference image components. Motion compensation may also include interpolation of sample values retrieved from reference picture memory (457) when accurate sub-sample motion vectors, motion vector prediction mechanisms, etc. are used.

[0072] Various loop filtering methods can be applied to the aggregator (455) output samples in the loop filter module (456). Video data compression technologies may include loop filter technologies driven by parameters included in the encoded video sequence (also referred to as the coded video bitstream) and available to the loop filter module (456) as symbols (421) from the analyzer (420), but may also be responsive to meta-information obtained in the course of decoding the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as to respond to previously reconstructed and contour filtered sample values.

[0073] The loop filter module (456) may provide a stream of samples that may be fed to the renderer (412) as well as stored in the reference picture memory (457) for use in future inter-picture prediction.

[0074] Some encoded pictures, when fully reconstructed, can be used as reference pictures for future prediction. For example, when the encoded picture corresponding to the current picture is fully reconstructed and identified as a reference picture (e.g., by the analyzer (420)), the current picture buffer (458) may become part of the reference picture memory (457) and a fresh current picture buffer may be re-allocated. until the reconstruction of the next encoded picture starts.

[0075] The video decoder (410) may perform decoding operations according to a predetermined video data compression technology, such as ITU-T Rec. H.265. The encoded video sequence may follow the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence may follow both the syntax of the video compression technology or standard and the profiles documented in the video compression technology or standard. In particular, a profile may select certain tools as tools available for use only under that profile, from among all the tools available in a video data compression technology or standard. It may also be necessary for consistency that the complexity of the encoded video sequence remains within the limits set by the level of technology or video compression standard. In some cases, the levels limit the maximum image size, the maximum frame rate, the maximum sampling rate for reconstruction (measured in megasamples per second, for example), the maximum reference image size, and so on. The limits set by the levels, in some cases, may be further limited by the specifications of the hypothetical reference decoder (HRD, Hypothetical Reference Decoder) and metadata for managing the HRD buffer signaled in the encoded video sequence.

[0076] According to an embodiment of the invention, the receiver (431) may receive additional (redundant) data with encoded video data. Additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (410) to properly decode the data and/or more accurately reconstruct the original video data. The side data may be, for example, temporal enhancement levels, spatial enhancement levels or signal noise ratio (SNR) enhancement levels, redundant slices, redundant images, forward error correction codes, and so on.

[0077] FIG. 5 is a block diagram of a video encoder (503) according to an embodiment of the present invention. The video encoder (503) is included in the electronic device (520). The electronic device (520) includes a transmitter (540) (eg, a transmission circuit). Video encoder (503) may be used instead of video encoder (303) in the example shown in FIG. 3.

[0078] The video encoder (503) may receive video data samples from a video data source (501) (which is not part of the electronic device (520) in the example shown in FIG. 5) that may capture the video image(s) to be encoding with a video encoder (503). In another example, the source (501) of video data is included in the electronic device (520).

[0079] The video data source (501) may provide the source video sequence to be encoded by the video encoder (503) in the form of a digital video data sample stream of any suitable bit depth (eg: 8 bits, 10 bits, 12 bits,...), any color space ( eg BT.601 Y CrCB, RGB,...) and any suitable sampling structure (eg Υ CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the source (501) of video data may be a storage device where previously prepared video is stored. In a videoconferencing system, the video data source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual images that give a sense of movement when viewed in sequence. The images themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sample structure used, color space, and so on. It is not difficult for a person skilled in the art to understand the relationship between pixels and samples. The following description is devoted to readings.

[0080] According to an embodiment of the invention, the video encoder (503) can encode and compress images of the original video sequence into the encoded video sequence (543) in real time or subject to any other time constraints imposed by the application. Establishing a proper coding rate is one of the functions of the controller (550). In some embodiments of the invention, the controller (550) controls other functional modules, as described below, and is operatively connected to other functional modules. The connection is not shown for simplicity. The parameters set by the controller (550) may include parameters related to frequency adjustment (picture skip, quantizer, lambda value used in rate-distortion optimization methods, etc.), image size, organization of groups of images (GOP, group of pictures), maximum motion vector search area, etc. The controller (550) may have other suitable functions related to the video encoder (503) optimized for a particular system design.

[0081] In some embodiments of the invention, the video encoder (503) is configured to operate in a coding loop. As a very simplified description, for example, an encoding loop may include a source encoder (530) (for example, responsible for creating symbols, for example, a symbol stream based on the input image to be encoded and the reference image(s)) and (local ) decoder (533) built into the video encoder (503). Decoder

(533) reconstructs the symbols to create sample data in a similar way as a (remote) decoder would (because any compression between the symbols and the encoded video bitstream is lossless in the video compression technologies discussed here). The reconstructed sample stream (sample data) is entered into the reference picture memory (534). Since decoding the symbol stream produces bit-accurate results that are independent of the position of the (local or remote) decoder, the content of the reference picture memory (534) will also be the same bit-accuracy for the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" as reference picture samples exactly the same sample values as the decoder would "see" if using prediction during decoding. This fundamental principle of reference picture synchronism (and eventually drift if synchronism cannot be maintained, for example due to channel errors) is also used in some related fields of technology.

[0082] A "local" decoder (533) may operate in the same manner as a "remote" decoder, such as the video decoder (410) detailed above with reference to FIG. 4. However, again, referring to FIG. 4, since the symbols are available and the encoding/decoding of the symbols into the encoded video sequence by the entropy encoder (545) and analyzer (420) can be lossless, the entropy decoding parts of the video decoder (410) including the buffer memory (415) and analyzer (420) can be not fully implemented in the local decoder (533).

[0083] It can be concluded that any decoding technology present in a decoder, with the exception of analysis/entropy decoding, must also necessarily be present in essentially identical functional form in the corresponding encoder. For this reason, the present invention is focused on the operation of the decoder. The description of the encoding technologies can be abbreviated as they are the inverse of the decoding technologies described in detail. Only in some areas a more detailed description is required, which is given below.

[0084] According to some examples, in operation, the source encoder (530) may perform motion-compensated predictive coding, in which an input picture is predictively encoded based on one or more previously encoded pictures from a video sequence, referred to as "reference pictures". Thus, the coding engine (532) encodes the differences between the pixel blocks of the input image and the pixel blocks of the reference image(s) that can be selected as the prediction reference(s) to the input image.

[0085] The local video decoder (533) may decode the encoded video data of pictures that can be specified as reference pictures based on the symbols generated by the source encoder (530). The operations of the encoding engine (532) may be predominantly lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 5), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (533) duplicates the decoding processes that may be performed by the video decoder on the reference pictures and may store the reconstructed reference pictures in the reference picture cache (534). Thus, the video encoder (503) can locally store copies of the reconstructed reference pictures having the same content as the reconstructed reference pictures that would be received by the remote video decoder (in the absence of transmission errors).

[0086] The predictor (535) may perform prediction searches for the encoding engine (532). Thus, for a new picture to be encoded, the predictor (535) may search the reference picture memory (534) for sample data (as pixel reference block candidates) or specific metadata such as reference picture motion vectors, block shapes, etc. .d., which can serve as a proper reference for predicting new images. The predictor (535) may operate on a "sample blocks χ pixel blocks" basis to find appropriate references for prediction. In some cases, according to the search results obtained by the predictor (535), the input picture may have prediction references extracted from the plurality of reference pictures stored in the reference picture memory (534).

[0087] The controller (550) can control the encoding operations of the source encoder (530), including, for example, setting the parameters as well as the subgroup parameters used to encode the video data.

[0088] The output signal of all the above functional modules can be entropy encoded in an entropy encoder (545). An entropy encoder (545) translates symbols generated by various functional units into an encoded video sequence by lossless symbol compression according to, for example, Huffman coding, variable run length coding, arithmetic coding, and so on.

[0089] The transmitter (540) may buffer the encoded video sequence(s) created by the entropy encoder (545) in preparation for transmission over a communication channel (560), which may be a hardware/software memory link where the encoded video data is stored. The transmitter (540) may combine the encoded video data from the video encoder (503) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (sources not shown).

[0090] The controller (550) may control the operation of the video encoder (503). During encoding, the controller (550) may assign to each encoded picture an encoded picture type, which may determine the encoding methods applicable to the corresponding picture. For example, images can often be assigned the following image types.

[0091] An intra-coded picture (I-picture) that can be encoded and decoded without using any other picture in the sequence as a source for prediction. Some video codecs allow different types of intra-coded images, including, for example, images in the Independent Decoder Refresh (IDR) format. The person skilled in the art will be aware of the varieties of I-images and their respective applications and features.

[0092] A predictive picture (P-picture) that can be encoded and decoded using intra prediction or inter prediction using at most one motion vector and a reference index to predict sample values of each block.

[0093] A bi-directional predictive picture (B-picture) that can be encoded and decoded using intra prediction or inter prediction using no more than two motion vectors and reference indices to predict sample values of each block. Similarly, multi-prediction pictures may use more than two reference pictures and associated metadata to reconstruct a single block.

[0094] Source images typically allow for spatial division into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coding on a block-by-block basis. The blocks may be predictively encoded with reference to other (previously coded) blocks defined by the coding assignment applied to the respective images of those blocks. For example, blocks of I-pictures may be encoded without prediction or predictively with reference to previously encoded blocks of the same picture (with spatial prediction or intra prediction). P-picture pixel blocks may be predictively encoded by spatial prediction or temporal prediction with reference to one previously encoded reference picture. Blocks of B-pictures may be predictively encoded by spatial prediction or temporal prediction with reference to one or two previously encoded reference pictures.

[0095] The video encoder (503) may perform encoding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In operation, the video encoder (503) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data may follow the syntax given by the technology used or the video coding standard.

[0096] According to an embodiment of the invention, the transmitter (540) may transmit additional data with encoded video data. The source encoder (530) may include such data as part of the encoded video sequence. The additional data may include temporal/spatial/SNR enhancement levels, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, and so on.

[0097] The video may be captured as a plurality of source images (video images) in time sequence. Intra-image prediction (often abbreviated as intra-image prediction) uses the spatial correlation within a given image, while inter-image prediction uses (temporal or otherwise) correlation between images. For example, a specific image to be encoded/decoded, which is called the current image, is divided into blocks. When a block in the current picture is similar to a reference block in a previously encoded and still buffered reference picture in a video, the block in the current picture may be encoded with a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension identifying the reference picture if multiple reference pictures are used.

[0098] In some embodiments of the invention, a dual prediction technique in inter-picture prediction may be used. According to the double prediction method, two reference pictures are used, for example, a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. The block may be predicted with a combination of the first reference block and the second reference block.

[0099] Additionally, the fusion mode method can be used in inter-picture prediction to improve coding efficiency.

[0100] According to some embodiments of the invention, predictions such as inter image prediction and intra image prediction are performed block by block. For example, according to the HEVC standard, an image in a video sequence is split into coding tree unit (CTU) packets for compression, CTU packets in an image have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU packet includes three coding tree blocks (CTBs), namely one luma CTB and two chrominance CTBs. Each CTU packet may be recursively divided on a quadtree basis into one or more coding packets (CUs). For example, a 64x64 pixel CTU packet may be divided into one 64x64 pixel CU packet or 4 32x32 pixel CU packets or 16 16x16 pixel CU packets. For example, each CU burst is parsed to determine the prediction type for that CU burst, eg, inter prediction type or intra prediction type. The CU burst is divided into one or more prediction bursts (PUs) depending on temporal and/or spatial predictability. In general, each PU packet includes a luminance prediction block (PB) and two chrominance prediction blocks PB. According to an embodiment of the invention, the encoding (encoding/decoding) prediction operation is performed in a prediction block unit. As an example of a prediction block, a luminance prediction block includes a matrix of values (e.g., luminance values) for pixels, e.g., 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, and the like.

[0101] In FIG. 6 shows a diagram of a video encoder (603) according to another embodiment of the invention. The video encoder (603) is configured to receive a processing block (eg, a prediction block) of sample values in the current video image in the video sequence and to encode the processing block into an encoded image that forms part of the encoded video sequence. For example, video encoder (603) is used instead of video encoder (303) in the example shown in FIG. 3.

[0102] In the HEVC example, video encoder (603) receives a matrix of sample values for a processing block, such as an 8×8 sample prediction block, or the like. The video encoder (603) determines whether the processing block is best encoded in intra prediction mode, inter prediction mode, or dual prediction mode using, for example, rate-distortion optimization. When the processing unit is to be coded in intra prediction mode, the video encoder (603) may use an intra prediction technique to encode the processing unit into an encoded picture, and when the processing unit is to be coded in inter prediction mode or dual prediction mode, the video encoder (603) may use an inter prediction technique. prediction or double prediction, respectively, to encode the processing block into the encoded picture. In some video coding technologies, the merge mode may be a sub-mode of inter-picture prediction where a motion vector is derived from one or more motion vector predictors without involving the encoded motion vector component outside of the predictors. In some other video coding technologies, there may be a motion vector component applicable to a given block. By way of example, the video encoder (603) includes other components, such as a mode determination module (not shown) for determining the mode of the processing units.

[0103] In the example shown in FIG. 6, the video encoder (603) comprises an outer encoder (630), an intra encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), a common controller (621), and an entropy encoder (625), connected to each other as shown in Fig. 6.

[0104] The outer encoding encoder (630) is configured to receive samples of the current block (e.g., processing block), compare the block with one or more reference blocks in reference pictures (e.g., blocks in previous pictures and later pictures), create outer prediction (eg, description of redundant information according to the inter-coding method, motion vectors, merge mode information); and calculation of inter-prediction results (eg, prediction block) based on the inter-prediction information using any suitable method. In some examples, reference pictures are decoded reference pictures that are decoded based on encoded video information.

[0105] The encoder (622) of the intra coding is configured to receive samples of the current block (for example, a processing block), in some cases comparing the block with blocks previously encoded in the same image, generating quantized coefficients after the transformation, and in some cases also information of the intra prediction (eg, intra prediction direction information according to one or more intra coding methods). By way of example, the intra encoder (622) also calculates intra prediction results (eg, prediction block) based on intra prediction information and reference blocks in the same picture.

[0106] The common controller (621) is configured to determine common control data and control other components of the video encoder (603) based on the common control data. For example, the common controller (621) determines the block mode and outputs a control signal to the switch (626) based on the mode. For example, when the mode is an intra coding mode, the common controller (621) controls the switch (626) to select the result of the intra coding mode to be used by the residual calculator (623) and controls the entropy encoder (625) to select the intra prediction information and include the intra prediction information in bitstream, and when the mode is an inter-coding mode, the common controller (621) controls a switch (626) to select an inter-prediction result for use by the residual calculator (623) and controls an entropy encoder (625) to select inter-prediction information and turn on inter-prediction information to the bitstream.

[0107] The residual calculator (623) is configured to calculate the difference (residual data) between the received block and the prediction results selected from the intra encoder (622) or the outer encoder (630). The residual encoder (624) is configured to operate on the residual data to encode the residual data to generate transform coefficients. For example, the residual encoder (624) is configured to transform residual data from the spatial domain to the frequency domain and generate transform coefficients. Then, the transform coefficients are subjected to quantization processing to obtain quantized transform coefficients. In various embodiments of the invention, the video encoder (603) also contains a residual decoder (628). The residual decoder (628) is configured to perform inverse transformation and generate decoded residual data. The decoded residual data may be appropriately used by the intra encoder (622) and the outer encoder (630). For example, an inter coding encoder (630) may generate decoded blocks based on the decoded residual data and inter prediction information, and an intra coding encoder (622) may generate decoded blocks based on the decoded residual data and intra prediction information. The decoded blocks are properly processed to generate decoded pictures, and the decoded pictures can be buffered in a memory circuit (not shown) and used as reference pictures in some examples.

[0108] The entropy encoder (625) is configured to format the bitstream to include a coded block. The entropy encoder (625) is configured to include various information according to a suitable standard, such as the HEVC standard. For example, the entropy encoder (625) is configured to include common control data, selected prediction information (eg, intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. It should be noted that, according to the invention, when encoding a block in the merge sub-mode for either of the inter-coding mode and the dual prediction mode, there is no residual information.

[0109] In FIG. 7 shows a diagram of a video decoder (710) according to another embodiment of the invention. The video decoder (710) is configured to receive encoded pictures that are part of the encoded video sequence and decode the encoded pictures to generate reconstructed pictures. For example, video decoder (710) is used instead of video decoder (310) in the example shown in FIG. 3.

[0110] In the example shown in FIG. 7, the video decoder (710) comprises an entropy decoder (771), an outer decoder (780), a residual decoder (773), a reconstruction unit (774), and an inner decoder (772) connected to each other as shown in FIG. 7.

[0111] The entropy decoder (771) may be configured to reconstruct, from the encoded image, some symbols that represent the syntax elements that make up the encoded image. Such symbols may include, for example, block coding mode (eg, intra coding mode, inter coding mode, dual prediction mode, the latter two in a merge submode or other submode), prediction information (eg, intra prediction information or inter prediction information), which can identify a certain sample or metadata used for prediction by the intra decoder (772) or the outer decode decoder (780), respectively, residual information in the form of, for example, quantized transform coefficients and the like. For example, when the prediction mode is an inter- or dual-prediction mode, the inter-prediction information is supplied to the inter-decoder (780), and when the prediction type is the intra-prediction type, the intra-prediction information is supplied to the intra-decoder (772). The residual information may be inverse quantized and fed to the residual decoder (773).

[0112] The inter-decoding decoder (780) is configured to receive inter-prediction information and generate inter-prediction results based on the inter-prediction information.

[0113] The intra decoding decoder (772) is configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0114] The residual decoder (773) is configured to perform inverse quantization to extract the dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require some control information (to include a Quantizer Parameter (QP)), and this information may be provided by the entropy decoder (771) (the data path is not shown since this may only be small volume control information) .

[0115] The reconstruction module (774) is configured to combine, in the spatial domain, the residual at the output of the residual decoder (773) and prediction results (at the output of the inter or intra prediction modules, depending on the situation) to form a reconstructed block that may be included into the reconstructed image, which, in turn, can be part of the reconstructed video. Note that other suitable operations, such as a deblocking operation and the like, may be performed to improve the visual quality.

[0116] It should be noted that video encoders (303), (503) and (603) and video decoders (310), (410) and (710) may be implemented using any suitable technology. According to an embodiment of the invention, video encoders (303), (503) and (603) and video decoders (310), (410) and (710) may be implemented using one or more integrated circuits. In another embodiment of the invention, video encoders (303), (503) and (503) and video decoders (310), (410) and (710) may be implemented using one or more processors that execute program instructions.

[0117] II. Transform Encoding Methods and Related Methods

[0118] 1. DCT-2 Primary Transform Examples

[0119] In some embodiments, 4-point, 8-point, 16-point, and 32-point DCT-2 transforms are used as primary transforms. Fig. 8A-8D show the kernel matrices of the 4-point, 8-point, 16-point, and 32-point DCT-2 transforms, respectively. The elements of these transformation kernel matrices can be represented using 8-bit integers, and thus these transformation kernel matrices are referred to as 8-bit transformation kernels. As shown, the transformation kernel matrix of the smaller DCT-2 transformation is part of the larger DCT-2 transformation.

[0120] DCT-2 transformation kernel matrices exhibit symmetry/antisymmetry characteristics. Accordingly, a so-called "partial butterfly" implementation can be supported to reduce the number of operations (multiplications, additions/subtractions, shifts). Using the partial butterfly implementation, identical matrix multiplication results can be obtained compared to when the partial butterfly implementation is not used.

[0121] 2. Subblock Transform Coding Examples

[0122] 2.1 Sub-block transform (SBT, sub-block transform)

[0123] In some embodiments of the invention, a subblock transform (SBT), also called a spatially varying transform (SVT), is used. The SBT transform is applied to inter prediction residuals in some embodiments of the invention. For example, a coding block may be divided into sub-blocks, and only a portion of the sub-blocks is processed in the remainder block. For the rest of the sub-blocks, a zero remainder is assumed. Therefore, the residual block is smaller than the coding block, and the transform size in SBT is smaller than the coding block size. For an area that is not covered by a remainder block, no transformation processing is performed.

[0124] FIG. 9A-9D show types of sub-blocks (SVT-H, SVT-V) (e.g. split vertically or horizontally), sizes and positions (e.g. left half, left quarter, right half, right quarter, top half, top quarter, bottom half , bottom quarter) that are supported in SBT. The shaded areas labeled "A" are residual blocks to be encoded with a transform, and other regions are assumed to be zero-residual without transformation.

[0125] 2.2 Intra-Subdivision Coding (ISP) Mode

[0126] In some embodiments of the invention, a subdivision intra coding (ISP) mode is used. In the ISP coding mode, an intra luminance prediction block may be divided vertically or horizontally into 2 or 4 sub-partitions. The number of subsections may depend on the block size. Fig. 10 shows the number of subsections depending on the block size. Fig. 11 shows a scenario in which a block is divided into two subsections. In FIG. 12 shows a scenario where the block is divided into four subsections. In the example, all subsections satisfy the condition of having at least 16 counts. In the example, ISP does not apply to chrominance components.

[0127] In the example, for each of the subsections received from the coding block, a residual signal is generated by entropy decoding the respective coefficients sent from the encoder, and then dequantizing and inverse transforming them. Then, the first of the sub-partitions is predicted by intra prediction to generate a prediction signal. The prediction signal is added to the corresponding first subsection residual signal to obtain the corresponding reconstructed samples. Thereafter, the reconstructed sample values of the first sub-section may be available to form a prediction of the second of the sub-sections. This process may be repeated from subsection to subsection until all subsections from the coding block have been reconstructed. In one example, the same intra-coding mode is used for all subsections.

[0128] In an embodiment of the invention, the ISP encoding mode is only tested with intra encoding modes that are part of the most probable mode (MPM) list. Accordingly, if ISP is used for the block, then it can be concluded that the MPM flag is equal to one. In addition, when ISP is used for a certain block, the corresponding MPM list will be changed to exclude the DC mode and give priority to horizontal intra coding modes for horizontal ISP division and vertical intra coding modes for vertical division.

[0129] In the ISP encoding mode, each sub-section can be considered as a transform sub-packet since the transformation and reconstruction are performed separately for each sub-section.

[0130] 3. Transform Coding with DCT-2 Extended Transforms and Transform Set Selection (MTS)

[0131] In some embodiments, when both the height and width of a coding block are less than or equal to 64 samples, the transform size is always the same as the coding block size. When the height or width of the coding block is greater than 64 samples, when performing transformation or intra prediction, the coding block is further divided into several sub-blocks, the width and height of each sub-block is less than or equal to 64, and transformation processing is performed for each sub-block.

[0132] 3.1 Transform Coding with DCT-2 Extended Transforms

[0133] In some embodiments, in addition to the 4-point, 8-point, 16-point, and 32-point DCT-2 transforms described above, 2-point and 64-point DCT-2 transforms may be used. Fig. 13A-13E show a 64x64 transform kernel matrix for a 64-point DCT-2 transform.

[0134] 3.2 Explicitly signaled transformation

[0135] In some embodiments of the invention, in addition to encoding with DCT-2 and 4x4 DST-7 transforms, Transform Set Selection (MTS) (also known as enhanced multiple transform (EMT) or adaptive a set of transformations (AMT, adaptive multiple transform)) for coding the rest of both blocks with external coding and blocks with internal coding. MTS uses a variety of selected transforms from the Discrete Cosine Transform (DCT) / Discrete Sine Transform (DST) families, in addition to the DCT-2 and 4×4 DST-7 transforms. The selection may be made at the encoder and explicitly signaled from the encoder to the decoder. For example, the selected transforms include DST-7 or DCT-8 transforms. Fig. 14 shows basic transform functions for DST/DCT. In some embodiments, the DST/DCT transform kernel matrices used in MTS use an 8-bit representation.

[0136] In some embodiments, the MTS may be applied to coding blocks with width and height less than or equal to 32 samples. Whether or not to apply MTS can be controlled by a flag denoted by mtsflag. For example, when the mtsflag flag is 0, only the DCT-2 transform is applied to encode the residual block. When the mts_flag is 1, indicating that MTS is applied, the selected transforms may be used. For example, an index denoted by mts_idx can also be signaled using 2 bins to specify the horizontal and vertical transformations to be used.

[0137] FIG. 15 shows a table (1500) illustrating the correspondence between the mtsidx value and horizontal or vertical transformations. Line (1501) with mts_idx set to -1 corresponds to a scenario where mts_flag is 0 (indicating that MTS is not applied) and DCT-2 is used. Lines (1502)-(1505) with mts idx having a value of 0, 1, 2, or 3 correspond to a scenario in which the mts_flag is 1 (indicating the use of MTS). In the two right columns of the table (1500), 0 represents the DCT-2 transformation type, 1 represents the DST-7 transformation type, and 2 represents the DCT-8 transformation type.

[0138] FIG. 16A-16D show DST-7 type transformation kernel matrices. In FIG. 17A-17D show DCT-8 type transformation kernel matrices.

[0139] In some embodiments of the invention, MTS can be enabled or disabled using high-level syntax elements (HLS). Each of the HLS elements may be a Video Parameter Set (VPS) syntax element, a Sequence Parameter Set (SPS) syntax element, a Picture Parameter Set (PPS) syntax element, a slice header syntax element, a tile header syntax element, or a tile group header syntax element, and etc. Fig. 18 shows an example of MTS control using SPS syntax elements. As shown, the SPS syntax element sps_mts_enabled_flag can be signaled to indicate whether MTS is enabled for a video sequence. When MTS is enabled, the two syntax elements sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag may be signaled to indicate whether MTS is enabled for encoding blocks with intra or inter prediction, respectively.

[0140] In an embodiment of the invention, an implicit MTS is applied in case the aforementioned signaling-based MTS (referred to as explicit MTS) is not used. With implicit MTS, transform selection can be made according to the block width and height rather than signaling. For example, with implicit MTS, a DST-7 transform may be selected for the shorter side of the transform block, and a DCT-2 transform may be selected for the longer side of the transform block.

[0141] 3.3 Scenarios where implicit transform selection is applied

[0142] In various embodiments of the invention, for certain scenarios, DST-7 and/or DCT-8 transformations can be used without explicit signaling. For example, DST-7 and/or DCT-8 transforms may be used implicitly based on information available to both the encoder and the corresponding decoder. These scenarios include:

[0143] 3.3.1 Intra-Subdivision Coding (ISP)

[0144] For a residual block encoded with the ISP mode, the horizontal transform is selected as DST-7 as long as the width of the residual block is greater than or equal to 4 and less than or equal to 16, and the vertical transform is selected as DST-7 as long as the height of the residual block is greater than or is 4 and less than or equal to 16.

[0145] 3.3.2 Subblock Transform (SBT)

[0146] In the SBT mode, for a transform subpacket located in the left half (or quarter) or right half (or quarter) of the current CU burst, the horizontal transform may be a DCT-8 transform or a DST-7 transform, respectively. Otherwise (the transform subpacket has the same width as the current CU packet), a DCT-2 transform may be used. For a transform subpacket located in the upper half (or quarter) or lower half (or quarter) of the current CU burst, the vertical transform may be a DCT-8 transform or a DST-7 transform, respectively. Otherwise (the transform subpacket has the same height as the current CU burst), a DCT-2 transform may be used.

[0147] 3.3.3 MTS disabled by HLS elements

[0148] For example, when sps_mts_enabled_flag is signaled as true, but both sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag are signaled as false, for intra prediction residuals, the horizontal transform may be selected as DST-7 as long as the corresponding block width is greater than or equal to 4 and is less than or equal to 16, and the vertical transformation can be chosen as DST-7 as long as the corresponding block height is greater than or equal to 4 and less than or equal to 16.

[0149] 4. Non-Separable Secondary Transformation (NSST)

[0150] 4.1 Initial NSST Scheme

[0151] In some embodiments of the invention, a mode-dependent non-separable secondary transform (NSST) can be applied between direct kernel transform and quantization (in the encoder) and between dequantization and inverse kernel transform (in the corresponding decoder). To keep complexity low, in some embodiments of the invention, NSST may only be applied to the low frequency coefficients after the primary transform. If both the width (W) and the height (H) of the transform coefficient block are greater than or equal to 8, then an 8×8 non-separable secondary transform may be applied to the upper left region of the 8×8 transform coefficient block. Otherwise, if either W or Η of the transform coefficient block is 4, a 4x4 non-separable secondary transform may be applied, and a 4x4 non-separable transform is performed in the upper left region min(8, W) × min(8, Η) of the transform coefficient block . The above transform selection rule applies to both luminance and chrominance components.

[0152] The implementation of the matrix multiplication nonseparable transform is described below using the 4×4 input block as an example. To apply a non-separable transformation, a 4x4 input block X

represented as a vector

[0153] The non-separable transform is computed as

указывает вектор коэффициентов преобразования, а Τ представляет собой матрицу 16×16 преобразования Т. Вектор

коэффициентов 16×1 впоследствии реорганизуется в блок 4×4 с использованием порядка сканирования для этого блока (горизонтальный, вертикальный или диагональный). Коэффициенты с меньшим индексом будут размещены с меньшим индексом сканирования в блоке коэффициентов 4×4. В примере преобразование HyGT (hypercube-givens) с реализацией бабочки используется вместо умножения матриц для уменьшения сложности неразделяемого преобразования.where

specifies a vector of transform coefficients, and Τ is a 16×16 matrix of T's transform. The vector

16×1 coefficients is subsequently reorganized into a 4×4 block using the scan order for that block (horizontal, vertical, or diagonal). Coefficients with a lower index will be placed with a lower scan index in a 4x4 coefficient block. In the example, a HyGT transformation (hypercube-givens) with a butterfly implementation is used instead of matrix multiplication to reduce the complexity of a non-separable transformation.

[0154] In the NSST example, there can be a total of 35x3 non-shared secondary transforms for both 4x4 and 8x8 block sizes, where 35 is the number of transform sets each corresponding to an intra prediction mode, and 3 is the number of NSST candidates for each intra prediction mode. The mapping from the intra prediction mode to the transform set is specified in the table (1900) shown in FIG. 19. For example, the set of transforms applied to the luminance/chrominance transform coefficients may be determined by the respective intra luminance/chrominance prediction modes according to Table (1900). For intra prediction modes greater than 34 (eg, diagonal prediction direction), the block of transform coefficients is transposed before/after the secondary transform at the encoder/decoder.

[0155] For each set of transforms, the selected non-shared secondary transform candidate is also determined by an explicitly signaled CU layer NSST index. The index is signaled in the bitstream once for each inner CU after the transform coefficients using truncated unary binarization. The truncated value is 2 in case of planar or DC mode and 3 in case of angular intra prediction mode. This NSST index is signaled only when there is more than one non-zero coefficient in the CU. If the value is not signaled, it defaults to zero. A null value for this syntax element indicates that no secondary transform is applied to the current CU, and values of 1-3 indicate which secondary transform from the set should be applied.

[0156] The NSST transform may not be applied to a block encoded in the transform skipping mode. When the NSST index is signaled for a CU and is non-zero, the NSST is not used for a component block that is encoded in the CU's Transform Skip Mode. When a CU with all component blocks is encoded in the transform skip mode or the number of non-zero coefficients of the non-skipping mode CB blocks is less than 2, the NSST index is not signaled to the CU.

[0157] 4.2 Reduced Size Transform (RST)

[0158] In some embodiments of the invention, a variant of NSST is used, called the reduced size transform (RST) or low-frequency non-separable secondary transform (LFNST, low-frequency non-separable secondary transform). The RST transformation uses a null transformation scheme. For entropy coding of the NSST index, it is checked whether the intra prediction mode is planar or DC.

[0159] In the example, 4 transform sets are applied, and each transform set includes three RST transform kernels. RST transform kernels can be 16x48 (or 16x64) (applicable for a transform coefficient block with a height and width greater than or equal to 8) or 16x16 (applicable for a transform coefficient block with a height or width equal to 4) . For convenience, a 16x48 (or 16x64) transform is referred to as RST8x8 and a 16x16 transform is referred to as RST4x4.

[0160] For RST8x8, two alternative transform processes (2000) and (2100) using transform kernels 16x64 and transform kernels 16x48 are shown in FIG. 20 and FIG. 21, respectively. The process that uses 16×48 transform kernels is adopted in VVC Draft 5 (VVC Draft 5).

[0161] In progress (2000) in FIG. 20, for example, at the encoder side, a direct primary transform (2010) may be performed first on the residual block, followed by a direct secondary transform (2012) of the coefficients generated from the direct primary transform (2010). In the forward secondary transform (2012), the 64 coefficients of the 4×4 A/B/C/D subblocks in the upper left corner of the coefficient block (2013) are represented as a vector of length 64 and multiplied by a 16×64 transformation matrix according to equation (3 ), resulting in a vector of length 16. The elements of the vector of length 16 again fill the upper left sub-block A of the 4×4 block of coefficients (2013). The coefficients in sub-blocks B/C/D may take zero values. The resulting coefficients after the direct secondary transform 2012 are then quantized in step (2014) and entropy encoded to generate coded bits in the bitstream (2016).

[0162] The encoded bits may be received at the decoder side and entropy decoded followed by dequantization (2024) to form a block (2023) of coefficients. An inverse secondary transform (2022) may be performed on the 16 coefficients in the top left 4×4 subblock E to obtain 64 coefficients that are again filled in the 4×4 B/F/G/H subblocks. Thereafter, the coefficients in the block (2023) after the inverse secondary transform (2022) may be processed with the inverse primary transform (2020) to obtain a reconstructed residual block.

[0163] The process (2100) in the example of FIG. 21 is similar to the process (2000), except that fewer, namely 48, coefficients are processed during the direct secondary transform (2012). In particular, the 48 coefficients in the A/B/C subblocks are processed with a smaller 16x48 transform matrix. The use of a smaller transformation matrix can reduce the size of the memory for storing the transformation matrix and thus the computational complexity.

[0164] FIG. 22 shows an example of a CU level syntax table (2200) where the lfnst_idx syntax element indicating the LFNST core selection is signaled at the end of the CU level syntax.

[0165] 4.3 RST Calculation Examples

[0166] The main idea of the reduced transformation (RT, Reduced Transform) is to map an N-dimensional vector to an R-dimensional vector of another space, where R/N (R<N) is the reduction factor.

[0167] The RST matrix is the R×N matrix shown below:

where the R transformation rows are the R basis elements of an N-dimensional space. The inverse transformation matrix for RT is the transpose of the forward transformation matrix.

[0168] FIG. 23 shows a shortened transform process (2301) and a shortened inverse transform process (2302). The matrix Τ is an RST transformation matrix having the size R×N, and the matrix T ^t is the transposed matrix T having the size N×R.

[0169] The RST8x8 conversion uses a reduction factor of 4 (size 1/4). For example, instead of a size of 64x64, which is the usual size of an 8x8 non-separable transform matrix, a 16x64 direct matrix is used. A 64x16 inverse RST is used at the decoder side to generate the (primary) transform kernel coefficients in the upper left 8x8 regions. The RST8x8 forward transform uses 16x64 matrices (or 8x64 for an 8x8 block) so that only the top left 4x4 region within a given 8x8 region generates non-zero coefficients. In other words, if the RST transform is applied, then the 8×8 region, except for the top left 4×4 region, will have only zero coefficients. For the RST4x4 transformation, 16x16 direct matrix multiplication (or 8x16 for a 4x4 block) can be applied.

[0170] In addition, for the RST8x8 transform to further reduce the size of the transform matrix, instead of all the top left 8x8 coefficients (shaded sub-blocks in FIG. 24A) of the residual block (2410) as input to calculate the secondary transform, the coefficients The top left 4×4 subblocks (shaded subblocks in FIG. 24B) of the residual block (2410) are used as input to calculate the secondary transform.

[0171] In an example, the inverse RST is applied when the following two conditions are satisfied: (i) the corresponding block size is greater than or equal to a given threshold (W>=4 && H>=4), and (ii) the transform skip flag is zero . For example, if the width (W) and height (H) of the transform coefficient block is greater than 4, then an RST8×8 transform is applied to the top left 8×8 region of the transform coefficient block. Otherwise, the RST4x4 transform is applied to the upper left region of size min(8, W) × min(8, Η) of the transform coefficient block.

[0172] In an example, when the RST index is 0, no RST transformation is applied. Otherwise, the RST transformation is applied and the kernel with the RST index is selected in the example. In the example, an RST transform is applied to the intra prediction CUs in both intra prediction slices and inter prediction slices for both luma and chrominance. If dual tree is enabled, the RST indexes for luma and chrominance are signaled separately. For an outer-encoded slice (dual tree disabled), one RST index is signaled and used for both luma and chrominance. When ISP mode is selected, RST conversion is disabled and the RST index is not signaled.

[0173] 4.4 RST transformation matrix selection example

[0174] In one example, the transformation matrix RST may be selected from four sets of transformations, each consisting of two transformations. Which set of transforms is applied can be determined based on the applied intra prediction mode as follows. In the example, the RST matrix can be selected from four sets of transformations, each consisting of two transformations. Which set of transforms is applied can be determined based on the applied intra prediction mode as follows. When one of the three cross component linear model (CCLM) modes is specified, a set of 0 transformations can be selected. Otherwise, the selection of a set of transformations may be performed according to the table (2500) shown in FIG. 25. An index for table access (2500), denoted IntraPredMode, may be in the range [-14, 83], which is a converted mode index used for wide-angle intra prediction, for example.

[0175] 5. Matrix-Based Intra Prediction (MIP) mode

[0176] In some embodiments of the invention, a matrix-based intra prediction (MIP) mode is used. Fig. 26 shows an example of a MIP mode process (2600). To predict samples of a rectangular block (2610) of width W and height Η, MIP uses one row of Η reconstructed adjacent boundary samples to the left of the block (2610) and one row of W reconstructed adjacent boundary samples above the block (2610) as input. If reconstructed samples are not available, reconstructed samples can be generated in the same way as in normal intra prediction.

[0177] Prediction signal generation may be based on the following three steps (2601)-(2603). In step (2601), from the boundary samples, four samples in the case of W=H=4 and eight samples in all other cases are extracted by averaging.

[0178] At step (2602), a matrix multiplication by the Ak'bdry _red vector is performed, followed by adding a bias bk with averaged bdry _red samples as input. As a result, a reduced prediction signal is obtained on the downsampled set (2621) of samples in the original block. Matrix A _k and offset b _k may be selected based on the index k of the MIP mode.

[0179] In step (2603), the prediction signal in the remaining positions (2622) is generated from the prediction signal in the downsampled set (2621) by linear interpolation, which is a one-stage linear interpolation in each direction.

i ∈ {0, …, 17), каждая из которых имеет 16 строк и 4 столбца, и 18 векторов смещения

i ∈ [0, …, 17} размером 16. Матрицы и векторы смещения этого набора используются для блоков размером 4×4. Набор S₁ состоит из 10 матриц

i ∈ [0, …, 9), каждая из которых имеет 16 строк и 8 столбцов, и 10 векторов смещения

i ∈ [0, …, 9), каждый размером 16. Матрицы и векторы смещения этого набора используются для блоков размером 4×8, 8×4 и 8×8. Наконец, набор S₂ состоит из 6 матриц

i ∈ [0, …, 5), каждая из которых имеет 64 строки и 8 столбцов, и 6 векторов смещения

i ∈ [0, …, 5} размером 64. Матрицы и векторы смещения этого набора или часть этих матриц и векторов смещения используются для всех других форм блоков.[0180] Matrices A _k and offset vectors b _k necessary to generate a prediction signal can be taken from three sets S ₀ , S _1; S ₂ matrices. The set S ₀ consists of 18 matrices

i ∈ {0, …, 17), each of which has 16 rows and 4 columns, and 18 displacement vectors

i ∈ [0, …, 17} of size 16. Matrices and displacement vectors of this set are used for blocks of size 4×4. Set S ₁ consists of 10 matrices

i ∈ [0, …, 9), each of which has 16 rows and 8 columns, and 10 displacement vectors

i ∈ [0, …, 9), each of size 16. Matrices and displacement vectors of this set are used for blocks of size 4×8, 8×4 and 8×8. Finally, the set S ₂ consists of 6 matrices

i ∈ [0, …, 5), each of which has 64 rows and 8 columns, and 6 displacement vectors

i ∈ [0, …, 5} of size 64. The displacement matrices and vectors of this set, or a portion of these displacement matrices and vectors, are used for all other block shapes.

[0181] As shown, for an 8×8 block (2610), MIP prediction uses four averages along each boundary axis. The resulting eight input samples fall into the multiplication of the matrix by the vector. The matrices are taken from the set S ₁ . This gives 16 samples (2621) at odd positions of the prediction block. Thus, in total, (8⋅16)/(8⋅8)=2 multiplications per sample are performed. After adding an offset, these samples are interpolated vertically using the reduced upper bound samples. Horizontal interpolation is performed using the original samples of the left border. In this case, the interpolation process does not require any multiplications.

[0182] 5.1 MIP mode signaling

[0183] In some embodiments of the invention, for each coding packet (CU) in the intra prediction mode, a flag may be transmitted in the bitstream indicating whether the MIP mode is applied to the corresponding prediction packet (PU) or not. If a MIP mode is applied, the MIP mode predmode index is signaled using an MPM list including 3 MPM modes.

[0184] Deriving the MPM modes may be performed using the upper and left PU intra coding modes as follows. There are three fixed mapping tables map_angular_to_mip _idx , idx ∈ {0,1,2}, with each table associating each traditional _Angular predmode intra-prediction mode with a specific MIP mode, as shown in the following formula.

where map_angular_to_mip is a fixed lookup table. The mapping table index is determined based on the width W and the height Η of the PU packet, and there are three indexes in total, as described below,

which indicate from which of the three sets the ΜΙΡ parameters above should be taken.

и левый режим ΜΙΡ, а именно

[0185] In some examples, in order to generate the MPM list for the current block, which is encoded in the ΜΙΡ mode, the above ΜΙΡ mode is first obtained, namely

and the left regime ΜΙΡ, namely

может быть получено следующим образом:[0186] Meaning

can be obtained like this:

- If the top PU, PU _above , is available and belongs to the same CTU as the current PU, and PU _above is MIP encoded using MIP mode,

- If the top PU, PU _above , is available and belongs to the same CTU as the current PU, and PU _above is encoded using normal intra prediction mode

- Otherwise

which means that this mode is not available.

получают тем же способом, что и

но без проверки, принадлежит ли левый PU тому же CTU, где находится текущий PU.[0187] Meaning

obtained in the same way as

but without checking if the left PU belongs to the same CTU as the current PU.

и трех заранее заданных фиксированных списка МРМ по умолчанию list_idx, idx ∈ [0,1,2], каждый из которых содержит три различных режима MIP, создается список МРМ. Список МРМ создается на основе заданного списка по умолчанию list_idx(PU) и

путем замены -1 значениями по умолчанию, а также удаления повторяющихся режимов MIP.[0188] Finally, taking into account the obtained

and three predefined fixed default MPM lists list _idx , idx ∈ [0,1,2], each containing three different MIP modes, an MPM list is created. The MPM list is created based on the given default list list _idx(PU) and

by replacing -1 with the default values, as well as removing duplicate MIP modes.

[0189] As an example, FIG. 27 shows a CU level syntax table where flags signaling MIP modes are shown in box (2701).

[0190] 5.2 Obtaining the MPM List for Regular Intra Prediction Modes

[0191] In some embodiments of the invention, the MIP modes are consistent with MPM-based coding of traditional intra prediction modes as follows. The luma and chrominance MPM list derivation processes for normal intra prediction modes use separate fixed tables map_mip_to_angular _idx , idx ∈ {0,1,2} that map the _{MIP mode's predmode MIP} to one of the normal intra prediction modes,

where map_mip_to_angular is the fixed lookup table. To obtain a luminance MPM list, whenever an adjacent luminance block is encoded using the MIP mode predmode _MIP , that block is processed as if it were using the traditional intra prediction mode predmode _Angu]ar . To obtain a chroma MPM list, whenever the current luminance block uses the MIP mode, the same mapping is used to convert the MIP mode to the normal intra prediction mode.

[0192] III. Enabling implicit conversion selection based on high-level syntax elements or block-level syntax elements

[0193] In some embodiments of the invention, two transform coding schemes, implicit transform (or referred to as implicit transform selection) and explicit transform (or referred to as explicit transform selection), can be used.

[0194] In implicit mapping, a group of non-DCT-2 transforms (eg, DST-1, DCT-5, DST-7, DCT-8, DST-4, DCT-4) can be selected without signaling the transform index. For example, a group of non-DCT-2 transforms may be selected using already encoded information that is available to both the encoder and the corresponding decoder. Already encoded information may include, but is not limited to, intra prediction mode (e.g., planar mode, DC mode, angle modes), block size, block width, block height, block aspect ratio, block area size, intra coding mode (e.g., whether reference line set (MRL), ISP, MIP), positions of selected spatial merge candidates (e.g., top merge candidate, left merge candidate), inter-prediction mode (e.g., position dependent inter-prediction combination (inter-PDPC, inter position dependent prediction combination), combined inter intra prediction (CIIP) mode).

[0195] In contrast, in an explicit transform, one transform may be selected from a group of possible transform types (e.g., DCT-2, DST-1, DCT-5, DST-7, DCT-8, DST-4, DCT-4) with an index signaled to indicate which type of transformation is selected.

[0196] 1. Enable implicit conversion when explicit MTS is disabled

[0197] In some embodiments of the invention, as described in Section II.3.2, for an intra prediction residual block that is not ISP encoded, implicit transform selection may be enabled when explicit MTS is disabled for both intra and inter prediction residual blocks. extrinsic prediction as indicated by High Level Syntax Elements (HLS). For example, when both sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag SPS syntax elements are equal to 0, a decoder may decide to enable implicit transform selection for pictures or blocks associated with SPS syntax elements.

[0198] As an example, FIG. 28A-28B collectively show text (2800) specifying a transform encoding process for performing implicit or implicit transform selection for the current block based on corresponding syntax elements received from the bitstream. Text (2800) may be used as part of a video coding standard. Two sections (2801) and (2802) are shown in FIG. 28A and one partition (2803) and two tables (2804) and (2805) are shown in FIG. 28V.

[0199] Section (2801) describes the input and output of the transform encoding process. In particular, the location, size, chrominance, dequantizing transform coefficients of the current block are inputs, and the residual samples after inverse transform processing using the selected transforms are outputs.

[0200] Section (2802) describes obtaining a variable denoted implicitMtsEnabled. The variable specifies whether implicit selection is enabled. As described, if the SPS syntax element, sps_mts_enabled_flag, is equal to 1 (indicating that MTS is enabled for images or blocks associated with that SPS syntax element), the value of implicitMtsEnabled is equal to 1 when one of the following three conditions is met: (i) ISP is used to encode the current block; (ii) SBT is on and both sides of the current block are less than or equal to 32, indicating the use of SBT; or (iii) sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag are both 0 (indicating that explicit MTS is disabled for both intra- and outer-encoding blocks) and the current block is an intra-encoding block. Otherwise, the implicitMtsEnabled value is set to 0, indicating that implicit transform selection is disabled for the current block.

[0201] As described in section (2802), according to condition (i), for an intra prediction block encoded in ISP mode, an implicit transform may be included to encode that intra prediction block. According to condition (iii), for an intra prediction block not coded in ISP mode, when explicit MTS is disabled for both intra prediction blocks and inter prediction blocks, an implicit transform can be enabled to encode the intra prediction block without encoded in ISP mode.

[0202] In section (2803), vertical and horizontal transformations are determined in accordance with the variable implicitMtsEnabled, the input to the process, and the corresponding syntax elements. For example, when the values of sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag are 0, the current block (that is, the luma component and intra-prediction block) can be given a horizontal transformation kernel (denoted trTypeHor) and a vertical transformation kernel (denoted trTypeVer) according to expressions (3-1) and ( 3-2), respectively.

[0203] In tables (2804) and (2805), the numbers 1 and 2 to indicate the horizontal or vertical types of the transformation core indicate the DST-7 transformation and the DCT-8 transformation, respectively, while the number 0 indicates the DCT-2 transformation. In some examples, DST-4 transforms may be used instead of DST-7 transforms.

[0204] In the examples in FIG. 28A-28B, for blocks controlled by sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_nter_enabled_flag HLS elements, explicit MTS for outer-encoded blocks and implicit transform for intra-encoded (non-ISP-encoded) blocks cannot be used together. However, the implicit transform selection in the intra prediction residual block need not depend on the HLS element (eg, sps_explicit_mts_inter_enabled_flag) controlling MTS enable for inter prediction residuals. Thus, in some embodiments of the invention, for an intra coding block that is not predicted in ISP mode, the possibility of applying an implicit transform depends on whether MTS can be applied to the intra prediction residual (for example, the value of sps_explicit_mts_intra_enabled_flag), but does not depend on whether whether MTS can be used for inter prediction residual (sps_explicit_mts_inter_enabled_flag value). With such a control mechanism, it is allowed to simultaneously enable the use of inter-coding MTS (an explicit MTS applied to an inter-predictive residual block) and an implicit transform (for intra prediction residuals).

[0205] For example, the first and second HLS elements may be received at a decoder. The first and second HLS elements manage the same set of pictures or areas (eg, video sequence, picture, slice, tile, etc.), which may include intra-coded residual blocks and outer-coded residual blocks. The first HLS element indicates whether explicit MTS is enabled or disabled for the corresponding intra-residual blocks, and the second HLS element indicates whether explicit MTS is enabled or disabled for the corresponding outer-encoded residual blocks. With regard to the term "Outer MTS", the explicit MTS applied to the Intra Residual Block may be referred to as Intra MTS.

[0206] When the first HLS element indicates that explicit MTS is disabled for intra-residual blocks, the decoder may accordingly decide to enable implicit transform (or implicit MTS) for intra-blocks, without regard to the value of the second HLS element. For example, the second element of the HLS may have a value of 0 or 1, which does not affect the inclusion of implicit conversion for the intra-residue block.

[0207] As an example, in FIG. 29 shows modifications (2900) of text (2800) that correspond to an implicit conversion inclusion scheme where implicit conversion for intra-residual blocks and explicit conversion for outer-coded residual blocks can be combined. In modifications (2900), deleted text is marked with a strikethrough, and added text is marked with an underline. As shown, the condition for the value of sps_explicit_mts_inter_enabled_flag to be 0 has been removed.

[0208] 2. Disabling implicit conversion when other encoding tools are enabled

[0209] In some embodiments of the invention, the possibility of applying an implicit transform to an intra-coding block (which is not predicted in ISP mode) depends on whether a particular encoding tool is included, as indicated by the HLS element.

[0210] 2.1 When NSST is enabled

[0211] In an embodiment of the invention, implicit conversion may be disabled for an intra-coded block (which is not predicted in ISP mode) when the HLS element indicates that NSST is enabled. For example, as described in section II.4.1, the transformation matrix Τ is used in equation (3) to process the secondary transformation. The transformation matrix Τ may include constant elements and may be generated assuming certain statistics of target coefficient blocks, such as coefficient blocks resulting from explicit transformation selection. Thus, the block of coefficients resulting from the implicit transformation selection may not match the NSST in terms of statistics of the corresponding coefficients. For example, applying NSST to the results of an implicit conversion may not improve encoding performance. For this reason, it may be desirable to disable implicit conversion when NSST is enabled.

[0212] In this description, the term NSST may be used to refer to a family of non-separable second transform coding schemes, such as the original NSST scheme, RST, LFNST, and the like.

[0213] For example, a decoder may receive two HLS elements: one HLS element (eg, sps_explicit_mts_intra_enabled_flag) indicates that explicit MTS is disabled, while another HLS element (eg, sps_lfnst_enabled_flag) indicates that NSST is enabled. Accordingly, the decoder may decide not to enable implicit transformation for the intra-blocks controlled by these two HLS elements. Conversely, if one HLS element indicates that explicit MTS is disabled and another HLS element indicates that NSST is also disabled, the decoder may decide to enable implicit transformation for the corresponding intra-blocks.

[0214] As an example, in FIG. 30 shows modifications (3000) of text (2800) that correspond to a scenario in which implicit conversion is enabled when NSST is disabled. As shown, condition (iii) has an additional constraint "the value of sps_lfnst_enabled_flag is 0" added to obtain the implicitMtsEnabled variable.

[0215] 2.2 When MIP is enabled

[0216] Similar to the scenario in which NSST is enabled, in some embodiments of the invention implicit conversion is disabled for an intra block (which is not predicted in ISP mode) when the HLS element indicates that MIP is enabled. For example, applying the MIP intra-coding mode to a block may result in the residual block having different statistics than residual blocks encoded with conventional intra-coding modes. Thus, a MIP-encoded residual block may not match the implicit transform, which may assume residual statistics resulting from conventional intra-coding modes. Therefore, disabling implicit conversion may be desirable when MIP is enabled.

[0217] For example, a decoder may receive two HLS elements: one HLS element (eg, sps_explicit_mts_intra_enabled_flag) indicates that explicit MTS is disabled, while another HLS element (eg, sps_mip_enabled_flag) indicates that MIP is enabled. Accordingly, the decoder may decide not to enable implicit transformation for the intra-blocks controlled by these two HLS elements. Conversely, if both explicit MTS and MIP are disabled, as indicated by two HLS elements, the decoder may decide to enable implicit mapping for the respective intra-coded blocks.

[0218] As an example, in FIG. 31 shows modifications (3100) of text (2800) that correspond to a scenario in which implicit conversion is enabled when MIP is disabled. As shown, condition (iii) has an additional constraint "the value of sps_mip_enabled_flag is 0" added to get the implicitMtsEnabled variable.

[0219] 3. Enabling implicit conversion based on indications of block-level syntax elements

[0220] In some embodiments, the ability to apply an implicit transform to an intra block (which is not predicted in ISP mode) depends on whether MTS, NSST, or MIP is applied to the intra block, as indicated by the block level syntax elements.

[0221] 3.1 When MTS and NSST do not apply

[0222] In an embodiment of the invention, the ability to apply an implicit transformation to an intra block depends on whether MTS and NSST are applied to the intra block, as indicated by the block level syntax elements.

[0223] For example, the decoder may receive first and second CU-level (or block-level) syntax elements associated with the current block, which is an intra-coded block and is not predicted in ISP mode. The current block can be located at the position with coordinates [x0] [y0] inside the image.

[0224] The first CU level syntax element (eg, tu_mts_idx[x0] [y0]) may indicate that DCT-2 transforms may be used for the current block instead of transforms used in MTS (eg, DST-7, DCT-8 , DCT-4, etc.). For example, in table (2804), when tu_mts_idx[x0][y0] is 0, the variables trTypeHor and trTypeVer are 0, indicating that the horizontal and vertical transformations are DCT-2 transformations and DST-7 or DCT-8 do not apply. The second CU level syntax element (eg, Ifnst_idx[x0][y0]) may indicate that NSST is not applied to the current block. Based on the aforementioned first and second CU layer syntax elements, the decoder may decide to include an implicit MTS for the current block.

[0225] On the contrary, if the first and second syntax elements of the CU level indicate that MTS is applied (tu_mts _idx[x0][y0] is 1, 2, 3 or 4), or NSST is applied (lfnst_idx[x0][y0 ] is non-zero), the decoder MAY decide to turn off the implicit MTS.

[0226] As an example, in FIG. 32 shows modifications (3200) of text (2800) that correspond to a scenario in which neither MTS nor NSST is applied to the current block. As shown, the original condition (iii) in text (2800) is replaced by the condition that the two CU level syntax elements tu_mts_idx[x0][y0] and lfnst_idx[x0][y0] are equal to 0.

[0227] 3.2 When MTS, NSST and MIP are not applied

[0228] In an embodiment of the invention, the ability to apply an implicit transformation to an intra block depends on whether MTS, NSST, and MIP are applied to the intra block, as indicated by the block level syntax elements. Compared to section III.3.1, one more coding tool, MIP, is additionally considered.

[0229] For example, the decoder may receive first, second, and third CU-level (or block-level) syntax elements associated with the current block that is intra-coded and not predicted in ISP mode. The current block can be located at the position with coordinates [x0[y0] within the image.

[0230] The first CU level syntax element (eg, tu__mts_idx[x0][y0]) may indicate that DCT-2 transforms may be used for the current block instead of the transforms used in the MTS. The second CU level syntax element (eg, lfhst_idx[x0][y0]) may indicate that NSST is not applied to the current block. The third CU layer syntax element (eg, tu_mip_flag) may indicate that MIP prediction is not applied to the current block. Based on the above three CU layer syntax elements, the decoder may decide to include an implicit MTS for the current block.

[0231] In contrast, if the above three CU level syntax elements indicate that MTS is applied (tu_mts_idx[x0][y0] is 1, 2, 3, or 4), NSST is applied (lfnst_idx[x0][y0] is non-zero) or MIP is applied (tu_mip_flag is set to 1), the decoder may decide to disable implicit MTS.

[0232] As an example, in FIG. 33 shows modifications (3300) of text (2800) that correspond to a scenario in which none of the MTS, NSST, or MIP is applied to the current block. As shown, the original condition (iii) in text (2800) is replaced by the condition that all three CU level syntax elements tu_mts_idx[x0][y0], intra_mip_flag[x0][y0] and lfnst_idx[x0][y0] are equal to 0.

[0233] 4. Examples of Transform Encoding Methods

[0234] FIG. 34-36 show flowcharts of transform coding methods (3400), (3500), and (3600) according to some embodiments of the invention. Methods (3400), (3500), and (3600) may be used in inverse transform processing at a decoder to generate a residual block for the reconstructed block. In various embodiments of the invention, methods (3400), (3500) and (3600) may be performed by processing circuits, such as processing circuits in terminal devices (210), (220), (230) and (240), processing circuitry that performs video decoder functions (310), a processing circuit that performs video decoder functions (410), and the like. In some embodiments, methods (3400), (3500) and (3600) may be implemented in program instructions such that when the program instructions are executed, the processing circuit performs methods (3400), (3500) and (3600).

[0235] 4.1 Method (3400): Enabling Implicit Translation When Explicit MTS Is Disabled

[0236] Method (3400) starts at step (S3401) and proceeds to step (S3410).

[0237] In step (S3410), the first element of the HLS is received at the decoder. The first element of the HLS may indicate whether explicit MTS is disabled for the intra-coded block. For example, an intra-coded block is in the process of being reconstructed and thus may be referred to as the current block. The first HLS element may be associated with a set of coding blocks that include the current block.

[0238] In step (S3420), a second HLS element is received at the decoder. The second element of the HLS may indicate whether explicit MTS is enabled for the outer-encoded block. For example, the second HLS element may be associated with the same coding block set as the first HLS element. Both the current block and the outer-encoded block are included in the coding block set.

[0239] In step (S3430), the decoder may decide to enable implicit MTS for the current block when the first HLS element indicates that explicit MTS is disabled for the intra-coded block and the second HLS element indicates that explicit MTS is enabled for the block with external coding. Method (3400) may proceed to step (S3499) and end at step (S3499).

[0240] 4.2 Method (3500): Disabling implicit conversion when other encoding tools are enabled

[0241] Method (3500) starts at step (S3501) and proceeds to step (S3510).

[0242] In step (S3510), the first element of the HLS is received at the decoder. The first element of the HLS may indicate whether explicit MTS is included for the intra-coded block. The intra-coded block may be a currently reconstructed block. The first element of the HLS manages a set of coding blocks including said intra coding block.

[0243] In step (S3520), a second HLS element is received at the decoder. The second element of the HLS may indicate whether NSST or MIP is disabled for the intra block. For example, the second HLS element may manage the same set of coding blocks as the first HLS element.

[0244] In step (S3530), the decoder may determine whether to enable the implicit MTS for intra block processing based on the first and second HLS elements. For example, a decoder may decide to enable implicit MTS for an intra-block when the first HLS element indicates that explicit MTS is disabled for the intra-block and the second HLS element indicates that NSST or MIP is disabled for the intra-block. . Alternatively, a decoder may decide to disable implicit MTS for the intra-block when the first HLS element indicates that explicit MTS is disabled for the intra-block, but the second HLS element indicates that NSST or MIP is enabled for the intra-block. internal coding. Method (3500) may proceed to step (S3599) and end at step (S3599).

[0245] 4.3 Method (3600): Enabling implicit conversion based on indications of block-level syntax elements

[0246] Method (3600) starts at step (S3601) and proceeds to step (S3610).

[0247] In step (S3610), an intra-coding block is received at the decoder. An intra-coded block may be associated with a first block level syntax element indicating whether MTS is applied and a second block level syntax element indicating whether NSST is applied. For example, an intra-coded block belongs to a CU that includes CU-level syntax elements, including first and second block-level syntax elements.

[0248] In step (S3620), the decoder may determine whether to enable the implicit MTS for the intra-coded block based on the first and second block-level syntax elements. For example, a decoder may decide to enable implicit MTS for an intra-coded block when the first block-level syntax element indicates that MTS is not applied and the second block-level syntax element indicates that NSST is not applied. Alternatively, a decoder may decide to disable implicit MTS for an intra-coded block when the first block-level syntax element indicates that MTS is not applied, but the second block-level syntax element indicates that NSST is applied. Method (3600) may proceed to step (S3699) and end at step (S3699).

[0249] IV. computer system

[0250] The above methods may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, in FIG. 37 shows a computer system suitable for practicing some embodiments of the invention.

[0251] Computer software may be coded using any suitable machine code or computer language that may be assembled, compiled, linked, or similarly processed to produce code containing instructions that can be executed directly or through interpretation, execution of microcode, etc. etc., one or more computer central processing units (CPU, central processing units), graphic processors (GPU, Graphics Processing Units), etc.

[0252] Instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[0253] The computer system components shown in FIG. 37 are illustrative in nature and are not intended to impose any limitation on the scope of application or functionality of computer software implementing embodiments of the present invention. The component configuration should also not be interpreted as having any dependency or requirement in connection with any component or combination of components shown in the exemplary computer system.

[0254] The computer system may include some human-machine interface input devices. Such an input device may be responsible for input by one or more users through, for example, tactile input (e.g., keystrokes, waving, glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), olfactory input (not shown). Interface devices may also be used to capture certain media not necessarily directly related to conscious human input, such as sound (e.g. speech, music, external sound), images (e.g. scanned images, photographic images received from a still camera), video (eg 2D video, 3D video including stereoscopic video).

[0255] The input human-machine interface of the device may include one or more of the following (shown one at a time): keyboard (3701), mouse (3702), touchpad (3703), touch screen (3710), information glove (not shown) , joystick (3705), microphone (3706), scanner (3707) and camera (3708).

[0256] The computer system may also include some human interface output devices. Such output devices may stimulate the senses of one or more users through, for example, tactile output, sound, light, and smell/taste. Such output devices may include tactile output devices (e.g., haptic feedback via touch screen (3710), data glove (not shown) or joystick (3705), as well as tactile feedback devices that do not serve as input devices), audio output devices (for example: speakers (3709), headphones (not shown)), visual output devices (for example, screens (3710), including CRT screens, LCD screens, plasma screens, OLED screens, each with touch screen input capabilities or without, each with or without haptic feedback capabilities, some of which are capable of 2D visual output or more than 3D output via, for example, stereographic output, virtual reality goggles (not shown), holographic displays, smoke tanks (not shown). ) and printers (not shown).

[0257] The computer system may also include human accessible storage devices and associated media, such as optical media including CD/DVD ROM/RW (3720) with CD/DVD media (3721) and the like, cards (3722 ) flash memory, removable hard drive or solid state drive (3723), traditional magnetic media such as tape and magnetic disk (not shown), dedicated ROM/ASIC/PLD based devices such as security dongles (not shown), and etc.

[0258] Those skilled in the art should also understand that the term "computer-readable media" as used in connection with the present invention does not encompass transmission media, carrier waves, or other time signals.

[0259] The computer system may also include an interface (3754) to one or more communication networks (3755). Networks can be, for example, wireless, wired, optical. Networks can also be local, global, urban, transport and industrial, real-time, latency, etc. Examples of networks include local area networks, such as Ethernet, wireless LANs, cellular networks, including GSM, 3G, 4G, 5G, LTE, etc., digital wide area networks of wired or wireless television, including cable television, satellite television and terrestrial broadcasting, transport and industrial networks, including CANBus, etc. Some networks typically require external network interface adapters that are connected to some general purpose data ports or peripheral buses (3749) (for example, computer system USB ports); others are typically built into the core of a computer system by connecting to a system bus as described below (eg, an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, a computer system can communicate with other entities. Such communication may be unidirectional with receive-only capability (eg broadcast), unidirectional with transmit-only capability (eg CANbus to some CANbus devices), or bidirectional, eg to other computer systems using a local area network or a wide area digital network. Several protocols and protocol stacks may be used on each of these networks and the network interfaces described above.

[0260] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be connected to the core (3740) of the computer system.

[0261] The core (3740) may include one or more central processing units (CPUs) (3741), graphics processing units (GPUs) (3742), specialized programmable processing modules in the form of Field Programmable Gate Arrays (FPGAs) ( 3743), hardware accelerators (3744) for some tasks, etc. These devices, together with read-only memory (ROM) (3745), random access memory (3746), high-capacity internal storage, such as internal hard drives that are not accessible to the user, SSD, etc. (3747), can be connected via the system bus (3748). In some computer systems, the system bus (3748) may be available in the form of one or more physical connectors to allow for expansions with additional CPUs, GPUs, or the like. Peripherals can be connected either directly to the system bus (3748) of the kernel, or via a peripheral bus (3749). Peripheral bus architectures include PCI, USB, and the like.

[0262] CPU (3741), GPU (3742), FPGA (3743), and accelerator (3744) devices may execute some instructions that together may constitute the aforementioned computer code. This computer code may be stored in ROM (3745) or RAM (3746). Temporary data may also be stored in the RAM memory (3746), while permanent data may be stored in, for example, an internal high capacity data storage (3747). Fast storage and retrieval from any storage device can be achieved through the use of cache memory, which can be closely associated with one or more processors CPU (3741), GPU (3742), mass storage (3747), ROM memory (3745) , RAM (3746), etc.

[0263] Computer-readable media may store computer code for performing various computer-executable operations. The media and computer code may be specially created for the purposes of the present invention or may be well known and available to those skilled in the art of computer software.

[0264] By way of example, and not limitation of the invention, a computer system having an architecture, and in particular, a core (3740), can provide functionality through execution by processor(s) (including(and) CPU, GPU, FPGA, accelerators, etc.) software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with the user-accessible mass storage described above, as well as some persistent kernel storage (3740), such as internal mass storage (3747) or ROM (3745). Software implementing various embodiments of the present invention may be stored on such devices and executed by the core (3740). The computer-readable medium may include one or more storage devices or microcircuits, in accordance with specific needs. The software may direct the core (3740), and in particular its processors (including CPUs, GPUs, FPGAs, and the like), to perform specific methods or specific portions of the specific methods described herein, including setting data structures stored in RAM memory (3746 ), and modifying such data structures according to the methods specified by the software. Additionally or alternatively, the computer system may provide functionality through logic hardwired or otherwise embodied in circuitry (e.g., an accelerator (3744)), which may operate in place of, or in conjunction with software, to perform specific methods or specific portions of the specific methods described herein. ways. A reference to software may cover logic, and vice versa when appropriate. Reference to computer-readable media may encompass a circuit (eg, an integrated circuit (IC)) where software is stored for execution, a circuit embodying logic for execution, or both circuits, as appropriate. The present invention encompasses any suitable combination of hardware and software.

Appendix A: Abbreviations

AMT: Adaptive Transform Set

ASIC: Application Specific Integrated Circuit

BMS: a set of standards

CANBus: controller network bus

CCLM: Linear Cross Component Model

CD: CD

COT: Composite Orthonormal Transform

CPU: central processing unit

CRT: cathode ray tube

CTB: coding tree block

CTU: coding tree package

CU: coding package

DVD: Digital Video Disc

EMT: extended set of transformations

FPGA: Field Programmable Gate Array

GOP: group of images

GPU: GPU

GSM: Global System for Mobile Communications

HDR: High Dynamic Range

HEVC: High Efficiency Video Coding

HLS: high level syntax

HRD: Hypothetical Reference Decoder

IBC: Intra-Coded Block Copy

IC: integrated circuit

IDT: identity transformation

ISP: subdivision intra coding

JEM: collaborative research model

JVET: Joint Video Research Team

KLT: Karhunen-Loeve transform

LAN: local area network

LCD: liquid crystal display

LFNST: Low Frequency Non-Separable Secondary Transform

LTE: long-term development

MIP: matrix-based intra prediction mode

MRL (or MRLP): multi-reference line prediction

MTS: Selecting Multiple Transforms

MV: motion vector

NSST: non-separable secondary transformation

OLED: organic light emitting diode

RW: prediction block

PCI: Peripheral Component Connection

PLD: Programmable Logic Device

PPS: Image Parameter Set

PU: prediction package

RAM: random access memory

ROM: Read Only Memory

RST: Reduced Size Transform

SBT: sub-block conversion

SDR: standard dynamic range

SEI: Additional Enhancement Information

SNR: signal to noise ratio

SPS: Sequence Parameter Set

SSD: solid state drive

SVT: Spatially Variable Transform

TSM: Skip Conversion Mode

TU: conversion package

USB: Universal Serial Bus

VPS: set of video options

VUI: Video Usability Information

VVC: Universal Video Data Coding.

[0265] Although some embodiments of the invention have been described here, changes, permutations, and various equivalent substitutions are possible within the scope of the invention. Thus, those skilled in the art can provide numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the invention and thus fall within its spirit and scope.

Claims (43)

1. Method for decoding video data in a video decoder, including: receiving a first High-Level Syntax Element (HLS) indicating whether Explicit Transform Set Selection (MTS) is enabled or disabled for the intra-block, wherein Transform Type information indicating the type of transformation is obtained at Explicit MTS; receiving a second HLS element indicating whether explicit MTS is enabled or disabled for an outer coding block, wherein the first and second HLS elements control the same set of coding blocks that include said intra coding block and said outer coding block, and enable implicit MTS for an intra-block when the first HLS element indicates that explicit MTS is disabled for the intra-block, regardless of whether the second HLS element indicates that explicit MTS is enabled or disabled for the outer-block , wherein the conversion type information is not signaled by the implicit MTS. 2. The method according to p. 1, also including: applying an implicit MTS to the intra block, wherein the transformation type for processing the intra block is determined according to the size of the intra block. 3. The method of claim 1, wherein the first or second element of the HLS is one of the following elements: a video parameter set (VPS) syntax element, a sequence parameter set (SPS) syntax element, a picture parameter set (PPS) syntax element, a slice header syntax element, a tile header syntax element, or a tile group header syntax element. 4. The method of claim 1, wherein the intra block is not encoded in an intra division subdivision (ISP) mode. 5. The method according to p. 1, also including: receiving a third HLS element indicating that MTS is enabled for each of the outer and inner coding blocks. 6. The method of claim 1, wherein the conversion type information is obtained based on the MTS index signaled at explicit MTS. 7. Method for decoding video data in a video decoder, including: receiving a first High-Level Syntax Element (HLS) indicating whether Explicit Transform Set Selection (MTS) is enabled or disabled for the intra-block, wherein Transform Type information indicating the type of transformation is obtained at Explicit MTS; receiving a second HLS element indicating whether non-separable secondary transform (NSST) or matrix-based intra prediction (MIP) is disabled or enabled for the intra block; and enabling implicit MTS for an intra-block when both the first condition and the second condition are met, the first condition being that the first HLS element indicates that explicit MTS is disabled for the intra-block, and the second condition being that the second element of the HLS indicates that NSST or MIP is disabled for the intra-coded block, and the transform type information is not signaled by the implicit MTS. 8. The method according to p. 7, also including: disable implicit MTS for an intra block when the first HLS element indicates that explicit MTS is disabled for the intra block and the second HLS element indicates that NSST or MIP is enabled for the intra block. 9. The method of claim 7, wherein said inclusion comprises: enabling implicit MTS for the intra block when the first HLS element indicates that explicit MTS is disabled for the intra block and both NSST and MIP are disabled for the intra block as indicated by the HLS elements. 10. The method according to p. 7, also including: applying an implicit MTS to the intra-coded block. 11. The method of claim 7, wherein the first or second element of the HLS is one of the following elements: a video parameter set (VPS) syntax element, a sequence parameter set (SPS) syntax element, a picture parameter set (PPS) syntax element, a slice header syntax element, a tile header syntax element, or a tile group header syntax element. 12. The method of claim 7, wherein the intra block is not encoded in an intra division subdivision (ISP) mode. 13. The method of claim 7, wherein the conversion type information is obtained based on the MTS index signaled at explicit MTS. 14. The method according to p. 7, also including: receiving a third HLS element indicating that MTS is enabled for the intra block. 15. A method for decoding video data in a video decoder, including: receiving an intra-encoded block associated with a first block-level syntax element indicating whether Transform Set Selection (MTS) is applied and a second block-level syntax element indicating whether a Non-Separable Secondary Transformation (NSST) is applied, and enabling an implicit MTS for an intra-coded block when both the first condition and the second condition are met, the first condition being that the first block-level syntax element indicates that no MTS applies, and the second condition being that the second block-level syntax element indicates that NSST is not applied, and the transformation type information is not signaled by the implicit MTS. 16. The method according to p. 15, also including: disable implicit MTS for an intra-coded block when the first block-level syntax element indicates that MTS is not applied and the second block-level syntax element indicates that NSST is applied. 17. The method of claim 15, wherein the intra block is associated with a third block level syntax element indicating whether matrix-based intra prediction (MIP) is applied, and said inclusion also includes the inclusion of an implicit MTS for an intra-coded block when the first block-level syntax element indicates that MTS is not applied, the second block-level syntax element indicates that NSST is not applied, and the third block-level syntax element indicates that MIP is not applied. 18. The method according to p. 17, also including: disable implicit MTS for an intra-coded block when the first block-level syntax element indicates that MTS is not applied and either the second block-level syntax element indicates that NSST is applied or the third block-level syntax element indicates that MIP is applied. 19. The method of claim 15, wherein the first block-level syntax element indicates that MTS is not applied by indicating that DCT-2 transforms are to be used to process the intra-coded block. 20. The method according to p. 15, also including: applying an implicit MTS to the intra-coded block. 21. The method of claim 15, wherein the intra block is not encoded in an intra division subdivision (ISP) mode. 22. The method according to claim 15, also including: receiving a High Level Syntax Element (HLS) indicating that MTS is enabled for the intra block.

Images